Archive-name: internet/web-research-faq Posting-Frequency: monthly Last-modified: variable:date URL: http://spireproject.com Copyright: (c) 2000 David Novak Maintainer: David Novak <[email protected]> Web Research FAQ Welcome. This FAQ introduces the concepts and tools of web research. Attention is focussed on how web research fits into the larger field of information research, but web research has many peculiarities all its own, quite obscure to the new user. This FAQ resides at SpireProject.com/webfaq.txt SpireProject.co.uk/webfaq.txt and http://cn.net.au/webfaq.txt This FAQ is just a small part of a much larger effort to help you with information research. The Spire Project is available as 3 website, mirrors, zip-file, and 3 other faqs (See our larger Information Research FAQ - http://spireproject.com/faq.txt) I have included here text versions of relevant webpages and sections of our other faqs. Enjoy, David Novak - [email protected] The Spire Project : SpireProject.com, SpireProject.co.uk, Cn.net.au Contents Web research starts with a vision. Don't Try to Search Everything Internet Information Theory ----- Articles from The Spire Project Finding A Webpage Discussion Groups ----- Excerpt from the Information Research FAQ What is Information Research? ----- Acknowledgements ___________________________________________________ Web research starts with a vision. The web is at the heart of a dramatic change to the information we receive. Even if we never touch the web, it will still affect what we read on account of how radically it permits new voices and new competition to the established ways we receive information. Web research, for those who wish to learn, is about finding more information, better information, & better answers from what exists on the web. There is a great deal of research about the relative strengths and techniques to maximize the use of search-engines. I do not follow this discourse closely. Concider reading the reviews offered by SearchEngineWatch (www.searchenginewatch.com) for this. My interest and excitement comes with developing and transplanting search techniques of a stronger nature. This FAQ covers topics like: - Boolean & field searches, - Anticipate what exists, - Judge webpage quality quickly, - Move to relevant nexus points, - An awareness of larger structures in the web, - Ask for directions, - Discussion archives as a search tool, - Information Clumps, so seek the clumps, - Information Research & its similarities. Before the web, we all had our private library, supplemented by the local public library, corporate library and perhaps local bookstore to provide our information needs. Add our daily paper and perhaps a magazine or two (and our 30 minutes of news broadcast on TV) and you have the sum total of most of our research needs. There were other sources available, but these were muted on account of the popularity and strength of these resources. Today, with the web, we have a vast new tool at our disposal. We can chose instead to follow the stockmarket live, read news direct from the newswires, browse our local library AND the British Library concurrently. We can read widely divergent views on an incident or activity. Perhaps best, we now have a vast slew of people and organizations to assist us in getting the information we seek. Each has different aims and focus and experience and bias. Navigation of the Internet is not simple. Those who tell you so are selling you something. Navigation (and research, a close cousin,) depends on your experience & practical understanding of how information is distributed on the Internet. Let's start with some theory. ___________________________________________________ Don't Try to Search Everything If a tree falls in a forest, and no one is around, does it make a sound? This is so relevant to the web. If a webpage exists, but no-one visits, does it really exist? This is more the norm for information on the web. People enjoy the publishing of information - but for most webpages, promotion is minimal at best. I may inform a search-engine or two, or tell a relative, a forum or another relevant website. Lets look at what this means for research. I inform a search-engine, so someone skilled in searching search-engines could find my webpage - provided they search for the right words. Unfortunately, from the point of view of a publisher, this is not a useful way to find readers. There are simply too many webpages and too few skilled researchers. If your webpage is fortunate enough to appear first on a popular search term, the search-engine may drive significant readers your way. Otherwise, you can expect very few visitors. From the researcher's point of view, the search-engine does not give you a clue which of the many webpages will be useful to you. The usual practise is to look for the top 30 and leave the rest. Thus, finding a particular webpage, lets say the best webpage on travelling Western Australia, depends a bit on luck (being in the top 30) and a bit on the skill of the reader (in choosing words which accurately describe the better webpages). When a writer informs a relative, they may tell a few other relatives, and some of this may snowball to a few more relatives or workmates. Again, it depends on the popularity of the webpage, but you should not expect considerable traffic. From the researchers perspective, to find this webpage you must either speak/write to one of the relatives, or look for a link in a relative's webpage (provided one of them added a link). Mentioning a webpage to a forum works similarly, but increases the likelihood a fine webpage will be linked into a collection of relevant webpages with existing traffic (perhaps some of the historically important documents - which we will explain later) and this in turn may snowball some. From the researcher's perspective, we could find my webpage by asking the forum, searching past forum messages, or by looking at those relevant webpages which link to fine webpages (nexus points - again explained later). Obviously there are more ways to promote that this. If you have money, you can purchase a top position in most of the search-engines. There is banner advertising. Promote away from the Internet too. But just from this rather lengthy example, can you see, no single tool will ensure you find a webpage, no single technique is enough. Many webpages will just never be found. Defeatist you say? No, its realism. Just as you can, theoretically find out everyone's first name with a telephone, in practise, you can't. So, to one of the first myths of internet research: You are not searching the web when you search AltaVista, Yahoo or AllTheWeb. You are only searching a database/directory - an incomplete tool that does a poor job selecting the best from the rest. The purpose of this narrative is to set the picture. In web research, we do not attempt to search everything. There is simply no good way to do this - and if we attempt to, we lose all hope of isolating the best information. No. We search the web in ways which provide good coverage AND link to quality information. We are, after all, search the web FOR something. Technically speaking, it is unwise to formulate your question as "search the web for everything about..." Oh, we will remain flexible and use blunt techniques when valuable or expedient, but our focus is on quality & depth. And this focus opens up a vast range of search techniques taking advantage of internet structures like nexus points and historically important documents. This focus vastly improves results. Lets turn our attention now to understanding structure on the web. ___________________________________________________ Internet Information Theory Lets agree the Internet is great fun to surf, but less valuable when you have a specific question in mind. To improve our search skills, we begin by understanding how information is arranged on the Internet. Contrary to myth, information is not disorganized but rather organized very carefully along clear patterns. Many patterns are specific to the information format (text document, webpage, email message, printed article). Further patterns match the way we become aware of information, or are specific to the information systems (mailing list, faq, peer-reviewed journal). Your understanding of the strengths and weaknesses of each pattern, each format, each system, guides your search for information. We shall start by shattering the Internet, and commenting on the many pieces. __ 32.1 three definitions of the Internet Let us be careful when we use the word 'Internet'. 1_ The Internet is a physical network; more than a million computers continuously exchanging information. The Internet allows us to transfer information around the world. 2_ The Internet is a landscape of information available on almost every topic imaginable. This information appears almost chaotically distributed to the world, but holds clear patterns. For instance, linking information together are various structures like government web links, search engines and FAQ documents. 3_ The Internet is a community of 100+ million individuals. These are real people who chose to interact, discuss and share information online. What we learn here is not so important as the technique - break the large seemingly chaotic system into smaller pieces: pieces that hopefully make more sense. Eventually, when we've made sense of the little bits, perhaps we can comment astutely on the big-picture. In this example, let me just draw your attention to the way most of our research effort focuses on the second definition: a landscape of information. Much of the best information originates in the third definition: the Internet is a community. Sometimes it is far more effective to ask real people than search the information cyberspace. Let us now illuminate more important facets of the Internet. __ 32.2 information, transaction, entertainment There is a triad of functions to all online activity: Function - Activity - Unit ---------------------------------------- Information - Research - The Fact or Conclusion Exchange - Business - The Transaction Entertainment - Play - The Experience Each Internet function grows at a different rate and moves in a different direction. The development of forums is firmly in the smallest segment dealing with information. This segment is quite poorly organized and confusing. The entertainment function in contrast is well financed and graphically innovative with clear, profitable opportunities. Much of the web is prepared with Exchange or Entertainment in mind. "Brochureware" (purely promotional webpages) is rarely required for research, but is critical to securing a transaction. Entertainment related, or just entertaining, websites abound. Let us recognize just how few webpages are information & research related. My own experience suggests we are just beginning to see the movements towards profiting from providing information. Direct sales of information is still chaotic and unrewarding. __ 32.3 information formats The way information is packaged has a great bearing on the content, quality and use of the information. This theme is evident throughout the work of The Spire Project, and is particularly applicable to Internet information. Webpages, text files, software, email and database entries each have particular qualities. Each shapes, constrains and restricts the informative content. These particular qualities apply irrespective of the information involved. Books are dense, factual, a little old. Articles are short, sharp, more recent. News is puff, introductory, immediate. Each way the information is packaged, each format, presents the information to set standards. Information formats on the Internet are the same. Webpages are graphical, technical to produce, and not easily updated. FAQs are easier to maintain, text only, and attract more peer review. Mailing lists are simpler still, text, short, immediate, very peer-reviewed, characterized by discussion and resource discovery. Newsgroups are characterized by extremely low costs, vulnerable to trashing, poorly managed. Email is simple use, one-to-one discussion. Lets look at books more closely. Books are created by authors who have something to write. Books are printed and marketed by Publishers to the bookstores that then provide it to the readers. Each facet of this process defines the resource. Books have quality, editorial vetting but minimal peer-review, marketable value and a potentially lengthy preparation time. When it comes to research, why look for a book when investigating digital money? Books would just have the wrong qualities - would present the information poorly. We need a more current format (digital money is a fast moving topic), and a more peer-reviewed format (books have editorial vetting, but not intrinsic peer-review). Why not search for a mailing list, an FAQ, or an association website. These formats have qualities more appropriate to our question. __ 32.4 information preparation Information flows also impress patterns on Internet information. Most information is transplanted to the web - first created elsewhere. The source of information imparts as much pattern as the eventual format the information takes. Information may appear as a webpage, and conform to our expectations for all webpages, but the information may have been prepared from the discussion on a mailing list - and thus enjoy a more topical, specific, timely and peer-reviewed quality. Lets look at FAQs. The best resource in the world on copyright law is the musings of a group of copyright lawyers who form the copyright mailing list. The copyright FAQ supported by this group is a logical document summarizing much of the discussion of this mailing list. FAQs are vetted by the news.answers team, then automatically mirrored around the world. From its origins in the mailing list, the FAQ is a peer-reviewed document, often full of links to further resources, topical, knowledgeable and factual. As an FAQ, the document is not immediate, graphical or financially rewarding (some FAQs stagnate). Only some Internet information is created within the Internet environment. The concept of 'brochureware' describes the common traits to promotional webpages directly prepared from paper promotional brochures. One of the more exciting trends is the movement of information from the dusty shelves of government offices and association libraries to their more accessible websites. The quality of information retained in your average government agency, from quality research reports, to detailed studies, to current industry monitoring is very high. These qualities are then brought over to the web format. Such web-documents tend to be isolated (not linked to other related resources) and perhaps a little behind the time line, but of a generally high quality. An exciting holistic view of the Internet information landscape is based on these descriptions. Imagine, for a moment, information flowing through a collection of systems. At certain points, information groups together, and generates new, perhaps higher quality information, which then flows in a different system, a different direction, to different people. The flow of information from one person to another, from one format to another, imprints qualities to the information along the way. Each organization, or subsequent re-organization, imparts specific styles and conventions and quality to the result. __ 32.5 publishing motivation Let us proceed to a third set of patterns. Information appears on the Internet for one very specific reason. Someone Publishes (DUH). The motivation behind publishing colours the information. Patterns we will use to better search for answers on the web. Ask yourself who is publishing, and why. One of the biggest publishing segment a year ago were individuals publishing documents derived from their personal expertise. A typical document would be one with minimal peer review, a list of aging links to further resources, simple graphics, variable to short length, prone to bias, but moderately reliable because the publisher knows their topic well. These pages are often located on web pages with private sub-directories (usually starting /~name/). Commercial sites publish mainly for the promotional value. Their secondary purpose is to provide sales information to prospective clients. Rarely do commercial sites go beyond this. Commercial webpages often reside on their own domain name, as a .com, or in sub-directories - without the tilde symbol. Commercial sites also tend to age badly. They are very noticeable from their front page. Government agencies are emerging as valued publishers. Slowly their dormant information becomes available through this new medium. Currently almost all government documents on the Internet also appear in print, meaning they are factual, exhaustively reviewed, tend to be a little old (but age well), and come from highly paid knowledgeable people who believe it is their duty to inform others. Such documents are lengthy and appear on .gov domains. These patterns are simple to see. Grant-funded projects create brilliant research resources and hold much promise in pushing the limits of this technology. I am eager to see the results of the US Patents project, and appreciate the value of having Supreme Court rulings on the Internet. Often such projects are short on money but deeply focused on content. Most projects reside on educational servers and are widely discussed within knowledgeable groups. Associations, publish association-kind-of-things. Most are initially just like the commercial webpages, but with time become much more factual and research-worthy. Most associations are dedicated to developing awareness of their chosen topic, albeit coloured by their chosen bias. Few associations are significant publishers yet, but this segment will begin to liberate dormant information within associations. Let's summarize. The key is to always watch who is the publisher. We can assume a great deal, quickly. We are unlikely to find the latest changes to patent law from government or commercial publishers. Such organizations are simply not motivated to present such information. __ 32.6 promoting information Publishing is one achievement, but you and I will never read any information until we learn it exists. This simple fact creates even more patterns to Internet information. Knowledge of information moves through set routes on its way from writer to reader. Promotion is not simple. It is a process that takes time, effort and perhaps money. Information without serious promotion tends not to be promoted far from the source. Another way to phrase this; you must search close to the source to find poorly promoted information. A search engine indexes pages relatively indiscriminately. This also means a site of quality is not likely to reach your attention. The odds are not good, and from a promotion point of view, search engines generate minimal traffic to your webpage. Search engines drop you rather randomly into a website. It is often necessary to move up a directory to understand the purpose and motivation of a site you find interesting. Information published through advertising tends to have a financial payoff for the promoter. This kind of information tends to be promotional information. Brochureware. The alternatives are to promote a webpage or website through one of the referral tools. Each such tool accepts links on some criterion. Each tool you use to locate information also selects particular types of information for your attention. If you arrive at a document by recommendation through a mailing list, the document is likely to be recent, on-topic, and specific to the purpose of the mailing list. Alternatively, (for poor mailing lists) it will be wildly off topic and trash. You are unlikely to see referrals to old documents or documents of historical importance. These are the qualities most acceptable to the mailing list environment. Directory trees, FAQs, guidebooks and related promotion tools all work as historically important documents. In the past, such resources list, describe and alert people to relevant information for the field. Slowly, over time, this function becomes acknowledged, reinforced and promoted. Time is the essence of this fame. Webpages or websites found through historically important documents, by their nature, tend to be long lasting websites with lasting importance in the field. Such documents point to other similar documents or websites that have achieved a long-lasting importance. You are unlikely to find specific documents, but rather sites that focus or bring together information. In short, there is little motivation to link to specific webpages, when a link to important websites is considered just as good. Similar generations can be made of each type of promotional tool, and become important in rapidly seeking our information which matches our intention, as well as summarizing the likely motivation - and bias - of webpages we are interested in. __ 32.7 information clumps Information Clumps. Information is created, nurtured, develops, gets transplanted, gets arranged and then becomes visible through a process which brings similar information together. As we have discussed, there are factors deeply affecting all information on the Internet. Motivation, Preparation, Format and Promotion defines the quality and content of any given item of information. With so many influences, we should not be surprised to learn information naturally groups together. In reality, there is nothing natural involved - it is a social phenomenon reinforced each time you and I visit or read one resource but not another. History can explain some aspects of Internet development. As a small collection of sites become dominant in particular fields, by collecting and delivering better content to more people, new sites find it progressively more difficult to capture attention. This dynamic works for websites reaching out for visitors, and discussion groups reaching out for subscribers. In each case, seniority counts. Seniority counts in several ways too. Promotion is directly related to quality, interest, traffic and time. The longer a site is active, the better the footpath develops, the more people visit. Secondly, quality content is directly related to access to quality content, peer review, and time/money. Important existing sites gain in every way. This results in a grand system where the first-in, best-dressed, can capture the high ground and secure a grand lead in awareness and footpath over competitors who follow. Yahoo is a prime example of a directory tree, not even the best in most areas, which has achieved unparalleled traffic & awareness. This competition is equally evident where no money is involved. Perhaps your association wishes to create a new referral website, or an open mailing list, or an informative guide. All sound concepts, effective projects. However, if older, established resources exist, the work will be long and arduous. Despite the marketing message, the Internet is not a world where the best information floats to the top. The Internet will not let you to reach millions. You must compete for the attention, participation, devotion and assistance in a manner very similar to building a business. In concrete terms, information clumps on the Internet. The best resource could appear on any Internet system (webpages, email mailing lists, ftp-archives, faqs, online databases, newsgroups...) but we can be fairly certain the best information will congregate in just one or two. Consider our article "Searching the Web" (http://cn.net.au/webpage.htm). We progressively search different web tools, looking for the most worthy. Searching the Internet is the same. You must touch each system to see which system is dominant, where the information is congregating for your topic. __ 32.8 bringing this together In summary, we have broken down and discussed various qualities of published information and promoted information. We have made sweeping generalizations and educated guesses about information on the Internet. Now what? When a painter begins to paint, they have already visualized some of the image. They already have a concept of the finished result. Internet research is no different. We start by building a vision of the information we seek. Who would publish it? Where would I find it? What is its motivation? How would we find it? We now have a practical vision. The address is the key. The url for any item of information gives us a surprising amount of information - particularly now we are making generalizations about information patterns. We can guess if information resides on a personal webpage, a funded university project, or a commercial project. The information resides on a .gov website? - the quality is likely to be higher and conform to our expectations of government resources. We use this new-found experience in three ways. First, we restrict our searches to the most likely sources. Second, we quickly jump through lists of resources (such as those generated by search engines) to the sources that match our expectations. Third, your understanding of the relative qualities of information guides your judgement of information value. Internet newcomers often expect to have instant access to the latest information at the touch of the button in beautiful colour and peer reviewed quality prose. Who is publishing this? Where is this information coming from? Who would help us find this? Such a vision is fantasy. If we were instead to look for an association website, dedicated to a certain type of research, or an informed newsgroup, maintained by people passionate about sharing this technology, then we have made four steps forward. We are clear about where to look for the answers we seek, and we will know quickly if the answers are online. ___________________________________________________ Two relevant webpages from The Spire Project, as text. Searching the Web ----------------- Webpages are often of unknown age, of only guessed at quality and potentially the easiest information to retrieve. There are many points of entry to web resources, but search tools differ. Try to match your search tool to your question. To start, you will need to learn something of the different tools - this is described below - and four basic search techniques: Boolean[1], Proximity[1], Field Searches[1] & Truncation[1]. Internet Global Search Engines [1] Altavista[1], among other tools, has a very large, fast search engine. Allows for Basic Boolean[1] AND + NOT - OR | Proximity[1] " " ~ (near - within 10 words of each other.) Several Fields[1]: title:"Spire Project" domain:gov url:edu link:cn.net.au and Truncation/Wildcard[1] (*) Of import, Capitals matter with Altavista. Read more here[2] and here[3]. [4] All-the-Web[4] is important because it is large - really large - with a flexible search facility. Allows Partial Boolean[1] + - Simple Proximity[1] " " and Several Fields[1] a title field search normal.title:spire url field url.all:.au link text and link url fields normal.atext:spire link.all:cn.net.au All-the-Web is not case sensitive. Read more here[5]. When searching for a topic with precise descriptive terms, use a broad search engines. Always place the Boolean +symbol before each search word (like this: +word1 +word2) to insist all words appear in the results. Quotes keep words together ("word1 word2"). These two simple steps dramatically improve results. Keep adding words and search limits until the number of hits is reasonable. [48] Inktomi[48] provides its substantial web directory through other companies, in this case, Yahoo. You may need to select "Web Page Matches". Accepts Partial Boolean[1] + - Simple Proximity[1] " " and Two Fields[1] title: and an index-date field through this form[6]. [53] Lycos[53] is a rapid search engine, again, one of the larger ones on the web. Accepts Partial Boolean[1] + - Simple Proximity[1] " " and Several Fields[1] through their advanced search form[7]. For more global search engines, consider visiting the W3 Search Engines[8] page at the University of Geneva. The Industry Research Desk also has a good search engines page[9] as does this site[10] by Paul Hopper and this page[11] from Search Engine Watch. Meta-Search Engines & Google If you know something of the destination already, like a title or company name or full name, try using a search tool which excels in finding named websites. There should be little difficulty in finding such sites with either Google or a Meta-Search engine, but don't get excited and use these on other occassions[1]. [2] Debriefing[2] is our meta-search engine of choice. Use this to find names & named websites. Accepts Partial Boolean[1] + - Simple Proximity[1] " ". Capitals matter. [12] Google[13] is a new style of search engine which ranks sites with more care and concern. This works well for sites you know a little about in advance. Allows Partial Boolean[1] + - Simple Proximity[1] " ". Unfortunately, No Truncation[1] not even for plurals! Read more here[14]. Categorized Lists When searching for information which lends itself to a particular category or topic, start with resources which group information in categories. With few exceptions, these resources index websites, not webpages. Also, keep your search words simple as these are small databases. [94] Yahoo[94] is the largest of this type of directory tree; the definitive site. Accepts Partial Boolean[1] + - Simple Proximity[1] " " Truncation[1] * and Several Field[1] t: (for titles) u: (for urls) and a date field through a form[6]. Read more here[15]. [16] The Open Directory Project[16] is a Netscape effort to, presumably, mute the strength of Yahoo. It is very good, and very similar to Yahoo. [46] Infoseek[46] gets my vote for the next best directory tree. [17] For an alternative, try the World Wide Web Virtual Library: Subject Catalogue[17], a distributed network of subject lists, not nearly as dominant as Yahoo, but far more "scholarly" shall we say. This virtual directory has been around many years, previously famous from www.w3.org. Reviewed Sites When seeking specific fields of study, when topics are clouded with many similar, low quality sites, start with resources with a greater degree of personal attention. Peer review and vetting produce resources with more quality but limited coverage, better suited to this situation. Also, keep your search words simple. [18] The Scout Report[74] is one of the oldest and most highly regarded e-newsletters introducing new Internet resources. Residing at the University of Wisconsin, the Scout Report describes research, education & topical sites. The Scout Report Signpost[18] provides a quick search[19] of previously featured sites: [20][21] BUBL[20] is a British site which reviews Internet resources then indexes by dewey decimal number[22]. I prefer their dewey presentation, but the collection is not large (though the largest of the library projects I have seen). Here is their keyword search[21]: [5][6] The Argus Clearinghouse[6] is a vast collection of Internet guidebooks. We can search the titles & descriptions, but then click on the highlighted keywords to find related guides. I suspect Argus is not successfully keeping pace with Internet development. [23] AlphaSearch[23] is similar to Argus. This one indexes important nexus sites and should be browsed. [24] The Britannica.com[25] (as in Encyclopedia Britannica) has been remolded as a free guide to books, periodicals, web and their encyclopedia. This encyclopedia is perhaps the best. Search from their search page[26]. [8] FAQs[8] can be searched from an FAQ database like this one at Oxford University. See also our Discussion Groups[12] article. [27][28] WebRings[29] list sites by topic. Each webring is maintained by a volunteer at an uninvolved site using standard software. The search here is of Webring.com[27], which lists the most, though there are other webring sites including bomis.com[28]. Specialty Tools For issues with a particular government, url or language origin, consider using tools designed with this in mind. [1] Altavista[1] can be limited to specific domains (gov edu au) with their "domain:domainname" field search[1]. "url:url-segment" is also useful. Read the Altavista Fancy Features for Typical Searches[2]. [30] GovBot[30], as developed by The Center for Intelligent Information Retrieval (CIIR) is a search engine which indexes exclusively a great number of government webpages, a unique resource. This is the CIIR gateway (there have been others). General Search: Title field: Url field: Accepts Simple Proximity[1] "" but not Boolean. Field searches[1] are optional, so just enter search terms in title box for a title search. [1] Altavista[1] also allows for a field search[1] by language. Searching for a Japanese site? Consider searching only webpages in Japanese. An alternative approach may be to search purely regional resources. Aussie.com.au[1], for example, is a search engine indexing only Australian websites. Yahoo maintains a moderate regional list[31] of indexes and search engines, but the list maintained by SearchEngineWatch - Regional Search Engines[3] is better. [32] Two further (but graphically intensive) lists: Search Engines WorldWide[4] and SearchEngineCollossus[5]. On a separate page, I have collected various entry points for Australia including search engines and government entry points. Commercial Commercial Databases There are commercial resources applicable to the study of the Internet. More are described separately in the topic brief: Software & IT Research[24]. NetFirst, is produced by OCLC, and delivers bibliographic data to Internet resources. Further descriptions can be found from FirstSearch[33]. Conclusion 3 Second Summary: Search the web with several tools in succession. No search will find everything. Different tools suit different questions. For many of us, searching the web is simply typing words into a search engine. This works until it doesn't, in which case we need something more. Contrary to myth, global search engines are not the best place to start most of the time - just some of the time. On other occasions, start with a directory, a meta-search engine, a guide, an faq... We should start thinking what tools excel at locating what kinds of webpages. (There is no simple search of everything.) The links on this page are arranged to follow this insight. There are more insights into effective Internet research. Information clumps; Information is not established in isolation but instead develops in context, is reinforced, and becomes a trend. The publishing motivation & promotion purpose can help us rapidly judge the content of a website. The webpage address can tell us a great deal about both the website structure and the type of publisher. These topics are covered in greater detail in Section 31[34] of the Information Research FAQ. Once skilled, you can segment and search the most promising areas of the web quickly and efficiently. If you do not quickly find your answers there may be other, more appropriate resources. Consider asking for help in an appropriate discussion group, or reviewing printed literature instead. The Web is only one resource among many. If your primary interest is Search Engines, consider reading A Higher Signal - To - Noise Ratio[35]: Effective Use Of Web Search Engines by Bob Bocher & Kay Ihlenfeldt, Sink or Swim[36]: Internet Search Tools & Techniques by Ross Tyner (alternative site[37]) and the recent The Search is Over[38] by Adam Page. For even more, read Searching the Internet[39] a publication in the Scout Toolkit[40] and browse Search Engine Watch[41]. Strategy Searching the web is more a skill than most of us acknowledge. The web is a manifestation of the demon professional researchers work with all the time in the commercial information market. There is constantly the fear you have missed that single important site with everything. Consider the researchers motto: Someone, somewhere, probably knows the answer. But how long do we search for gems, and where do we look? To decide, we must learn about Internet structure and organization. Why is information published on the web? Why is it promoted? Lets review the reasoning behind effective Internet research. There is so much more than putting words into search engines. #1 Motivation We can make some very astute generalizations about a webpage very quickly if we can judge the reason it was published. Not only is this an important step in analyzing any information, but this tells us a great deal about the contents of the webpage. Yes, merely determining a site belongs to an association actually specifies the quality, motivation and type of information we will find. Associations either publish what is termed 'brochureware' (promotional material), or if well advanced, present research work previously restricted to the association library: important research studies & the like. Commercial interests have much more difficulty delivering useful resources. The importance of projecting a corporate image comes first (lots of 'brochureware'), and service descriptions come second. On occasion, commercial interests will support a worthwhile service tied closely to their own service - thus banks present interest rates - bookstores present their book database. The certainty with which we can make these judgments will astound you. Corporate websites never publish "changes to patent law". They simply don't have the motivation. Only an individual would publish this, most likely not on the web but though a mailing list. Information is not distributed randomly. Consider Format, Preparation, Motivation and Promotion. Consider this, then Visualizethe information you seek. #2 Promotion We can make further snap judgments about web information from the way you get there. Promotion is very difficult on the web, and it is hard to find poorly promoted information. The tools you use to reach information pre-determines the type and quality of information you will find. Search engines index webpages indiscriminately. Advertised websites must have a pay-off. Directories focus on established websites (not webpages). Link pages also link to established websites but put more thought into the selection of resources. Both usually focus on general sites. For specific or current resources, we need to move to mailing lists or active nexus point. Yes, when we find a webpage through the Scout Report (a prominent resource discovery newsletter), we can assume the webpage has a high quality of information, is reasonably current and has a general appeal (within the interest of the newsletter readers). Lets put this in reverse. If we are looking for a recent document by a prominent library committee, we will not find it through AltaVista, Yahoo, or normal link pages (except accidentally). We may find it through specialist newsletters, active nexus points, or through mailing lists. #3 Visualize This discussion continues in Section 32[42] of the Information Research FAQ. When an artist begins to paint, they visualize the image. They already have a concept of the finished result. Internet research is no different. We start by building a vision of the information we seek. Who would publish it. What is their motivation. Who would promote it. Where would I find it. Information Clumps. Information is created, nurtured, develops, gets transplanted, gets arranged and becomes visible through a process which brings similar information together. Your understanding of this process, including motivation and promotion, must guide your search of the web. Only then will we will know where to look, and quickly know if the answers are on the web. This article comes from The Spire Project. Advice welcome : email [email protected] [1] http://spireproject.com/webpage.html# [2] http://www.altavista.com/av/content/help.htm#fancy [3] http://www.altavista.com/av/content/help_advanced.htm [4] http://www.alltheweb.com [5] http://www.alltheweb.com/advsearch [6] http://search.yahoo.com/search/options [7] http://lycospro.lycos.com [8] http://cuiwww.unige.ch/meta-index.html#INF [9] http://www.rbbi.com/links/sengine.htm [10] http://members.xoom.com/PRHopper/Search.html [11] http://searchenginewatch.internet.com/facts/major.html [12] http://www.google.com [13] http://google.com [14] http://www.google.com/help.html [15] http://search.yahoo.com/search/syntax? [16] http://dmoz.org [17] http://vlib.org/Overview.html [18] http://www.signpost.org/signpost [19] http://www.signpost.org/signpost/quicksearch.html [20] http://link.bubl.ac.uk [21] http://link.bubl.ac.uk/isc1 [22] http://link.bubl.ac.uk/isc2 [23] http://www.calvin.edu/library/as [24] http://www.ebig.com [25] http://www.britannica.com [26] http://search.britannica.com/bcom/search [27] http://www.webring.org/#ringworld [28] http://bomis.com [29] http://www.webring.org [30] http://ciir2.cs.umass.edu/Govbot [31] http://dir.yahoo.com/Computers_and_Internet/Internet/World_Wide_Web/Sear ching_the_Web/By_Country_and_Region [32] http://spireproject.com/note05.htm [33] http://www.oclc.org/oclc/man/6928fsdb/netfirst.htm [34] http://spireproject.com/faq.htm#31 [35] http://www.msstate.edu/Dept/hrdc/sum98/search2.html [36] http://www.lboro.ac.uk/info/training/finding/sink.htm [37] http://gatsby.tafe.tas.edu.au/sinkorswim/search.htm [38] http://www.zdnet.com/pccomp/features/fea1096/sub2.html [39] http://wwwscout.cs.wisc.edu/toolkit/searching/index.html [40] http://wwwscout.cs.wisc.edu/toolkit/index.html [41] http://www.searchenginewatch.com [42] http://spireproject.com/faq.htm#32 Discussion Groups ----------------- Mailing Lists, Newsgroups, Associations - each are focal points of discussion, exchange of information and professional development. Collectively called Special Interest Groups (SIGs), these are the original sources of many fine research resources. Brilliant research sites in their own right, a mailing list, newsgroup or association can also be a fine contact point for experts, or the site of focused, specialized libraries. [1] Internet The copyright mailing list is a group of more than 100 lawyers who focus on copyright. This list, and their Copyright FAQ, are the best resources on copyright law in the world; current, factual, and peer-reviewed. This is not unusual for a mailing list. As a source of experts, I once found an accomplished but poorly published scientist from an old message in a mailing list archive. Locating Mailing Lists [1] Tile.Net/Lists (tile.net/lists[1]) has both a searchable and directory style index to mailing lists. This has overtaken others to become the best, most helpful place to start. Here is an abrieviated gateway: [2][2] Liszt is the second place to look, perhaps the more definitive. Again, there is a database and a subject directory. [3] The Directory of Scholarly and Professional E-Conferences, known also as the Kovacs Lists[3], is a long standing service in the world. Again, a fine subject-based[4] listing. [5][6] The Argus Clearinghouse indexes subject guides and webpages[6], but almost all refer to relevant mailing lists. Search several list directories for more rewarding results. Also keep in mind some lists have too little or too much traffic for your purpose. Find a list with a manageable number of messages and a wide enough membership. This takes a little effort in interrogating the list management software for the number of forum members, a look at past discussion, perhaps a look for supporting websites. Complete information on list commands can be found below in the strategy[7] section. The FAQ may be a brilliant informative document in itself, or the definitive pointer to further tools and resources. By virtue of its public origin, FAQs are far more likely to attract the peer review often very lacking from other resources. They are also open invitations to communicate with the knowledgeable FAQ maintainers. Searching FAQs by Name [8] Search for FAQs from an official FAQ archive like this FAQ database[8] at Oxford University. [9] Alternatively, the Universiteit Utrecht (Netherlands) expressly provides a search of FAQs and PIPs[9]. [10] If you know the newsgroup, consider visiting an html faq archive like this one[10] at www.faqs.org Newsgroups, (also known as Usenet Discussion or Network News), are large discussion grounds where resources and ideas are shared, and sometimes discussed. Messages are archived, available for searching or sifting. As a public notice board, non-commercial queries/briefs are often welcome. Internet Newsgroups (Usenet) (Network News) [11] You can get a good list of newsgroups from your own computer (search for news.rc - it should be in your newsreader directory). Perhaps wiser, undertake a word search of current newsgroup discussion with AltaVista or Deja.com (below), then focus on matching newsgroups. [12] Newsgroups are not carried everywhere; This webpage at Duke University will help you find additional newsgroups. Approach your Internet Service Provider to bring it in (a simple task). For low volume newsgroups there is an email alternative. [13] Liszt.com also maintains a searchable list of newsgroups. Searching Newsgroup Discussion can be very rewarding, but also can be a shortcut method to find newsgroup of interests you. Digital's Altavista allows searches of recent newsgroup messages. Deja.com has an even larger archive (to before March '95). [14] Deja.com[14] Search of Usenet Discussion Deja.com's Power Search[15] form is a must-see, and allows for author profiles and field searches and more. [1] Altavista[1] Search of Usenet Discussion. Library Associations are more involved than their Internet companion. Associations are also more into paper publishing, conferencing and colating specialist statistics. As an example, the Australian Booksellers Association publishes the best benchmark statistics on this topic. When approaching an association, consider asking for their publications list. Directory of Associations The definitive way to find an association is through certain large national directories. Internet alternatives are not nearly as valuable, but are more immediate. The [US] Encyclopedia of Associations, produced by Gale Research, is the definitive source for addressing and contact numbers to American Associations. All the primary libraries will certainly have a copy, as will many smaller libraries. Further description can be found for the database format thanks to SilverPlatter[16], Dialog[17]. The Directory of Australian Associations is the definitive Australian source for addressing and contact numbers. All the primary libraries will certainly have a copy, as will many smaller libraries. National Association Directories exist for many countries: Directory of Associations in Canada (further description thanks to SilverPlatter[18].) Directory of Association of Asia 1997/1998 by Bing Chang. Finding Associations Online A Directory of the American Society of Association Executives is online[19]. Unfortunately, the database is small & americanocentric. A search for 'book' did get me the address of the American Booksellers Association, but not others. [2] Of course, if you have a name, use a large search engine to find an address. We recommend a meta-search engine called Debriefing (www.debriefing.com[2]), as it also suggests a home-site. The last online source is a bit of work, but involves searching for associations which have published, by searching the large national libraries. The Library of Congress experimental search system[20] allows us to search for "association" as an author, and book as a keyword. Incidentally, this process is similar to searching for theses - search normally but add 'theses' to the query. Another prominent source are the local service directories, such as InfoLink in Western Australia. Most communities have a public directory of local associations and government authorities. If in doubt, ask a local librarian for directions to such a directory. Conclusion 3 Second Summary: Easily search past discussion via archives & FAQs. For value, search for the private, moderated forums. Associations are listed in national print directories. There are three important research applications for mailing lists.1) Research through past discussion, 2) Directly ask members for assistance, 3) Become a participative member to pick up and exchange information. On a personal side, mailing lists are easy to use and a minimal investment in time (the information comes to you). However, mailing lists are difficult to develop and maintain. Few reach the potential brilliance of this form of communication, so many of the forums you come across will be non-existent or on their death-bed. Mailing lists depend on four vital ingredients - Content, Participation, IT-support, and Management. Often, one of these go wrong and the forum dies. As a member, there are important obligations starting with participation, and ending with forum etiquette. The better forums are private. Membership is not automatic, the list manager has more control, and often, more control and effort is expended developing interesting content and discussion. If you find a closed or private forum, persevere. Associations When a group of like-minded individuals come together to achieve an aim, they often create an association. What better place to research. Even better, associations often interpret their purpose as a place to pool and distribute information. Larger associations often maintain a small library of their own and many associations publish documents about their area of interest. Furthermore, if you are seeking an expert in a given field, associations are sure to have one, or two, or many. For the smaller associations, be polite but firm in describing your interest and be ready to buy whatever small book they do publish in your quest for further information. The FAQ An FAQ is created to enhance the discussion of a newsgroup. After a time, the initial members of a newsgroup would have discussed many of the standard topics to death, which newcomers will still find interesting. To prevent only discussing introductory topics (and annoying long-term members) an FAQ is created to record answers to standard questions. Because one of the primary functions of a special interest group is resource discovery - and because FAQs are collectively created, they are valuable and generally reliable. I consider the Official Copyright FAQ[21] the best document in the world on copyright law. As an aside, many FAQs are also available as web pages. Trouble is, without an system to vet true newsgroup FAQs, you are far more likely to encounter FAQs which have not been vetted by the news.answers team. The Official Copyright FAQ is 70+ pages of topical and factual detail with links to further information. There are several other copyright FAQs with less than 10 pages, (and not particularly concerned with providing information). Access an established FAQ archive for your FAQs. www.faqs.org[10] has a small list[22] (but is elegant as a source of FAQs). Another longer list resides midway down this document[23]. Strategy Discussing the mailing list, I thought long and hard on how to simplify the task of communicating with list software. Not only are there five prominent list software packages, but each package allows us to accomplish different things. The email interface predates popular use of hypertext, and is a little clumsy at first - especially if you are interacting with different mailing lists as a researcher will. Our solution is threefold: [24]Firstly, James Milles of the Saint Louis University Law Library has graciously permitted us to include his grand table of Mailing List Commands[24] divided by list package. Very comprehensive and easy to use lookup file. Also found online[25]. Secondly, hypertext allows us to add information into the subject of an email message. With this in mind, we have added shortcut email links to our articles for the more common tasks. You must move the subject information into the body of the message, then post. Here are two examples: BusLib-l (Business Librarians' Electronic Discussion List) subscribe[26] | post to Buslib-l[27] | index the archive[26] | retrieve from archive[26] | subscribers list[26] | digest[26] | cancel digest[26] | unsubscribe[26]. * Access also available as the newsgroup bit.listserv.buslib-l[28](see Deja.com's usenet archive[29]). Libref-l (Government Documents List) subscribe[30] | post to Buslib-l[31] | index the archive[30] | retrieve from archive[30] | subscribers list[30] | digest[30] | cancel digest[30] | unsubscribe[30]. These are usually willing to field polite focused questions about your research project. For your convenience, this form will create the html used above for mailing lists of your choice. Save the file this generates for convenient use later. Your Name: List name: List address: Thirdly, retrieve the technical help files for the list software. Listserv - send help, info refcard, & info database to [email protected][32] Listproc - send help & help listproc to [email protected][33] Majordomo - send help to [email protected][34] Mailserv - send help to [email protected][35] Mailbase - send help to [email protected][36] This article comes from The Spire Project. Advice welcome : email [email protected] [1] http://tile.net/lists [2] http://www.liszt.com [3] http://www.n2h2.com/KOVACS [4] http://www.n2h2.com/KOVACS/Sindex.html [5] http://www.clearinghouse.net/searchbrowse.html [6] http://www.clearinghouse.net [7] http://spireproject.com/discuss.html#5 [8] http://www.lib.ox.ac.uk/Excite/AT-LAS_WWW_Serverquery.html [9] http://www.cs.ruu.nl/cgi-bin/faqwais [10] http://www.faqs.org/faqs [11] http://spireproject.com/discuss.html#snn [12] http://www.duke.edu/~mg/usenet/newsgroups.html [13] http://www.liszt.com/news [14] http://www.deja.com [15] http://www.deja.com/home_ps.shtml [16] http://www.silverplatter.com/catalog/eass.htm [17] http://library.dialog.com/bluesheets/html/bl0114.html#AB [18] http://www.silverplatter.com/catalog/daca.htm [19] http://www.asaenet.org/Gateway/OnlineAssocSlist.html [20] http://lcweb2.loc.gov/resdev/ess/booksquery2a.html [21] http://www.cs.ruu.nl/wais/html/na-dir/law/Copyright-FAQ/.html [22] http://www.faqs.org/#FAQHTML [23] http://www.faqs.org/faqs/news-answers/introduction [24] http://spireproject.com/mailser.htm [25] http://lawwww.cwru.edu/cwrulaw/faculty/milles/mailser.html [26] mailto:[email protected] [27] mailto:[email protected] [28] news:bit.listserv.buslib-l [29] [30] mailto:[email protected] [31] mailto:[email protected] [32] mailto:[email protected] [33] mailto:[email protected] [34] mailto:[email protected] [35] mailto:[email protected] [36] mailto:[email protected] ___________________________________________________ Excerpt from the Information Research FAQ 1. What is Information Research? I prefer to think of Information Research as an effort to locate answers, efficiently. Information Research is not vague browsing of available information for something that interests you, browsing the library bookshelf, reading the newspaper, nor is it Internet Surfing. Information Research is research with a purpose ... and it is hard work. Information Research is also an art form. The skills, tools, and resources we work with are only the canvass and paints of an artist. It extends from commercial, legal, reporting, through the skills of interviewing, database searching, and research analysis using books, articles, experts, patents. Research is so large a field, involving so many skills, tools and resources, you will quickly find you do not wish to learn it all. At the heart of Information Research lies a simple motto: "Someone, somewhere, probably knows the answer." To quote The Information Broker's Handbook (Sue Rugge and Alfred Glossbrenner): "As information brokers, we shouldn't consider ourselves capable of providing solutions... What we 'can' provide, and what sets a really good information broker apart from the rest, are resources. We can provide the client with the kinds of information he or she needs ... that make it possible for individuals to solve their problems." In this FAQ, I will try to inform you about this exciting field most people do occasionally, and occasionally do well. We include articles excerpts from The Spire Project to add depth and distinguish this field from the more general task, research. ___________________________________________________ 3. A Quick Introduction to Information Research Let us start with the four tenets of information research: 1) Researchers work hard to properly frame the question, 2) Researchers know the technology and know where to look, 3) Researchers drink coffee, and 4) Researchers ask for help. __ 3.1 properly frame the question Your question is critical. There is a galaxy of difference between a young student asking, "I am interested in trees", and a specific, attainable question like "Where would I find a tree surgeon I could talk to?" The information sphere is very large, and rather confusing. Each item of information has aspects of authenticity, accuracy, reliability, and bias. Information comes in many formats: interviews, books, articles, statistics. We learn about information from many sources: literature, discussion, resource lists, experience. There are also personal issues: budget, time, depth and purpose. With all this to think about, we must be very careful about each question we ask. This issue is vital once we start an article search, and can easily mean the difference between 5 concise articles, and hundreds of general articles. The essence of research is the manner with which we approach the information sphere. Research is an art, much like painting or photography. The true mark of an artist, and the primary step wanna-be artists miss, is visualizing what you want, before you begin. It is the same with research. Sit down and visualize what a successful search would look like in this situation. How many pages? How many documents? What kind of authors and what kind of quality of document? Go through the whole gamut of different types of research tools and describe it. Would a simple three-line newspaper article be a success? Would a 20-year-old dissertation be acceptable? Would a short conversation with an expert suffice? Would all three together suffice? (This approach works exceptionally well with internet research too.) If you can phrase a question in a way that lends itself to your resource, you are far more likely to get the answers desired. Oddly, this often means you are asking for places where the information resides rather than asking directly for the information. "Where do I find a definitive list of associations?" (or a search for "+association +directory") works much better than, "What association works with exceptional children?" What about, "Who would know of associations for exception children?" and, "Are there pamphlets of advice to exceptional children?" and, "What umbrella organizations/specialist libraries exist for exceptional children?". Questions are not right or wrong, just better or worse at illuminating certain aspects of the answer. Make sure your questions illuminate something useful. There are ways to frame questions for commercial databases, for research assistance, for interviews, for getting the truth from to your children. Your skill in phrasing the question has a lot to do with the result. Poor questions tend to come back and haunt us later when we miss relevant information. Try to set aside time to refresh and reframe your questions. __ 3.2 know where to look Information Research rests on understanding the technology and an awareness of the resources. In the example above, a directory of associations does exist. Here in Australia it is the "Directory of Australian Associations", found in most important Australian libraries. The Australian "Department of Education" has a major interest in promoting exceptional students. In Western Australia, Infolink, a community information service, should have a record of major community groups for exceptional students. I have no direct knowledge of umbrella organizations or specialist libraries, though I expect both the education department and Infolink would. A quick search of some large libraries may help us find some of the pamphlets but certainly not all that exist. Knowing of specific resources is helpful. Knowing the tools to help you find resources, the meta-resources, is vital. So what if we do not know exceptional students come under the Department of Education. Do we know who to ask to find the government department involved? If you do not know of the directory of associations, who or where would you look for one? Being unfamiliar with meta-resources is a serious handicap - you will find yourself searching hours for something a professional would do on the phone while drinking coffee. Keep in mind this website is dedicated to providing you some of this experience. Our research articles should suggest directions to look. This FAQ should help, but there are limits to what we can accomplish. At some point you simply must sit down with the Kompass Directory, or the Gale Directory of Databases, or browse the Australian Bureau of Statistics library, and become familiar with getting to all the relevant information. A researcher also needs much experience with searching electronic databases with complex research queries - a difficult task only made better with practice. As a general rule, if you don't use boolean search terms, you are doing it wrong. __ 3.3 drink coffee Researchers Drink Coffee. Internet connections work better at odd times of the day. As some information comes from international resources, and the internet is cheaper than direct dialing, researchers drink coffee. We also work too much on computers and have bad eyesight. __ 3.4 ask for help There is very little mystery about professional research. Lots of people are experienced in different aspects of this field. My personal weak point is in direct interviewing where as I am a pioneer in secondary resource research - and this is OK. In fact I use this liberally to determine the skill of professional researchers - do they know their own limits? The field is much too large to be an expert in all its aspects. The positive site to this is many many people welcome requests for help. I enjoy asking librarians questions. I also ask my customers, my suppliers and other professional researchers. Never get caught in the trap of feeling you know what to do. The joy in this profession is that most people do not expect you to be an expert in their field, just an expert in your particular field: particularly the meta-resources. Even if it requires a polite reminder, customers will appreciate you asking them for likely keywords in difficult searches. I always make a habit of asking librarians if I am missing something. A librarian is always fluent in their collections and I frequently locate real gems this way. (As an example, my state library arranges computer books in two sets, one dewey and another in an alternative structure. Without asking a librarian I would miss so much.) Even if you are just a student, always keep your ears open. You will frequently find yourself in the presence of someone expert in some facet of research telling you something you already know. Consider carefully before you interject... your expert may be about to explain something new to you. In summary, information research is a dedication to learning. At its heart is a collection of specific research skills, an awareness of research tools, and a gifted mind. - Oh, and a large amount of coffee. Without knowledge of and access to relevant research-worthy resources, your research will be severely limited and doubtful. This is why much of your work becoming an effective researcher involves learning about the resources and meta-resources for your field. Much of our work in The Spire Project is drawing your attention to relevant resources. A large list of resources comes next. ___________________________________________________ ___________________________________________________ ----- Acknowledgements I would like to thank my wife Fiona, whom I love and cherish. The spire project is the culmination of several years bridging information research and internet development. We trust this guide to web research enriches your efforts on the web. The spire project (http://cn.net.au), including this faq, forms the most advanced information guide today. Thanks to the many readers who assist in building and refining this information. ___________________________________________________ Legalities: Information and data put forward here are supplied in good faith and entirely without expressed or implied warranty or fitness for use. The content of this FAQ is simply a collection of information intended but not promised to be correct and useful, gathered from many sources with limited editorial checking. Further, included articles are the thoughts of the authors alone and may not represent the beliefs of Community Networking or any sponsoring organization. If you find a mistake or claim copyright infringement, contact David. Copyright (c) 1998 by David Novak, all rights reserved. This FAQ may be posted to any USENET newsgroup, on-line service, website, or BBS as long as it is posted unaltered in its entirety including this copyright statement. This FAQ may not be included in commercial collections or compilations without express permission from the author. Please post permission requests to [email protected] ----------------------------------- David Novak - [email protected]
Закладки на сайте Проследить за страницей |
Created 1996-2024 by Maxim Chirkov Добавить, Поддержать, Вебмастеру |