Exploring the Invisible Web

The invisible web hides in plain sight. There is no dividing line between the visible web and the invisible one. It's all out there. We've always known that we couldn't reach some sites through general search engines; that we had to go to them in order to search them...Amazon.com, eBay, library catalogs. But until recently, no one realized how large this body of documents was, and how little of the visible web the general search engines could index. A study last summer by Bright Planet states that the inaccessible part of the web is about 500 times larger than what search engines like Alta Vista, Google, Hotbot provide access to. What kind of information are we talking about? Topic-specific databases with specialized content such as company information, census data, government documents, reference works. Material with special formats, like pdfs, sound sites, video files. These sites are searchable but not by a general search engine.

Where to go first: Gateways to the Invisible Web

  • Direct Search Impressive list of databases drawn from sources invisible to general search engines. Compiled by an academic librarian with emphasis on research
  • Google Although a general search engine, Google has developed a way to search for the .pdf format, using the code inurl:pdf after the keywords in a search
  • Those dark hiding places The invisible web revealed
  • Librarians Index to the Internet Information from 8000 web resources, annotated, compiled by librarians, including only the "best" in categories

    Some Very Interesting Databases Out There

  • Dauphin County Library System You will have to indicate which library you have a card with. Go to their site and click on the Power Library icon. If you have a library card from this library, use that barcode to log in. No card from another library will work. Ebsco's vast holdings of magazines and newspapers, ERIC, AP's Photo Archive which is called AccuNet/AP...all free and you can search over a wide range of them
  • York County Library System Look for York County Libraries, and then About Us. You will get a list of library links in this system. Click on the Red Land Community Library, click on the Access PA icon and use your library card's code. Here is the site for The Cumberland County Library System, and here's the site for The Lebanon County Library System
  • FedStats Good site for government statistics from over 100 government agencies
  • Firstgov.com The U.S. government's search engine for its vast network covering local, state and federal government sites
  • GPO Access This is a service of the U.S. Government Printing Office and gives access to a wealth of important information produced by the federal government. Allows you to search multiple databases from a single page.
  • The Virtual Technical Reports Center Links to institutions that provide technical reports on the Web.

    Great tool for journalists!

  • Sources and Experts Compiled by a researcher at the St.Petersburg Times News for journalists, a strong preference was given to those organizations that make it possible to search for experts by area of specialty, that provide contact information (phone and e-mail) and that publish the expert's credentials.
  • Research beyond Google Remember "the invisible web"? The invisible web comprises databases and results of search engines that popular search engines are not able to index. This site is an annotated list from 2006 of authoritative websites on the invisible web. Check it out.

    Sources used in a 2001 staff presentation by librarian, updated

  • Invisible Web This was a valuable starting place on about.com. Much of the material was compiled by Chris Sherman who is writing a book with Gary Price on the invisible web.
  • Discovering the Invisible Web A presentation made by Kay Benjamin on September 29, 2000 for the South Central Regional Library Council.
  • The Invisible Web: Database contents rarely found in search engines Very helpful from Teaching Library Internet Workshops, University of California, Berkeley.
  • Research Wire Exposing the Invisible Web A column on LLRX.com by Diana Botluk.
  • The Invisible Web Gets Deeper By Danny Sullivan from The Search Engine Report, August 2, 2000.
  • Search Engine Watch Very helpful for an understanding of how search engines work in any number of articles on the site.
  • The Internet's Hidden Content and How to Find it Online, May/June 2000, by Bonnie Snow.
  • Spotlight on the Invisible Web By Michael Dahn, Online, July 2000.
  • Google Does PDF & Other Changes From The Search Engine Update, Feb 6, 2001, subscription area.

    Presented by Dee Mills, Chief Librarian at The Patriot-News on March 15, 2001.

    Updated 10/28/08

  • Return to Librarians' Links