In our effort to track down information on the Web, we employ an arsenal of
tools. We may browse the handpicked sites of a directory like Yahoo! or the results of a search engine like Google. Some cast an even wider net by using metasearch tools like Search.com or ixquick. And yet we still don't always find what we need.
This material has been referred to as the Hidden Internet, the Deep Web or
the Invisible Web. The Invisible Web is comprised of information stored in
databases, according to Chris Sherman, Webmaster of About.com's
Web Search. Spiders and robots cannot enter these
"It's as if they've run smack into the entrance of a massive
library with securely bolted doors," Sherman said. "Spiders can record the
library's address, but can tell you nothing about the books, magazines or
What else makes up the Invisible Web?
Non-HTML files (PDF files, etc.)
Sites requiring registration or login
Archives (newspapers and magazines, etc.)
Dynamically created Web pages
Interactive tools (calculators, etc.)
Thankfully, gateways have been developed to mine these hidden troves. Think
of them as database or search tool finders. Here are a few of the best:
Maintained by librarian Gary Price, this site provides direct access to the
search interfaces of Invisible Web resources.
A Yahoo!-like directory of more than 10,000 specialized databases (business, legal, travel, etc.). Includes free- and fee-based resources.
Search forms and direct links to hundreds of search engines, from general
purpose directories to niche topic indexes. Categories of databases include
Reference and Research, Medical, Maps, Legal and Government.
Researchers share their secrets to searching the invisible Web at the