Many people don't know much about the Deep Web, often confusing it with the Dark Web (you don't want to go there!). Sometimes called the Invisible or Hidden Web, this mysterious place is estimated to be several orders of magnitude larger than our everyday surface Web.
In 1999, a colleague and I went to the Emeryville offices of Ask Jeeves to explore partnering with them for our startup. We were astonished to see dozens of people sitting at terminals doing research and answering questions. They had designed an elaborate database matrix to track and link everything, isolating all types of question prefixes (which was giving them a headache at the time of our visit). When we were leaving, we looked at each other and said something like, "This will never scale."
I've been interested in how information is published online ever since that encounter opened my eyes to the power of databases. In the simplest terms, that is what the Deep Web is all about. Good researchers know how to search virtually bottomless repositories on practically every subject under the sun. Their secret is knowing how to get into the databases.
I have a master list (my own database!) with hundreds of entry sites and I'm sure my list pales in comparison to the complete one, if anyone has it. Since I starting keeping track, nearly half of the sites are now dead links. Some companies have cashed in on the Deep Web by offering access only by subscription. Some require academic credentials or they require signing up with email info. But there are plenty of open sites, too.
Lately, I have fallen into the habit of visiting the same few "surface" sites and using standard search engines to find what I'm looking for. (I confess, my most common searches are for "free" and "how to".) When I get into these slumps, I go treasure hunting. That is how I found new inspiration from old books with hummingbirds and Cuban illustrations to restore. A few weeks ago, I found some African scenes and gave them new life.
I realize we are sophisticated people here so Deep Web research is probably not new to you! But I love to hear what secret research sites people know about, so please share your tips if you are willing to!