Advice for HackerPenetration Testing

What is the Deep Web and Why Is It Worth Exploring?

The words “deep web” and “darknet” are sometimes used to refer to the same thing, even though they are very different things. The simple answer is that most people only use the “surface web,” which is made up of the most popular and well-connected websites on the Internet. This is why “deep” and “dark” are sometimes mixed up. So there could be a lot of private websites that no one has officially classified or “indexed,” like how Google and Bing do by crawling public websites. Only a few people might have seen them.

Some individuals have likened the deep and dark web to trying to fish in the ocean, or perhaps an iceberg that rises to the surface but actually goes thousands of leagues under the ocean. As far back as 2001, the deep web was said to be “orders of magnitude” larger than the surface web, and modern 2016 reports suggest that hidden websites and databases are 500+ times greater in number than everything we see on the surface web.

Now theoretically, there are millions of websites that you would ordinarily never find unless, of course, you were able to penetrate the deep and dark web.

What we see on our favorite sites (Facebook, YouTube, etc.) could be called filtered media. You will have access to only filtered information and news information, as well as entertainment pages, which have been carefully selected for your viewing. This is a similar practice to how executives plan network or cable television.

Search sites like Google and Bing may be able to link you to obscure pages that have very low popularity, thanks to a longtail keyword search such as “Historical Records from the 1800s.” It may not be a well-known page but you can still find it with some diligence.

Sites like LiveLeak and 4Chan are known for sometimes posting controversial content that is later investigated and taken down, but even these sites are still considered the surface web, because they have high traffic (and many people who can report illegal or questionable content) and because many of their pages are searchable.

The deep web refers to sites and pages that have been intentionally blocked from any search engines indexing the content. Also called the invisible or hidden web, these pages and sites are usually not illegal or even dangerous; they’re simply not made to be searched. This could include anything from non-public entertainment, or perhaps private content that is password-protected, such as online banking records, webmail, pay-on-demand video or subscription magazines, and various medical or legal documents that are considered non-public information.

It should be noted that there are two ways to avoid being indexed by a search engine; either manually install code (or download an automatic plug-in) that blocks search robots from crawling the webpage, or do not externally link the page (or site) to any other external pages.

Search engine bots find new sites by following a linking trail, page to page, onsite and offsite. Even a website that doesn’t advertise and that avoids registering with any public search engine or domain name can still be found via search if it is linked to other public pages.

Deep web pages cannot be searched and thus offer greater privacy than public surface websites. However, these websites still answer to authority since they are hosted by well-known internet service providers that offer private or shared server hosting. While they do have privacy, anything you see on these pages is far from anonymous. There is a long paper trail. Deep web pages can limit access by these technologies:

  • Registration and login required
  • Unlinked pages
  • Non-HTML content, such as multimedia with highly specific file formats
  • Scripted content that is only accessible through JavaScript, Flash, or another special scripting.

In simpler words, the deep web is still theoretically possible to surf, if inconvenient.

The Difference Between the Deep Web and Darknet?

What differentiates the deep web from the “dark web” is that the dark web is hidden from the public Internet and is not accessible at all without special software. You cannot accidentally stumble onto the Dark Web. You must take deliberate steps to download the software.

Darknet refers to an overlay network with restricted access, one requiring the use of different communications protocols and ports in configuration. There are various darknets you can use to access the “Dark Web”, the otherwise impenetrable web you’re not supposed to find.

Dynamic content or contextual webpages, which are only returned through a specific form, query, or access context Search-prohibited code, such as Robots Exclusion programming, or even using CAPTCHAs to disqualify searching bots You may have heard of file-sharing software programs like Napster or LimeWire some years ago, and these are examples of one “darknet” type; a peer-to-peer network of users that share files. The “Deep Web” tends to encompass the darknet but the main difference is in the deliberate steps you must take to connect to the darknet.

The idea of the darknet actually precedes the “Deep Web”, since the Darknet concept was devised in the 1970s, and referred to networks that were kept apart from ARPANET, which later became the worldwide connected Internet. The Deep Web resulted because of the need for resistance to an all searchable Internet. The Darknet concept simply evolved with the preference for privacy and more exclusive traffic.

The main difference between Deep and Dark is simply that a darknet doesn’t have the classic “searchability” or even the compatibility to be loaded by unsuspecting users. If you tried to load darknet URLs provided by underground sources, you would not be able to load them simply by clicking the link in Internet Explorer, Firefox, Safari, Chrome, and so on. The darknet is sometimes considered a portion of the Deep Web in technical terms, but in semantics, it is something far more guarded.

Later in this book, we’re going to review the best ways to connect to the Dark Web as well as which ways are safest, which come with a risk, and which programs are ideal for surfing anonymously.

Why the Surface Web is Extremely Limited

The question many people have when finding out that a “dark side of the Internet” exists, is why would you want to search for information that is considered “secret”?

The easy answer is that the surface web is extremely limited in terms of providing information, and much in the same way as a TV network plans its schedule, there isn’t much “exploring” to do on the surface web.

The surface web is highly guarded and filtered for public use. While some people from English- speaking countries may think that Darknet software programs are only useful to those living in communist countries, where the government controls Internet viewing, consider the facts about capitalist or socialist countries.

Google and Bing, as well as other English search engines, have all been known to manipulate their keyword search results for esoteric business reasons. Many believe that smaller websites are being suppressed by larger search engines in favor of well-known brand-name businesses. Furthermore, many of the news stories you see on the surface web are presenting one-sided information, as this stands to help various commercial or governmental purposes.

Part of the excitement of exploring the Dark Web comes from the idea that you could be reading stories or web pages that present a completely new perspective of life. Remember that in order to host a website on the surface web, most of the content published must be considered generally “mainstream”.

No hosting provider will allow a user to host content that is considered too taboo, controversial or “alternative” in viewpoint.

Knowing this, it comes as no surprise that many users search darknets for underground political news, especially since many whistleblowing websites are now being censored from the surface web. Some hackers dream of being the next “big whistleblower”, and imagine what information they might find by searching “secret” files that no one is supposed to read.

Some users prefer using a darknet to protect their own privacy and escape what they consider “mass surveillance” from invasive surface websites as well as the government.

Lastly, yes, the Darknet is sometimes used for illegal purposes, which might include hosting illegal stores, committing hacking crimes, sharing illegally acquired multimedia (new music and movies), or even distributing illegal pornography or scenes of a real-life murder. There is also much in the way of counterfeit software, identity theft, and spam operations.

Last but not least, some enjoy surfing the dark web simply for the joy of finding something new and unexpected. Many sites on Darknet are harmless and somewhat trivial excursions into niche topics. Some web hosting users simply don’t want to be searched or indexed and prefer anonymity. Others use the Darknet market to discover hard-to-find books that would probably not be published on Amazon,, and other traditional stores.

BBS and Usenet Predecessors

Years ago in the 1980s and 1990s, Bulletin Board Systems were the predecessors of the online world. These were computer servers running custom software allowing anyone with a PC to connect to the system using a terminal. Once logged in, users were given access to a small community of users. Here, they could chat with each other, play games, read news items, and share content without any third-party company presiding over the process.

The BBS “Systems Operator” presided over the cyber-community using just a computer server, software, and a phone line that directly connected users to the BBS. Sometimes the “sysops” who owned the BBSs would list phone numbers to other BBSs that you could call and connect to, for more exploring, file trading, and news sharing.

Usenet groups were the next hike in technology and were repositories for discussion groups, connecting users from multiple locations and allowing file downloads and uploads and text discussions. Unlike the BBS or the modern-day web servers, Usenet technology did not require a central server or dedicated administration. It existed via a forwarding news system that made use of multiple servers.

The deep and dark web essentially relive those old Usenet / BBS days, where users had access to an unfiltered, unregulated, and largely unknown online world free from filtering and corporate-sponsored ads.

Is Surfing Darknet Legal or Illegal?

In surfing the Dark Web, you may discover something new that no one has ever seen before. You may find out something you never knew before or may catch a glimpse into an alternative lifestyle or political/religious point of view that you never knew existed.

You may stumble upon some illegal content, unfortunately, but there are ways to avoid that as we will discuss later. Lastly, you may and probably will discover many, MANY dead links. That’s all just part of being an explorer into deep undiscovered territory.

The thrill of exploring the darkest parts of the web comes from venturing boldly into the unknown. Perhaps it’s the closest we human beings can imagine to exploring space or the deepest part of the ocean. Half the thrill is searching for something undiscovered or stumbling upon a truly unique or mysterious page.

There is absolutely nothing illegal or unethical about exploring the darknet with a curious mind. You have every right to download software for exploring these “locked sites” and must use discretion if and when you do find something illegal. When in doubt, it’s probably best to leave the page behind. And for more information on “what can go terribly wrong” be sure to check out our final chapter, which shares some of the more outrageous but still possible horror stories of deep web exploration.

For now, let’s proceed to the first chapter which discusses the only way to surf anonymously and safely if you’re venturing into Darknet.

William Keener

William is a tech writer at Hacktoday. He's been writing how-to guides for about 4 years now and has covered many topics. He loves to cover topics related to Windows, Android, and the latest tricks and tips.

Related Articles

Back to top button