The truth about the deep web

By Greg Stevens

The internet has never been decentralised. One of the most pervasive and popular myths about the internet is that it cannot be controlled, censored or shut off by any single action or source, because it is a “distributed network”. Those who understand how the internet works know that this is not, and never has been, the case.

This is why Prime Minister David Cameron’s threats of mass internet censorship in the United Kingdom have real weight behind them. From a purely technical standpoint, it would not be difficult at all to monitor, censor or even shut down vast portions of the internet at the government’s command.

At least, that is true for most of the internet. There are places out there – networks in cyberspace separate from the places where most people usually go – that are not so easily controlled. Politicians do not approve of these places. The media enjoys using vague and mysterious labels such the “Darknet” or the “Deep Net”, and Mr Cameron has said that he intends to “crack down” on them.

Depending on who you ask, networks such as these may either be a disgusting moral sinkhole where the worst dregs of society fester and thrive, or the future salvation of the internet.


Freenet is one of the networks that makes up the “Darknet”. It was created in the 1990s by Ian Clarke as a way to share and consume information anonymously in a completely decentralized way, so that no organization or government could easily censor it or shut it down.

“I started by asking myself, what would it take to build the type of system that everyone thinks the internet is already?” Clarke explains that his motivation for creating Freenet was partially political and partially academic. He was studying complex dynamic systems, also called emergent systems, when he realised that the properties of these systems were exactly the properties that a network would need in order to be truly beyond the reach of government spying and control.

Complex dynamic systems are most well-known from the study of artificial neural networks in artificial intelligence, or the study of swarms and flocks of animals in the study of artificial life. In these systems, a large number of relatively simple parts can accomplish very complex goals simply by the way that they interact.

These systems are remarkably robust. You can damage large groups of brain cells, and the brain finds a way to compensate and keep functioning. Disrupt a flock of geese or an anthill, and they adjust and continue on their way. There is no single location or “switch” in these massively distributed systems from which everything can be controlled or monitored.

Clarke says that this formed the inspiration for the underlying architecture of Freenet. By having a system where all information was shared and stored through a true network of local interactions between nodes, instead of there being any core set of “central databases” or “central servers”, the Freenet becomes almost impossible to censor or “switch off”.

To look at it less philosophically for a moment, the way that you get on to Freenet is by installing special software on your computer. It acts as peer-to-peer software, setting your computer up as a “node” from which you can connect to other nodes in a Freenet network.

Politicians don’t want to target the Darknet simply because it harbours illegal activities that, after all, exist in plain view all over the internet. Politicians have a problem with the Darknet for a different reason.

How do you get started? You simply have to know at least one other node in a Freenet network. If you have a friend who is on Freenet, you can connect your node to your friend’s node, and you are part of the network. Or, there are “seed nodes” that you can use to get started. Once you are connected, you are part of the network.

Information is transmitted, stored and shared in a completely distributed way. In the same way that information about a party might be passed from friend to friend in a large social group, information is passed between nodes in Freenet. “There is no company, there is no central server that is Freenet,” Clarke says. “Freenet is the collective behaviour of a large number of computers that all collectively form this network.”

Clarke’s political motivation was simple: freedom of speech. “I believe in the concept of democracy. I believe that a government should be regulated by the people it governs. If you think about what that implies, one of the things it takes for the governed population to effectively regulate its government is they’ve got to know what’s going on. And so, I believe at a fundamental level, people’s freedom to exchange information and ideas is fundamental to the legitimacy of democratic government.”

Anonymous networks may attract freedom-fighters and political dissidents, but they will also attract child pornographers, and weapons and drug dealers. When asked about this, Clarke suggests that it is simply a reflection of what it means to have free expression in society.

“Freenet is like the internet,” Clarke insists. “A system like this, much like any tool, becomes a reflection of society as a whole. Society has good aspects and bad aspects. I think Freenet is very similar. There are certainly are people who are using Freenet for disseminating political information that might otherwise be censored, and then there are also people who use Freenet to distribute pornography and hate speech. But the same is also true of the internet more generally.”

But then he adds: “Because of the anonymity, Freenet can be a little more extreme.”


The media’s coverage of the “Darknet” is, frankly, disgusting. It is overly-sensationalized and histrionic. When addressing the topic of the Darknet, or the related “deep net”, writers can’t seem to help themselves but to pepper their prose with words such as lurking, underbelly, lawless, cloaked and swamp-like.

A report from PC World practically fawns over the scandal of it all: “You’ll quickly find links to credit-card scammers, forged documents and currency, weapons dealers, gambling sites, marketplaces for every vice imaginable, hacker havens, the types of illegal and disgusting porn that get chased off the Surface Web…”

Never mind that it’s impossible to tell at a glance how many of these illegal products are actually being sold, and how many are scams posing as the real thing. Never mind that there is no way to tell how many of the people bragging knowledge of illicit activities may actually be pimply 13-year-olds passing off second-hand stories because it makes them feel cool.

Certainly, never mind the fact that credit-card scammers, gambling sites, hacker havens and weapons dealers exist in abundance on the “Surface Web” as well. As Clarke observed, the Darknet is a reflection of society. But strip away the media hype and evil-sounding metaphors, and the Darkweb is essentially just the internet – but more extreme.

Politicians don’t want to target the Darknet simply because it harbours illegal activities that, after all, exist in plain view all over the internet. Politicians have a problem with the Darknet for a different reason.

Remember that although “dark” can imply shady or unsavoury, it also has a much simpler meaning: not illuminated, or difficult to see. That is essentially the problem that politicians have with Darknet: it is difficult to monitor and control.

As part of Clarke’s quest to protect the freedom of speech, Freenet has the option of operating in a highly secure mode that it refers to as “Darknet mode”. When operating in “Darknet mode”, Freenet nodes will only connect to a known and specified list of “trusted nodes”.

A Freenet network operating in this secure mode will still be able to grow, and information will still be able to pass across the entire network. However, information will be forced to pass through short, local “hops” between trusted connections. No computer will be able to connect to every node in the network and monitor them all.

This makes the Darknet operation of Freenet different from most peer-to-peer networks, in which nodes will connect to any computer that wants to connect to them. Computer scientists whimsically call these “promiscuous” nodes. Networks with promiscuous nodes are easy to tap into and monitor, because a government agency could easily set up a node that connected to every other node in the network, and then sit back and take notes.

Clarke’s idealistic vision for the purpose and goal of Freenet is really the idealist’s argument in favour of the Darknet more generally. “I don’t think that a democratic government should be in the business of restricting how people share information, because sharing information is the way society thinks collectively. Just as most people would recoil at the idea of a government telling you what to think as an individual, I don’t think government should interfere with a society’s ability to think, as a group.”

This kind of talk is somewhat reminiscent of paranoiacs and conspiracy theorists, so I could not help but ask Clarke whether he really thought that the US was in danger of an immediate dictatorial crack-down on free speech on the internet.

“To be honest, I’m a lot more concerned about European countries than I am the United States,” he said. “I think that in the United States, people really do believe in the first amendment, for the most part. People really do believe that freedom of speech is a fundamental right.

“But I’m concerned about the UK, which is where I went to university and lived for many years. They are actually instituting an internet censorship system already.”


Ironically, a story in Forbes last week describes Freenet – or a system very much like it – as the only possible hope and salvation for the future of the internet. The argument from Forbes, however, has nothing to do with censorship or the freedom of speech.

According to this article, one of the biggest problems that the internet will face in upcoming decades is the problem of scalability. Because the internet is not distributed, it must rely on ever-increasing power and efficiency in a small set of central data centres and servers to support the entire infrastructure.

The solution? Radically change the way that the internet works, so that instead of relying on a small set of datacentres through which most information is passed, it acts as a truly distributed, peer-to-peer network. The article cites a project called Pursuit Internet, funded by the European Union, which has been taking steps in this direction. They are described as having achieved a “proof of concept”.

Freenet, of course, has been doing this since 2000.

“It’s a general frustration of mine,” Clarke says, “that while Freenet has achieved a vast amount of attention in the mainstream press over the years, our work has often been ignored or misunderstood – and then duplicated – by the more academic community.”

He explains that Freenet was specifically designed to be extremely scalable from the beginning. “Freenet’s routing relies on it forming a small-world network, which has no dependency on a small number of nodes doing most of the work.”

So perhaps in the end the motivation for a decentralised internet will not be ideological concerns about freedom of speech, but a much more mundane motivation of the problem of scale.

Either way, however, Clarke may end up living to see a day when the internet is more like Freenet, or as he imagined from the beginning: “the way everyone thinks the internet is already”.