The TAD!

Once upon a time, a man named Tav came up with a system called the Plexnet, and had a heck of a time explaining what it is. After several patient years, I think I vaguely understand what he's getting at, and so this document both encapsulates my understanding and serves as a guide for people working on the 24 Weeks project to implement parts of the system.

The Plexnet Unfolded

A rapid, five minute explanation of a product is called an "elevator pitch" in America; in Britain I suppose it's called a "lift oration". What follows is what I'd gabble to you if you were going up a very tall skyscraper with me.

The Plexnet is like an Operating System that lives on the internet. The contemporary idiom that it most resembles is the much overhyped and touted WebOS, only the Plexnet rewrites a lot more of the fundamental architecture so it's more of an InternetOS. The Plexnet works by evolving and rewriting many existing technologies, such as HTTP and P2P, in such a way that they're all designed to better work with one another.

The aim of the Plexnet is to enable various simple but effective things that can't easily be envisaged using today's technology. For example, take something as simple as Flickr—a website for sharing photos. Before Flickr there was the Web; and before the Web there was the Internet. Though the Web and the Internet are grand and very abstract technologies, they enable lots of very small and interesting things to happen such as Flickr. In the same way, the Plexnet is quite a bit and abstract system, but it's specifically designed to enable more of these small and interesting things to happened. As such, it does help to keep in mind the goals when trying to understand the Plexnet as a system.

The system itself can be divided into seven specific technologies, which together we're calling The Plexnet Heptarchy. Each of the technologies in the Heptarchy resembles an existing technology to some degree, only crafted in such a way as to make it work better was part of the wider system. So for example, Plexnet Storage is a bit like current P2P technologies, but it focusses on only being efficient for what's demanded of it by other parts of the system.

It's very important, therefore, to understand how all of the technologies in the Heptarchy work together as a system; merely understanding a technology by itself is not enough to implement it effectively! Let's take a look at the overall system in all its glory before looking at the individual technologies in the Heptarchy.

Plexnet Diagram

Here's a diagram of how the components interact:

Plexnet Entities: Entities are cryptographically-secure private databases that store information created by a single agent, individual, or organisation &c. Their backend is basically just a queryable set (the Entity itself) of hashtables (called Units). The information stored is metadata, so for example Entities should not be used to store binary data.

Units only store property/value mappings. They may be made more useful with Schemas, which are like rules or contracts that affirm some condition, such as "this Unit should have an 'age' property; if it doesn't, consider the 'age' property value to be 0". A Unit that has one or more associated Schemas is called an Object.

Entities may be queried to return the set of all Units that contain a specific property and value pair, for example all the Units containing {age: 18}. These resulting sets are called Indexes.

Plexnet Events: Any dynamism in the system, anything that changes such as a Unit being modified or so on, will have an associated sensor. If that sensor is on, it'll trigger an Event with any registered listener. As with Units, the Sensors may have conditions on them.

Plexnet Storage: There are four verbs in the Plexnet Storage mechanism: GET, PUT, EXISTS, INFO, and DELETE. These accord to the HTTP verbs, except for EXISTS which is like a simple status request, and INFO, which is like HTTP's HEAD. These verbs work on identifiers, which are combined (md5, sha1, sha256, tiger, whirlpool) digests of the content.

The storage system is backend agnostic: it can be implemented over anything from the filesystem to a distributed hash table.

Plexnet Services: Plexnet Services are essentially simple remote procedure calls, allowing you to access functions on other systems. For example, there may be services to convert between currencies. The services will use the unix philosophy of being small modular units which emit straightforward easily parsable typed output, which can then be linked up to make larger services.

Calls to these services will be embeddable inside documents and so forth, to facilitate the inclusion of live data.

There are other parts of the Heptarchy too, but, it's difficult to document them because of several problems...

Problems and the Anti-TAD

There are several specific problems with the design of the Plexnet, from the smallest design details up to the overall rationale of the Plexnet itself.

1. Names of things in the system are not clear. We call cryptographically secure private databases "Entities", which makes no sense. What are Units? What is Disco? Not what you think they are, that's for sure!
2. There's too much to do. Even the storage layer would take a lot programmers a long time to accomplish, even if they has a specification for it to implement. But to innovate takes time, and to implement takes time. It's a subjective judgement, but even a single component of the Plexnet would take too long to implement even if there were several programmers; and in fact there are none.
3. Too many details are hashed out already. There is information, for example, on how to get around the thousands of files problem in storage nodes. But there's no consideration of why this matters; what about leveraging off a database system instead? The details are being designed before the larger level choices.
4. It's not clear how the Plexnet drives the Toman system. The Toman System could be implemented with perhaps a half dozen programmers working on a Reddit/eBay/Craiglist sized site. Why is the Plexnet needed when it seems possible to build it using conventional technology? tomanr.com!
5. The Toman System isn't revolutionary. People already rate transactions. We already have currency. Contracts exist. The commons exists. There's no barriers between these parts. Having a central site to do all this on might be useful to a small extent, but it won't enable a global revolution.
6. People don't innovate, they concentrate on details. By leaving most of the details of each of the technologies mainly undefined, it leaves the door open for people to squabble about implementational details whilst ignoring the overall design; this happened massively with the Semantic Web, much to its detriment. Even if you appoint someone to quash it, it can take a lot of work; and TimBL hasn't succeeded, for example, in his role of spearheading the Semantic Web.
7. Documentation isn't accessible to enough people. There's no point in hiding a system when hardly any of it is designed yet. If after seven years of exposure you can't get enough people interested to build a system, bringing it behind closed doors won't make people suddenly interested; it'll just stop any opportunity whatsoever of people being interested.
8. The system as a whole doesn't work because it's not designed. Insofar as the Plexnet is specified, it doesn't do anything. What's missing are usecases and requirements. What specific things can someone accomplish on the Plexnet, and what bits of the technology do what, to a quite fine grained level, to acheive that? Nobody has outlined a single thing, unless the Toman System counts. "Great things", or "revolutionary new messaging protocols" or other handwavings don't count because, well, they're handwavings.

The biggest of the enumerated problems is 8), the accusation that the Plexnet simply doesn't do anything in particular. It's a lot easier to disprove than prove, but the fact that not a single Kalati or Plexnet design document that I have to hand describes a use case is a warning. There need to be not only several use cases (and it sounds like there should be several hundred, given the lofty goal of the system: to change the world), but also intelligent technical sketches of how each of those use cases will be achieved.

Though to a great extent this is really Tav's job, here are some possible ways of approaching solutions to the eight problems listed:

1. Use clear names. If entities are really databases, then call them databases. If they're not quite databases, call them something that's like "database". Or go more specific about their components and call them "hashtable sets". Anything but entities! Repeat for all technologies. There should be people debating things like this but there aren't. We need to encourage that kind of culture. We need more participants.
2. Focus on only the essentials. When we have zero coders and no specification, we cannot create the Plexnet. We need to concentrate still on getting people interested. Once people are interested, we need to either a) work on a very, very ramshackle system; or b) work very slowly for many years doing each part in a merely ramshackle way. The journey is probably going to be more valuable than the goal; so document that and enjoy and learn and share.
3. Don't hash out the details before the general system. Just don't. Do things in their proper order; and listen more to the inputs and criticism.
4. Describe how the Plexnet drives the Toman system. This is something that nobody but Tav can do; if he wants to do it, he's going to have to sit down for just a few hours and give us a few thousands words on it. He rarely does that. It is hard to do tedious specificationeering. But if he wants this to succeed, then he has to do that a lot more.
5. - No solutions thought of yet for #5.
6. Don't rely on people to innovate. Rely on them to work out the details! Tav's the innovator, and yet he's encumbering himself with the details. Again this links into the point above; Tav's going to just have to be more disciplined about this.
7. Make all proceedings public. Value and treasure the data that we create. Have a single ESP website and encourage creation of information on there. Put everything the project has ever done on the website. Mirror the data as much as possible. Spread links to the data. Do nothing behind closed doors.
8. Design use cases first, technical walkthrough of how the use cases can be answered second, and then design the system based on those walkthroughs third. This is rather trivial level system design here, and you can kinda see how not following for eight years or so doesn't get you very far! I think that this is actually a much larger problem, but this is one possible way of fixing it; there are likely others, and this one mightn't even succeed. We need to encourage a culture of thinking about and tackling this problem. A lot of the motivation for that needs to come from Tav.

Note that one of the metaproblems that has sprung up is that the culture really isn't very strong at the moment. We have a good website and a good project going on, but not much in the way of people sitting down and really interchanging ideas and details like mad and recording them in the public web. Basically we have no frenzied set of intelligent people in MadScientistMode, which is what you need for a project like this to work.

Note also that there's an order to which these problems must be tackled, and the ordering above is merely as I thought of them. We'll have to discuss somewhat the order in which they need to be tackled, but for example it's no good fixing the names before the system is designed, and for the system to be designed we need to work on the whole use cases thing, and for us to do that we need to fix the culture and get people more interested. Writing this TAD and raising issues and causing a controversy actually seems like a good way of doing that, so we're already making a start by admitting that there are problems. There are also some things that can be done without any dependence on order, such as making all the proceedings public and setting up a decent central website.

So, in summary: 1) we need to have a better culture, and more programmers, which includes openness of proceedings; 2) we need to develop a more use case oriented Plexnet, perhaps (other solutions are just as likely), for people to be able to understand what it is and in order to be able to develop it, the problem being that at the moment the system is undesigned at the high level; and 3) we need to think about all the other problems such as how to surmount the insurmountable, how to stop the bubbling up of detailism, and we still need more information from Tav about his vision.

Social Architecture

See the SocialArchitecture document on the 24weeks wiki for a further continuation of the ideas started in this document; and the TAD Chat for further reference information. Consider those documents required adjuncts to the TAD, enabling its purpose of explaining how the Plexnet is to be developed.