How would you connect 10 million peers with decentralized p2p?

[TankGirl]

How would you connect 10 million peers with decentralized p2p?



When Napster was living its heyday it provided a central meeting place for about a million online music lovers. Two years ago this million-seat filesharing venue was big enough to host almost the entire mp3 trading community. The p2p scene has evolved considerably since. A recent Odyssey study suggested that the number of online music swappers is already 40 million in US alone. Worldwide there might be perhaps 100+ million p2p users, and they are already consuming even 60 % of the bandwidth of some broadband ISPs. Comparing these figures to the total number of people on Internet which is about 560 million, we see that already every fifth or sixth Internet user is taking part in the fun.

Napster’s achievement to connect a million people to each other’s shares and chat reach was remarkable – both technically and socially. In terms of technical connectivity decentralized p2p is still about two magnitudes below Napster’s centralized performance. There may be two million people simultaneously online on FastTrack or WPN but not in any singular space where they could see each other. No two users are guaranteed to find each other when they connect to the network - actually they have a low chance of ending up to the same shared space (under the same supernode or under two linked supernodes). With each peer seeing just a fraction (perhaps 10.000 other peers) from the total population of millions, nice community-binding things like hotlists become unfeasible. And as each peer’s view to the content pool is similarly limited, the large network’s potential to provide rare and diverse content for everybody becomes seriously crippled. In other words there is plenty of room for innovation and improvements!

The present decentralized connectivity level has been achieved with a self-organizing supernode architecture where a number of powerful nodes volunteer to form a second connectivity level above the grass root level of 1-to-1 communications. The end result is not that different from Napster’s centralized architecture. P2p networks also have their own ‘server’ nodes (Gnutella’s ultrapeers, FastTrack’s supernodes, WinMX’s ‘primary connections’) acting as connectivity hosts, message brokers and search engines for the rest of the nodes. And just like Napster’s servers, these volunteering p2p supernodes are linked to each other to provide their local clients a wider view to the community and its resources.

Building connectivity on these two different platforms requires two very different approaches and architectures. It is almost like building two machines to do the same job, one working on Newtonian mechanics and the other on quantum mechanics. In the first case we have a deterministic system with components we know and control. Performance and reliability are built on powerful, reliable and well-maintained hardware. In the latter case we have a non-deterministic collective of unknown components interacting more or less freely (within the boundaries of the software) with each other. They keep appearing and disappearing at random, perform unpredictably and are subject to constant bandwidth fluctuations. There is plenty of collective power - technical and social - in large decentralized networks but it must be collected from a dispersed population of peers and its reliability (if any) must be based on redundancy and statistical behaviour of peer groups.

How to proceed from the supernode system? How to establish a smarter, more efficient connectivity infrastructure that could collect the growing megacommunities back into shared social spaces, into each other’s reach? The design target for next generation p2p should be connectivity for millions rather than for thousands. Should there be a number of 'hypernodes' selected from among the supernodes to form yet another layer of connectivity? If the present supernode architecture fails to scale up due to the increasing 'long distance' search and browse traffic between the linked supernodes, how would you regulate bandwidth usage with the possible additional connectivity layers? Or would you try some other approach altogether?

Think of it as a game with ten million active pieces in it. The pieces represent a wide spectrum of different computing powers, bandwidths and shared contents. You can instruct all pieces how to act in different situations but you cannot change the inherent properties (bandwidth etc.) of any particular piece. Nor can you control the online presence of any given piece although you can observe and analyse it. To make the challenge even tougher, many of the pieces will be shielded with firewalls which prevent them from responding to externally initiated contacts. How would you create a working decentralized hotlist in such an environment, so that an arbitrary peer A would find its hotlisted contact B in a reasonable time (even minutes would be acceptable in a large network) after popping online? And how would you handle the situation where peer B would not want to be found by A (ignore list)?

Your creative thoughts and reflections please, dear Napsterites.

- tg