What Will They Think of Next?

[TankGirl]

Originally posted by Scyth
Say you have an X byte file. If you download in 10,000 byte (hundreth of a megabyte) chunks from each source, then there will be X/10,000 chunks to verify. Assuming a 32 bit = 4 byte hash for each chunk, that's X/100,000*4 bytes worth of hash information. So, hash information will make up only X/10,000*4/X=.04% of the total download, which is a trivial amount, even if it has to be retrieved for each source.

Good maths, Scyth.

Your calculation demonstrates how just a small stream of extra hash information can give us safe multisourced downloads from partially completed files. That is just about as good as it can get at the file transfer level and makes possible fast cascade-like distribution of huge files.

It is enough that there is a single source with the complete original file to calculate the hash numbers for all data chunks included. With that information available the potential downloaders can start searching for the content itself and - as soon as some content becomes available - form a cascade of simultaneously downloading and uploading nodes.

In case of large files the hash numbers themselves could be packed into separate, sharable metaobjects verified by their own hash numbers. For example a 650 MB movie would produce 65.000 10 kB data chunks and with 4 byte hash size this would be approximately 65.000 x 4 = 260 kB of hash data - enough to justify a separate, multisourcable object.

eDonkey must use some sort of chunk hashing mechanism and I also remember reading recently about a Gnutella client experimenting with similar technology.

- tg