Partial hashing subjects FastTrack to fake file flooding

[TankGirl]

From Zeropaid:

Found this on the k-lite forum. It explains the floud of damaged mp3s. Pure horror. We can't trust the verifieds anymore.

Author: Hasnain
Date: 08-26-03 16:13

Foreword:

I am listing this topic here, because more and more people are beginning to download fake or corrupted files from the FastTrack network, despite using a verified hash. Some people have asked me how this is possible, the main reason being that Kazaa does not use each and every byte of a file to determine its hash.

Vulnerability:

I had noticed this vulnerability when saw the source code of sig2dat. Essentially, Kazaa calculates the hash of a file in the following way:

1. It reads the filesize and if the filesize is less than 300Kb, it hashes the whole file. If the filesize is greater than 300Kb it reads the first 300Kb from the file. The hash method is md5 one way hash.
2. After hashing the first 300Kb, it now calculates the offset of the next block as the offset of the previous block t shifted by two i.e.
New Offset = Old Offset << 1 (Shifting a number left by one means multiplying it by two).

------------------------------
Block 1, Size = 300Kb
Start Offset = 1
------------------------------
Block 2, Size = 300Kb <--This block will be
Start Offset = 300K <-- skipped
------------------------------
Block 3, Size = 300Kb
Start Offset = 600K
------------------------------
. <- Other skipped block(s) here
.
.
------------------------------
Block N, Size = 300Kb
Start Offset = N*300K
------------------------------

Now its obvious from the above procedure that after the first 300Kb is hashed, the next will be skipped, because the offset is multiplied by two. That means 300Kb will not be hashed. This will get even worse, as the next time the offset is multiplied by 2, 600Kb will be skipped.

Now all a miscreant needs to do is damage those parts of the file which are not hashed, like the second block or 300Kb, or the fourth block of 600Kb, etc. This means that Kazaa will think that they are the same files, and will download the damaged file as part of the regular file. This will result in incorrect files downloaded, even if the content was supposed to be verified.

Why was it done?
----------------------
the basic idea behind such an algorithm to calculate hash would be speed. This method allows Kazaa to hash files of gigabyte sizes in a few milliseconds, even on slower machines. But in doing so, they miss out a very important aspect, integrity.

Corrective Measures:
---------------------------
Unfortunately, its the way Kazaa calculates the hash and *nothing* can be done about it. The only way to correct this is that Sharman Networks actually changes the calculation algorithm to use all the bytes available in the file.

Partial hashing is usable to spot identical files in a friendly environment, and was already used by Napster. In hostile environment only complete hashes can be trusted though. Not only can mp3 and movie files be damaged due to this vulnerability - it is also possible to insert viruses to shared software without changing the partial hash numbers.

- tg

[goldie]

very very bad news there TG.

I can imagine the riaa and mpaa just drooling at the mouth.

[pod]

Frankly I'm surprised it took this long to notice partial hashing as a problem, and to exploit it on a large scale. Anyone giving serious thought to file integrity would notice right away, as soon as they run Kazaa for the first time 3 years ago, that there is no way to fully hash a 20gb share in just a few seconds. Ironically, the biggest original complaint against eDonkey was that it took a long time to add files to a share. Duh, no shit.

The only way to fix the situation now is for Sharman Networks to rewrite the FastTrack protocol (and it really would be a major overhaul) and have everyone rehash all their files. Might as well switch to Overnet now

And full hashing doesn't even take THAT long. On my lowly P3 1GHz, eMule takes about 30 seconds to hash a 700MB movie file, and of course these are cached so you only do it once. Even if you remove a file from your share and re-add it later, the hash is remembered. If you keep your files on a remote file server, this will be a bit of an issue, but again, you only gotta do it once.

[napho]

It goes to show how innocent those days were. It never occurred to anyone that those holes could lead to security problems even though alot of redirects to porn websites were embedded in movie clips 3 or 4 years ago.

[JackSpratts]

Originally posted by pod
there is no way to fully hash a 20gb share in just a few seconds.

so true. says a hobitt, "some shortcuts make longer ones".

this is just the begining. the dark minions of the riaampaa are working on some really sinister ways to sabotage p2p.

- js.

[hrabn]

despite reading a number of explanations and tutorials on hashes, sig2dat, K-sig, etc, there is a basic question that I am not able to find a direct answer to. i infer from what i've read that the hash information is embedded in the file itself (eg. mp3, avi, etc). is this indeed the case?

if so, can the information be extracted and a new hash (complete hash or what-have-you) be inserted?

ancillary question - what happens if the file is renamed? i gather that some programs cannot then find the file based on the hash (kazaa), but that others can (kazaa lite).

can someone point me to a source for this type of information?

thanks!

[Stoepsel]

The hash code of a file is not stored in the file. The file itself is used as the basis for calculating its hash code.

A hash code is generated by treating the file as a long sequence of 0's and 1's, running them through some form of hashing algorithm and thereby turning them into a more manageable (read: shorter) sequence of 0's and 1's. In that sense, the hash code is like a fingerprint of a human person. A small part of the human body is used to identify the whole person to a good degree of accuracy.

One of the most common hashing algorithms is called MD5. You can read on MD5 hashing here.

Renaming a file does not change the hash code of a file. The file name is not part of the file contents and therefore isn't considered in the hashing process.

Hope this gets you headed in the right direction,
Stoepsel

[JackSpratts]

fasttrack includes id3 track info when it creates a hash code so changing any part of the id3 file after you've downloaded a song changes the hash code too, preventing the file from multi-sourcing, until enough of your "new" files seed the system again.

- js.

[cindy]

stoepsel good information. JS i like ur reference to LOTR with the 'shortcut' reference.

Now let's get down and dirrrty ..into the nitty gritty of this...anybody tried spoofing or faking a hash?

if so...how'd ya do it? i have my own ideas...but once again...its like national security and working in area51...ya wanna tell somebody bout dem 'aliens' or in this case..."how ta do it"

but by the same token you don't want a lot of "copycat" serial killahs (scipt kiddies) running loose imitating "ha><ors" and totally fudging up fasttrack (anymore than its already been fudged)

so should information like this be made public? my thoughts on the matter are no...unfortunately the temptation to cure curiosity will often time lead to trying to exploit/copy/reproduce a 'new'' (or old) vulnerability in something.

botton line or rather 'closing thoughts':

so is 'the donkey' the only network (p2pwise) that does complete file hashing? are there others? do any exist?

as someone else once said...(referring to the internet and 'getting caught') the only way to be 'totally safe' is to just rip out yer modem or network card and not go online...

for the rest of us there is 'comon sense'. it will be your biggest weapon against such maladies. Perhaps an alternative p2p program which supports full hashing?

or you can continue the Prussian roulette with KaZaA and take ur chances...u never know when u might be the lucky wiener who gets caught or finds a virus.

Only the prize isn't desirable as the ones found in 'crackerjacks'

-Ciao

-

[multi]

i wonder if they may have a program for creating a spoofed file from the original..
like they get the original file and copyit thru a prgram that uses that hashing algorithm to create a looped or white noise version of that file but with as exact matching file signiture..i guess they allready do this..

i do reckon fasttrack gnutella..ed2k ect and all involved with programming start looking at alternate ways of coding the way the networks work..easy to say i know..but theres plenty of good ideas starting to suface..maybe theres way
full hashing can be worked into it with out making the searches slow bad..

[Stoepsel]

From what I read elsewhere about how FastTrack partially hashes its files, it is utterly simple to take a given file and polute it with white noise and still retain the original hash. And with all the reports that I've been reading on other boards about the increasingly high rate of MP3s I bet that the RIAA and the companies they hire already make good use of this technique.

Apart from ed2k, I know that BitTorrent also uses complete hashing. With Piolet/Blubster I'm pretty positive that they use complete hashes. I'm not sure if WinMX does or not.

Stoepsel

[Ðiego]

To me all this hash business is a no-brainer..

I don't use them and never will. They mean nothing. When I download a tune I listen to a bit of it. If it's what I was after I keep downloading, if not i stop it and delete. When it's done I listen to it. If it's what I was after I keep it, if not I delete. In any case, I have NEVER had a 'bad' Mp3..

Some people reliance on others in some reguards confounds me. To me, hashing is just a tool for multi-sourcing, which I feel is foolish.


Ð

[hrabn]

thanks stoepsel and jackspratts for the info that clears things up considerably. i hope you don't mind a few more questions.

is the information contained in the dat file sourced from the hash or from the file? is the dat file created by say kazaa lite when i click on the file for download or is it read from somewhere else?

if i create a file and offer it in my shared directory and do not use sig2dat, k-sig, or something to create a hash, how is a signature created?

as you can probably deduce from my questions, it is some of the basic underlying concepts that i am missing.

thanks for the help.

[JackSpratts]

the kazaa clients create the dat files on demand, and usually piece them together on bits of unused stuff lying around your hard drive so until they're complete your dats can contain some interesting chunks, but they're made form actual file parts aquired in multi-sourcing using the hash number, not hash bits. the kazaa client will also create any new hashes by itself, you don't need an outside client to do it. simply putting a file in your shared folder starts the process automatically at your next start-up. it's probably how most music hashes are made, since most people only use the ft client and aren't aware of doing them any other way. movies are a different matter entirely.

- js.

[hrabn]

thanks js

there are 2 things i've read that i'm having a hard time integrating with this new information.

first is the business about riaa forensics examining hashes and determining that certain files were originally shared at a specific time and by a specific service (eg napster).

the other is the experience of finding a file in a particular users share, initiating a download, actually getting a piece of the file, and then sitting for 1-2 weeks with no additional progress -- the accelerator reinitiates the process, the source is identified (the original user), but then things eventually terminate with "more sources needed".

btw, the files in question are cult tv (mgp & avi) & i am using k++lite 1.4.2