The Much wanted commodity

[Snarkridden]

It seems the fakers are at it again, re-coding MP3's of popular artists, latest discovered is the Norah Jones "Come away with Me"
album, a whole bundle of 320k files, some good some bad, some just plain aweful.

You might be better off going for 256k copies of these files, they seem mostly genuine, out of 7 users with 320k copies, there were three different file size groups, so any attempt at multi-sourcing (WinMx) was useless, two of these three groups, have files so obviously re-coded from 128k or 160k, as to be easily detected by ear, the third was so heavily normalised as to make it unlistenable except perhaps in the subway at the morning rush hour.

As mentioned in a much earlier post the ENCSPOT utility will point out the subtle details of your MP3 files, show up original rips, and report on the overall quality, compression, bit-rate, use of joint Stereo etc.

If the link is still valid it was www.guerillasoft.com/encspot

Give it a try, you may be disgusted at your collection, there is some real rubbish out there, old Xing and Blade encoded stuff, and a host of other less than state of the art, codings.

For a real good job... Lame 3.92 and RazorLame seem the best
quality utilities around today..

www.dors.de/razorlame and www.mp3dev.org

Snark...

[JackSpratts]

thanks snark for your tireless work "improving the breed".

- js.

[StereoDeluxe]

I hate re-encoded files

[StereoDeluxe]

got Norah Jones "Come away with Me" Album... very nice, Thank you Snark

oh, Norah is the daughter of Ravi Shankar... I din know that

[Mazer]

Remember way back when people were complaining that Napster would not share certain files without any rhyme or reason? I looked into the matter and discovered that the files in my collection that Napster would not share all had misplaced headers. The Winamp decoder said all the files had headers that started at some random place like 27 bytes or 49 bytes, rather than at 0 bytes were they were supposed to be and that's why Napster rejected them. I suspect it was their way of filtering out the bad MP3's (even thoug Winamp plays those files just fine).

Maybe it's time for all the p2p's out there to start implimenting quality filters that will detect recompressed MP3's and not only reject them but tell the user that they are bad files. People are obviously misinformed and p2p's should take up the responsibility of educating the public.

[StereoDeluxe]

Originally posted by Mazer
Remember way back when people were complaining that Napster would not share certain files without any rhyme or reason? I looked into the matter and discovered that the files in my collection that Napster would not share all had misplaced headers. The Winamp decoder said all the files had headers that started at some random place like 27 bytes or 49 bytes, rather than at 0 bytes were they were supposed to be and that's why Napster rejected them. I suspect it was their way of filtering out the bad MP3's (even thoug Winamp plays those files just fine).

Memories... Ye I had same problems back in Napster era, it was all ripped by RealJerk Box adding 64K ID3V2 tag (front of MP3 file) for no reason, had to use mp3Trim to clean offset.

Is there way to detect re-encoded files other than ears?

[TankGirl]

Originally posted by StereoDeluxe
Is there way to detect re-encoded files other than ears?

Not really. If the forger converts a low-bitrate mp3 to WAV and then uses any popular encoder to make a higher bitrate mp3 out of this WAV, it will look like an original high-bitrate rip even if it sounds probably slightly worse than the original low-bitrate mp3 (due to the two extra conversions).

In practice you can sometimes spot fakes by having a look at an album that you have downloaded from somebody with EncSpot and seeing if the details of all tracks are similar. When collecting albums from open networks people may end up with an almost complete set of high-bitrate tracks and be tempted to 'upgrade' one or two lower bitrate tracks to make the set look nicer. These tracks often differ in some technical detail (ripping software, joint stereo, tagging etc) and suggest multiple sources instead of a single source original rip and a more thorough quality check is in place.

- tg

[butterfly_kisses]

Originally posted by TankGirl

Not really. If the forger converts a low-bitrate mp3 to WAV and then uses any popular encoder to make a higher bitrate mp3 out of this WAV, it will look like an original high-bitrate rip even if it sounds probably slightly worse than the original low-bitrate mp3 (due to the two extra conversions).

In practice you can sometimes spot fakes by having a look at an album that you have downloaded from somebody with EncSpot and seeing if the details of all tracks are similar. When collecting albums from open networks people may end up with an almost complete set of high-bitrate tracks and be tempted to 'upgrade' one or two lower bitrate tracks to make the set look nicer. These tracks often differ in some technical detail (ripping software, joint stereo, tagging etc) and suggest multiple sources instead of a single source original rip and a more thorough quality check is in place.

- tg

your wealth of knowledge always impresses me, TG i think it is ur mind that i love you for now and not ur body.

btw i don't know a lot about this stufrf (encoding and mp3's) but I am a little concerned with all the xtra sh!t that can be bundled with mp3's now...namely when i use real player to play a track and it tries to access my web connection...what is up with that?

is this just querying one of those databases to get the more info on the song like title and artist so that it will scroll that info while playing?

now tell me...should this practice be encourage/discouraged as doesn't this allow for some tracking of what you have on your pc (music wise) i've seen articles posted in the news about these companies particuliarly that one research firm that told about eminem's new album being available on people's harddrive's before it was sold in the stores....

how can we combat this "market research" firms? i'd rather not have a database queried surreptitiously in the background to give me title and track info...i know what it is...that's why i'm playing it

i smell conspiracies everywhere...but that doesn't mean that some of them aren't there.

cheers, tankgirl

[Snarkridden]

Originally posted by StereoDeluxe


Memories... Ye I had same problems back in Napster era, it was all ripped by RealJerk Box adding 64K ID3V2 tag (front of MP3 file) for no reason, had to use mp3Trim to clean offset.

Is there way to detect re-encoded files other than ears?

The only way I know is to analyse the frequency spectrum, there are some pretty convincing spectra graphs for 96k, 128k, 160k, 192k, 224k, 256k & 320k, though the 256 & 320 spectra's are very close, so close that diferent encoders will blur the differences visually, however a 128 to 256 conversion will stand out a mile (or a kilometre for others!) .

Firstly you need to convert the MP3 file into a WAV file, either with DBPowerAmp converter or directly with Sound Forge (versions 4.5 up will read MP3 directly) then apply the FFR analysis filter on a small section of the most active part of the music, I use 30-40 second samples of the most complex parts of the file.

Screen print these spectra for reference purposes, using a known good rip from one of your own CD's as a yardstick, I used the superb Live Nils Lofgren super audio CD for the comparitive tests here.

Ok so its a long tedious job, but if you intend to make a compilation then best to get as many copies of the wanted tracks as you can, select the best versions.

File size, and playing time are dead givaways, as are the odd encodes at other than 44,100 Hz.

Of course, the final result should always be judged by your ear, if it sounds nice to you, then fine.

Some like normalised (all boom and LOUD) personally I prefer to hear dynamic music where the quiet is QUIET and the loud is LOUD..

The normal CD player has a dynamic capability of 96 dB, the complete studio, mixers, recorders, patches etc, usually exceed 75 dB overall, so why compress CD's to a range of 10 - 20 dB?
certainly NOT for Hi-Fi home listening anyway, it may be diferent for you car player, or portable player, but the compression should be in the player NOT on the CD..

I will be posting some typical spectra taken here of 96 to 320k samples as examples, they are in the form of a Word 2000 docthen file so I hope such things are allowed as attachments, if not then it will have to be a sequence of JPG files.

Snark..

[indiana_jones]

some additional thoughts about spectra:

as i know, the ear is a lot more sensible than the eye.

especially the ear is not only sensible to amplitude but also to phase of the signal and through this to stereophonic location of the source.

thus you may already hear artefacts and distortion long before you can see anything in the spectra. mostly the spectra are amplitude only and do not say anything about phase because its difficult to explain effects related to phase distortion.

harmonic distortion of every single frequency may also be hidden by the power of the other parts of the signal, if it isn't just a single sinewave (but who likes to listen to single sinewaves)

some thoughts to the 96 db:
from the 16 bit coding follows that the signal/noise ratio of a cd is 96 db for a sinewave using the full range.
but the signal/noise ratio lowers if the signal becomes more quiet.

if you have a longer period of quiet signal, the ear starts to get more sensible and starts to hear noise, because of the lower SNR.

additionally through quantization effects at low bit quantization, the noise produced is not white noise anymore, but becomes correlated to the signal such starting to produce artefacts which are more annoying than white noise.

those 2 effects are contraproductive.

this way the 16 bit of a cd are not that much, especially for music with longer quiet periods.

one way to hide this, is to have NO long quiet periods and the best results you get, if you have long equally full ranged loud periods.

[Mazer]

Snark is right about the frequency cutoff, but his process is more work than simply listening to the forgeries. I'm sure a simple utility could be written that would scan the first 10 seconds of an MP3 for the lowpass cutoff and compare that number to the default cutoff for the MP3's bitrate, and maybe even look for quantization noise that shouldn't be there.

[TankGirl]

Originally posted by Mazer
Snark is right about the frequency cutoff, but his process is more work than simply listening to the forgeries. I'm sure a simple utility could be written that would scan the first 10 seconds of an MP3 for the lowpass cutoff and compare that number to the default cutoff for the MP3's bitrate, and maybe even look for quantization noise that shouldn't be there.

Different musical styles, instrumentations and mixing styles all produce such different frequency spectrums that I am rather skeptical of any spectral data being enough to spot fakes without proper reference data. To make reliable quality tests you would actually need a high-quality reference rip from a trusted source to compare its spectrum to that of the suspect rip. But if you have a trusted source who is willing to deliver you reference quality rips, why would you download anything from the suspicious guys in the first place...

I see two good practical approaches to handling the quality/fake problem. One is to publish the unique content signatures (hash numbers) for the rips that a trusted person or group has checked and verified to be good - something that our skillful Morphia members have already demonstrated to work in practice. The other approach is to allow peers to form trust chains and groups among themselves with their p2p client and use these trusted channels to establish more quality-conscious distribution chains within the community. Not all people care for quality and there will probably always be some low-quality and faked stuff in circulation in the open p2p networks. But even then both above approaches would help a quality-conscious collector to keep the bad stuff out of his/her own library.

- tg

[JackSpratts]

here's an mp3 quality test that will give you some idea of the differences apparent in even lower bitrate conversions using several popular methods, including mp3pro, ogg, wma and lame. as you will hear, your preferred encoding type may not be in agreement with the reviewer. the science of sound is ultimately more art than empirical, more subjective than objective, with numbers taking one only so far.

setting up a standard p2p q-meter may not be possible or even desirable, any more than having files marked as "good songs" and "really good songs" conveys practical, usable information. suffice it to say that more bits have more information but more bits take up more space and increase u/l-d/l time. do they also sound better? maybe. is more always better? probably not.

i listened to a pre-recorded cd of joni mitchells’ “hits” (‘96) during this post. it was remastered using a high definition method unavailable a decade ago and it simply takes your breath away on some selections, under some circumstances, yet other songs on this meticulously crafted cd sound as they always have. that’s the way sound is; personal, indefinable, ephemeral, mystical. wonderful. and so impossible to cage.

- js.

[Snarkridden]

Thanks for the link Js.

Interesting that was done some while ago, would be more interesting if it were re-done to day with the newest Lame (3.92)

I think we have got past the hype of all these so called NEW coders, seems anything foisted on us by the "Big boys" is suspect, and possibly carries a threat to P-2-P on the if you cannot stop them sharing, make what they share rubbish, theme.

I was once said that the original restrictions with KaZaa to 128k was possibly something to do with an alternate agenda?

Coming back to a point raised earlier, the question of harmonics causing spectra to lie, when 128ks are re-coded, even this is easy to spot, as a pure original encode will have a steep cut off and a low noise floor above the cut off, any raggedness in the above cut off range noise floor level can show that the material was re-coded, Ok its not a total proof, as so many different encoders perform differently in the high frequency region, the original Xing encoder applied High frequency cut before encoding as a way to minimise "muddyness" it didn't work?

There are still loads of old encodes on the network, and will be for ages to come.

I really like TG's trusted idea, surely this is an ideal topic for Napsterite users ? We could perhaps start it off, at least guide others onto the right "trusted" path...

There must be many that read the open forums wihout being joiners, or posters, maybe the influence will spread,,

Snark...

[TankGirl]

Originally posted by Snarkridden
There must be many that read the open forums wihout being joiners, or posters, maybe the influence will spread

That is so. Our public forums have a lot of visiting readers nowadays so whatever information and insights you can provide will have an influence beyond our registered membership.

- tg

[BuzzB2K]

Originally posted by TankGirl
I see two good practical approaches to handling the quality/fake problem. One is to publish the unique content signatures (hash numbers) for the rips that a trusted person or group has checked and verified to be good - something that our skillful Morphia members have already demonstrated to work in practice. The other approach is to allow peers to form trust chains and groups among themselves with their p2p client and use these trusted channels to establish more quality-conscious distribution chains within the community...
- tg

I use eDonkey and a number of Websites that post hashes for Full Album CD rips - They have ranged in size from 16MB (5 song CD Single) to 777MB (Led Zeppelin 10 Cd Box Set)...
Takes a while to aquire this way, but worth the wait...

[Snarkridden]

I've had all sorts of problems trying to create a zipped HTML file that would be easy to use with a browser, as attaching HTML is frowned upon, i've left it as a word2000 type .DOC file, if this does not work, then I need to find out how to post a link to it, and set it up on my web page (when its finished)

Meanwhile, here we try the impossible... Don't complain if it does not work eh?

Snark..

[JackSpratts]

worked perfectly sr. excellent job!

edit: snark, do you have a graph of the original .wav for comparison? i'd like to see it if you do, thanks.

- js.

[Snarkridden]

I knew someone would ask that! I was concerned the doc with embedded pictures would be getting too large, so saved these for the final part, but to make it easy they are JPG's as no text is really neccessary...

So, the straight Rip spectrum first at 320k...

Snark..

[Snarkridden]

I intend to put the whole document together as one, and make a PDF file of it, also an HTML web page, and maybe a compiled HTML page, when I can find a decent HTML compiler to take the sweat out of doing it by hand..

I'm getting to hate <html> and </html> legends...?

Snark..