FAQ: Food for Thought - BLIND TESTING

[monkumonku]

Jul 27, 2010 20:49:12 GMT -5 klinemj said:

I don't understand why it will be complicated and costly.
As stated in the link in my previous post, this scenario could have been perfect

I had read that particular link about a year ago, and I did find it amusing - no doubt.

But, separate from this amusing story, the point of my post is that any "n=1" blind test is not statistically valid. The point is that large base sizes (read: a lot of people participating in the test) is required, and that's what is costly and complicated to execute.

And Lawrence - thanks for your positive comments to dyohn and I. Much appreciated...

Mark

Well n=1 is a valid sample size if you are conducting a test only for your own edification, which is what I think the thepcguy is trying to say (don't mean to put words in his mouth or lounge post, though!) - i.e., prove it to your own self by doing your own blind test. But you are right, if you are trying to prove something with a statistically valid sample that can be generalized to the population, then you need a lot bigger sample size and that gets costly.

This has been a good thread and I appreciate what everyone has to say. There's good points on both sides and it certainly is food for thought!

[thepcguy]

i.e., prove it to your own self by doing your own blind test.

it certainly is food for thought!

you got me.

n=1 is also n=me + someone to switch the cable sometimes

[thepcguy]

I have noticed that the naysayers never discuss imaging, depth, dimension, air, etc....all of which are legitimate things on recordings, not audiophilespeak. Is this a possible reason they hear no differences?

Have you ever wondered why there is no blind testing of car tires? Not every query can be answered with a blind test, no matter what the little Dexters of the audio world say.

I believe blind testers are critical listeners too. Sine waves and measuring instruments I believe are not part of the equation. If you would rely on meters, then what's the use for blind testing? Just read the measurements, voila! you're done. They don't just listen to the loud passages, the devil is in the details.

[jlafrenz]

Jul 28, 2010 1:30:52 GMT -5 thepcguy said:

I have noticed that the naysayers never discuss imaging, depth, dimension, air, etc....all of which are legitimate things on recordings, not audiophilespeak. Is this a possible reason they hear no differences?

Have you ever wondered why there is no blind testing of car tires? Not every query can be answered with a blind test, no matter what the little Dexters of the audio world say.

I believe blind testers are critical listeners too. Sine waves and measuring instruments I believe are not part of the equation. If you would rely on meters, then what's the use for blind testing? Just read the measurements, voila! you're done. They don't just listen to the loud passages, the devil is in the details.

Specs and white papers don't tell you how something sounds. I think that might be what your essentially saying...?

[thepcguy]

Specs and white papers don't tell you how something sounds. I think that might be what your essentially saying...?

Specs and technical papers are part of the DESIGN equation. I do respect and honor the Engineers, Scientists and Inventors without whom these 'toys' we enjoy will never see the light of day.

Voltage, amperes, ohms, resistance - all of these can be measured and they're all part of and seriously considered in the design.

Specs are important. Even Emotiva has this to say (emphasis is mine) ..........

A typical receiver is a well engineered series of compromises. It's give and take...size, cost, power, performance - you can't have it all. Don't get us wrong, there are some great receivers on the market today, with great features and great prices. Unfortunately, something's gotta give, and it's usually in the power amp.

Take a close look at the specs of most popular receivers. Now, take the top cover off, and ask yourself, how did they do it? How could that small transformer, a couple of capacitors and that little heat sink and fan be generating and dissipating all the power claimed?

Here's a typical power spec from a popular name-brand receiver that sells for around $2,000:

120W x 7 (8 Ohm 20-20kHz 0.09%THD) - Sounds great, doesn't it? But look a little deeper into the specifications, and you'll see it again, but listed as a 'Channel Power Rating'. What does that mean? It's called 'fine print...a disclaimer.

It means that one (maybe two) channels can make that rated output, at that specification for a finite amount of time. It doesn't mean that all seven channels, working at the same time, and at the same power level, can come anywhere close to the cumulative power rating of 840 watts that the specs indicate at first glance (7 x 120 = 840). You get the idea. Something had to give. That something was power.

[klinemj]

Well n=1 is a valid sample size if you are conducting a test only for your own edification, which is what I think the thepcguy is trying to say

monkumonku...It is true that n=1 is just fine if you are the n=1 and you are trying to decide for yourself if you like something better than something else or not. That's not what I read in the original set of posts, but that's more clear in his later posts.

In any case, I agree with all those saying to not rely on blind test reports of others...UNLESS you happen to see one with, say, 200-300 independent samples, that is. And...I'm betting you won't see any of those on cables or other AV equipment! And...even then...just because a sample of 200-300 people does reveal an audible difference does not mean that every single person (as their own n=1) will. In any given test, there are groups of people who think A is better than B, B is better than A, and that A=B. Any given person just might be one of those who draws the opposite conclusion as the masses or might find A=B.

So moral of the story remains: take guidance from others as a starting point, but always listen and decide for ones' self.

Mark

[thepcguy]

This one's a good read too:

Blind Listening Tests are Flawed

www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

[monkumonku]

Jul 28, 2010 19:30:11 GMT -5 klinemj said:

Well n=1 is a valid sample size if you are conducting a test only for your own edification, which is what I think the thepcguy is trying to say

monkumonku...It is true that n=1 is just fine if you are the n=1 and you are trying to decide for yourself if you like something better than something else or not. That's not what I read in the original set of posts, but that's more clear in his later posts.

In any case, I agree with all those saying to not rely on blind test reports of others...UNLESS you happen to see one with, say, 200-300 independent samples, that is. And...I'm betting you won't see any of those on cables or other AV equipment! And...even then...just because a sample of 200-300 people does reveal an audible difference does not mean that every single person (as their own n=1) will. In any given test, there are groups of people who think A is better than B, B is better than A, and that A=B. Any given person just might be one of those who draws the opposite conclusion as the masses or might find A=B.

So moral of the story remains: take guidance from others as a starting point, but always listen and decide for ones' self.

Mark

You are so right, to thine own self be true!

[DYohn]

Jul 28, 2010 20:35:57 GMT -5 thepcguy said:

This one's a good read too:

Blind Listening Tests are Flawed

www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

As much as I respect Mr. Harley, his article is a load of bollocks. His conclusion that all blind testing is bad because some of it is is just as wrong headed and the conclusion that all amplifiers sound the same because of one limited test set. People who discredit blind testing tend to be those who really do not understand it. What they are really judging are overly broad conclusions drawn from limited testing. Blind testing when done correctly is a scientifically valid method for evaluating subjective phenomenon.

[jlafrenz]

Jul 28, 2010 20:35:57 GMT -5 thepcguy said:

This one's a good read too:

Blind Listening Tests are Flawed

www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

I am a bit confused... in one thread you consistently talk about blind testing and how it is the best way to determine what the ears like and eliminate other variables such as sight. In this thread you are now linking to how blind tests flawed. Where are you going with this and which side of the fence are you on?

I am not trying to challenge you on any of this, basically just trying to figure out where exactly you are coming from.

[johndavidson]

The thing that continues to amaze me is how so many will claim HUGE sonic differences that are undeniable but when put into a double blind test can not make any distinguishable score above a guess.

You would think if these differences are so great that these same people could easily score in the 90% range when asked to identify one cable or amp from another.

I really don't think its the test that is the issue. I think it is the refusal of the testers to accept the obvious conclusion.

[thepcguy]

I am not trying to challenge you on any of this, basically just trying to figure out where exactly you are coming from.

I am sorry for the confusion. I posted Mr. Harley's editorial to be fair to everybody who will read/participate in this thread. To hear both sides, think and decide. I totally agree with DYohn's comments and that's basically my point in posting Mr. Harley's arguments: to illuminate Mr. Harley's FLAWED reasoning and if you digest the back and forth debate on that editorial, you'll notice there were considerable knowledgeable people who don't agree with Mr. Harley. I am a blind tester on a personal basis. Meaning I haven't participated in a professionally controlled setting. I do my personal testing for myself only. I do it coupled with critical listening. That's where I believe blind testers win. While the other side ONLY do critical listening, blind testers do critical listening coupled with 100% elimination of BIAS. Mr. Harley can assure us all he want that he is a professional and can turn off his bias at will, but it's just another way of saying "TRUST ME". There is one person who most of the time says "TRUST ME". And I trust him. His name is Jack Bauer.

[klinemj]

I agree fully w/dyohn...blind testing when done correctly is a scientifically valid method for evaluating subjective differences.

The issue with the article from Mr. Harley is that the data referred to is not scientifically valid. First, with only 60 panelists, the sample size is likely too small to declare a statistical difference yet is definately far too small to declare parity (sameness, equivalence). Said another way, a 60 panelist test proves nothing at all. That there are 20,000 evaluations in total (as opposed to 1 per panelist) is a good start, but given subjective measures are highly panelist dependent - the effective base size is merely 60 as it is highly likely the multiple replicates within a panelist are highly correlated. This is a statistical issues that most would not understand, but it real and hard (mathematically) to deal with (esp. with data in which many hear no difference and some hear imporant differences to them, which is typical with audio).

That Mr. Harley points out that the listening conditions were effectively garbage further discredits the research he cites.

And then beyond how the data is collected, how one statistically analyzes it is key. If I could only share a specific example from my work I would, but what I can say is that one way of analyzing a set of data I was looking at could not statistically confirm that a control product was equal to itself (even though we KNOW it is...), yet another mathematical approach could. Weird, eh?

So once again...the link just proves the point: don't discredit blind testing as a whole, but beware of trusting most blind test data, look at the details behind the testing & the data analysis, and above all - listen for yourself.

Mark

[Mr. Ben]

Thank you Mark, for describing the statistical limitations of the arguments on both sides. You've been able to state it in a way which is much more understandable than I ever could.

[klinemj]

Thx Mr. Ben...glad to provide the info. I've been living this topic for years in my job, but more than ever the last year. Ran into a really tough statistical problem...it's pushing me to new levels, as well as my statistical experts I consult with. I've learned more on the topic in the past year than in all my years applying and studying it - enlightening it has been, for sure.

[bearfan51]

Jul 29, 2010 8:36:01 GMT -5 jlafrenz said:

Jul 28, 2010 20:35:57 GMT -5 thepcguy said:

This one's a good read too:

Blind Listening Tests are Flawed

www.avguide.com/forums/blind-listening-tests-are-flawed-editorial?page=1

I am a bit confused... in one thread you consistently talk about blind testing and how it is the best way to determine what the ears like and eliminate other variables such as sight. In this thread you are now linking to how blind tests flawed. Where are you going with this and which side of the fence are you on?

I am not trying to challenge you on any of this, basically just trying to figure out where exactly you are coming from.

I'm also confused. If such a large sample is needed for any kind of real statistical conclusion, how is blind testing considered valid for audio testing? It was never used when stereo was originally developed and tested.


I also believe that blind testers have not eliminated their biases. Most of them that I have talked to, don't believe any substantial differences in sound exist between cables/amps/transports/cd players, etc......I personally made a believer out of two people, using an ERC-1 over a very decent older Yamaha, that neither believed would be a significant difference. Frankly, I wasn't sure there would be one either.

We didn't need to be blinded to hear a very distinct difference.


I found a post on another board that goes into some history, very interesting stuff. It's kind of long, but well worth the read, and it's obvious a lot of effort and research went into it.
www.polkaudio.com/forums/showthread.php?t=104701

[bitsandbytes]

"THE KIND OF WIRE MADE NO DIFFERENCE

It can be solid, stranded, copper, oxygen free copper, silver, etc.--or even "magic" wire--as long as the resistance is kept to be less than 5% of the speaker impedance. There is no listening difference as long as the wire is of adequate size."

Hi to everyone at the Emotiva lounge. This beginner trying to separate the fact from fiction in audio and this seems like a great place to start. I read this article online before I purchased speaker wire, I seriously skimped and used a cheap 16 gauge from Best Buy. After all, I had first checked out this table to find the wire resistances:

hyperphysics.phy-astr.gsu.edu/hbase/tables/wirega.html

My speaker impedance is 8 ohms - so 5% of that is .4 ohms - and subtracting .1 ohms for connections means could have a resistance of .3 ohms for the wire itself without an audible difference.

16 gauge wire has a resistance of 4.016 ohms over 1000 feet according to the table. 1000/4.016*.3 = 74.7 feet. As my footage to each speaker is less than 35 feet, it seemed the 16 gauge was way more than adequate. Even 19 gauge wire would have sufficed with this figuring (1000/8.051*.3=37.26').

I felt I was not getting the quality sound I should be getting so decided to try the Emotiva Ultra speaker cables - even though I honestly thought I would probably be returning them. To my amazement, my sound system came alive. Cleaner, quicker, more presence... In movies, could handle sound effect peaks much better. For music, a slight improvement on my Blu-Ray, significant for the Pandora listening, & way better sound with my cd player. I even began using a THX setting on my receiver instead of direct for some music.

So what happened here? Was it the gauge, the connectivity or the shielding - or a combination? Or did my wishful thinking lead me to a mental breakdown where I became so self deluded that I did not know what I was listening to?

My view is that facts, stats, expert opinions, user thoughts, electronic testing, blind testing and personal listening in audio stores are valuable in determining whether or not to take the time to try a product at home. How it sounds can only be fairly determined in your listening environment by your ears. The ability to return and/or upgrade seem most critical.

So what to do with my old speaker wire? Seems inadequate as a clothesline, dog leash or jump rope. Maybe I'll just let it sit on a basement shelf and oxidize and ferment with age like wine and become the Mad Dog or Night Train of speaker wire. ;D

[thepcguy]

I felt I was not getting the quality sound I should be getting so decided to try the Emotiva Ultra speaker cables - even though I honestly thought I would probably be returning them.

Please don't get the impression I am challenging you, but I think the above quote is a sign of BIAS.

So what happened here? Was it the gauge, the connectivity or the shielding - or a combination? Or did my wishful thinking lead me to a mental breakdown where I became so self deluded that I did not know what I was listening to?

If you really want an honest answer to these questions, ask somebody (a friend, spouse, kins) to help you switch the wires. Critically listen to the improvements you've heard WITHOUT knowing which wire is connected. You can do this as many times as you want to really hear the difference. IMHO, this is the only way you yourself can answer these questions.

[bitsandbytes]

Thank you, thepcguy, for your frank, courteous and prompt response to my post. Especially from an old Sansui person like myself .... I remember the days when my Sansui AU-20000 integrated amp rocked my world.

Bias certainly exists in listening tests. Maybe my sarcasm really undermined how critically I listened to the differences in my speaker wiring. In my view, when an A/B listening test is close, a blind test would be valuable. In this instance, my Emotiva speaker cable so far outshined my old wire that I had a hard time believing it. So I stripped the old wires and reconnected them then compared the results again with several selections using various cd's.

It was honestly night and day in favor of the Emotiva speaker cable. That is with MY ears in MY listening environment. I would absolutely know the difference between them in a blind test. An exception occurred when I used some of my poorer quality cd's. Those sounded a mess before - and maybe they sounded even worse now. I could conduct a blind test on those to know for sure ... but why bother?

Would I say the Emotiva speaker cable be better for everyone who uses lower quality speaker wire? Absolutely not. Our ears and listening environments are all different. Would it be worth a free trial test? Without a doubt IMHO. Yes, I was THAT impressed!

[bearfan51]

Welcome bitsandbytes!

Thanks for the insight on Emotiva wires. I'd love to try a pair sometime. Maybe some of the IC's too. (That is, as long as they aren't pushing snake oil....) ;D