FAQ: Food for Thought - BLIND TESTING

[thepcguy]

Have you ever wondered why there is no AUDIO cable company conducting and officially publishing results of BLIND TESTING of their superior products?

[thepcguy]

Popular Mechanics BLIND TESTING.

Comparing a $13 HDMI cable against brand named cables costing $200, $300.
www.popularmechanics.com/technology/gadgets/tests/4235717

[thepcguy]

Transparent Audio ( Jack Summer, President) - The only company I knew who ALMOST did an official company blind testing:
www.vxm.com/21R.64.html

[flamingeye]

Now that`s to funny

[RuggSkins]

The Information below is taken from this website: www.roger-russell.com/wire/wire.htm#gordongow

Gordon Gow's Speaker Wire Listening Test

I have read several magazine articles and papers expressing the findings and opinions about the various kinds of speaker wire. Some engineers have applied their expertise to make measurements to prove conclusively that there ARE differences between wires. A few authors have devoted their entire paper to the measurements and never mention whether they have actually made any listening tests or if they could hear any difference. Despite all the measurements and opinions, the final test is whether you can hear any difference or not. Obviously, this must be done under controlled conditions where you don't know which wire is connected and there is no delay in switching.

In the early 1980's, special speaker wires were beginning to appear on the market. Some of the claims were totally unbelievable and had prices to match. Realizing that wire resistance was the critical factor in speaker wire, Gordon Gow, President of McIntosh Laboratory, used a speaker cable demonstration to show there was no listening difference between these wires and plain line cord. He delivered his presentation about the truth in speaker wire using a reel of Monster cable to stand on. Fifty-foot lengths of wire were used in the comparison. The setup consisted of a master control relay box and two slave relay boxes. A three-position switch was used to select one of three different speaker wires of equal length. One was line cord. The other two wires were from popular manufacturers. 8-ohm speakers were selected to be used in the test. The two other brand name wires were heavier than the line cord.

The boxes now show some signs of wear. This is from being handled and traveling around in a large fiberglass case along with all of the speaker wires and connecting cables.

A slave box was positioned at each speaker. Power to drive the relays in each slave box was provided with separate cables. The speaker wires were switched at both the power amplifier and the speaker so that only one kind of wire was connected at a time. Short pieces of heavy wire were run from the speakers and amplifier to the relay boxes. No other devices were used in the speaker line. The relay contact resistance was measured to be less than 0.1 ohms. No consistent listening differences were heard by customers or dealers.

The test proved his point. When I took the test, I was unable to hear any differences using several different 8-ohm speaker systems. BUT, when I deliberately played one particular 4-ohm speaker and I switched to the line cord position, I could hear differences. I knew this system dipped down to 2.6 ohms in one frequency range, and 3 ohms in another. It verified that differences can be heard if the wire is too light for a lower impedance system. A system this low in impedance requires heavier wire. After replacing the line cord with a heavier line cord of equal length, differences could no longer be heard.


THE KIND OF WIRE MADE NO DIFFERENCE

It can be solid, stranded, copper, oxygen free copper, silver, etc.--or even "magic" wire--as long as the resistance is kept to be less than 5% of the speaker impedance. There is no listening difference as long as the wire is of adequate size.

Of course, we are not personally able to establish the truth of everything for ourselves and it's not easy to set up a similar wire listening test. Very few people are able to make speaker impedance measurements or wire resistance measurements down to 0.1 ohms. Like many other things in life, we rely on indirect sources of information, such as sales literature, reviews and opinions. This is called Authority Belief, which is part of our belief system. An interesting article about the belief system is described in ETC: A Review Of General Semantics Sept. 1964 titled Images Of the Consumer's Mind by Milton Rokeach.

Gordon Gow's cable demonstration provided a personal experience for customers that could replace the Authority Beliefs they had relied on earlier. The demonstration was controlled. It was an instant comparison and the listeners did not know the wire identification. Gordon held many such demonstrations in dealer showrooms and at shows.



The Truth about Speaker Wire

Despite the effectiveness of Gordon's cable demonstration and the truth about speaker wire, people visiting the McIntosh room at the shows, who had not experienced the cable demonstration, were disturbed that we were using ordinary heavy zip cord instead of one of the popular brands of speaker wire. Instead of listening to the McIntosh speakers and electronics, they recalled "bad" things they had been told about "common" speaker wire and this promoted questions about the "inferior" wire being used. When we changed the wire to a popular brand of wire, customers were happy with the setup, and directed their attention to the McIntosh equipment.

The demand for high quality speaker wire was increasing and appeared to be a new marketing area for several companies. McIntosh did not make or sell speaker wire. The solution seemed very obvious--rather than spend time and effort to create negative sales for McIntosh dealers who were beginning to sell speaker wire, it seemed best to encourage the speaker owner/customer to consult with the dealer about what speaker wire to use. Consequently, I no longer recommended the kind of wire or wire sizes in the speaker manuals.

By 1988, McIntosh no longer supplied audio interconnects with the electronics. Again, many kinds of special audio cables were available to the customer/owner. The dealer could also be consulted about what cables to use.

I credit the success of the speaker wire industry to their expert sales and marketing ability. However, it is my experience that ordinary copper wire, as long as it's heavy enough, is just as good as name brands.

Looking at this from a different perspective, there will always be those who will want expensive wire, not because there is an audible difference, but because they may value pride of ownership and prestige in a similar way to that of owning a Tiffany lamp or a Rolex watch.



www.roger-russell.com/wire/wire.htm#gordongow

[bearfan51]

Jul 26, 2010 14:26:45 GMT -5 thepcguy said:

Have you ever wondered why there is no AUDIO cable company conducting and officially publishing results of BLIND TESTING of their superior products?

I have noticed that the naysayers never discuss imaging, depth, dimension, air, etc....all of which are legitimate things on recordings, not audiophilespeak. Is this a possible reason they hear no differences?

Have you ever wondered why there is no blind testing of car tires? Not every query can be answered with a blind test, no matter what the little Dexters of the audio world say.

[DYohn]

Jul 27, 2010 16:17:26 GMT -5 @bearfan51 said:

Jul 26, 2010 14:26:45 GMT -5 thepcguy said:

Have you ever wondered why there is no AUDIO cable company conducting and officially publishing results of BLIND TESTING of their superior products?

I have noticed that the naysayers never discuss imaging, depth, dimension, air, etc....all of which are legitimate things on recordings, not audiophilespeak. Is this a possible reason they hear no differences?

Have you ever wondered why there is no blind testing of car tires? Not every query can be answered with a blind test, no matter what the little Dexters of the audio world say.

All the ideas you mention are subjective factors and cannot be measured with any consistency. They vary from listener to listener. This is why in a legitimate test to find distinctions between things like hook up wire, as many subjective factors as possible must be taken out of the mix. And even then, if what you end up with is based on preference, then it's still just a subjective choice.

There is nothing wrong with making decisions based on subjective criteria : indeed, this is what we all MUST do to find an audio system we "like." But using your subjective criteria (or anyone else's) to tell anyone else what they will find as "best" is always going to be wrong. We do not all hear things the same way. It's just that simple.

[ronm1]

Nothing wrong with a blind test as long as its used as another tool in the box. I've picked a couple of items where the blind test fooled me sad to say. Later on extended listening I detested the product. IMHO anyone who buys on just a blind test is an audiophool not phile.

[bearfan51]

Jul 27, 2010 16:23:17 GMT -5 DYohn said:

All the ideas you mention are subjective factors and cannot be measured with any consistency. They vary from listener to listener. This is why in a legitimate test to find distinctions between things like hook up wire, as many subjective factors as possible must be taken out of the mix. And even then, if what you end up with is based on preference, then it's still just a subjective choice.

There is nothing wrong with making decisions based on subjective criteria : indeed, this is what we all MUST do to find an audio system we "like." But using your subjective criteria (or anyone else's) to tell anyone else what they will find as "best" is always going to be wrong. We do not all hear things the same way. It's just that simple.

Which is precisely why a blind test shouldn't be the one deciding factor. When important differences between the cables lie in these factors, why discard them? Everything about music and the enjoyment of it is subjective. Think about that......

Why try to objectify, if it forces you to discard portions of it, which are enjoyable? IMO, if you have to do that, then maybe you are trying to objectify the wrong things. All I know, is that since I've been upgrading my cables, I have fewer and fewer bad recordings.....lots more of them image to the degree that is seems some sounds are coming from my surround speakers, except that I'm running in 2 channel.

[DYohn]

Jul 27, 2010 17:14:08 GMT -5 @bearfan51 said:

Jul 27, 2010 16:23:17 GMT -5 DYohn said:

All the ideas you mention are subjective factors and cannot be measured with any consistency. They vary from listener to listener. This is why in a legitimate test to find distinctions between things like hook up wire, as many subjective factors as possible must be taken out of the mix. And even then, if what you end up with is based on preference, then it's still just a subjective choice.

There is nothing wrong with making decisions based on subjective criteria : indeed, this is what we all MUST do to find an audio system we "like." But using your subjective criteria (or anyone else's) to tell anyone else what they will find as "best" is always going to be wrong. We do not all hear things the same way. It's just that simple.

Which is precisely why a blind test shouldn't be the one deciding factor. When important differences between the cables lie in these factors, why discard them? Everything about music and the enjoyment of it is subjective. Think about that......

Why try to objectify, if it forces you to discard portions of it, which are enjoyable? IMO, if you have to do that, then maybe you are trying to objectify the wrong things. All I know, is that since I've been upgrading my cables, I have fewer and fewer bad recordings.....lots more of them image to the degree that is seems some sounds are coming from my surround speakers, except that I'm running in 2 channel.

You and I are talking apples and oranges here. You're talking about the process of discovering what you like. I'm talking about a process that allows the user to determine what component is actually causing them to like something. If you set up a proper A/B test so that only one factor changes, then it must be the reason for any resultant difference. But in order to remove the placebo effect of "knowing" that it's now the orange cable instead of the blue one and thus expecting a change, it must be a blind test. The listener can only be certain that what they think they hear actually has a root cause if they do not know what has changed.

I have conducted many blind tests where the listeners did not know what was changing, if anything. They were told to begin listening, then told that change A had occurred, then change B, etc. At each change they documented the differences, what they liked and what they didn't like. The most telling of these tests was one I did in 2004 with an audience of 20 people where we changed absolutely nothing, we only told the audience a change was happening. Fully 60% of the audience recorded significant differences and preferences for changes that did not occur. They expected to hear something different, so they did. In other tests, the choices made were either not statistically significant or were inconclusive or counter to expectations.

The bottom line is you should be happy with your system. But my experience tells me that differences reported by some users are all in their minds, or are due to expectations rather than actual electrical changes, and so I am always skeptical of these reports.

YMMV.

[klinemj]

To the op's original question, I can tell you why nobody does and publishes blind tests. It's quite simple: to get any statistically valid data would be really, really costly. And, that cost is likely not worth it.

I'll elaborate on why, but let me start by saying that my personal career and area of expertise lies in developing consumer products. A key part of that is evaluating them to make sure they are better in ways we intend and no worse in measures we don't intend to change. A lot of the research we do has to do with subjective measures (consumer preference) as well as technical measures - each of which can vary person-to-person.

So, here's the deal: sound preferences vary by person (due to differences in person to person hearing ability, as well as the "do they give a #!@% factor). If you want to measure and prove that your new (pick a thing...cable, amp, CD player, etc.) is better than prior (pick equivalent thing that is either being replaced or is a competitive offering), then you have to have enough samples to statistically prove there's a difference. And, likewise if you want to claim the two things are the same (except far more samples are required to prove this...).

For sound, while a very few samples of a test tone can tell you if the item in question yields a difference electrical response, if one wants to know if consumers actually hear AND prefer the difference - the testing gets far tougher. This is because the lab test measures on test tones have low variation test to test, but subjective measures vary significantly test to test (within person AND person to person). Given this higher variability, large #'s of samples are required to confirm a difference exists AND even larger sample sizes are required to statistically confirm that no differences exist. The smaller the difference to be detected, the larger the base size required to prove the point being proven. (Note: this is just a statistical fact, not some opinion...)

As one with 24 years experience in subjective consumer statistics, I read the reports in which a blind test with 1 listener declares "no difference exists" and make the presumption that because they didn't hear it that nobody in the world could, and I just have to laugh. While they can accurate report that 1 listener heard no difference, the data mean nothing for the population. The data from n=1 listener has a greater chance of being wrong than right.

And, even with 12-25 subjects (the most I see reported in audio tests and most of those include non-valid repeats of panelists), the difference has to be HUGE to be statistically detected (and even larger to say with statistical confidence that "no difference exists").

I have searched long and heard to find anything I would call statistically valid data on AV equipment, and I have never found any at all.

That all said...do I really care about this? Simply put: no. I buy what I think I will like, listen to it and decide if I personally like it or not. If I don't, I return it. (That one reason I love Emo so much! Great trial system!)

Also, to the topic of cables which is being debated here and there...I have confirmed that I can hear differences between cables. By that, I don't mean the usual that I think cable "a" is "far more airy and spatially blah, blah, blah..." than cable "b". I have had situations in which a reputable cable simple sounded bad...grainy, harsh, nasty...while other similarly reputable cables sounded fine. And, I have had reputable cables that would shut down my XPA-2 and XPA-5 but some dime-store cheapos would not and played fine.

So, that last point is added to simply say that in many things, there's garbage and there's "good enough". Beyond that, I don't worry much about it...I have to do that far too much in my job!

Mark

[DYohn]

Mark, you and I are in accord.

[thepcguy]

"IMHO anyone who buys on just a blind test is an audiophool not phile. "
NOBODY demands a blind test at the store. You can, but do you? As consumer, You do a blind test yourself, your own pace. You have two choices here: the Blind test people or the "trust me" people. It's all up to you.

[klinemj]

^ ;D

Another tidbit...a lot of audio testing has gravitated to ABX testing, in which subjects have to determine if sample "a" or sample "b" matches sample "x". This is a 3-way comparison, and humans are inherently better at 2-way comparisons (is "a" better than, worse than, or the same as "b"). So, in addition to low sample sizes being used in ABX comparisons, beware of conclusions due to the 3-way comparison being a more blunt tool than a 2-way.

Mark

[Wideawake]

Mark, that was a beautiful post. David makes very good points too. This thread should be made into a stickie, IMO. Good stuff here!

[LCSeminole]

After having read through this highly subjective topic, I now have to say that the 12g home depot lamp cord speaker wire connecting my XPA-5 to my Klipsch Legends is sounding even better than it did last year, just as it has for the last 13 years. I have run a blind test each year, on the same day each year, for the last 14 years and have come to the conclusion that the $38.50 + tax, I payed for the spool of 250 ft of the Home Depot 12g lamp cord speaker, that with age it has produced better sound each year, comparable to wine I might add. At this rate I have extrapolated that the remaining appoximately 125 ft of my spool of lamp cord should be worth plenty, again the similarity to aging wine. Anyone want to buy some from me before the price goes up? ;D

[LCSeminole]

On a serious note. Mark and David have given some well thought out responses. These types of responses are good examples of how to be civil and respectful while still holding ones beliefs without having to bash another members contributions in a potentially volatile thread subject.

[LCSeminole]

Jul 27, 2010 18:12:31 GMT -5 Wideawake said:

Mark, that was a beautiful post. David makes very good points too. This thread should be made into a stickie, IMO. Good stuff here!

Good idea! Done.

[thepcguy]

To the op's original question, I can tell you why nobody does and publishes blind tests. It's quite simple: to get any statistically valid data would be really, really costly. And, that cost is likely not worth it.

I don't understand why it will be complicated and costly.
As stated in the link in my previous post, this scenario could have been perfect: it would be done in Jack Summer's own turf, his own people doing the critical listening and Tom would just switch the cables.

Posting it again here:
www.vxm.com/21R.64.html

excerpt....

In front of all present that night, including myself, Summer jumped up, and immediately challenged Nousaine to come to Chez Transparent in Hollis, Maine. Tom would then have an opportunity to do one of his cable comparison tests, using Transparent cables. Summer publicly stated that Nousaine would definitely hear a big difference in cable 'sound' . Nousaine grew testy, and rather red in the face, as he sparred back and forth with Summer. Things proceeded to get rather tense.

Like everyone else there that night, I watched all this with grim fascination. Quite frankly, given the players involved, it should not have been a totally unexpected development. Finally, everyone calmed down, and the evening's agenda proceeded.

Immediately following the meeting in Boston, Summer wrote the following letter to the Boston AES. Please note that Summer once again restates he is prepared to participate in a comparison test.

"To the Editor: January 18, 1995 Last night's Section meeting with Tom Nousaine was an interesting experience for me. I needed the experience to prepare for a talk that I am going to give to the New York AES section later this spring.

My undergraduate degree is in physics and my doctoral work was done in statistical analysis and research design. It is important to examine the validity of double blind testing at revealing subtle differences which audiophiles consider important. I intend to conduct an experiment in an area outside of audio to see what level of difference must exist for a double blind to statistically validate it. Perhaps someday I can share the results of this experiment with the Boston Section.

We invited Tom Nousaine to come to Maine to hear the difference in cables in our reference systems. I would like to extend the invitation to your membership, not for the purpose of comparing cables, although we would certainly do that if anyone wanted. The reason for anyone coming would be to hear a very good audiophile sound system in (a) very good room. The room was designed by Ed Bannon of TAJ Soundworks. It is about 29' x 19' with no parallel walls and solid construction. We have a variety of equipment to cover the upper range of audiophile tastes, and best of all, cable by Transparent.

Anyone can contact me at 207-929-4553 to set up a listening session. I look forward to attending more of your sessions this season and I intend to become a member.

Jack Summer
Transparent Audio
Hollis, Maine"

[klinemj]

I don't understand why it will be complicated and costly.
As stated in the link in my previous post, this scenario could have been perfect

I had read that particular link about a year ago, and I did find it amusing - no doubt.

But, separate from this amusing story, the point of my post is that any "n=1" blind test is not statistically valid. The point is that large base sizes (read: a lot of people participating in the test) is required, and that's what is costly and complicated to execute.

And Lawrence - thanks for your positive comments to dyohn and I. Much appreciated...

Mark