Why Do Audiophiles Fear ABX Testing?

[harsh]

Aug 3, 2015 8:14:22 GMT -5 DYohn said:

A/B/X relies on memory, not on comparison.

Then, just do a BX test.
Don't even bother to listen to A, just B then X.
Sounds the same, choose B.
Sounds different, choose A.
Repeat this a few time and see if you stay around 50% accurate, or more.

What can be objected to this protocol?

[DYohn]

Aug 3, 2015 13:08:33 GMT -5 harsh said:

Aug 3, 2015 8:14:22 GMT -5 DYohn said:

A/B/X relies on memory, not on comparison.

Then, just do a BX test.
Don't even bother to listen to A, just B then X.
Sounds the same, choose B.
Sounds different, choose A.
Repeat this a few time and see if you stay around 50% accurate, or more.

What can be objected to this protocol?

Because you are still relying on your memory of how A sounded. The best test is a back and forth A/B comparison. Actually, the BEST test is just to play a track you like and if it sounds good BUY IT. You really never know if you truly like something until you live with it for a while.

And no, I am not an audiophool who bases audio choices on belief. I am an engineer and designer and measurement tech since 1973, but I know enough to tell yo that measurements and "objectivity" only tell you part of the story. All your senses are involved when you listen to music, so they should be involved when you choose gear.

[garbulky]

Why do they fear? Well I don;t know about fear but I do know the implications doesn't reflect reality. If it DID, we won't be buying any Emotiva products except maybe a mini-x.

[405x5]

"Why Do Audiophiles Fear ABX Testing?"

Just be an "Average Joe" audio enthusiast (like me!)
Enjoy and TRUST YOUR EARS...then...ain't nothing to fear!!!!

Sorry, I just hate that word, audiophile. Anything with "phile" at the end of it makes me think something's already wrong.

Bill

[novisnick]

Aug 3, 2015 14:15:16 GMT -5 405x5 said:

"Why Do Audiophiles Fear ABX Testing?"

Just be an "Average Joe" audio enthusiast (like me!)
Enjoy and TRUST YOUR EARS...then...ain't nothing to fear!!!!

Sorry, I just hate that word, audiophile. Anything with "phile" at the end of it makes me think something's already wrong.

Bill

Yep!,,,,phile does sound like pile ya know!!    

                                                                                                          


Dont one forget to scrape your shoes if you step in it! 

[405x5]

 Uhh well, that wasn't EXACTLY what I was  thinking, (but I like it!)

[yves]

Aug 3, 2015 14:30:38 GMT -5 novisnick said:

Aug 3, 2015 14:15:16 GMT -5 405x5 said:

"Why Do Audiophiles Fear ABX Testing?"

Just be an "Average Joe" audio enthusiast (like me!)
Enjoy and TRUST YOUR EARS...then...ain't nothing to fear!!!!

Sorry, I just hate that word, audiophile. Anything with "phile" at the end of it makes me think something's already wrong.

Bill

Yep!,,,,phile does sound like pile ya know!!    

                                                                                                  View Attachment        


Dont one forget to scrape your shoes if you step in it! 

In France, Audio Piles are hearing aid batteries.
www.audio-piles.com/piles-a10-rayovac/48-piles-auditives-rayovac-a10-10-plaquettes-50037179.html

[Boomzilla]

The other problem with ABX testing, IMHO, is that the preponderance of statistical evidence is null and void. If 100 people take an ABX test and 99 of them can't tell a difference, but the 100th can tell every single time, then the 99% are wrong and the listener who hears the difference has proven (beyond any doubt whatsoever) that A & B are absolutely different

Yes, you might have to retest the one who got it right every time just to make sure it wasn't random chance, but if even one can hear reliable differences, then A & B are not the same.

[DYohn]

Aug 3, 2015 16:16:43 GMT -5 Boomzilla said:

The other problem with ABX testing, IMHO, is that the preponderance of statistical evidence is null and void. If 100 people take an ABX test and 99 of them can't tell a difference, but the 100th can tell every single time, then the 99% are wrong and the listener who hears the difference has proven (beyond any doubt whatsoever) that A & B are absolutely different

Yes, you might have to retest the one who got it right every time just to make sure it wasn't random chance, but if even one can hear reliable differences, then A & B are not the same.

Actually statistically that is not what youe example tells you at all. If you got results like that the test was likely not valid... and the conclusion would be that there was no difference since that's what 99% of the sample population reported.

[Boomzilla]

Ah but statistics DO lie. In this example, the 99% couldn't tell a difference. Maybe they couldn't hear high frequencies? Maybe they had presbyacusis? Maybe they didn't want to be taking the test at all & just didn't care?

But if even ONE person can reliably hear a difference, then the difference DOES exist. The other 99 just didn't hear it. The one might have been lucky on the first pass & guessed correctly, but if the one can consistently tell A from B, then it can NOT be ascribed to random chance - A & B ARE different - Period.

[DYohn]

Aug 3, 2015 16:57:58 GMT -5 Boomzilla said:

Ah but statistics DO lie. In this example, the 99% couldn't tell a difference. Maybe they couldn't hear high frequencies? Maybe they had presbyacusis? Maybe they didn't want to be taking the test at all & just didn't care?

But if even ONE person can reliably hear a difference, then the difference DOES exist. The other 99 just didn't hear it. The one might have been lucky on the first pass & guessed correctly, but if the one can consistently tell A from B, then it can NOT be ascribed to random chance - A & B ARE different - Period.

NO. If 99% could tell no difference there was no (significant) difference. Welcome to science.

[Boomzilla]

You're waffling - The word "significant" is IMPORTANT. I'll agree that if 99% can't tell a difference then there is no significant difference. But if the one can reliably tell the difference between A & B, THEN THERE IS A DIFFERENCE.

What may be totally insignificant to the 99% may well be VERY significant to the one who can hear the difference. And if 1% of the general population shares that same capability, then the preferred product may be worth more money to those who have the ability to not only hear, but also prefer that sound.

Boom

[klinemj]

KeithL essentially what you are describing in your proposed test is a specific execution of an A-B (often called a "paired comparison" test). In other words, you are describing what I said is the better approach.

And I knew yves would find this thread and comment on bias and other effects. And he is right that bias can exist in any test, so the test must be properly set up to minimize bias, properly administered, and analyzed accordingly. If not "gigo"... (Garbage in, garbage out).

And regarding the statistical point dyohn and boom are debating, dyohn is right that the proper conclusion is "no statistical difference was detected". Boom could be considered correct in that it would be valid to report that 1 person claimed to hear a difference. But, that does not override the statistical conclusion for the population. So, it really depends on how one words their conclusion as to whether the statement is valid relative to the data.

Mark

[DYohn]

Aug 3, 2015 17:21:41 GMT -5 Boomzilla said:

You're waffling - The word "significant" is IMPORTANT. I'll agree that if 99% can't tell a difference then there is no significant difference. But if the one can reliably tell the difference between A & B, THEN THERE IS A DIFFERENCE.

What may be totally insignificant to the 99% may well be VERY significant to the one who can hear the difference. And if 1% of the general population shares that same capability, then the preferred product may be worth more money to those who have the ability to not only hear, but also prefer that sound.

Boom

Waffling? No. It's science. It's how statistics works. Who cares if there is an actual difference if 99% of people cannot hear it? If 99% cannot hear it then it does not exist. At least not in any SIGNIFICANT way. And what if in reality it's your 1% who is wrong?

[brutiarti]

Looks like boomzilla is talking about facts and dyohn is talking statistics. For example pharmaceutical trials they use statistics, if the drug shows no significant difference it will not be developed, even if the drug worked for 1 of the 100 volunteers.

[Boomzilla]

I understand your position, DYohn - and from a statistical standpoint, you are completely correct. I've stated my disagreements, so there's no reason to go over them again. On this we can agree to disagree.

Respectfully - Boom

[Cogito]

Aug 3, 2015 9:41:43 GMT -5 DYohn said:

In my experience, A/B/X testing results are almost always statistically equivalent to flipping a coin.

And so it should. Well made modern electronics pretty much sound the same unless one design has obvious flaws in it's. ABX testing supports this.

[garym]

Aug 3, 2015 10:18:15 GMT -5 KeithL said:

If the subject can't statistically tell any better than a guess whether he or she is listening to one source repeatedly, or they're switching back and forth, then we will have determined that they are "indistinguishable".

Not bad, Keith. Though your test may be more sensitive than ordinary ABX testing, it still relies on memory --- when the subject hears a click he must compare the ensuing sound to the previous sound, which he only has in memory.

Unfortunately, unlike the colored tiles, we can't do side-by-side comparisons of sounds. Two or more nearly identical sounds cannot be presented, perceived, and processed simultaneously. All but one of the comparables must be retrieved from memory.

But we are fairly sensitive to changes in a continuing sound. Your test might work better if you eliminate the clicks --- the subject pushes the button and the component changes or does not change. After each push of the button the subject writes down whether or not he heard a change; his responses are compared to the record saved by the device. Eliminating the clicks allows the sound to continue seamlessly, and may eliminate some expectation bias.

[garym]

Aug 3, 2015 16:57:58 GMT -5 Boomzilla said:

But if even ONE person can reliably hear a difference, then the difference DOES exist. The other 99 just didn't hear it. The one might have been lucky on the first pass & guessed correctly, but if the one can consistently tell A from B, then it can NOT be ascribed to random chance - A & B ARE different - Period.

You're right. A person with extraordinary auditory acuity might perceive differences 99% of the population can't. As I suggested above, there is probably some "threshhold of difference" for each listener, below which that listener will not perceive a difference. But a difference may indeed exist, and be detectable with instruments (and that golden-eared listener).

[garym]

Aug 3, 2015 16:36:56 GMT -5 DYohn said:

... and the conclusion would be that there was no difference since that's what 99% of the sample population reported.

That conclusion would be unwarranted. The only conclusion that can be drawn from that result is that any differences present are undetectable by 99% of listeners. Whether a difference actually exists is not dependent upon the judgments of listeners, no matter how many of them you test.