Why double-blind testing is completely worthless.

[klinemj]

Dec 25, 2016 13:31:27 GMT -5 Jim said:

You can't possibly correlate professional research experience to DAC comparisons.

I can, but I have to be warmed up first.



Mark

[Boomzilla]

Thanks , Mark - I believe you. But your qualifier "...method error becomes vitally important," is the core of what I'm driving at.

I'm still thinking that the method itself is flawed for COMPLEX AUDITORY comparisons.

So to expound, if you're asking me which tint for a photo is more realistic between two samples, I can probably tell with few comparisons. If you're asking me which fabric texture is more pleasant to the fingers, I can also probably tell within few comparisons. If you're asking me to differentiate between two flavors, I can probably do it with few comparisons. If you're asking me to differentiate between two tones that differ by a semitone, I can probably tell within a few comparisons.

But asking which two playbacks (of the SAME original material - and where I've never heard the original performance at all) are different from each other is a challenge of a completely different magnitude.

[DYohn]

Boom, you are apparently not seeing the forest for the trees here. Complex auditory perceptions - or any other perception - is exactly what DBT is for and what it is best at. But you cannot draw too fine a conclusion from the results, nor can you attribute cause and effect except in very limited ways. You seem to want the result to lead to some sort of grand unified truths... nothing short of basic technical measurements can be that. Perception is about the furthest away from genuine truth as you can get. Like I said, just relax and enjoy the music.

[bootman]

Dec 25, 2016 13:07:40 GMT -5 Jim said:

All DACs sound the same. There. I said it.

Now we don't need to dispute the validity of DBT of DACs.

Happy now? :-)

DACs sound the same, it's the anolog stage after them that can sound different.

...and this is due to how well they drive the next component in the audio chain, and how you are positioned in the room, and environmental factors like temp and humidity.
...how the listener is feeling that day.

etc etc etc.

bottom line, it your toes are tapping to the music, it is sounding good.

However DBT is a valid testing methology. As an engineer I cannot refute this.

Happy Holidays everyone.

[ÈlTwo]

I'm trying to work out a protocol for next month when I'm planning to perform double blind-drunk testing.

So far my problems are identifying which components were heard, since everyone involved in the selection and the listening will be blind-drunk.
I guess we could video every step, send the video to a secure server and prohibit viewing of the video until the following weekend, when at least some of the participants might be sober.

[brand]

Dec 25, 2016 17:42:17 GMT -5 ÈlTwo said:

I'm trying to work out a protocol for next month when I'm planning to perform double blind-drunk testing.

So far my problems are identifying which components were heard, since everyone involved in the selection and the listening will be blind-drunk.
I guess we could video every step, send the video to a secure server and prohibit viewing of the video until the following weekend, when at least some of the participants might be sober.

Better yet sell the video to "Audiophiles Gone Wild"

[klinemj]

Dec 25, 2016 16:48:56 GMT -5 Boomzilla said:

Thanks , Mark - I believe you. But your qualifier "...method error becomes vitally important," is the core of what I'm driving at.

I'm still thinking that the method itself is flawed for COMPLEX AUDITORY comparisons.

So to expound, if you're asking me which tint for a photo is more realistic between two samples, I can probably tell with few comparisons. If you're asking me which fabric texture is more pleasant to the fingers, I can also probably tell within few comparisons. If you're asking me to differentiate between two flavors, I can probably do it with few comparisons. If you're asking me to differentiate between two tones that differ by a semitone, I can probably tell within a few comparisons.

But asking which two playbacks (of the SAME original material - and where I've never heard the original performance at all) are different from each other is a challenge of a completely different magnitude.

If you think audio is complex, try sorting out preference for other consumer products...Like diapers. Really.

Audio is easy by comparison. And I am not joking.

Mark

[Jim]

Dec 25, 2016 17:42:17 GMT -5 ÈlTwo said:

I'm trying to work out a protocol for next month when I'm planning to perform double blind-drunk testing.

So far my problems are identifying which components were heard, since everyone involved in the selection and the listening will be blind-drunk.
I guess we could video every step, send the video to a secure server and prohibit viewing of the video until the following weekend, when at least some of the participants might be sober.

Oh man. Can you post on YouTube after it's all complete?

[pedrocols]

So if I am able to identify differences, small and big, with my eyes open why do I need to cover my eyes for?

[sahmen]

Dec 25, 2016 17:42:17 GMT -5 ÈlTwo said:

I'm trying to work out a protocol for next month when I'm planning to perform double blind-drunk testing.

So far my problems are identifying which components were heard, since everyone involved in the selection and the listening will be blind-drunk.

I'm afraid you will, first,  have to get through the double-blind "drink" testing, i.e. the "do all wines taste the same?" double blind trial, which may just complicate things even further, given all the uncharted mysteries and secrets of winophilia that remain to be unlocked and proven, beyond all scientific doubt, reasonable or not...


 

are the uncharted mysteries of audiophilia not already complicated enough for you?

[garbulky]

ÈlTwo : also you have to go undergo the "Am I Drunk?" test. Here it's been proven that subjective "I am not really drunk" statements can be not entirely accurate. Therefore a strict prescription of alcohol consumption must be adhered. Also for good measure, there is a mandatory swig needed every time somebody mentions cable lifters or Bose.

[Jim]

Dec 25, 2016 21:10:07 GMT -5 garbulky said:

ÈlTwo : also you have to go undergo the "Am I Drunk?" test. Here it's been proven that subjective "I am not really drunk" statements can be not entirely accurate. Therefore a strict prescription of alcohol consumption must be adhered. Also for good measure, there is a mandatory swig needed every time somebody mentions cable lifters or Bose.

Also, breathalyzers always lie. So don't bother with those for testing drunk-ness. Actually, science is a lie too.

But bonus points if you can pronounce schitt while blind drunk!

[Deleted]

Dec 25, 2016 10:45:21 GMT -5 Boomzilla said:

Hi Chuck -

I contend that the double-blind method ITSELF is flawed (even your method). Why? LMDB itself has insufficient comparisons for the listener to "learn" what to listen for. The Chuckienut test is flawed due to listener fatigue.

The ONLY way to determine subtle differences is with extended listening. This allows the listener to compare again and again without the stress of having to decide in a short window, and without the fatigue of comparison after comparison. It also allows the listener time to identify subtle differences and to then compare back and forth to determine whether the differences are real or perceived. Over that time, the inherent "expectation bias" is abandoned (for me, within a week), and the listener can focus on what (if anything) differentiates the two devices.

The problem with extended listening is that it is less quantifiable than the more documentable tests because it (of necessity) occurs both over time and at the listener's convenience. But these "flaws" that make extended listening less amenable to documentation ALSO make it far, far more accurate.

Therefore ANY test that subjects the listener to short, stressful, and/or repetitive sessions has no credibility. Period.

Cordially - Boomzilla

Listener fatigue, that's cute Boom!  

I am in your own room, with your equipment except one or more DAC's., you sit on your sofa, you can even choose your own CD track if you like.  You can have a third neutral observer of your choice to make sure I don't pull any tricks on you.  However you know if you are smart at all that I might select the same DAC 5 times in a row which really is a mini test in itself of your true hearing.  The details of the test which are amazingly simple will be explained in advance.  You casually stroll in from the other room after I select one DAC unknown to you.   You can even ask that the volume be turned up or down to your preference for each listening.  Each track shouldn't take more than a minute max, again your preference.  There is no level matching or quick back and forth switching.  You should be very relaxed.  I'm very low keyed, friendly and non-pressure in person.  No booze allowed before or during test.  This should be completely fatigue free!

We can take a break between the two main sections if you like.  No pressure, no fatigue here.

The only reason you might feel fatigued (due to stress) is that you realize the test might end up indicating that you cannot consistently identify in this relaxed environment which DAC is playing more than compared to random chance.  This is the reason those with the golden ears ("I know what I hear and no damn quadruple blind test it going to change my mind") won't submit to such a simple test is the inner prophetic fear that they will in fact flunk the test.  I knew you and the Gar would not be agreeable to a test like this because you would avoid like the plague any test that might invalidate your amazing ability to identify such minuscule sound differences, such as:   The Schiit had the most midrange detail, but maybe the least midrange dynamics.    (huh?) 

Speaker listening tests are a whole different ballgame for obvious reasons.


 

[Deleted]

Dec 25, 2016 17:59:35 GMT -5 klinemj said:

Dec 25, 2016 16:48:56 GMT -5 Boomzilla said:

Thanks , Mark - I believe you. But your qualifier "...method error becomes vitally important," is the core of what I'm driving at.

I'm still thinking that the method itself is flawed for COMPLEX AUDITORY comparisons.

So to expound, if you're asking me which tint for a photo is more realistic between two samples, I can probably tell with few comparisons. If you're asking me which fabric texture is more pleasant to the fingers, I can also probably tell within few comparisons. If you're asking me to differentiate between two flavors, I can probably do it with few comparisons. If you're asking me to differentiate between two tones that differ by a semitone, I can probably tell within a few comparisons.

But asking which two playbacks (of the SAME original material - and where I've never heard the original performance at all) are different from each other is a challenge of a completely different magnitude.

If you think audio is complex, try sorting out preference for other consumer products...Like diapers. Really.

Audio is easy by comparison. And I am not joking.

Mark

I was a father of very young ones many years ago and changed many diapers (the real cloth ones, not Pampers).  It's easy dummy.  Haven't you ever heard of the blind smell test?

[Jim]

Hahaha. I'm just thinking..... percentage of men buying diapers has to be small. And most of them are probably going "what brand did she say, again?"

[Axis]

Dec 25, 2016 6:43:01 GMT -5 Jim said:

Dec 24, 2016 19:51:32 GMT -5 Axis said:

Double-blind testing may not be necessary but makes for a great thread topic on a Christmas eve day for the members of the Emotiva Lounge.

But the real question is, will the thread slow down on Christmas Day?

Will anyone have any new equipment to do testing with?
Will anyone set up a scientific DBT with their family?

Would you trust your in-laws to give honest answers to your test questions?!? Who claims to have golden ears?

Inquiring minds want to know.

All day Christmas day.


I have seen the future.

[knucklehead]

Dec 25, 2016 13:51:08 GMT -5 DYohn said:

DBT is just as valid a method for "small difference detection" as any other human perceptual-based testing method. But it's only as valid as the experimental setup and the results are only as valid as the interpretation. You cannot "prove" anything with perceptual-based tests, and this is the error most test advocates make: they jump to conclusions based on their very small, very specific-to-the-subject results.

Bottom line, boomzilla, is you are wrong when you say it is completely invalid, just like advocates are wrong when they say it is the "one true religion" for testing results of audio setups.

I say f*ck it man, relax, enjoy the music! If you are doing anything else you are wasting your time.

I'm still trying to figure out why this thread has lasted as long as it has.

Well said!

Enjoy the music.

[Deleted]

Dec 25, 2016 22:25:08 GMT -5 Jim said:

Hahaha. I'm just thinking..... percentage of men buying diapers has to be small. And most of them are probably going "what brand did she say, again?"

Well Jim, as some folks know here I have already way passed over the senior citizen level and I'm quickly approaching the very old fart level.  I read that I might need to be back in diapers again myself in not too long.  Nori-Nut ask me if I knew of a good brand yet for large size older men.  I told her yes.  She said how long would I wait to buy a supply to have here at home if and when quickly needed.  I responded:  Depends.

I also read I'll soon be repeating that joke often.   

[monkumonku]

I sure hope through all of this that y'all didn't overlook the fact that it was (and still is here in the PST) Christmas Day!

Me, I'm on vacation this week and plan to enjoy some two-channel and home theater stuff and I know it's gonna be good.

[RichGuy]

Dec 25, 2016 10:45:21 GMT -5 Boomzilla said:

The Chuckienut test is flawed due to listener fatigue.

Yes, lots of people get tired when listening to Chuckienut but that's another story and no blind testing is needed.