|
|
Post by Jim on Dec 25, 2016 6:43:01 GMT -5
Double-blind testing may not be necessary but makes for a great thread topic on a Christmas eve day for the members of the Emotiva Lounge.  But the real question is, will the thread slow down on Christmas Day? Will anyone have any new equipment to do testing with? Will anyone set up a scientific DBT with their family? Would you trust your in-laws to give honest answers to your test questions?!? Who claims to have golden ears? Inquiring minds want to know. |
|
|
|
Post by brand on Dec 25, 2016 8:22:20 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part.
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 25, 2016 8:32:25 GMT -5
The feeling [that your method has a very logical and common sense approach] is the culprit. Do you have a method to test whether this feeling is actually correct? Yves, Oui ... Va te faire foutre |
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 25, 2016 8:40:02 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part. Brand said: I have been lusting over Schiit products for a while.
Oh but Brand, you are obviously very neutral and I didn't start this thread. You should go back and read this thread again. Obviously the Boom has softened his views very significantly. Garbulky on the other hand, well all I can say is he is very, very stubborn! Have you tired my Nut Test? Suggest you don't after you buy the Schiity stuff!  |
|
|
|
Post by brand on Dec 25, 2016 8:46:17 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part. Brand said: I have been lusting over Schiit products for a while.
Oh but Brand, you are obviously very neutral. You should go back and read this thread again. Obviously the Boom has softened his views very significantly. Garbulky on the other hand, well all I can say is he is very, very stubborn! Have you tired my Nut Test? Suggest you don't after you by the Schiity stuff! I will tonight just before I go to bed. This thread will do the job! |
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 25, 2016 8:52:56 GMT -5
Double-blind testing may not be necessary but makes for a great thread topic on a Christmas eve day for the members of the Emotiva Lounge. But the real question is, will the thread slow down on Christmas Day? Will anyone have any new equipment to do testing with? Will anyone set up a scientific DBT with their family? Would you trust your in-laws to give honest answers to your test questions?!? Who claims to have golden ears? Inquiring minds want to know. Jim, I think you should have posted this in the: The Dumb Questions/Dumb Statements Thread
|
|
klinemj
Emo VIPs 
Official Emofest Scribe
Posts: 15,096
|
Post by klinemj on Dec 25, 2016 9:33:40 GMT -5
Will anyone set up a scientific DBT with their family? I tried...they all said I am a nut and walked away shaking their heads in disbelief. Mark |
|
|
|
Post by Jim on Dec 25, 2016 9:51:32 GMT -5
But the real question is, will the thread slow down on Christmas Day? Will anyone have any new equipment to do testing with? Will anyone set up a scientific DBT with their family? Would you trust your in-laws to give honest answers to your test questions?!? Who claims to have golden ears? Inquiring minds want to know. Jim, I think you should have posted this in the:   The Dumb Questions/Dumb Statements Thread  Â
If you're trying to auction off family members at Christmas.... and it's a blind auction. Who will fetch the highest bids? Does a double silent test work well for speaker testing? |
|
|
|
Post by Boomzilla on Dec 25, 2016 10:45:21 GMT -5
Hi Chuck -
I contend that the double-blind method ITSELF is flawed (even your method). Why? LMDB itself has insufficient comparisons for the listener to "learn" what to listen for. The Chuckienut test is flawed due to listener fatigue.
The ONLY way to determine subtle differences is with extended listening. This allows the listener to compare again and again without the stress of having to decide in a short window, and without the fatigue of comparison after comparison. It also allows the listener time to identify subtle differences and to then compare back and forth to determine whether the differences are real or perceived. Over that time, the inherent "expectation bias" is abandoned (for me, within a week), and the listener can focus on what (if anything) differentiates the two devices.
The problem with extended listening is that it is less quantifiable than the more documentable tests because it (of necessity) occurs both over time and at the listener's convenience. But these "flaws" that make extended listening less amenable to documentation ALSO make it far, far more accurate.
Therefore ANY test that subjects the listener to short, stressful, and/or repetitive sessions has no credibility. Period.
Cordially - Boomzilla
|
|
|
|
Post by Boomzilla on Dec 25, 2016 10:48:28 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part. Everyone's entitled to their own opinions. Nobody's entitled to their own facts. |
|
|
|
Post by Jim on Dec 25, 2016 11:37:46 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part. Everyone's entitled to their own opinions. Nobody's entitled to their own facts. This thread is interesting because I've never seen anyone challenge DBT as being meaningless like this. If it's good enough for peer reviewed academic journals and legitimate research, it's good enough for me. It all comes down to methodology. Your methodology is either very very good and you control for most factors or it's crap. And if you're somewhere in the middle - you might learn something...but some people might not think it's credible. Read up on some "real" research that's been done using DBT. I think you might learn a lot. Clearly if Harmon (Dr. Toole, etc) spent the time and money to build an environment for testing where they could control a lot of the things that are hard to control.... there must have been SOME measure of useful information that's obtained. And what if you have trained listeners?! Because some researchers use trained listeners - people who can repeatably distinguish frequencies and artifacts. But I digress. It sounds like you already know all the answers. |
|
|
|
Post by brand on Dec 25, 2016 12:42:20 GMT -5
Why even start this thread? I've read through the whole thing and there is zero new information in it (as to be expected in a topic this ancient) and there is NOBODY in this thread who changed his/her mind about this topic in the slightest bit so really what's the point? (Besides a bit of banter of course which I don't mind) I see no discussion here just people stubbornly sticking to their agenda for the most part. Everyone's entitled to their own opinions. Nobody's entitled to their own facts. I don't see a whole lot of facts in this thread. Just lots of people presenting their opinions as facts which is usually the case whenever somebody pulls the old Patrick Moynihan quote out of the bag. This is not a topic that can be discussed factually to the point where nobody could argue otherwise (not even close!) because like many other discussions in the audio world it has a giant backdoor I call "you can't prove what I hear in my head" people love to use. It will usually sound like: "I see your XYZ proof/testing method, here is the flaw with it" and then proceeds to talk about the minor changes (insert audiphile language) he/her can hear (or in this case that the XYZ proof/testing isn't conclusive). I don't mind the thread I just don't see the point. |
|
|
|
Post by Boomzilla on Dec 25, 2016 12:43:21 GMT -5
The training is KEY. If you KNOW what to listen for, then LMDB testing is OK, because the learning curve has already been met. And I'd note that Dr. Toole's DBT routines were designed for SPEAKERS (where differences are MUCH greater than with DACs, etc.).
The concept of DBT is valid for large difference comparisons - it ISN'T for small differences. The smaller the difference, the less the DBT is valid.
I've always been uneasy about the validity of DBTs that supposedly "proved" that there were no differences between components, when I could CLEARLY hear that there were. This thread is my attempt to quantify exactly WHY the DBT method wasn't / isn't matching what I hear. I don't claim to have all the answers - I'm just thinking out loud about an issue that's bothered me for many years.
I'd love to hear Dr. Toole's responses to my contentions. He might convince me that my speculations are wrong. In which case, I'd feel obligated to search for other reasons why DBTs don't match my perceptions.
But one way or another, I'm convinced that DBT isn't functioning well at discriminating between small differences.
Boomzilla
PS - If anyone feels that this thread is worthless, then feel free not to read it. Nobody's forcing you...
|
|
|
|
Post by garbulky on Dec 25, 2016 12:56:00 GMT -5
Forget the extended period of time with your DAC. If there is in fact a warm up period or burn in period (which I don't agree with, maybe a hour or so should be 55 minutes too long). Once you listen in a controlled manner you should come away with the same impressions now as 6 months from now. I don't have any questions except why all the changing your tune from one post to the next. Go back and read your posts again and see all the conflicting comments. See that's not been my experience - at least with the Schiit gear. Why should I accept that? You haven't heard it. But even if you did, so have I. Whose findings matter more? One's own. Also you don't really seem to be interested in my responses. Everything I write, you are simply trying to find holes in it. You are convinced. Which is fine. I am happy you are. I haven't changed my mind either. But I do try to read what the other person is trying to communicate in the way they try do it. Not trying to be rude here. You did spend time illustrating clearly what it is. I like that it's not level matched. But I have no desire to do so. It sounds boring and all I can think is what good is this going to do? Regardless of the result it has little bearing with how I experience audio. Maybe I'm not giving this method a fair shot. Or maybe, I'm unwilling to give my listening an unfair shot. Depends on one's perspective. Do what works for you. I'll do the same. There's room for differing methods! Ah your opionion is any subjective review of electronics with results of differences is PURELY IMAGINARY. Then why waste time on it? I guess I can't help but feel a little put off here. You come from an approach or experience that is the opposite of where I'm coming from and criticize pretty much most of the stuff in the review - which was just two dudes listening to some gear. As Bzilla says different universes. It's like a scientist criticizing an artist. It's too different. It makes no sense to argue about it. Scientist "why did you paint the tree blue? When I measured the tree that you drew it was six feet. But here it is only six inches and blue. The tree was brown dangit. I measured it with my color meter. You have imagined it." Artist "But the blue feels better no?" Scientist: "What?! You are so delusional. My color meter says nothing about this. And I even closed my eyes while using it to measure your painting. My phd says absolutely nothing about the feeling this tree gives. You are full of it! Let me tell you a bit more about spectrophotometry. I think you really need to hear this!" |
|
|
|
Post by Jim on Dec 25, 2016 13:07:40 GMT -5
All DACs sound the same. There. I said it.
Now we don't need to dispute the validity of DBT of DACs.
Happy now? :-)
|
|
klinemj
Emo VIPs
Official Emofest Scribe
Posts: 15,096
|
Post by klinemj on Dec 25, 2016 13:18:03 GMT -5
The concept of DBT is valid for large difference comparisons - it ISN'T for small differences. The smaller the difference, the less the DBT is valid. In professional research (which I have done for 30 years), the first part of the statement is wrong. In truth, the smaller the difference, the more important it is to use a technique like a DBT. That said, the smaller the difference, the harder it is to detect and prove there is a difference. Hence, the second part is basically right even though I could quibble with your wording of "less valid". Harder to be statistically valid but not necessarily is it less valid. It is key to hear enough of a selection to assess the sound and to not create fatigue. But, used properly, the technique is excellent. As I mentioned, I have used versions for 30 years professionally...for products other than A/V types. Mark |
|
|
|
Post by Jim on Dec 25, 2016 13:31:27 GMT -5
The concept of DBT is valid for large difference comparisons - it ISN'T for small differences. The smaller the difference, the less the DBT is valid. In professional research (which I have done for 30 years), the first part of the statement is wrong. In truth, the smaller the difference, the more important it is to use a technique like a DBT. That said, the smaller the difference, the harder it is to detect and prove there is a difference. Hence, the second part is basically right even though I could quibble with your wording of "less valid". Harder to be statistically valid but not necessarily is it less valid. It is key to hear enough of a selection to assess the sound and to not create fatigue. But, used properly, the technique is excellent. As I mentioned, I have used versions for 30 years professionally...for products other than A/V types. Mark You're not a professional DAC tester. Therefore everything you've just said is invalid. Including your research experience. You can't possibly correlate professional research experience to DAC comparisons. Crazy man. |
|
|
|
Post by Boomzilla on Dec 25, 2016 13:45:18 GMT -5
The concept of DBT is valid for large difference comparisons - it ISN'T for small differences. The smaller the difference, the less the DBT is valid. In professional research (which I have done for 30 years), the first part of the statement is wrong. In truth, the smaller the difference, the more important it is to use a technique like a DBT. That said, the smaller the difference, the harder it is to detect and prove there is a difference. Hence, the second part is basically right even though I could quibble with your wording of "less valid". Harder to be statistically valid but not necessarily is it less valid. It is key to hear enough of a selection to assess the sound and to not create fatigue. But, used properly, the technique is excellent. As I mentioned, I have used versions for 30 years professionally...for products other than A/V types. Mark Hi Mark - Yes - I agree that statistical validation may be more IMPORTANT for small differences than for large ones. But what I'm (apparently poorly) trying to hypothesize is that the DBT technique itself is more reliable for large differences than for small ones. We agree that any such test would need to be "enough...to assess the sound and not create fatigue," but my contention, here, is that such a test, while sufficient to detect large differences, may not be sufficient for the subject to identify small ones. There is no disagreement that when used properly, the technique IS excellent. But the technique, itself, may invalidate any results intended to discern small differences. In other words, exactly what I'm questioning is whether the LMDB technique is inherently inappropriate for the "small-difference detection" that we're trying to use it for. |
|
DYohn
Emo VIPs
Posts: 18,494
|
Post by DYohn on Dec 25, 2016 13:51:08 GMT -5
DBT is just as valid a method for "small difference detection" as any other human perceptual-based testing method. But it's only as valid as the experimental setup and the results are only as valid as the interpretation. You cannot "prove" anything with perceptual-based tests, and this is the error most test advocates make: they jump to conclusions based on their very small, very specific-to-the-subject results.
Bottom line, boomzilla, is you are wrong when you say it is completely invalid, just like advocates are wrong when they say it is the "one true religion" for testing results of audio setups.
I say f*ck it man, relax, enjoy the music! If you are doing anything else you are wasting your time.
|
|
klinemj
Emo VIPs
Official Emofest Scribe
Posts: 15,096
|
Post by klinemj on Dec 25, 2016 14:56:29 GMT -5
In professional research (which I have done for 30 years), the first part of the statement is wrong. In truth, the smaller the difference, the more important it is to use a technique like a DBT. That said, the smaller the difference, the harder it is to detect and prove there is a difference. Hence, the second part is basically right even though I could quibble with your wording of "less valid". Harder to be statistically valid but not necessarily is it less valid. It is key to hear enough of a selection to assess the sound and to not create fatigue. But, used properly, the technique is excellent. As I mentioned, I have used versions for 30 years professionally...for products other than A/V types. Mark ...hypothesize is that the DBT technique itself is more reliable for large differences than for small ones... ...But the technique, itself, may invalidate any results intended to discern small differences... In other words, exactly what I'm questioning is whether the LMDB technique is inherently inappropriate for the "small-difference detection" that we're trying to use it for. EVERY technique to assess small differences in anything is inherently more challenging because method error becomes vitally important. However, the DBT method is inherently one of the best to assess small differences. So, it is not inherently inappropriate. It is inherently appropriate. That is why it is commonly used on a wide variety of sensory-related research in many types of products...from food, to perfumes/scents, visual appeal of objects, and even what I do...diaper research...both technical and consumer preference testing. Mark |
|
