|
|
Post by Boomzilla on Dec 24, 2016 9:35:54 GMT -5
So let's put this to rest once and for all - Level-matched, double-blind (LMDB) testing is ABSOLUTELY WORTHLESS.
Why? Because, in a nutshell, one, two, or even twelve comparisons are (likely) insufficient for the listener to identify differences. That's why such tests so seldom show any difference at all.
To hear subtle differences, the listener must hear the material repeatedly until the listener "learns what to listen for."
For this to happen in LMDB testing, the listener might have to hear the same material for weeks (or maybe even months) to learn what specific audible characteristics differentiated A from B. NOBODY wants to hear the same song over and over and over for that amount of time.
So ultimately, LMDB testing not only NEVER HAS worked, but also NEVER WILL work - period. The method itself is fantastically flawed. The "test sequence" is too brief to identify any but the most glaring differences in the limited number of comparisons available.
Looking for subtle differences? garbulky is RIGHT - You need to listen over time.
|
|
|
|
Post by beardedalbatross on Dec 24, 2016 10:27:57 GMT -5
Put to rest once and for all? Well I don't know about all that but lets dig in.
LMDB is completely worthless as compared to what exactly? I would agree that ideally you would be listening to music you already know, and on the fly switching is also useful.
|
|
klinemj
Emo VIPs 
Official Emofest Scribe
Posts: 15,096
|
Post by klinemj on Dec 24, 2016 10:34:29 GMT -5
If you are specifically referring to testing using an ABX comparator, I agree. If you are referring to all possible forms of level matched double blind testing, I disagree. There are many ways a valid test could be run and provide useful data. I have elaborated on that before in other threads. It's just that most people who claim to have done it have not done it well, and most would not bother to take the time to do it right.
Mark
|
|
|
|
Post by monkumonku on Dec 24, 2016 10:48:21 GMT -5
I'm not going to comment on the merits of DBT per se, but just say that I don't find it useful for my own audio evaluations.
It's too much trouble and whether I like gear because it truly does sound "better" (as in to my own ears) because physically it is, or if it is purely psychological, then what difference does it make. I suppose someone may say if it is psychological then why waste my money but then on the other hand how many things do we buy for purely psychological reasons?
If we didn't buy things for psychological factors then there wouldn't be competing brands of the same product.
The main thing is to be able to enjoy whatever you spend your hard earned money on. If you have to keep second-guessing yourself then what's the good in that (unless that's what brings you joy, rather than whatever the product is intended to do).
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 24, 2016 10:50:58 GMT -5
BINGO Monk
|
|
|
|
Post by vneal on Dec 24, 2016 10:56:26 GMT -5
I agree that double blind tests are not really accurate. You need to live with a component for a while.
|
|
DYohn
Emo VIPs
Posts: 18,494
|
Post by DYohn on Dec 24, 2016 11:22:18 GMT -5
DBT is absolutely valid and valuable for testing certain things like preference or specific responses to one variable. It is pretty close to worthless for trying to prove some cause and effect or for choosing personal audio gear. People who advocate it for audio gear are types who do not want to believe that personal preferences or inputs other than audible ones are valid. They are, generally, wrong.
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 24, 2016 11:37:41 GMT -5
You IMO are completely incorrect! Those who badmouth blind testing are mostly the people who think they can detect all the minuscule or even non-existing differences in sound from products that at most offer a very slight actual difference. There are many forms of blind testing and some have arguably some possible defects especially the more complicated ones. There are some very simple to conduct like my own Nut Blind Test as I posted in Garbulky's Shiit DAC reciew, which might blow what you think you hear right out of your room. However, the real reason the golden ear folks don't like blind tests is because the blind tests embarrass them and prove them wrong in most cases. The subjective listening tests like from Boom and Garbulky are simply full of faults. Look at the recent one and comments by both about the Schiit DAC. They both admit it is full of mistakes with a huge lack of any controls and thus basically invalid and useless IMO. Such reviews are mainly 100% subjective with no reasonable sound evaluation controls to avoid the common psychoacoustic tricks our brains can play on us when we are aware pre-test of the brand, model, price and other facts which affect our brains ability to deliver a 100% unbiased sound evaluation. Many of the reviews seem to be to promote the reviewer's self-established esteemed status at this and other forums for their expertise in speaker wire, cables, DAC's and other products, etc. that never seem to be presented with any or very few test/review controls. The one DAC review I have read that was blind was very controlled and confirmed that very few if any folks could distinguish high end DAC's except when listening to high end headphones. These product identified reviews can be very sloppy due to the lack of controls and are filled with creative writing type prose such as Grabulky's: The important thing in a DAC is can it communicate music? What the hell does that mean? Isn't that one of the purposes of all gear? My Nut Blind Test is relatively quick and quite effective. It tests if the subject can really hear any real sound differences in their own room. No other changes are made to their gear or room. Extremely minute differences they think they might hear in their own review suddenly disappear. If they think brand-A DAC is superior to brand-B DAC or even to just the built-in DAC in their pre-pro or CD player then they should be aware of it in their own system anytime they walk in and listen to music. If not the "upgrade" to an external DAC is meaningless other than to say I have a schiity DAC. My Nut Test: I go into my friends room when he is outside and he has already tweaked everything so his Shiit DAC sounds at its best. I have the ability to switch back and forth anytime between it and the DC-1. He comes in an cannot see which DAC is being used. I play a track from one of my best high quality CD's. He tells me which DAC he thinks is being used. I make a note whether he is correct or not. He leaves the room for a few minutes and I switch or do not switch the DAD randomly. I play the same CD track. We do this 20-30 or more times. Each time he hears the same music cut and has to tell which DAC is being used. Then for the next part of the test, when he comes in each time I play a different music track and he has to decide which DAC is being used, which I chose randomly. We do this again 20-30 times or more. I will bet 100,000 Yen he can't consistently choose the correct DAC more than by simple chance. I'll go one step further and switch back and forth from the Shiit Gungnirschwizer to the simple DAC in my Emo ERC-2 (which I will bring for this test). This time I'll bet 75,000 Yen he can't again tell the difference more than chance. No quick A/B switching here and variable volume level (med to loud) as I select. This is a real world audio hearing test and if he can't hear any difference when he casually walks into his sound room then all DAC comparison are out the window, plus the money spent on them. I consider Boom and Garbulky my Lounge friends and nice guys but we obviously strongly disagree on our review methods. I hope no personal offense is taken. My opinion and thoughts are seldom voiced here and I wanted to finally post them. I wait for someone to try my very simple blind test and report back. I unfortunately think it might be a long time coming! It's 1:20 am here on Christmas morning. I'm going outside for awhile to see if I can see Santa Craus over Sendai. They say he rides in a red Honda and stops for a break at the Kentucky Flied Chicken.  PS: I have to get up at 6:30am to watch the Seahawks/Arizona game online/NFL Game Pass. |
|
|
|
Post by novisnick on Dec 24, 2016 11:41:03 GMT -5
I firmly believe in the triple blind test!!  |
|
hemster
Global Moderator 
Particle Manufacturer
...still listening... still watching
Posts: 51,952
|
Post by hemster on Dec 24, 2016 11:43:37 GMT -5
Respectfully I disagree with the OP. klinemj and @chuckienut are correct. Except that Santa now rides in a Toyota!  |
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 24, 2016 11:48:52 GMT -5
I'm not going to comment on the merits of DBT per se, but just say that I don't find it useful for my own audio evaluations. It's too much trouble and whether I like gear because it truly does sound "better" (as in to my own ears) because physically it is, or if it is purely psychological, then what difference does it make. I suppose someone may say if it is psychological then why waste my money but then on the other hand how many things do we buy for purely psychological reasons? If we didn't buy things for psychological factors then there wouldn't be competing brands of the same product. The main thing is to be able to enjoy whatever you spend your hard earned money on. If you have to keep second-guessing yourself then what's the good in that (unless that's what brings you joy, rather than whatever the product is intended to do). Monku, I have to mention one point here. I know of many folks who have spent for mainly psychological reasons thousands of IMO useless dollars on $1000 DAC's for example plus expensive speaker connectors, cables, and other very questionable gear, way more than I have spent on my entire system! |
|
|
|
Post by copperpipe on Dec 24, 2016 12:46:49 GMT -5
Agreed with chuckienut and monku and others.
Look at the extremes to see why you are wrong. DBT a dayton audio $50 speaker against your axioms. How many trys / DBT's do you think you need to spot the differences? I bet you nail the speaker each time. Is DBT flawed, even though you CAN spot the difference here?
Now to the other extreme, DBT schiit X versus schitt Y. Much harder ain't it? That's because the differences are non existent or negligible. Is DBT flawed, even though you can't spot the difference here? No, DBT is accurate and telling you what you unfortunately don't want to accept: the reason you need to spend months finding a difference is a red neon honking sign telling you there ARE NO (notable) differences.
Respectfully, having a hobby is fine, but you can push it into an unhealthy obsession if you're not careful.
|
|
|
|
Post by monkumonku on Dec 24, 2016 13:04:48 GMT -5
I'm not going to comment on the merits of DBT per se, but just say that I don't find it useful for my own audio evaluations. It's too much trouble and whether I like gear because it truly does sound "better" (as in to my own ears) because physically it is, or if it is purely psychological, then what difference does it make. I suppose someone may say if it is psychological then why waste my money but then on the other hand how many things do we buy for purely psychological reasons? If we didn't buy things for psychological factors then there wouldn't be competing brands of the same product. The main thing is to be able to enjoy whatever you spend your hard earned money on. If you have to keep second-guessing yourself then what's the good in that (unless that's what brings you joy, rather than whatever the product is intended to do). Monku, I have to mention one point here. I know of many folks who have spent for mainly psychological reasons thousands of IMO useless dollars on $1000 DAC's for example plus expensive speaker connectors, cables, and other very questionable gear, way more than I have spent on my entire system! Count me among them. Not thousands of dollars, but certainly a fair amount! |
|
|
|
Post by garbulky on Dec 24, 2016 13:07:17 GMT -5
Well double blind testing....forgets the one thing. Positive results are true and testable. NULL RESULTS - DO NOT MEAN NO DIFFERENCE in real world use. And also it does not mean not if there are differences they are not worthwhile.
But the people who love DBT for audio...like thinking that it is. That that's what a null result means. DBT says nothing about those last two conclusions which they assume is the logical conclusion for a null test.
|
|
Deleted
Deleted Member
Posts: 0
|
Post by Deleted on Dec 24, 2016 13:13:46 GMT -5
Well double blind testing....forgets the one thing. Positive results are true and testable. NULL RESULTS - DO NOT MEAN NO DIFFERENCE in real world use. And also it does not mean not if there are differences they are not worthwhile. But the people who love DBT for audio...like thinking that it is. That that's what a null result means. DBT says nothing about those last two conclusions which they assume is the logical conclusion for a null test. Lul wut??? |
|
|
|
Post by garbulky on Dec 24, 2016 13:15:04 GMT -5
Well double blind testing....forgets the one thing. Positive results are true and testable. NULL RESULTS - DO NOT MEAN NO DIFFERENCE in real world use. And also it does not mean not if there are differences they are not worthwhile. But the people who love DBT for audio...like thinking that it is. That that's what a null result means. DBT says nothing about those last two conclusions which they assume is the logical conclusion for a null test. Lul wut??? was that unclear |
|
|
|
Post by beardedalbatross on Dec 24, 2016 13:16:25 GMT -5
Clearly  |
|
|
|
Post by copperpipe on Dec 24, 2016 13:23:26 GMT -5
Well double blind testing....forgets the one thing. Positive results are true and testable. NULL RESULTS - DO NOT MEAN NO DIFFERENCE in real world use. And also it does not mean not if there are differences they are not worthwhile. But the people who love DBT for audio...like thinking that it is. That that's what a null result means. DBT says nothing about those last two conclusions which they assume is the logical conclusion for a null test. I think you're making this more complicated than it needs to be. Either you can consistently spot the difference, or you can't. If you can spot the difference the vast majority of the time, then it proves there is a difference. If you can't spot the difference, it doesn't prove there is none, but it does tell you there is no point in worrying about any possible difference; pick the item based on price, brand, or looks, or anything other than sound at that point. You're blaming the testing methodology because you don't like what the results are telling you, rather than finding a flaw in DBT itself. |
|
|
|
Post by garbulky on Dec 24, 2016 13:32:55 GMT -5
Well what Chuckienut says (and I do like him too!) ... Golden eared audiophiles don't like it because they aren't able to identify even which gear sounds like he one they've been raving about because they will get a null result identifying things like different electronics. Showing that they are not amazing at hearing as they pride themselvers to be or that their setup isn't amazing. But I digress But how is that not getting a positive result a bad thing? The only thing bad about is that it fails a positive result which makes it definite without a doubt that there is a difference. It doesn't actually tell you that the product is not better. But a null test isn't bad. It's null. What it means is that you are correct no more often than somebody simply randomly guessing in that specific test situation. It may seem strange but that doesn't imply things like 1. They didn't hear it. 2. They can't also hear it in a non level matched real world listening. 3. The only way they'd be able to hear a difference would take so many tries in real world situations because it took them that many tries here. 4. Since they are correct so few times in THAT testing situation then those differences are not worthwhile. copperpipe5. The difference is not audible in a different situation - real world listening. All the above conclusions seem to make sense....except that is not what a null result means. They've made these up to say somebody really doesn't hear that in real world use. The two main differences things are .... 1. Null doesn't mean none. 2. And a null result means equal to random chance in THAT TESTING SITUATION. Level matched. A/B comparisons. Play one thing over and over. Blind. Not the same thing as how we actually use audio gear. (And not it's more or less the same.) 3. A double blind test usually requires extensive training. Usually totally ignored here. |
|
|
|
Post by audiobill on Dec 24, 2016 13:42:34 GMT -5
So there's a difference. Do you choose the component you like better or you've convinced yourself is more "accurate" whatever that means?
|
|
