|
|
Post by geebo on Mar 11, 2013 22:04:22 GMT -5
The brain is part of the hearing system. No machine known to mankind can perfectly measure everything that goes on inside the brain, re perception and the conscious present. OTOH, there are measuring devices that can detect things no unaided human sense can. |
|
|
|
Post by garym on Mar 11, 2013 22:44:10 GMT -5
However, the very first thing he points out is if you listen for high frequencies, you might not hear low frequencies and, if you listen for low frequencies, you might not hear high frequencies. Meaning, our perception can be altered, or "steered", depending on what aspect of sound we try to listen for, or "focus on". You are merely describing attention. If you are attending to a certain stimulus in the environment, other stimuli that might be present are ignored or relegated to "background noise." That has no bearing on ABX testing. ABX testing does not require focusing attention on any particular aspect of the sound. Indeed, it requires attention to *all* the sound presented, since the subject has no idea what aspects of the sound will differ (if any). It merely requires critical listening --- the same attention paid by a recording engineer placing mics or mixing tracks. Critical listening is not "bias." You bet. Our cognitive apparatus strives for pattern completion. If details of a familiar pattern are missing in a percept, the brain fills them in. That fact has no bearing on the validity of ABX testing either. The argument here is not whether the brain can infill data that is missing from a percept, but whether the sensory system, including the brain, can detect data that is (allegedly) present in the percept. That argument of yours merely affirms the brain can be fooled into "hearing" things that are not there in fact. Which we all already know.  |
|
|
|
Post by arthurz on Mar 11, 2013 23:44:12 GMT -5
The brain is part of the hearing system. No machine known to mankind can perfectly measure everything that goes on inside the brain, re perception and the conscious present. You are switching the argument on me. My reply was to this: Further, there is no reason to assume test instruments are sufficiently sensitive to measure every important aspect of sound The brain has nothing to do with it!!! Not unless you're suggesting ESP. |
|
|
|
Post by yves on Mar 12, 2013 5:02:36 GMT -5
You are merely describing attention. If you are attending to a certain stimulus in the environment, other stimuli that might be present are ignored or relegated to "background noise." That has no bearing on ABX testing. Humans constantly steer their own attention both consciously and subconsciously. Perception is constantly altered by the brain not only because of that, but also because of numerous other natural processes that occur inside the brain. As explained by Poppy Crum in that video I linked, an important part of the bias that steers our perception is coming from the brain, and it simply cannot be eliminated by ABX testing. In the experiment she used in her presentation, the test subjects hadn't been told what the test sound was, so yes they were forced to listen blindly, yet they all got it wrong. This proves beyond a shadow of a doubt it does indeed have bearing on ABX testing. I never said ABX testing requires focusing on specific aspects of the sound. All I was saying is it's human nature to focus on specific aspects of the sound. We just do it all the time. It's hard to predict what those specific aspects will be. Basically, all we know is that it's a natural phenomenon that's not as easy to control as you might think. It's even harder to predict the impact this phenomenon will have on our perception. It all requires special training. People who are specialized in psychoacoustics will tell you that. The argument here is the outcome of ABX testing is steered by the brain itself because like I said the brain is a very important part of the hearing system. Since ABX testing forces the test subjects to listen specifically for differences whereas a normal listening session doesn't, the brain is effectively altered by ABX testing. The only logical conclusion is you cannot prove it has no bearing. Experiments in psychoacoustics indicate the opposite is true. Yes, it merely affirms the brain can be fooled even during ABX testing, as some of us already found out. |
|
|
|
Post by yves on Mar 12, 2013 5:19:49 GMT -5
The brain has got everything to do with sound, because we require it to be able to perceive sound. If we want to try and understand what aspects of sound are important to us and by how much, we not only have to study measurements of sound, but we also have to study how we hear. One is meaningless without the other. If you're going to assume measurements are always sufficiently accurate to describe all aspects of sound except ones that are unimportant to the listener, you'll have to substantiate your claim. The brain is part of the hearing system. No machine known to mankind can perfectly measure everything that goes on inside the brain, re perception and the conscious present. You are switching the argument on me. My reply was to this: Further, there is no reason to assume test instruments are sufficiently sensitive to measure every important aspect of sound The brain has nothing to do with it!!! Not unless you're suggesting ESP. |
|
KeithL
Administrator 
Posts: 10,273
|
Post by KeithL on Mar 12, 2013 10:09:00 GMT -5
I'm afraid you're missing the entire point. The point of ABX (or any other double blind testing) is that the subject is blind TO WHICH PRODUCTS OR SUBJECTS ARE BEING TESTED. It does NOT mean that they are blind to the other aspects of the situation. What you're describing would be equivalent to handing someone a pill, having them swallow it, and then asking "do you feel any different". When we test drugs we do NOT do that. Our subjects know what the pill is supposed to do; the ONLY thing they are blind to is which pill they got. Likewise, if we were to swap interconnects while our subjects were out of the room, then ask "do you hear anything different", it is true that the focus of their attention would influence whether they noticed the change or not. (I say it would still have some validity). However, we don't do that either. Instead, we tell them "we're going to try two different interconnects"; so they know what to listen for within reasonable parameters. And, yes, they are going to apply the same criteria to the situation that they usually do, so one subject may be more concerned with frequency response while another notices mostly the noise floor. The point is that, since they know they are testing an audio component, they use the same criteria to judge it as they normally do when they listen to music. [If you have to TELL them what to listen for, then you are artificially altering the situation. If your UUT (unit under test) changes something which the subject doesn't notice unless it is pointed out to him/her, then it really doesn't matter. Not to mention the obvious problem that, by "planting" expectations, you encourage subjects to hear things that aren't there because they expect to hear them.] [Did you ever wonder if when, after installing some new component, and asking your spouse "gee, honey, can you hear any difference?", maybe she can hear by the hopeful tone of your voice that you're hoping she'll say "gee, honey, that sure sounds better"?] The fact is that anything you can claim about ABX testing adding any sort of unfairness or bias is 1000x worse when you make the test NON-BLIND. If you do that, you just add in all the problems with expectation bias - which pretty much overshadows ANY ability to judge ANYTHING for real. [And you CAN do an ABX test without a time limit, and allow subjects to listen in their own homes, with their own music, and even allow them to use their own audio system to do the listening. It just takes a bit more effort.] You can also CONTROL FOR many of the issues in ABX testing. For example, if I were to do an ABX test, but actually provide the SAME sample every time, it is indeed true that some subjects would imagine differences that weren't there. However, those imaginary differences would statistically "cancel out" and we would get a null result. (The fact that most subjects might "claim" to have a favorite would be interesting but not statistically significant.) Likewise, if we TOLD the subjects that two different interconnects were the same, they might hear a difference, or their expectation not to could bias them to NOT hear a difference that really existed. In that situation, they might end up with a null response (not hear a difference) or they might hear a difference less often (because they expect not to). HOWEVER, and this is the important thing, none of those biases would lead to a result that showed a difference that didn't really exist. We would be "protected" from that possibility. In other words, having an ABX test end with "no result" doesn't PROVE BEYOND DOUBT that a difference doesn't exist. It does, however, strongly suggest that, if a difference does exist, it is relatively minor. I leave it to you whether you would buy a $100 bottle of wine instead of a $10 bottle IF, in a blind taste test done by 100 people, the $100 bottle was NOT preferred on average over the $10 bottle. (Even if most people aren't wine connoisseurs, I would expect at least some significant bias towards the $100 bottle if it was really better.) Of course, we are still dealing with people, so we can consider a test of whether they hear a difference or not to be scientific, but an expression of PREFERENCE will always be a survey of popular opinion. (In other words, if the cheap wine rated better, it could just be that most people PREFER the taste of cheap wine even though, by some "higher standard of criticism" it is "worse".) For example, it could be that the reason people preferred MP3 files to WAV files in a recent survey is that they are more used to poorly compressed files, and so the MP3 files are more in tune with their expectations of what a fie "usually" sounds like. The fact that they expressed a preference shows that they could hear a difference (which "proves" that MP3 files are NOT identical to the original); the fact that they preferred MP3s demonstrates a matter of opinion amongst the survey group. You are merely describing attention. If you are attending to a certain stimulus in the environment, other stimuli that might be present are ignored or relegated to "background noise." That has no bearing on ABX testing. Humans constantly steer their own attention both consciously and subconsciously. Perception is constantly altered by the brain not only because of that, but also because of numerous other natural processes that occur inside the brain. As explained by Poppy Crum in that video I linked, an important part of the bias that steers our perception is coming from the brain, and it simply cannot be eliminated by ABX testing. In the experiment she used in her presentation, the test subjects hadn't been told what the test sound was, so yes they were forced to listen blindly, yet they all got it wrong. This proves beyond a shadow of a doubt it does indeed have bearing on ABX testing. I never said ABX testing requires focusing on specific aspects of the sound. All I was saying is it's human nature to focus on specific aspects of the sound. We just do it all the time. It's hard to predict what those specific aspects will be. Basically, all we know is that it's a natural phenomenon that's not as easy to control as you might think. It's even harder to predict the impact this phenomenon will have on our perception. It all requires special training. People who are specialized in psychoacoustics will tell you that. The argument here is the outcome of ABX testing is steered by the brain itself because like I said the brain is a very important part of the hearing system. Since ABX testing forces the test subjects to listen specifically for differences whereas a normal listening session doesn't, the brain is effectively altered by ABX testing. The only logical conclusion is you cannot prove it has no bearing. Experiments in psychoacoustics indicate the opposite is true. Yes, it merely affirms the brain can be fooled even during ABX testing, as some of us already found out. |
|
|
|
Post by yves on Mar 12, 2013 11:52:21 GMT -5
I am already fully aware of all that, but I still disagree that differences that cannot be detected in a double-blind listening test are necessarily always too small to worry about. My point is double-blind listening tests are flawed, as carefully explained by Robert Harley www.avguide.com/forums/blind-listening-tests-are-flawed-editorial and Bob Stuart www.theabsolutesound.com/articles/tas-194-meridian-audios-bob-stuart-talks-with-robert-harley-1I'm afraid you're missing the entire point. The point of ABX (or any other double blind testing) is that the subject is blind TO WHICH PRODUCTS OR SUBJECTS ARE BEING TESTED. It does NOT mean that they are blind to the other aspects of the situation. What you're describing would be equivalent to handing someone a pill, having them swallow it, and then asking "do you feel any different". When we test drugs we do NOT do that. Our subjects know what the pill is supposed to do; the ONLY thing they are blind to is which pill they got. Likewise, if we were to swap interconnects while our subjects were out of the room, then ask "do you hear anything different", it is true that the focus of their attention would influence whether they noticed the change or not. (I say it would still have some validity). However, we don't do that either. Instead, we tell them "we're going to try two different interconnects"; so they know what to listen for within reasonable parameters. And, yes, they are going to apply the same criteria to the situation that they usually do, so one subject may be more concerned with frequency response while another notices mostly the noise floor. The point is that, since they know they are testing an audio component, they use the same criteria to judge it as they normally do when they listen to music. [If you have to TELL them what to listen for, then you are artificially altering the situation. If your UUT (unit under test) changes something which the subject doesn't notice unless it is pointed out to him/her, then it really doesn't matter. Not to mention the obvious problem that, by "planting" expectations, you encourage subjects to hear things that aren't there because they expect to hear them.] [Did you ever wonder if when, after installing some new component, and asking your spouse "gee, honey, can you hear any difference?", maybe she can hear by the hopeful tone of your voice that you're hoping she'll say "gee, honey, that sure sounds better"?] The fact is that anything you can claim about ABX testing adding any sort of unfairness or bias is 1000x worse when you make the test NON-BLIND. If you do that, you just add in all the problems with expectation bias - which pretty much overshadows ANY ability to judge ANYTHING for real. [And you CAN do an ABX test without a time limit, and allow subjects to listen in their own homes, with their own music, and even allow them to use their own audio system to do the listening. It just takes a bit more effort.] You can also CONTROL FOR many of the issues in ABX testing. For example, if I were to do an ABX test, but actually provide the SAME sample every time, it is indeed true that some subjects would imagine differences that weren't there. However, those imaginary differences would statistically "cancel out" and we would get a null result. (The fact that most subjects might "claim" to have a favorite would be interesting but not statistically significant.) Likewise, if we TOLD the subjects that two different interconnects were the same, they might hear a difference, or their expectation not to could bias them to NOT hear a difference that really existed. In that situation, they might end up with a null response (not hear a difference) or they might hear a difference less often (because they expect not to). HOWEVER, and this is the important thing, none of those biases would lead to a result that showed a difference that didn't really exist. We would be "protected" from that possibility. In other words, having an ABX test end with "no result" doesn't PROVE BEYOND DOUBT that a difference doesn't exist. It does, however, strongly suggest that, if a difference does exist, it is relatively minor. I leave it to you whether you would buy a $100 bottle of wine instead of a $10 bottle IF, in a blind taste test done by 100 people, the $100 bottle was NOT preferred on average over the $10 bottle. (Even if most people aren't wine connoisseurs, I would expect at least some significant bias towards the $100 bottle if it was really better.) Of course, we are still dealing with people, so we can consider a test of whether they hear a difference or not to be scientific, but an expression of PREFERENCE will always be a survey of popular opinion. (In other words, if the cheap wine rated better, it could just be that most people PREFER the taste of cheap wine even though, by some "higher standard of criticism" it is "worse".) For example, it could be that the reason people preferred MP3 files to WAV files in a recent survey is that they are more used to poorly compressed files, and so the MP3 files are more in tune with their expectations of what a fie "usually" sounds like. The fact that they expressed a preference shows that they could hear a difference (which "proves" that MP3 files are NOT identical to the original); the fact that they preferred MP3s demonstrates a matter of opinion amongst the survey group. |
|
|
|
Post by garym on Mar 12, 2013 13:41:46 GMT -5
Humans constantly steer their own attention both consciously and subconsciously. Perception is constantly altered by the brain not only because of that, but also because of numerous other natural processes that occur inside the brain. As explained by Poppy Crum in that video I linked, an important part of the bias that steers our perception is coming from the brain, and it simply cannot be eliminated by ABX testing. There is no need to eliminate it. That phenomenon is universal, and thus in no way compromises the validity of ABX testing. In order for it to create a "bias," you'd have to show that the "steering" you mention when the "A" unit is played differs from that when the "B" unit is played. Since the subject has no idea which unit is online, it is difficult to see how that change in "steering" could be triggered. You could argue that the mere fact that the subject knows he is in a test scenario alters the way his brain "steers" the information, and does it in such a way that differences he would notice in a "normal" listening scenario become undetectable in the test situation. But that leads back to the unfalsifiability problem. It shows no such thing. It shows nothing whatever about ABX testing; it only shows how human audio perception works, during ABX testing or any other time Yes. So how does that fact invalidate ABX testing? Mr Harley claims that "blind listening tests fundamentally distort the listening process." But he has shown no such thing. All he has show is that the mechanisms of human audio perception work the same way during ABX testing as during any other listening experience. If you are convinced that neither ABX testing nor test equipment can reveal the differences you claim exist, then the burden falls upon you to produce another means of *objectively* demonstrating those differences. Otherwise your claim becomes unfalsifiable, and thus vacuous *prima facie*. |
|
|
|
Post by yves on Mar 12, 2013 19:07:48 GMT -5
There is no need to eliminate it. That phenomenon is universal, and thus in no way compromises the validity of ABX testing. In order for it to create a "bias," you'd have to show that the "steering" you mention when the "A" unit is played differs from that when the "B" unit is played. Since the subject has no idea which unit is online, it is difficult to see how that change in "steering" could be triggered. You could argue that the mere fact that the subject knows he is in a test scenario alters the way his brain "steers" the information, and does it in such a way that differences he would notice in a "normal" listening scenario become undetectable in the test situation. But that leads back to the unfalsifiability problem. The experiment shows the brain is perfectly able to fill in missing portions of sounds using nothing but its own interpretation of what things SHOULD sound like, based on memory imprints of what they DID sound like in the past. An ABX test requires people to listen to one sound first and then another sound after that. Listening to one sound first will create a memory imprint even before the listener gets a chance to listen to the other sound. Meaning, listening to the other sound next will be biased by this memory imprint, as shown in the experiment. Quid est demonstrandum. As per my above comment, human audio perception works in such way that it causes ABX testing to be fundamentally flawed. ABX testing changes the way we focus our attention because it forces us to focus our attention on whether we can hear a difference rather than allowing us to focus on nothing but the sound itself. This causes stress, and it has been proven elsewhere that stress keeps us from being able to tell the difference between quite surprising things. Perhaps he has not. However, Poppy Crum and I both have. ...Which is exactly the reason why ABX testing is fundamentally flawed. If you had the memory of a goldfish, maybe it would work. If my uncle had breasts, he'd be my aunt. |
|
|
|
Post by garym on Mar 12, 2013 20:02:28 GMT -5
The experiment shows the brain is perfectly able to fill in missing portions of sounds using nothing but its own interpretation of what things SHOULD sound like, based on memory imprints of what they DID sound like in the past. An ABX test requires people to listen to one sound first and then another sound after that. Listening to one sound first will create a memory imprint even before the listener gets a chance to listen to the other sound. Meaning, listening to the other sound next will be biased by this memory imprint, as shown in the experiment. No, Yves, that is not how that works, and not what Poppy's demo shows. The memory infill to which Poppy is referring is from long-term memory, i.e., memories of sounds in place before the testing begins. Her demo is an example of expectation bias, or pattern completion. There is no expectation or pattern established by a sound heard moments before that prevents you from perceiving a change in that sound, if one exists. That hypothesis would preclude understanding speech, which is a sequence of quickly changing, unpredictable sounds! We do not fail to perceive an "s" sound because we heard a "th" sound immediately before it (as in "this"). "Focusing on the sound" and, "focusing on differences in sounds" are not mutually exclusive. Moreover, "focusing" is the wrong word. There is nothing there at the outset upon which to focus (as one might focus, or attend, to the sax and thereby miss a sour note on the piano). In ABX testing you must "focus" on all of the music presented, in order to hear any differences there might be, since you don't know when they will appear or what they will be. You need to deal with that unfalsifiability problem, Yves. |
|
|
|
Post by arthurz on Mar 12, 2013 21:13:10 GMT -5
Your argument about needing to listen to A first and B second in an ABX test is strange. You're basically saying that no comparisons can ever be made (not even using machines!). Let's theoretically assume you're right. Now what? You end up in this extremely confused world where no meaningful statement about audio quality can be made.
Previously you brought up the ability to hear digital artifacts 100 dB below the signal. Are you just suggesting uncertainty or are you claiming a positive result of some sort, and if so, on what basis (if not ABX testing)?
|
|
|
|
Post by yves on Mar 13, 2013 19:21:50 GMT -5
There is no expectation or pattern established by a sound heard moments before that prevents you from perceiving a change in that sound, if one exists. Got proof? Because, Bob Stuart of Meridian Audio, who has a Ph.D in auditory neuroscience, who is an expert in psychoacoustics, and who is a respected member of the Audio Engineering Society, totally disagreed in that TAS interview I linked earlier in the thread. I have pointed this out on another forum in the past. Wanna know what happened after I did? Robert Harley, the guy who interviewed Bob Stuart, got accused of probably having skewed Bob Stuart's words. Worse, the guy who made this accusation never deemed it necessary to provide objective evidence to support this accusation. Following the same logic, it would be fair to conclude that any statements made by self-proclaimed objectivists, who are probably subjectivists in denial, must be far more from the truth than merely subjective statements. en.wikipedia.org/wiki/Confirmation_biasThe fact perception can be shaped by memory is what keeps people from being able to "focus" on all of the music presented. Straw man argument. |
|
|
|
Post by yves on Mar 13, 2013 19:45:08 GMT -5
Your argument about needing to listen to A first and B second in an ABX test is strange. You're basically saying that no comparisons can ever be made (not even using machines!). Let's theoretically assume you're right. Now what? You end up in this extremely confused world where no meaningful statement about audio quality can be made. Exactly. Audio is part science, part subjectiveness. Like I said, it was just a metaphor I used to illustrate my point that there is not a linear relationship between the measured size of an error in an audio signal and the reduction in musical satisfaction (if any) this error engenders. Try reading that TAS interview with Bob Stuart again, it very very carefully explains a whole lot of the reasons why neither ABX testing nor measurement techniques are necessarily always the best tool for the job if our job is to improve the quality of sound. |
|
|
|
Post by garym on Mar 13, 2013 22:34:27 GMT -5
The burden of proof rests with he who holds the affirmative, Yves. That means that if Dr Stuart contends that a sound heard seconds before can cause us not to hear another sound seconds later, it is up to him to produce evidence for that claim. Pre-existing memories of familiar material can cause us to fill in information that is missing or has been altered in a new presentation. There is no evidence that a sound heard seconds ago will prevent us from hearing a new sound seconds later. As I said, if that were the case, we could not understand a spoken sentence, since the phonemes heard first would cause us to miss or mis-hear phonemes uttered later. Moreover, there is no evidence that memory feedback works any differently during an ABX session than at any other time. You've advanced a claim which, apparently, no conceivable empirical observation could show to be false. That means it has zero information content, i.e., the world would look exactly the same whether it were true or false. en.wikipedia.org/wiki/Falsifiability |
|
KeithL
Administrator
Posts: 10,273
|
Post by KeithL on Mar 13, 2013 23:10:36 GMT -5
I'm not sure whether it's a perfect example of taking something out of context, or a legitimate instance of confusion...  Certainly, at the long end of the scale, it has been shown that memories are often largely built from "generic information". In other words, you specifically remember a few details, but "fill in the background" with general purpose information. This can lead to interesting (and scary) mistakes in memory. (For example, if you think that all muggers belong to one racial group, and you see a suspicious person out of the corner of your eye, it is reasonably likely that your mind will "fill in" the missing details with what it expects to see - quite possibly leading to a detailed but entirely wrong description of "the mugger".) It wouldn't surprise me to learn that this may extend to music - SPECIFICALLY IN THE CONTEXT that you may not actually remember every note of the song you heard five minutes ago. You probably remember the tune in general, modified by details of that particular listening session (for example "Beethoven's Fifth, a bit louder than usual, and a hit faster"). As I recall, however, this generally occurs later, when you are trying to recall the experience, and not so much right afterwards. To me, it also suggests just how flawed such memories are likely to be, since the "modifiers" persist out of context (it may have seemed "loud" because you heard it in a quiet church, which may skew your perception of the actual objective loudness it was playing at). [Remember the experiment where you put one hand in hot water, and one in cold, and then put them both in the same luke warm water - and the two hands experience different temperatures due to the contrast from previous temperatures. That would be a great example of a BADLY run ABX test.] Unfortunately for SOME folks, this is all a great argument IN FAVOR of ABX testing since ABX testing at least tries to remove the "background fill" and force you to experience more of the current input (without expectations, you have no ready place to find "generic memories" to use). Also, by asking for results right away, it minimizes the "drift" caused by trying to remember hours or days later what you think something sounded like. Now, in the short term, it IS well established that sounds mask other sounds that have a specific relationship in time and frequency to them (usually at the same or higher frequencies). Usually sounds mask other sounds that occur AFTER them, but I believe the effect extends to VERY short times before in certain cases. (The masking sound prevents the brain from processing the masked sound that occurred VERY shortly before it. This is the basis of discussions about certain DAC filters that reduce pre-ringing and replace it with post-ringing because the post ringing is less audible to most people.) All of these masking effects persist for time spans numbering in milliseconds - and much of "perceptual" compression schemes like MP3 is based on its working. If you look up how MP3s are processed, you can find all the graphs and charts of how it works, and how long it lasts. (MP3's sometimes sound very bad because they "miscalculate" the situation - because their models are very sophisticated but far from perfect. When they get it right they usually sound OK.) Unfortunately, the simple fact appears to be that TAS is, shall we say, less than critical about the validity of tests they publish. I leave it to you whether they are just sloppy, or whether they have discovered (again) that "controversy sells papers". The burden of proof rests with he who holds the affirmative, Yves. That means that if Dr Stuart contends that a sound heard seconds before can cause us not to hear another sound seconds later, it is up to him to produce evidence for that claim. Pre-existing memories of familiar material can cause us to fill in information that is missing or has been altered in a new presentation. There is no evidence that a sound heard seconds ago will prevent us from hearing a new sound seconds later. As I said, if that were the case, we could not understand a spoken sentence, since the phonemes heard first would cause us to miss or mis-hear phonemes uttered later. Moreover, there is no evidence that memory feedback works any differently during an ABX session than at any other time. You've advanced a claim which, apparently, no conceivable empirical observation could show to be false. That means it has zero information content, i.e., the world would look exactly the same whether it were true or false. en.wikipedia.org/wiki/Falsifiability |
|
KeithL
Administrator
Posts: 10,273
|
Post by KeithL on Mar 13, 2013 23:19:33 GMT -5
Oh, I agree with you.... ABX testing is flawed. The only problem, for the context of this discussion, is that it is far LESS flawed than any OTHER type of testing. ABX testing at least TRIES to eliminate things like perception bias and the effect of our poor acoustic memory. It HAS been pretty well proven that we have trouble remembering "what something sounds like" for more than a very few minutes - if even. If I "remember" what a certain DAC sounds like, it is because I remember certain ways in which it differs from my generic mental image of what "a DAC" sounds like. So, when I remember what my favorite DAC sounds like, rather than remember every nuance of what it sounds like, I actually remember that it sounds "like <MY IDEA OF A GENERIC DAC>, plus a little clearer, and with a little bit brighter high end". The problem there is that, if that description happens to be "in the memory bank" for two different DACs, which I may have listened to at different times and under different circumstances, it doesn't provide any useful way of accurately comparing them to each other. It's like trying to decide whether the hot water in your apartment is hotter than the hot water in the men's LAV on the subway. You can make a fair guess, but forget about accuracy unless you compare them NEXT TO EACH OTHER. (And, if you have to walk from one to the other, the differences in room temperature and other extraneous factors will render your guess very unlikely to be accurate at all.) I am already fully aware of all that, but I still disagree that differences that cannot be detected in a double-blind listening test are necessarily always too small to worry about. My point is double-blind listening tests are flawed, as carefully explained by Robert Harley www.avguide.com/forums/blind-listening-tests-are-flawed-editorial and Bob Stuart www.theabsolutesound.com/articles/tas-194-meridian-audios-bob-stuart-talks-with-robert-harley-1I'm afraid you're missing the entire point. The point of ABX (or any other double blind testing) is that the subject is blind TO WHICH PRODUCTS OR SUBJECTS ARE BEING TESTED. It does NOT mean that they are blind to the other aspects of the situation. What you're describing would be equivalent to handing someone a pill, having them swallow it, and then asking "do you feel any different". When we test drugs we do NOT do that. Our subjects know what the pill is supposed to do; the ONLY thing they are blind to is which pill they got. Likewise, if we were to swap interconnects while our subjects were out of the room, then ask "do you hear anything different", it is true that the focus of their attention would influence whether they noticed the change or not. (I say it would still have some validity). However, we don't do that either. Instead, we tell them "we're going to try two different interconnects"; so they know what to listen for within reasonable parameters. And, yes, they are going to apply the same criteria to the situation that they usually do, so one subject may be more concerned with frequency response while another notices mostly the noise floor. The point is that, since they know they are testing an audio component, they use the same criteria to judge it as they normally do when they listen to music. [If you have to TELL them what to listen for, then you are artificially altering the situation. If your UUT (unit under test) changes something which the subject doesn't notice unless it is pointed out to him/her, then it really doesn't matter. Not to mention the obvious problem that, by "planting" expectations, you encourage subjects to hear things that aren't there because they expect to hear them.] [Did you ever wonder if when, after installing some new component, and asking your spouse "gee, honey, can you hear any difference?", maybe she can hear by the hopeful tone of your voice that you're hoping she'll say "gee, honey, that sure sounds better"?] The fact is that anything you can claim about ABX testing adding any sort of unfairness or bias is 1000x worse when you make the test NON-BLIND. If you do that, you just add in all the problems with expectation bias - which pretty much overshadows ANY ability to judge ANYTHING for real. [And you CAN do an ABX test without a time limit, and allow subjects to listen in their own homes, with their own music, and even allow them to use their own audio system to do the listening. It just takes a bit more effort.] You can also CONTROL FOR many of the issues in ABX testing. For example, if I were to do an ABX test, but actually provide the SAME sample every time, it is indeed true that some subjects would imagine differences that weren't there. However, those imaginary differences would statistically "cancel out" and we would get a null result. (The fact that most subjects might "claim" to have a favorite would be interesting but not statistically significant.) Likewise, if we TOLD the subjects that two different interconnects were the same, they might hear a difference, or their expectation not to could bias them to NOT hear a difference that really existed. In that situation, they might end up with a null response (not hear a difference) or they might hear a difference less often (because they expect not to). HOWEVER, and this is the important thing, none of those biases would lead to a result that showed a difference that didn't really exist. We would be "protected" from that possibility. In other words, having an ABX test end with "no result" doesn't PROVE BEYOND DOUBT that a difference doesn't exist. It does, however, strongly suggest that, if a difference does exist, it is relatively minor. I leave it to you whether you would buy a $100 bottle of wine instead of a $10 bottle IF, in a blind taste test done by 100 people, the $100 bottle was NOT preferred on average over the $10 bottle. (Even if most people aren't wine connoisseurs, I would expect at least some significant bias towards the $100 bottle if it was really better.) Of course, we are still dealing with people, so we can consider a test of whether they hear a difference or not to be scientific, but an expression of PREFERENCE will always be a survey of popular opinion. (In other words, if the cheap wine rated better, it could just be that most people PREFER the taste of cheap wine even though, by some "higher standard of criticism" it is "worse".) For example, it could be that the reason people preferred MP3 files to WAV files in a recent survey is that they are more used to poorly compressed files, and so the MP3 files are more in tune with their expectations of what a fie "usually" sounds like. The fact that they expressed a preference shows that they could hear a difference (which "proves" that MP3 files are NOT identical to the original); the fact that they preferred MP3s demonstrates a matter of opinion amongst the survey group. |
|
KeithL
Administrator
Posts: 10,273
|
Post by KeithL on Mar 13, 2013 23:29:24 GMT -5
Absolutely correct.... and it would be almost useless to listen to one sound once, then listen to a second sound once. You get all sorts of biases about the order as well. That's why, instead of listening to a piece of equipment today, and trying to compare it to what we remember something sounded like last week, we go back and forth multiple times and look for consistent PATTERNS of preference. A properly run ABX test will let the listener listen to each UUT as long as they like, go back and forth as many times as they like, and NOT insist on a difference (at least some runs should include multiple "dummy" runs with the SAME UUT, to sort out imagined differences as well). And, if you think an ABX test "generates stress", how about listening to an expensive piece of gear, AFTER putting down your credit card, and trying to be sure whether the difference you think you hear is real, BEFORE your 30 day trial period expires, and remembering vividly the return shipping it will cost you to return it.... AND how embarrassed you'll be to admit to your friends that you goofed....now THAT'S stress Or imagine listening to that $2000 cable, while five of your audiophile friends (or, if you're a reviewer, your readers) all stand around asking.... "You DO hear a difference, RIGHT? "  Finally, if anything, I would expect a "stressful test situation" to encourage people to hear MORE differences that perhaps they wouldn't have noticed if they weren't expecting them. This should make ABX tests MORE sensitive to differences than "casual" listening sessions. There is no need to eliminate it. That phenomenon is universal, and thus in no way compromises the validity of ABX testing. In order for it to create a "bias," you'd have to show that the "steering" you mention when the "A" unit is played differs from that when the "B" unit is played. Since the subject has no idea which unit is online, it is difficult to see how that change in "steering" could be triggered. You could argue that the mere fact that the subject knows he is in a test scenario alters the way his brain "steers" the information, and does it in such a way that differences he would notice in a "normal" listening scenario become undetectable in the test situation. But that leads back to the unfalsifiability problem. The experiment shows the brain is perfectly able to fill in missing portions of sounds using nothing but its own interpretation of what things SHOULD sound like, based on memory imprints of what they DID sound like in the past. An ABX test requires people to listen to one sound first and then another sound after that. Listening to one sound first will create a memory imprint even before the listener gets a chance to listen to the other sound. Meaning, listening to the other sound next will be biased by this memory imprint, as shown in the experiment. Quid est demonstrandum. As per my above comment, human audio perception works in such way that it causes ABX testing to be fundamentally flawed. ABX testing changes the way we focus our attention because it forces us to focus our attention on whether we can hear a difference rather than allowing us to focus on nothing but the sound itself. This causes stress, and it has been proven elsewhere that stress keeps us from being able to tell the difference between quite surprising things. Perhaps he has not. However, Poppy Crum and I both have. ...Which is exactly the reason why ABX testing is fundamentally flawed. If you had the memory of a goldfish, maybe it would work. If my uncle had breasts, he'd be my aunt. |
|
|
|
Post by garym on Mar 13, 2013 23:51:14 GMT -5
Usually sounds mask other sounds that occur AFTER them, but I believe the effect extends to VERY short times before in certain cases. (The masking sound prevents the brain from processing the masked sound that occurred VERY shortly before it . . . All of these masking effects persist for time spans numbering in milliseconds . . . . Yes; that would be going to the other extreme (milliseconds). Masking is not a memory phenomenon, though. It is a matter of a new sound interfering with processing of a sound heard an instant before. But that would have no differential effect on sounds heard during an ABX session, compared to a sequence of sounds heard any other time. Yves's argument seems to be that your memory of a few bars of a song heard seconds ago via the "A" device prevents you from "correctly" hearing the next few bars played through the "B" device. Thus ABX testing is "flawed" (correct me if that is not your argument, Yves). |
|
|
|
Post by GreenKiwi on Mar 13, 2013 23:52:35 GMT -5
I think Keith put it perfectly. Every test/comparison has its problems, but a properly run ABX test attempts to remove as many of those problems as possible.
If I could do it easily, I'd ABX all my new gear, if only to see if I could hear differences.
I don't think that ABX is about trying to figure out the best, but rather whether there are noticeable differences, and then you can decide what you think of those differences.
|
|
|
|
Post by yves on Mar 14, 2013 5:36:31 GMT -5
The burden of proof rests with he who holds the affirmative, Yves. That means that if Dr Stuart contends that a sound heard seconds before can cause us not to hear another sound seconds later, it is up to him to produce evidence for that claim. Not only did he claim that a sound perceived seconds before can cause us not to perceive another sound seconds later, but also that a sound perceived seconds before can cause us to DO perceive another sound seconds later ("perceive" as opposed to "hear", i.e. due to the fact our memory, among other things, shapes our perception of what we hear). Every expert in auditory neuroscience (every sane expert in auditory neuroscience, that is...) will tell you there's nothing questionable about this claim. This example Bob Stuart described in the TAS interview shows that our memory CAN cause us to hear sounds that aren't actually there : As a matter of fact, that is exactly my whole point to begin with. ABX testing cannot cause memory feedback to work differently, which is the very reason why ABX testing is flawed. For ABX testing to not be flawed, you'd have to fully eliminate memory feedback altogether. Hence that sentence "If you had the memory of a goldfish, maybe it would work" : The earth once used to be flat until proven round. Today, however, that doesn't make the earth any less round. |
|
