A short review here of the QSC ABX comparator.
The latest and greatest ABX comparator is now being made by Audio by Van Alstine. AVA is a long established perfectionist audio company in USA that is most know for its preamplifiers.
David Clark invented the ABX comparison method around 1980 and published it in JAES in 1981. The ABX method addresses long-standing objections by audiophiles to objective testing methods by permitting the Subject to compare the X (randomly chosen from A and B) against A and B as long as they wish and switching among them as many times as they choose before making a decision. The decision is a forced-choice objective question of whether X is A or B so there are only two possible answers, objectively verifiable (and not a "which is better" which might rest on multiple shifting dimensions). The purpose is specifically to find whether an audible difference even exists, since well before the 1980's many engineers and a number of famous audio reviewers had claimed that electronic components such as amplifiers and preamplifiers were already good enough that no audible differences existed between objectively good ones, while others claim to find huge differences--often favoring components with less good measured performance. The ABX comparator was created expressly to investigate such claims regarding whether differences are actually audible. (Of course, to true-believing subjectivists who believe in their previous sighted tests, to them it is axiomatically In Your Face and useless. But true belief is the opposite of science. Science requires blind testing.)
From 1981-87 an ABX Company made ABX comparators (apparently not many). I believe this company was run by David Clark and his friends in the SWMWTS audio society in Michigan, one of the few (along with the Boston Audio Society) which favored scientific methods.
QSC was a company that started by making instrument amplifiers for musicians but graduated to mostly making general purpose professional amplifiers (for musicians, PA systems, etc.). Generally such amplifiers were inexpensive for each watt of power produced, and especially compared with the exotic and expensive amplifiers used by well heeled audiophiles. So it stands to reason they would be interested in a comparator that would show that their amplifiers sound just as good as much more expensive audiophile amplifiers. This is to suggest why they were motivated to build an ABX comparator in the absence of overwhelming customer demand, and not to suggest they "cheated" in any way. I assume and it has always appeared that they made the best comparator they could, and it was relatively high priced compared with their amplifiers. It appears that their comparators were designed to meet an emerging international standard.
The QSC ABX comparator came out in 1998 and QSC likely made more comparators than anyone else. The manufacturing run has been said to be 200-300, and it does not appear than any more were made after that.
This appears to be a product type which most audiophiles don't want to bother with, and in many cases they might understandably positively hate the idea of having their sighted comparisons shown to be weak or wrong. They are satisfied with their sighted and unscientific comparison methodologies and/or they fear that testing devices themselves will corrupt their highly purified systems (and/or the latter may be used as an excuse initially).
There exists a smaller (?) group of audiophiles who embrace ABX testing and it's known history of ubiquitous negative results which tend to discredit many audiophile claims, but this group of audiophiles may not generally spend kilobucks on equipment like this either.
I think this is sad…and I have always felt every audiophile who contemplates changing electronic equipment in their lifetimes (which means ALL) should own and use some kind of DBT device. (I own two QSC ABX comparators, one is on permanent loan to a subjectivist audiophile friend who has never used it. Sadly I haven't used it much either. And for audiophiles of perfectionistic bent (like myself) there is no reason why an ABX comparator can't be made or modified to the highest quality level.
Using a comparator is not necessarily identical to performing an ABX test. Actually I have mainly used my ABX comparator simply to perform sighted tests with quick switching. If I don't think I can hear a difference in a sighted test with quick switching…it's not worth bothering with an ABX test as I'd only be guessing. Or sometimes I want to make a few blind tests. I don't always need statistical verification. If I feel as though I'm guessing…I don't need to go any further.
What are the arguments that can be made AGAINST an ABX testing methodology?
A Negative Result is Not Proof. (Of course, experts already know this, but sometimes exuberant objectivists forget.) A negative result in a particular ABX comparison does not prove that an audible difference does not exist between items being tested. It only judges the evidence presented in a particular testing run. There is a long list of reasons why a negative result may have been obtained even when there are audible differences:
1) Not enough trials were predetermined to reach significance. Note: the number of trials MUST BE predetermined in a classic ABX test and most statistical methods. Hard-to-hear differences require larger numbers of trials. Of course, subjectivist audiophiles believe nearly all the differences they hear, even the ones from their latest tiny tweak, are enormous and life-changing, so they overestimate the audibility of effects (or imagine them) and don't think they need many trials. In tests I have conducted, one subject had no doubt up front that he would get every single identification correct so the minimum number of trials would suffice, and he didn't want to bother with more than that. This overestimation of effect itself has a Bayesian interpretation which is not kind to the subject's POV when he is shown wrong. In fact, it adds to the evidence the effect is imaginary in it's effect or at least much more limited than believed….therefore requiring even more trials to really confirm next time.
2) Not sufficiently trained or capable listener(s) (if the finding is to generalize to others or the same person in the future).
3) Listener lacking patience, motivation, or in bad mood and therefore unable to do test correctly.
4) Musical selections not completely representative of what is possible. Or switching simply not done in a way as to display the possible differences.
5) Equipment errors. (Of course, this is what subjectivist audiophiles have almost always argued when confronted with a negative result. In particular, the comparator itself is blamed for obscuring the differences.)
Errors 2,3,4 are often discounted by advocates of ABX. They say "well, they had all the time they wanted," etc. But that kind of dismissal is inadequate for science. In real science, we must take 2,3,4 very seriously even when Subjects believe they did OK. This is not a contest to see who is more manly.
The Peter Aczellian "All Decent or Better Amplifiers Sound the Same" does not come from any one ABX test, but rather over 3 decades of failure to produce an ABX or equivalent test showing the reverse (with countless ABX tests having been performed--mainly by people who believe much like Aczel). And in that, I think he has a fairly good case (though at the same time, another contrary argument, that an amalgamation of endless poor tests does not equal one good test, arises).
But where I have to come down is at the very nature of scientific truth. It is not ever absolute! It is conditional and contingent and endlessly subject to revision. So I do not believe that even decades of research has "proven" anything except that a contrary proof has not yet arisen. But obviously what decades of ABX test failures to show differences has robustly shown is that "If Audible Differences Exist, they Must Be Rather Hard to Hear or Very Inconsistently Heard." This actually is very important in itself and not a meaningless quibble with TBS's.
So ultimately it's a matter of the degree and effect strength. It is actually very hard to discriminate between weak effects and nonexistent effects. And that's where we are now in The Big Question.
If you wish to relate this back to scientific philosophy, the basic problem is that an open ended question such as "What Can Humans Hear" is not a scientific question. The conditioned question "What Can Humans Hear Most of the Time" is a scientific question. Routinely such conditions are left out, but leaving them out without understanding they are, finding in science the solution to unconditioned questions--therefore all questions--that is scientism, not science.
So then suppose you are the designer of the new ultimate audio system, intended to actually be (but not necessarily make people think) the best ever, and you have an unlimited budget. Do you design your amplifier with approximately 0.5% distortion because anything less than that has not been shown to be audible in ABX testing? Hell No! You do the best that you can in every way. You throw every idea at it that doesn't conflict with a more well established idea. You make it perform as objectively well as possible, or at least far better than needed, while not breaking suspected audiophile superstitions more than necessary either. You don't ignore things merely because they haven't been proven necessary.
This is what artists and competitors do, within their budgets, within their capabilities, they don't hold back, they do all they can.
The Objectivist Triumphalists claim to represent the ordinary cost-conscious audiophile. They want to get the best value, and not waste their money on the unnecessary. Audio is simply the machinery that enables people to enjoy recorded music, they say.
Then look at their systems and how they spend their time. It shows as much as music something else--a cause, a hope, a body of shared understandings. Much as it is to subjectivists. I would certainly side with the objectivists has having the better understandings nearly complete, but not fully. And however misguided in ways…subjectivists have good times as much as others. There is far more to life than truth.
Clearly audio itself is a human art, as well as human society, not merely science or engineering. It should serve music, but what all it might serve cannot be known or studied by science.
This is not that many shouldn't rely on cost-conscious and scientific engineering when as it is called for. On tap if not on top.
The latest and greatest ABX comparator is now being made by Audio by Van Alstine. AVA is a long established perfectionist audio company in USA that is most know for its preamplifiers.
David Clark invented the ABX comparison method around 1980 and published it in JAES in 1981. The ABX method addresses long-standing objections by audiophiles to objective testing methods by permitting the Subject to compare the X (randomly chosen from A and B) against A and B as long as they wish and switching among them as many times as they choose before making a decision. The decision is a forced-choice objective question of whether X is A or B so there are only two possible answers, objectively verifiable (and not a "which is better" which might rest on multiple shifting dimensions). The purpose is specifically to find whether an audible difference even exists, since well before the 1980's many engineers and a number of famous audio reviewers had claimed that electronic components such as amplifiers and preamplifiers were already good enough that no audible differences existed between objectively good ones, while others claim to find huge differences--often favoring components with less good measured performance. The ABX comparator was created expressly to investigate such claims regarding whether differences are actually audible. (Of course, to true-believing subjectivists who believe in their previous sighted tests, to them it is axiomatically In Your Face and useless. But true belief is the opposite of science. Science requires blind testing.)
From 1981-87 an ABX Company made ABX comparators (apparently not many). I believe this company was run by David Clark and his friends in the SWMWTS audio society in Michigan, one of the few (along with the Boston Audio Society) which favored scientific methods.
QSC was a company that started by making instrument amplifiers for musicians but graduated to mostly making general purpose professional amplifiers (for musicians, PA systems, etc.). Generally such amplifiers were inexpensive for each watt of power produced, and especially compared with the exotic and expensive amplifiers used by well heeled audiophiles. So it stands to reason they would be interested in a comparator that would show that their amplifiers sound just as good as much more expensive audiophile amplifiers. This is to suggest why they were motivated to build an ABX comparator in the absence of overwhelming customer demand, and not to suggest they "cheated" in any way. I assume and it has always appeared that they made the best comparator they could, and it was relatively high priced compared with their amplifiers. It appears that their comparators were designed to meet an emerging international standard.
The QSC ABX comparator came out in 1998 and QSC likely made more comparators than anyone else. The manufacturing run has been said to be 200-300, and it does not appear than any more were made after that.
This appears to be a product type which most audiophiles don't want to bother with, and in many cases they might understandably positively hate the idea of having their sighted comparisons shown to be weak or wrong. They are satisfied with their sighted and unscientific comparison methodologies and/or they fear that testing devices themselves will corrupt their highly purified systems (and/or the latter may be used as an excuse initially).
There exists a smaller (?) group of audiophiles who embrace ABX testing and it's known history of ubiquitous negative results which tend to discredit many audiophile claims, but this group of audiophiles may not generally spend kilobucks on equipment like this either.
I think this is sad…and I have always felt every audiophile who contemplates changing electronic equipment in their lifetimes (which means ALL) should own and use some kind of DBT device. (I own two QSC ABX comparators, one is on permanent loan to a subjectivist audiophile friend who has never used it. Sadly I haven't used it much either. And for audiophiles of perfectionistic bent (like myself) there is no reason why an ABX comparator can't be made or modified to the highest quality level.
Using a comparator is not necessarily identical to performing an ABX test. Actually I have mainly used my ABX comparator simply to perform sighted tests with quick switching. If I don't think I can hear a difference in a sighted test with quick switching…it's not worth bothering with an ABX test as I'd only be guessing. Or sometimes I want to make a few blind tests. I don't always need statistical verification. If I feel as though I'm guessing…I don't need to go any further.
What are the arguments that can be made AGAINST an ABX testing methodology?
A Negative Result is Not Proof. (Of course, experts already know this, but sometimes exuberant objectivists forget.) A negative result in a particular ABX comparison does not prove that an audible difference does not exist between items being tested. It only judges the evidence presented in a particular testing run. There is a long list of reasons why a negative result may have been obtained even when there are audible differences:
1) Not enough trials were predetermined to reach significance. Note: the number of trials MUST BE predetermined in a classic ABX test and most statistical methods. Hard-to-hear differences require larger numbers of trials. Of course, subjectivist audiophiles believe nearly all the differences they hear, even the ones from their latest tiny tweak, are enormous and life-changing, so they overestimate the audibility of effects (or imagine them) and don't think they need many trials. In tests I have conducted, one subject had no doubt up front that he would get every single identification correct so the minimum number of trials would suffice, and he didn't want to bother with more than that. This overestimation of effect itself has a Bayesian interpretation which is not kind to the subject's POV when he is shown wrong. In fact, it adds to the evidence the effect is imaginary in it's effect or at least much more limited than believed….therefore requiring even more trials to really confirm next time.
2) Not sufficiently trained or capable listener(s) (if the finding is to generalize to others or the same person in the future).
3) Listener lacking patience, motivation, or in bad mood and therefore unable to do test correctly.
4) Musical selections not completely representative of what is possible. Or switching simply not done in a way as to display the possible differences.
5) Equipment errors. (Of course, this is what subjectivist audiophiles have almost always argued when confronted with a negative result. In particular, the comparator itself is blamed for obscuring the differences.)
Errors 2,3,4 are often discounted by advocates of ABX. They say "well, they had all the time they wanted," etc. But that kind of dismissal is inadequate for science. In real science, we must take 2,3,4 very seriously even when Subjects believe they did OK. This is not a contest to see who is more manly.
The Peter Aczellian "All Decent or Better Amplifiers Sound the Same" does not come from any one ABX test, but rather over 3 decades of failure to produce an ABX or equivalent test showing the reverse (with countless ABX tests having been performed--mainly by people who believe much like Aczel). And in that, I think he has a fairly good case (though at the same time, another contrary argument, that an amalgamation of endless poor tests does not equal one good test, arises).
But where I have to come down is at the very nature of scientific truth. It is not ever absolute! It is conditional and contingent and endlessly subject to revision. So I do not believe that even decades of research has "proven" anything except that a contrary proof has not yet arisen. But obviously what decades of ABX test failures to show differences has robustly shown is that "If Audible Differences Exist, they Must Be Rather Hard to Hear or Very Inconsistently Heard." This actually is very important in itself and not a meaningless quibble with TBS's.
So ultimately it's a matter of the degree and effect strength. It is actually very hard to discriminate between weak effects and nonexistent effects. And that's where we are now in The Big Question.
If you wish to relate this back to scientific philosophy, the basic problem is that an open ended question such as "What Can Humans Hear" is not a scientific question. The conditioned question "What Can Humans Hear Most of the Time" is a scientific question. Routinely such conditions are left out, but leaving them out without understanding they are, finding in science the solution to unconditioned questions--therefore all questions--that is scientism, not science.
So then suppose you are the designer of the new ultimate audio system, intended to actually be (but not necessarily make people think) the best ever, and you have an unlimited budget. Do you design your amplifier with approximately 0.5% distortion because anything less than that has not been shown to be audible in ABX testing? Hell No! You do the best that you can in every way. You throw every idea at it that doesn't conflict with a more well established idea. You make it perform as objectively well as possible, or at least far better than needed, while not breaking suspected audiophile superstitions more than necessary either. You don't ignore things merely because they haven't been proven necessary.
This is what artists and competitors do, within their budgets, within their capabilities, they don't hold back, they do all they can.
The Objectivist Triumphalists claim to represent the ordinary cost-conscious audiophile. They want to get the best value, and not waste their money on the unnecessary. Audio is simply the machinery that enables people to enjoy recorded music, they say.
Then look at their systems and how they spend their time. It shows as much as music something else--a cause, a hope, a body of shared understandings. Much as it is to subjectivists. I would certainly side with the objectivists has having the better understandings nearly complete, but not fully. And however misguided in ways…subjectivists have good times as much as others. There is far more to life than truth.
Clearly audio itself is a human art, as well as human society, not merely science or engineering. It should serve music, but what all it might serve cannot be known or studied by science.
This is not that many shouldn't rely on cost-conscious and scientific engineering when as it is called for. On tap if not on top.
No comments:
Post a Comment