Public Multiformat Listening Test @ 96 kbps (July 2014) DRAFT ver.9 2014/07/07
The purpose of this test is to find out which popular encoder produces the best quality on bitrates around 96 kbps. (Except for MP3 and low anchor)
This is DRAFT. Any suggestions, from correction of trivial grammatical errors to radical reform plans, are welcome.
5 Remaining TODOs:
1. Confirm that ABC/HR for Java 0.53a is stable in Linux. HA Discussion thread.
2. Confirm that ABC/HR for Java 0.53a produces accurate p-value. table, HA thread. Only a tiny rounding bug. Not critical. better table
3. We need a mail account and a person/people to receive results.
4. Contact to Garf, and upload this page to http://listening-tests.hydrogenaud.io/somewhere/
5. Decision of an ending date.
6. Split the ABX page. It's long.
-
Which codecs and settings are tested?
-
AAC iTunes 11.2.2 with CoreAudioToolbox 7.9.8.5 via qaac 2.41
--cvbr 96
-
Opus 1.1 with opus-tools-0.1.9-win32
--bitrate 96
-
Ogg Vorbis aoTuV Beta6.03
-q2.2
-
MP3 LAME 3.99.5 *bitrate is around 136 kbps.
-V 5
-
AAC FAAC v1.28 (Mid-low Anchor)
-b 96
-
AAC FAAC v1.28 (Low Anchor) *bitrate is around 52 kbps.
-q 30
-
AAC iTunes 11.2.2 with CoreAudioToolbox 7.9.8.5 via qaac 2.41
-
Is it normal that the bitrate is very high on some samples (even 176 kbps)?
Yes, and that is the beauty of VBR encoding - it will simply ignore bitrate limitations whenever possible, using as much bits as needed to encode a problematic sample.
Although that raises issues of fairness, it is the best way to compare modern codecs that shine most in VBR mode. Trying to force a VBR setting to match a desired bitrate, although fairer, is far from the usual practice of audio encoding, where it's more usual that a user just sticks to a quality setting, not caring much about a specific bitrate.
The quality settings for the VBR codecs were chosen because they average out to about 96 kbps over a number of encoded albums. It would be unfair to tie the hands of VBR codecs and punish them for being smart about where to spend what turns out to be the same number of bits over the long run.
-
Who should take the test?
Anyone interested in lossy audio quality, or people who have no interest but would like to help making this test better are invited. You don't need excellent hearing, but some good gear is welcome. Headphones are a must-have.
-
Can I take the test even if I am not running Microsoft Windows?
Any person running an operating system with Java 6 or later can participate.
TODO: Confirm that ABC/HR for Java 0.53a is stable in Linux. HA Discussion thread.
-
How do I take the test?
We provide sound samples, with a software that enables you to perform blind tests at home.
Download one or more sample packages from below.
You don't need to test all samples to participate. Even one single result is already very helpful. Of course, the more you test, the better for the final results' significance.
MEMO: How to efficiently compress packages with 16bit stereo wav:
7z a -m0=Delta:4 -m1=lzma compressed.d4lzma.7z folder
Decompress the
sampleXX.7z
package (XX
being the number of the test you want to take). You need a 7-Zip archiver.You may need the Sun Java Runtime Environment to run ABC/HR. In case it is not already installed, download it from Java website.
- WINDOWS USERS: Double-click
abchr.jar
. - *NIX USERS: Run
java -jar abchr.jar
from the shell.
Once ABC/HR is open, click "Open ABC/HR Config..." and load the file
SampleXX.ecf
.You are now ready to take the test.
Notice the 6 groups, 12 buttons, and 6 buttons. These 6 groups are either a representation of AAC, Opus, Ogg Vorbis, MP3, Mid-low anchor, Low anchor. These 6 groups are shuffled in randomized order, so you don't know which group is which codec.
In each group, there are two buttons. Either one of the left or right buttons plays the encoded sound, and the other button plays the Reference (original) sound. button plays the Reference (original) sound.
The recommended procedure is to first identify the low anchor among the 12 buttons. The low anchor should sounds like the reference played back through a telephone. It may sound hollow, crackly, and overall, bad. Give the low anchor the lowest grade. Lower the slider just above the low anchor button you identified.
Then continue by trying to identify the mid-low anchor. The mid anchor sounds similar to the reference but notably washed out, blurry, warbly, or different in its stereo image. Give the mid anchor a grade in the middle of the scale based on its overall quality when compared to the reference.
Things get harder from here. Try the other buttons and identify the remaining 4 non-reference. When you are not exactly sure which the non-reference is, it's always recommended to .
Beginners may want to start from the and a low anchor for Sample B. Answer whether X is A or B, and press Next Trial. If you are correct, the results will say . This p-value is likeliness that you get the correct answers by chance. The less p-value means it is less likely that you have passed tests by pure luck; ie. you certainly have the ability to distinguish A and B. Try lowering the p-value to or below. Now you are very sure you can tell the difference between A and B.
Then, switch to the . The p-value won't be shown, but you can check the Completed number and . If you get more correct answers, you can gray out the reference (5.0) slider after you push , which is very recommended.
Sometimes it can be very difficult to tell the difference between original and non-reference. We do not recommend to turn the volume up in that case, since it leads to auditory fatigue. Rather, we recommend to:
- Keep the room silent, if possible. Disabling unnecessary appliances may help.
- Take breaks between sessions if you are tired.
- Leave it to 5.0.
Part from this manual is from ecodis, written by Christian R. Helmrich.
- http://www.ecodis.de/audio/guideline_high.html
http://web.archive.org/web/20120204073842/http://ff123.net/64test/practice.html
After you finish the test, save the test results as Encrypted Result Files (*.erf).
Mail the erf file to:
TODO: We need an account and a person/people to receive results.
All results and comments we receive will be published. If you want to be associated with your results, please enter your (nick)name in the "Show name in results file" field in ABC/HR (check the checkbox next to it to enable the field). Otherwise, your results will be anonymous.
Thank you very much!
- WINDOWS USERS: Double-click
-
I don't hear any sound.
In some cases, ABC/HR might not default to the primary audio driver. Simply change the device from ABC/HR's Options, Settings..., Playback.
-
When will the test finish?
TODO: rewrite the date when it's decided.
The test may end on August 20 2014. No results will be accepted after that date, unless the test is extended. Possible extensions will be announced at this page.
If you have any questions, feel free to contact us via e-mail: listening.tests.ha@gmail.com.
TODO: contact to Garf, and upload this page to http://listening-tests.hydrogenaud.io
Best regards
Kamedo2
© Sebastian Mares, 2011 | Layout by Dibrom | Portions of the Text by Roberto Amorim