Page 309

TABLE 10.1 An example of critical bands in the human hearing range showing increase in bandwidth with absolute frequency. A critical band will arise at an audible sound at any frequency.

Critical band
number
(Bark)

Center
freuqency
(Hz)

  Critical
  band
  (Hz)

Lower cutoff
frequency
(Hz)

Upper cutoff
frequency
(Hz)

  1

    50

 —

   —

    100

  2

  150

  100

    100

    200

  3

  250

  100

    200

    300

  4

  350

  100

    300

    400

  5

  450

  110

    400

    510

  6

  570

  120

    510

    630

  7

  700

  140

    630

    770

  8

  840

  150

    770

    920

  9

1000

  160

    920

   1080

10

1170

  190

   1080

   1270

11

1370

  210

   1270

   1480

12

1600

  240

   1470

   1720

13

1850

  280

   1720

   2000

14

2500

  320

   2000

   2320

15

2900

  380

   2320

   2700

16

2900

  450

   2700

   3150

17

3400

  550

   3150

   3700

18

4000

  700

   3700

   4400

19

4800

  900

   4400

   5300

20

5800

1100

   5300

   6400

21

7000

1300

   6400

   7700

22

8500

1800

   7700

   9500

23

10,500

2500

   9500

12,000

24

13,500

3500

12,000

15,500

25

18,775

6550

15,500

22,050

(Tobias)

Dissonance tends to increase at low frequencies; for example, musicians tend to avoid thirds at low frequencies.

The bark (named after German physicist Georg Heinrich Barkhausen) is a unit of perceptual frequency; specifically, a bark measures the critical band rate; a critical band has a width of one bark; 1/100 of a bark equals 1 mel. The bark scale relates absolute frequency (in hertz) to perceptually measured frequencies such as pitch or critical bands (in bark). Using a bark scale, the physical spectrum can be converted to a psychological spectrum. In this way, a pure tone (a single spectrum line) can be represented as a psychological masking curve.

The pitch place theory further explains the action of the basilar membrane. Carried by the surrounding fluid, a sound wave travels the length of the membrane and the wave stops at particular places along the length of the membrane, where the greatest vibration of the membrane occurs, corresponding to different frequencies. Specifically, high frequencies are sensed at the membrane near the middle ear while low frequencies are sensed at the far end. The wave excited by a high-frequency sound does not reach the far end of the basilar membrane. However, a low-frequency sound will pass through

Page 308

vibrations are converted to electrical impulses and sent to the brain as neural information along a bundle of nerve fibers. The brain decodes the period of the stimulus and point of maximum stimulation along the basilar membrane to determine frequency; activity in local regions surrounding the stimulus is ignored.

Examination of the basilar membrane shows that the ear contains roughly 30,000 hair cells arranged in multiple rows along the basilar membrane, roughly 32 mm long; this is the Organ of Corti. The cells detect local vibrations of the basilar membrane and convey audio information to the brain via electrical impulses. Frequency discrimination dictates that at low frequencies, tones a few hertz apart can be distinguished; however, at high frequencies tones must differ by hundreds of hertz. In any case, hair cells respond to the strongest stimulation in their local region; this is called a critical band, a concept introduced by Harvey Fletcher. Experiments show that critical bands are much narrower at low frequencies than at high frequencies; three-fourths of the critical bands are below 5 kHz; the ear receives more information from low frequencies and less from high frequencies. Critical bands are approximately 100 Hz wide for frequencies from 20 to 400 Hz and approximately 1/5 octave in width for frequencies from 1 to 7 kHz. Alternatively, bands can be assumed to be 1/3 octave for frequencies from 300 Hz to 20 kHz; an error of less than 1.5 dB will occur. Still other research shows that critical bands can be approximated with the equation:

where F = center frequency in kHz.

Eberhard Zwicker modeled the ear with 24 arbitrary critical bands for frequencies below 15 kHz; a 25th band occupies the region from 15 to 20 kHz. An example of critical bands is listed in Table 10.1. Physiologically, each critical band occupies a length of about 1.3 mm, with 1300 primary hair cells. The critical band for a 1-kHz sine tone is about 160 Hz in width; thus a noise or error signal that is 160 Hz wide and centered at 1 kHz is audible only if it is greater than the same level of a 1-kHz sine tone. As another example, the loudness of a band of noise at a constant SPL remains constant as the bandwidth increases; however, when the critical bandwidth is exceeded, loudness increases. Critical bands are analogous to a spectrum analyzer with variable center frequencies. Importantly, critical bands are not fixed; they are continuously variable in frequency and any audible tone will create a critical band centered on it. The critical band concept is an empirical phenomenon. Looked at in another way, a critical band is the bandwidth at which subjective responses abruptly change.

Interestingly, critical bands have been used to explain consonance and dissonance. Tone intervals with a frequency difference greater than a critical band are generally more consonant; intervals less than a critical band tend to be dissonant with intervals of about 0.2 critical band being most dissonant.

Page 310

all the high frequency places to reach the far end. Because hair cells tend to vibrate at the frequency of the strongest stimulation, they will convey that frequency in a critical band, ignoring lesser stimulation. This excitation curve is described by the cochlear spreading function, an asymmetrical contour. This explains, for example, why broadband measurements cannot describe threshold phenomena, which are based on local frequency conditions. There are about 620 degrees of differentiable frequencies equally distributed along the basilar membrane; thus a resolution of 1/25 Bark is reasonable. Summarizing, critical bands are important in perceptual coding because they show that the ear discriminates between energy in the band, and energy outside the band; in particular, this promotes masking.

Threshold of Hearing, and Masking

Two fundamental phenomena that govern human hearing are the minimum hearing threshold, and amplitude masking, as shown in Fig. 10.4. The threshold of hearing curve describes the minimum level (0 sone) at which the ear can detect a tone at a given frequency. The threshold is referenced to 0 dB at 1 kHz. The ear is most sensitive in the 1 to 5 kHz range, where we can hear signals several decibels below the 0-dB reference. Generally, two tones of equal power and different frequency will not sound equally loud. Similarly, the audibility of noise and distortion varies according to frequency. Sensitivity decreases at high and low frequencies. For example, a 20-Hz tone would have to be approximately 70 dB louder than a 1-kHz tone to be barely audible. Perceived loudness can be expressed in sones; one sone describes the loudness of a 40 dB SPL sine tone at 1 kHz. A loudness of 2 sones corresponds to 50 dB SPL; similarly, any doubling of loudness in sones results in a 10-dB increase in SPL. For example, 64 sones corresponds to 100 dB SPL. A perceptual coder compares the input signal to the minimum threshold, and discards signals that fall below the threshold, because the ear cannot hear these signals.

Amplitude masking occurs when a tone shifts the threshold curve upward in a frequency region surrounding the tone. The masking threshold describes the level where a tone is barely audible. When tones are sounded simulta-

Figure 10.4
The threshold of hearing  describes the  softest sounds audible across the human
hearing range. A masker tone or noise will raise the threshold of hearing in a
local region, creating a masking curve. Masked tones or noise, perhaps otherwise
audible, that fall below the masking curve during that time will not be audible.

Page 307

reduction. The original signal cannot be reconstructed exactly. A data reduction system reduces entropy; by modeling the perceptual entropy, only irrelevant information is removed, hence the reduction can be inaudible. A perceptual coder must use psychoacoustic models to identify irrelevant content in the audio signal.

Traditionally, audio system designers have used objective parameters as their design goals—flat frequency response, minimal measured noise, and so on. Designers of perceptual coders recognize that the final receiver is the human auditory system. Following the lead of psychoacoustics, they use the ear's own performance as the design criterion. After all, any musical experience—whether created, conveyed and reproduced via analog or digital means—is purely subjective.

Physiology of the Human Ear and Critical Bands

The ear performs the transformation from acoustical energy to mechanical energy and ultimately to the electrical impulses sent to the brain, where information contained in sound is perceived. A simplified look at the human ear's physiological design is shown in Fig. 10.3. The outer ear collects sound, and its intricate folds help us to assess directionality. The ear canal resonates at about 3 kHz, providing extra sensitivity in the frequency range critical for speech intelligibility. The eardrum transduces acoustical energy into mechanical energy; it reaches maximum excursion at about 120 dB SPL, above which it begins to distort the waveform. The three bones in the middle ear, colloquially known as the hammer, anvil, and stirrup (the three smallest bones in the body) provide impedance matching to efficiently convey sounds in air to the fluid-filled inner ear. The vestibular canals do not affect hearing, but instead are part of a motion detection system providing a sense of balance. The coiled basilar membrane detects the amplitude and frequency of sound; those

Figure 10.3
A simplified look at the design and construction  of the human ear. The coiled
cochlea and basilar  membrane are straightened  for clarity of illustration.

Page 311

neously, masking occurs in which louder tones can completely obscure softer tones. For example, a tone of 500 Hz can mask a softer tone of 600 Hz. In other words, the physical presence of sound certainly does not ensure audibility and conversely can ensure inaudibility of other sound. The strong sound is called the masker and the softer sound is called the maskee. Masking theory argues that the softer tone is just detectable when its energy equals the energy of the part of the louder masking signal in the critical band; this is a linear relationship with respect to amplitude. Generally, depending on relative amplitude, soft (but otherwise audible) audio tones are masked by louder tones at a similar frequency (within 100 Hz at low frequencies).

The mechanics of the basilar membrane explain the phenomenon of amplitude masking. A loud response at one place on the membrane will mask softer responses in the critical band around it. Unless the activity from another tone rises above the masking threshold, it will be swamped by the masker. Fig. 10.5A shows four masking curves (tones masked by narrow band noise) at 60 dB SPL, on a logarithmic scale in hertz. Fig. 10.5B shows seven masking curves on a bark scale; using this natural scale, the critical band rate can be seen. Masking thresholds are sometimes expressed as an excitation level; this is obtained by adding a 2 to 6 dB masking index to the sound pressure level of the just-audible tone. Low frequencies can interfere with the perception of higher frequencies. Masking can overlap adjacent critical bands when a signal is loud, or contains harmonics; for example, a complex 1-kHz signal can mask a 2-kHz signal. Low amplitude signals provide little masking. Narrow band tones such as sine tones also provide relatively little masking. Likewise, louder, more complex tones provide greater masking with masking curves that are broadened, and with greater high-frequency extension.

Figure 10.5
Masking curves describe the threshold where a tone or noise is just audible
in the presence of a masker. Threshold width varies with frequency when
plotted logarithmically. When plotted on a bark scale, the widths and slopes
are similar, reflecting response along the basilar membrane.A. Masking
thresholds plotted with logarithmic frequency. B. Masking thresholds
plotted with critical band rate. (Zwicker)

Page 306

hearing can perceive. For example, a barely audible 30-Hz tone would be 60 dB louder than a barely audible 4-kHz tone. The response varies with respect to loudness; the louder the sounds, the flatter the response. The contours are rated in phons, measuring the SPL of a contour at 1 kHz.

All characteristics of sound except one can be perceived by one ear; localization can only be accomplished with two ears. When sound originates from the side, the ear-brain uses cues such as intensity differences, waveform complexity and time delays to determine the direction of origin. When equal sound is produced from two loudspeakers, instead of localizing sound from the left and right sources, the ear-brain interprets sound coming from a space between the sources. Because each ear receives the same information, the sound is stubbornly decoded as coming from straight ahead. Similarly, stereo is nothing more than two different monaural channels. The rest is simply illusion.

There is probably no limit to the complexity of psychoacoustics. For example, consider the musical tones in Fig. 10.2A. A major scale is played through headphones to the right and left ears. Most listeners hear the pattern in Fig. 10.2B, where the sequence of pitches is correct, but heard as two different melodies in contrary motion. The high tones appear to come from the right ear, and the lower tones from the left. When the headphones are reversed, the headphone formerly playing low tones now appears to play high tones, and vice versa. Other listeners might hear low tones to the right and high tones to the left, no matter which way the headphones are placed. Curiously, right-handed persons tend to hear high tones on the right and lows on the left; not so with lefties. Still other listeners might perceive only high tones and little or nothing of the low tones. In this case, most right handers perceive all the tones, but only half the lefties do so.

The ear perceives only a portion of the information in an audio signal; this could be called the perceptual entropy. Small entropy signals can be efficiently reduced; large entropy signals cannot. For this reason, a coder might output a variable bit rate that is low when information is poor, and high when information is rich. The output is variable because although the sampling rate of the signal is constant, the entropy in its waveform is not. Using psychoacoustics, irrelevant portions of a signal can be removed; this is known as data

Figure 10.2
When a sequence of two-channel tones is presented to
a listener, perception might depend on handedness.
(Deutsch)

Page 312

Many masking curves have been derived from studies in which both single tones or narrow bands of noise are used as the masker stimulus. Generally, single tone maskers produce dips in the masking curve near the tone due to beat interference between the masker and maskee tones. Narrow noise bands do not show this effect. In addition, tone maskers seem to extend high frequency masking thresholds more readily than noise maskers. Generally, it is agreed that these differences are artifacts of the test itself. Tests with wide band noise show that only the frequency components of the masker that lie in the critical band of the maskee are effective at masking.

Many masking studies examine masking of a tone by another tone or by noise; in perceptual coding, it is reversed; we are concerned with quantization noise that is masked by a music tone. As noted, noise bands provide masking curves that differ from sine tone stimulus; generally, music is more tonelike in its masking properties. Sine-tone masking is used in masking models because it provides the least (worst case) masking of noise; complex tones provide greater masking. Generally, a tone is inaudible when it is about 4 dB below a 1/3-octave masking noise in a critical band. Conversely, when a 1/3-octave band of noise is masked by a pure tone, the noise must be 24 dB below the tone; it is 20 dB harder to mask noise. Ultimately, careful listening is needed to tune any masking criteria. In addition, coders presume a threshold of quiet that is below the standard curves, because it cannot be known what the listening level will be. Relatively little scientific study has been done with music as the stimulus. However, it is generally agreed that tone and noise curves are valid models for music coding. Clearly, one musical sound can mask another, but future work in the mechanics of music masking will result in better masking algorithms.

Simultaneous masking curves are asymmetrical. The slope of the shifted curve is less steep on the high-frequency side. Thus it is relatively easy for a low tone to mask a higher tone, but the reverse is more difficult. Specifically, the lower slope is about 27 dB/bark; the upper slope varies from –20 to –5 dB/bark depending on the amplitude of the masker. Low-level maskers influence a relatively narrow band of masked frequencies. However, as sound level of the masker increases, the threshold curve broadens, and in particular its upper slope decreases; its lower slope remains relatively unaffected. Fig. 10.6 shows a series of masking curves produced by a narrow band of noise centered at 1 kHz, sounded at different amplitudes. Clearly, the ear is most discriminating with low amplitude signals.

Temporal masking occurs when tones are sounded close in time, but not simultaneously. A signal can be masked by a noise (or another signal) that occurs later; this is premasking (sometimes called backward masking). In addition, a signal can be masked by a noise (or another signal) that ends before the signal begins. This is post masking (sometimes called forward masking). In other words, a louder tone appearing just before (pre-masking), or after (post-masking) a softer tone overcomes the softer tone. Just as simultaneous masking increases as frequency differences are reduced, temporal masking increases as time differences are reduced. Given an 80-dB tone, there may be

Page 305

no change in pitch with loudness. Also, with musical tones, the effect is much less.

Beat frequencies occur when two nearly equal frequencies are sounded together. When the difference in frequency between tones is itself an audible frequency, a difference tone can be heard. The effect is especially audible when the frequencies are high, the tones fairly loud, and separated by not much more than a fifth. Although debatable, some listeners claim to hear sum tones. An intertone also can occur, especially below 200 Hz where the ear's ability to discriminate between simultaneous tones diminishes. For example, simultaneous tones of 65 and 98 Hz will be heard not as a perfect fifth, but as an 82-Hz tone. On the other hand, when tones below 500 Hz are heard one after the other, the ear can differentiate between pitches only 2 Hz apart.

The ear can accommodate a very wide dynamic range; the threshold of feeling at 120 dB SPL has a sound intensity 1,000,000,000,000 times that of the threshold of hearing at 0 dB SPL. For convenience of expression, it is clear why the logarithmic decibel is used when dealing with that extreme range. Although the dynamic range is vast, the ear's sensitivity is frequency dependent. Maximum sensitivity occurs at 1 to 5 kHz, with relative insensitivity at low and high frequencies. Through testing, equal-loudness contours such as the Robinson-Dadson curves have been derived, as shown in Fig. 10.1. Each contour describes a range of frequencies that are perceived to be equally loud. The lowest contour describes the minimum audible field, the minimum sound pressure level across the audible frequency band that a person with normal

Figure 10.1
The Robinson-Dadson equal-loudness contours show that the ear is nonlinear with respect
to frequency, and loudness. These contours are based on psychoacoustic studies, using sine tones.

Page 313

Figure 10.6
Masking thresholds vary with respect to sound pressure level; this test uses a narrow
 band masker noise centered at 1 kHz. The  lower slope remains essentially unchanged.

40 dB of post masking within 20 ms, and 0 dB of masking at 200 ms. Premasking can provide 60 dB of masking for 1 ms, and 0 dB at 25 ms. This is shown in Fig. 10.7. Temporal masking decreases as the duration of the masker decreases. In addition, a tone is post masked by an earlier tone when they are close in frequency or when the earlier tone is lower in frequency; post masking is slight when the masker has a higher frequency. Logically, simultaneous masking is stronger than either pre- or post masking because the sounds occur at the same time.

Temporal masking suggests that the brain integrates sound over a period of time, and processes the information in bursts at the auditory cortex. Alternatively, perhaps the brain simply processes loud sounds faster than soft sounds. Whatever the mechanism, temporal masking is important in frequency domain coding. These coders have limited time resolution because they operate on blocks of samples, thus spreading error over time; temporal masking can overcome audibility of artifacts caused by transient signals. Ideally, filter banks should provide a time resolution of 2 to 4 ms. Acting together, amplitude and temporal masking form a contour that can be mapped in the time-frequency domain, as shown in Fig. 10.8. Sounds falling under that contour will be masked; it is the obligation of perceptual coders to identify this contour for changing signal conditions, and code the signal appropriately.

Multichannel playback presents additional opportunities for data reduction. Joint stereo coding takes advantage of interchannel redundancy between

Figure 10.7
Temporal masking occurs before, and in particular, after a masker sounds; the
threshold decreases  with time. Dashed line indicates threshold for test
 tone impulse without a masker signal. (Zwicker)

Page 304

seem to have a slightly lower subjective pitch than pure tones of the same frequency? All of this and more is the domain of psychoacoustics, the study of human auditory perception, ranging from the biological design of the ear, to the brain's interpretation of aural information. Sound is only an academic concept without our perception of it. Psychoacoustics explains the subjective response to everything we hear. It is the ultimate arbitrator in acoustical concerns because it is only our response to sound that fundamentally matters. Psychoacoustics seeks to reconcile acoustical stimuli and all the scientific, objective, and physical properties that surround them, with the physiological and psychological responses evoked by them.

The ear and its associated nervous system is an enormously complex, interactive system. Over the past several million years, the physiology of the human hearing system has evolved incredible powers of perception. At the same time, as with even complex systems, it has its limitations. On one hand, the ear is astonishingly acute in its ability to detect nuance or defect in a signal; on the other hand, it is surprisingly casual with portions of the signal it deems irrelevant. The accuracy of a coded signal can be very low, but this accuracy is very frequency and time dependent.

The ear is a highly developed physical organ (the eye, for example, can only receive frequencies over one octave), but the ear is useful only when coupled to the interpretative powers of the brain. Those mental judgments form the basis for everything we experience from sound and music. The left and right ears do not differ physiologically in their capacity for detecting sound, but their respective right and left brain halves do. The two halves loosely divide the brain's functions. There is some overlap, but the primary connections from the ears to the brain halves are crossed; the right ear is wired to the left brain half and the left ear to the right brain half. The left cerebral hemisphere processes most speech (verbal) information. Thus, theoretically the right ear is perceptually superior for spoken words. On the other hand, it is mainly the right temporal lobe that processes melodic (nonverbal) information. Therefore, we are better at perceiving melodies heard by the left ear.

The ear's response to frequency is logarithmic; this can be demonstrated through its perception of musical intervals. For example, the interval between 100 and 200 Hz is perceived as an octave, as is the interval between 1000 and 2000 Hz. In linear terms, the second octave is much larger, yet the ear hears it as the same interval. For this reason, musical notation uses a logarithmic measuring scale. Each four and one-half spaces or lines on the musical staff represent an octave, which might be only a few tens of hertz apart, or a few thousands, depending on the clef and ledger lines used.

Frequency is a literal measurement. Pitch is not. It is a subjective, complex characteristic based on frequency, as well as other physical quantities such as waveform and intensity. For example, if a 200-Hz sinewave is sounded at a soft then louder level, most listeners will agree that the louder sound has a lower pitch. In fact, a 10% increase in frequency might be necessary to maintain a listener's subjective evaluation of constant pitch at low frequencies. On the other hand, in the ear's most sensitive region, 1 to 5 kHz, there is almost

Page 314

Figure 10.8
When simultaneous and temporal masking are  combined, a time-frequency contour
results. A  perceptual coder must place quantization noise  and other artifacts
within this contour to ensure  inaudibility. A Simultaneous masking. B. Temporal
masking. C. Combined masking effect in  time-frequency. (Beerends and Stemerdink)

stereo (or multiple) channels to increase efficiency. The data rate of a stereo signal is double that of a mono signal, but most stereo programs are not dual-mono. Rather, the channels share some level and phase information to create phantom images. Joint stereo coding codes common information only once. A 256-kbps joint stereo coding channel will perform better than two 128-kbps channels.

Using diverse and dynamically changing psychoacoustical cues and signal analysis, inaudible components can be removed with acceptable degradation. For example, a loud sound in one channel can mask other softer sounds in other channels. Above 2 kHz, localization is achieved primarily by amplitude; because the ear cannot follow fast individual waveform cycles, it tracks the envelope of the signal, not its phase. Thus the waveform itself becomes less critical; this is intensity localization. In addition, the ear is limited in its ability to localize sounds close in frequency. To convey a multichannel surround field, the high frequencies in each channel can be divided into bands and combined band by band into a composite channel; the bands of the common channel are reproduced from each loudspeaker, or panned between loudspeakers, with the original signal band envelopes. Use of a composite channel achieves data reduction; in addition, other masking principles can be applied prior to forming the composite channel. Many multichannel systems use a 5.1 format with three front channels, two independent surround channels, and a subwoofer channel. Very generally, the number of bits required to code a multichannel signal is proportional to the square root of the number of channels. A 5.1 channel coder, for example, would theoretically require 2.26 times the number of bits needed to code one channel.

Although a maskee signal exists acoustically, it does not exist perceptually. It might seem quite radical, but aural masking is as real as visual masking.