import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

11.2 Sample-based synthesis

A brief history

Sample-based synthesis predates both digital audio and sampling. Pierre Schaeffer, a pioneer of musique concrète, built the phonogène, a tape machine whose keyboard changed the playback speed of a tape loop, and with it the pitch. Instruments like the Mellotron later put this idea into mass production. The Fairlight CMI (1979), the first commercially available digital sampler, was marketed as a way to play realistic sounds, like a piano, that the synthesizers of the day could not produce. Memory was scarce: each voice had just 16 kilobytes for its 8-bit samples, at most about two seconds of audio.

Synthesis techniques like frequency modulation were motivated in part by the goal of producing realistic imitations of musical instruments. This turns out to be extremely difficult! As we saw with the trumpet in Chapter 6, the harmonics of real instruments evolve in intricate ways over the course of every note. For example, below we attempt to imitate a clarinet playing A#5 for about five seconds, using the tools of Chapters 3 and 4: a wavetable of two harmonics at the right fundamental frequency, multiplied by an ADSR envelope. The result has the right pitch, duration, and overall shape, but a listener is unlikely to mistake it for the real thing:

Additive synthesis

A real clarinet

An imitation of a clarinet with additive synthesis (left), and a recording of a real clarinet (right), each playing A#5 for about five seconds. Each is shown in full (top) and in a 5 ms close-up (bottom). 356904 by MTG, License: Attribution 3.0.

With sampling, we can just “cheat”! Rather than reverse engineering an instrument’s sound in order to synthesize it, why not just record the instrument and play it back when needed? This was prohibitively expensive in the early days of computer music, when computers could store only a few seconds of audio in memory. However, these days, it’s trivial to store many hours of audio.

But sampling has a key limitation: we are stuck with the expressive parameters that we pre-recorded. Our clarinet recording has one pitch (A#5), one duration (about five seconds), and one loudness. With additive synthesis, we can play any combination of pitch, duration, and loudness simply by changing its parameters. With samples, we can only play back the combinations that we happened to record:

Fig. 76 Additive synthesis (blue) can produce any combination of pitch and duration by changing its parameters. Samples (red) exist only for the combinations that were recorded, e.g. the three cells shown here, and every other combination (?) is unavailable.

Recording everything

An alternative workaround is to simply record the full space of combinations, as commercial sample libraries often aspire to do. These libraries can span hundreds of gigabytes, with recordings of the world’s finest instruments played by top studio musicians in world-class recording studios. In practice, however, even these libraries come with features for manipulating expressive parameters on the fly.

We need some way to modify samples on the fly, adapting their high-level expressive parameters like pitch, duration, and loudness. This is the idea behind sample-based synthesis: pre-recording musical material, and applying manipulations on the fly to change its expressive parameters. In the rest of this section, we will develop manipulations for (1) pitch, (2) duration, and (3) loudness.

Compared to the other synthesis techniques we have studied, sample-based synthesis is:

• Highly realistic: every note starts from a recording of the real instrument.

• Computationally cheap: the manipulations we develop below take only a handful of operations per audio sample.

• Easy to implement: as we will see, its core is just a few lines of code.

• Very general: it works for any tonal sound, from a clarinet to a sung vowel to a car horn.

But it also has disadvantages:

• High memory usage: convincing instruments still require many recordings, often many gigabytes of samples on disk or in memory.

• Limited range: the further a manipulation pushes a recording from its original parameters, the less realistic the result sounds.

Modifying pitch via resampling

Recall wavetable synthesis from Chapter 3, where we stored a single cycle of a waveform in a table. We saw that changing the rate at which we read through the table changes the perceived pitch of the output. We can use the same idea to change the pitch of not just a wavetable, but an entire recorded sample stored on disk! Reading through a recording faster raises its pitch, and reading through it slower lowers it.

As a running example for the rest of this chapter, we will use our recording of a clarinet playing A#5. It comes from the Good-sounds dataset, a collection of single notes played by professional musicians on a variety of instruments, recorded by the Music Technology Group (MTG) at Universitat Pompeu Fabra and available on Freesound. Here is the recording repitched to a range of notes:

A#4

D5

F5

A#5 (the original recording)

D6

A single recording of A#5 on a clarinet (red), repitched by \(k\) steps to play A#4, D5, F5, and D6 (blue). We trimmed the silence from the recording, normalized its level, and fine-tuned its pitch.

If you listen closely, you may notice that the lower notes also last longer, and the higher notes are shorter. We will return to this side effect soon.

Let’s make this precise. Recall resampling from Chapter 7, which maps a vector of \(N\) audio samples \(\mathbf{x}\) to a new vector of \(M\) audio samples \(\mathbf{y}\) by reading \(\mathbf{x}\) at fractional positions \(p\), interpolating between neighboring audio samples. To change the playback speed of a recording from an original duration of \(T_1\) seconds to a new duration of \(T_2\) seconds, we keep the sample rate fixed, set the new length to \(M = N \cdot \frac{T_2}{T_1}\), and read

\[y[m] = \text{Interpolate}\!\left(\mathbf{x},\; p = m \cdot \frac{T_1}{T_2}\right).\]

What does this do to the frequencies in the recording? Suppose the recording contains a component at frequency \(f_1\). Playback covers \(T_1\) seconds of the recording in \(T_2\) seconds, so it advances through the recording at \(\frac{T_1}{T_2}\) input seconds per output second. Working through the units, the output contains that component at

\[f_2 \left[\frac{\text{cycles}}{\text{output second}}\right] = f_1 \left[\frac{\text{cycles}}{\text{input second}}\right] \cdot \frac{T_1}{T_2} \left[\frac{\text{input seconds}}{\text{output second}}\right] = f_1 \cdot \frac{T_1}{T_2}.\]

Every frequency in the recording is multiplied by the same factor \(\frac{T_1}{T_2}\), the ratio of the original duration to the new one.

This raises a key question: how should we configure the ratio \(\frac{T_1}{T_2}\) to change pitch in units of steps? Recall from Definition 9 in Chapter 4 that raising a frequency by \(k\) steps multiplies it by \(2^{k/12}\). Setting \(f_2 = f_1 \cdot 2^{k/12}\), we need

\[\frac{T_1}{T_2} = 2^{k/12}, \qquad \text{or equivalently} \qquad T_2 = \frac{T_1}{2^{k/12}}.\]

For example, to shift a recording of C4 up to G4 (\(k = 7\) steps), we read it \(2^{7/12} \approx 1.498\) times faster than normal. If the C4 recording lasts \(2\) seconds, the resulting G4 lasts \(\frac{2}{1.498} \approx 1.33\) seconds. To shift a recording down, \(k\) is negative, so the ratio is less than \(1\) and we read the recording slower.

The code below implements repitching. The function resample(x, ratio) reads x at ratio times its original speed with linear interpolation, keeping the output sample rate the same. It is just like the wavetable reader of Chapter 3, minus the wraparound. Then, repitch(x, k) shifts the pitch of x by \(k\) steps by resampling it with a ratio of \(2^{k/12}\). Finally, clarinet(pitch) is an instrument that plays any pitch by repitching our single recording of A#5:

# hide
import numpy as np
import pyquist as pq
from pyquist.helper import pitch_name_to_pitch

def resample(x: pq.Audio, ratio: float) -> pq.Audio:
    """Read x at `ratio` times its original speed, with linear interpolation."""
    samples = np.asarray(x.samples)
    N = len(samples)
    M = int(N / ratio)                        # the duration scales by 1 / ratio
    p = np.arange(M) * ratio                  # fractional read positions in x
    p_floor = np.floor(p).astype(int)
    alpha = (p - p_floor)[:, np.newaxis]
    p_next = np.minimum(p_floor + 1, N - 1)
    y = (1 - alpha) * samples[p_floor] + alpha * samples[p_next]
    return pq.Audio(y, x.sample_rate)


def repitch(x: pq.Audio, k: float) -> pq.Audio:
    """Shift the pitch of x by k steps."""
    return resample(x, 2 ** (k / 12))


def clarinet(pitch: str) -> pq.Audio:
    """Play any pitch by repitching a single recording of a clarinet playing A#5."""
    recording = pq.Audio.from_file("./assets/clarinet.wav")
    k = pitch_name_to_pitch(pitch) - pitch_name_to_pitch("A#5")
    return repitch(recording, k)


arpeggio = pq.Score([(0.5 * i, {"pitch": pitch})
                     for i, pitch in enumerate(["A#4", "D5", "F5", "A#5", "D6"])])
pq.play(arpeggio.render(clarinet), normalize=True)

Recall from Chapter 7 that reading a recording faster can also push its highest harmonics past the Nyquist frequency, where they alias. High-quality samplers avoid this by low-pass filtering the recording before reading it faster, much like Pyquist’s bandlimited pq.Audio.resample.

In theory, a single recording can now be shifted to any pitch. In practice, the larger \(|k|\) is, the less realistic the result sounds. So sample-based instruments typically pre-record several pitches across the range of the instrument, and resample whichever recording is closest to the target pitch, a practice called multisampling.

Resampling gives us control over pitch. However, there’s a catch: duration and pitch are now coupled! Repitching our A#5 recording down an octave doubles its duration, and repitching it up an octave halves it. When we play music, we want to control the duration of each note independently of its pitch, just like for the other synthesis methods we have studied.

Modifying duration via looping

Recall from Chapter 4 that the notes of real instruments pass through multiple phases: an attack, a decay, a sustain, and a release. Suppose we have a single recording of a note, repitched or otherwise, that lasts some fixed duration. How can we adjust its duration?

The key insight comes from how these phases behave as duration changes. When the same instrument plays notes of two different durations, the attack, decay, and release are roughly the same in both. Only the sustain is elongated (or shortened), and this elongation is quasi-periodic at the note’s fundamental period. This suggests a strategy: keep the attack and release of the recording as they are, identify a portion of the recorded sustain, and loop it, repeating it as many times as needed to reach the desired duration:

Fig. 77 Our clarinet note (top), shaded by phase, with a sustain loop (orange) in the steady part of its sustain. To play a note of any duration (bottom), everything before the loop plays once, the loop repeats as many times as needed, and playback then jumps to a point shortly before the release, skipping the rest of the recorded sustain.

Let’s formalize this. We choose a sustain loop spanning the time range \([t_s, t_e)\) of the recording, and let \(T_R\) be the duration of the end of the recording that we jump to, which contains the release. To play a note lasting \(T\) seconds, we play the first \(t_s\) seconds of the recording, then the loop as many times as needed, then the end of the recording. So the loop repeats

\[n = \frac{T - (t_s + T_R)}{t_e - t_s}\]

times, which we round to the nearest whole number.

For example, suppose a recording has a loop from \(t_s = 0.5\) to \(t_e = 0.75\) seconds, and the end of the recording that we jump to lasts \(T_R = 1\) second. Then a note lasting \(T = 4\) seconds needs \(n = \frac{4 - (0.5 + 1)}{0.25} = 10\) repetitions of the loop.

Where exactly should the loop begin and end? For the loop to repeat seamlessly, the waveform at its end should flow smoothly back into the waveform at its start. Within the sustain, the waveform is quasi-periodic, so ideally the loop spans a whole number of periods of the fundamental. The catch is that the loop must be a whole number of audio samples long, but a period is rarely a whole number of audio samples.

For example, our clarinet’s A#5 has a fundamental frequency of \(f_0 \approx 932.33\) Hz, so one period lasts \(\frac{f_s}{f_0} = \frac{44{,}100}{932.33} \approx 47.30\) audio samples. A loop of either \(47\) or \(48\) audio samples would not line up with the waveform, so every wrap would jump by a fraction of a period, more than \(900\) times per second.

There are two common workarounds. The first is to loop over many periods instead of one, choosing the loop points by inspecting the waveform for a start and an end whose surroundings match closely.

We chose the loop points for our clarinet this way, along with the point where it jumps to the release. Putting it all together with repitching, here is our clarinet with independent control over pitch and duration. Notice the order of operations: we loop first, at the recording’s original pitch, so that the loop points are exact audio samples. Only then do we repitch, because resampling a loop can make its seams worse. Since repitching scales the duration by \(2^{-k/12}\), we loop to a duration of \(T \cdot 2^{k/12}\) seconds before repitching:

# hide
import numpy as np
import pyquist as pq
from pyquist.helper import pitch_name_to_pitch


def resample(x: pq.Audio, ratio: float) -> pq.Audio:
    """Read x at `ratio` times its original speed, with linear interpolation."""
    samples = np.asarray(x.samples)
    N = len(samples)
    M = int(N / ratio)                        # the duration scales by 1 / ratio
    p = np.arange(M) * ratio                  # fractional read positions in x
    p_floor = np.floor(p).astype(int)
    alpha = (p - p_floor)[:, np.newaxis]
    p_next = np.minimum(p_floor + 1, N - 1)
    y = (1 - alpha) * samples[p_floor] + alpha * samples[p_next]
    return pq.Audio(y, x.sample_rate)


def repitch(x: pq.Audio, k: float) -> pq.Audio:
    """Shift the pitch of x by k steps."""
    return resample(x, 2 ** (k / 12))

LOOP_START, LOOP_END = 0.586803, 0.766916   # seconds: audio samples 25,878 and 33,821
RELEASE = 3.321361                          # seconds: audio sample 146,472, where we jump to the release


def clarinet(pitch: str, duration: float) -> pq.Audio:
    """A sampled clarinet with independent control over pitch and duration."""
    recording = pq.Audio.from_file("./assets/clarinet.wav")
    x, f_s = np.asarray(recording.samples), recording.sample_rate
    k = pitch_name_to_pitch(pitch) - pitch_name_to_pitch("A#5")
    # Loop first, at the original pitch, so the loop points are exact audio samples. Repitching
    # will then scale the duration by 2^(-k/12), so we loop to duration * 2^(k/12) seconds.
    target = duration * 2 ** (k / 12) * f_s
    s, e, r = (round(t * f_s) for t in (LOOP_START, LOOP_END, RELEASE))
    # Play up to the loop, repeat the loop to fill out the duration, then jump to the release
    n = max(0, round((target - s - (len(x) - r)) / (e - s)))
    looped = pq.Audio(np.concatenate([x[:s]] + [x[s:e]] * n + [x[r:]]), f_s)
    return repitch(looped, k)


# The same pitch at three durations, then three pitches at the same duration
notes = [("A#5", 2.5), ("A#5", 4.0), ("A#5", 8.0), ("D5", 4.0), ("F5", 4.0), ("D6", 4.0)]
onsets = np.cumsum([0.0] + [duration for _, duration in notes[:-1]])
melody = pq.Score([(t, {"pitch": pitch, "duration": duration})
                   for t, (pitch, duration) in zip(onsets, notes)])
pq.play(melody.render(clarinet), normalize=True)

The second workaround addresses a subtler problem. Real sounds are only quasi-periodic, so the end of a loop never matches its start exactly. To smooth over the remaining mismatch, samplers often crossfade at each wrap: they gradually fade out the end of the loop while fading in the audio that originally led into the loop’s start:

Fig. 78 Crossfading a loop. Just before each wrap, the end of the loop (red) fades out while the audio just before \(t_s\) (gold) fades in. Each pass therefore ends on the audio that originally led into \(t_s\), so wrapping back to \(t_s\) is seamless.

Modifying loudness

Finally, we want to control the loudness of each note with a parameter \(\ell \in [0, 1]\), from silent (\(\ell = 0\)) to the full level of the recording (\(\ell = 1\)). Recall from Chapter 7 that we perceive loudness on a logarithmic scale, which is why we measure it in decibels. So a simple approach maps \(\ell \in (0, 1]\) linearly onto a range of decibels, from \(-80\) dB (nearly inaudible) up to \(0\) dB (the level of the recording), and maps \(\ell = 0\) to \(-\infty\) dB (silence):

\[\begin{split}\text{dB}(\ell) = \begin{cases} 80 \cdot (\ell - 1) & \text{if } 0 < \ell \leq 1, \\ -\infty & \text{if } \ell = 0. \end{cases}\end{split}\]

We then multiply the note by the corresponding amplitude, \(10^{\text{dB}(\ell) / 20}\). Here is the final version of our sampled clarinet, with independent control over pitch, duration, and loudness:

# hide
import numpy as np
import pyquist as pq
from pyquist.helper import pitch_name_to_pitch


def resample(x: pq.Audio, ratio: float) -> pq.Audio:
    """Read x at `ratio` times its original speed, with linear interpolation."""
    samples = np.asarray(x.samples)
    N = len(samples)
    M = int(N / ratio)                        # the duration scales by 1 / ratio
    p = np.arange(M) * ratio                  # fractional read positions in x
    p_floor = np.floor(p).astype(int)
    alpha = (p - p_floor)[:, np.newaxis]
    p_next = np.minimum(p_floor + 1, N - 1)
    y = (1 - alpha) * samples[p_floor] + alpha * samples[p_next]
    return pq.Audio(y, x.sample_rate)


def repitch(x: pq.Audio, k: float) -> pq.Audio:
    """Shift the pitch of x by k steps."""
    return resample(x, 2 ** (k / 12))

LOOP_START, LOOP_END = 0.586803, 0.766916   # seconds: audio samples 25,878 and 33,821
RELEASE = 3.321361                          # seconds: audio sample 146,472, where we jump to the release


def clarinet(pitch: str, duration: float, loudness: float) -> pq.Audio:
    """A sampled clarinet with independent control over pitch, duration, and loudness."""
    recording = pq.Audio.from_file("./assets/clarinet.wav")
    x, f_s = np.asarray(recording.samples), recording.sample_rate
    k = pitch_name_to_pitch(pitch) - pitch_name_to_pitch("A#5")
    target = duration * 2 ** (k / 12) * f_s                               # duration
    s, e, r = (round(t * f_s) for t in (LOOP_START, LOOP_END, RELEASE))
    n = max(0, round((target - s - (len(x) - r)) / (e - s)))
    looped = pq.Audio(np.concatenate([x[:s]] + [x[s:e]] * n + [x[r:]]), f_s)
    gain = 0.0 if loudness == 0 else 10 ** (80 * (loudness - 1) / 20)    # loudness
    return repitch(looped, k) * gain                                     # pitch


# A crescendo: four notes, each louder than the last
notes = [("D5", 0.7), ("F5", 0.8), ("A#5", 0.9), ("D6", 1.0)]
melody = pq.Score([(3.0 * i, {"pitch": pitch, "duration": 3.5, "loudness": loudness})
                   for i, (pitch, loudness) in enumerate(notes)])
pq.play(melody.render(clarinet), normalize=True)

Real instruments also sound brighter when played louder, because playing harder strengthens their upper harmonics. To explore further, try low-pass filtering each note with a cutoff frequency that rises with \(\ell\), using the filters from Chapter 9. The complete instrument is also in code/sampler.py.