import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

10.3 Granular synthesis

We can now extract and reassemble frames, but so far the exercise has been been somewhat pointless: worst case we lose information, and best case we get back exactly what we started with. The interesting possibilities open up when we manipulate the frames before reassembling them.

This is the idea behind granular synthesis: chop a sound into many tiny slices, called grains (typically tens of milliseconds long), then transform and rearrange those grains to build something new. It is a bit like making a collage out of a photograph, cutting it into little pieces and gluing them back in a new arrangement.

Fig. 65 Granular synthesis in three steps: extract short grains from the source (each multiplied by a smooth window), then reassemble them, possibly reordered, resized, or otherwise transformed, into a new sound.

Because a grain is so short, it loses much of the recognizable character of the original sound. And a raw grain, sliced out with a hard rectangular window, has abrupt edges that produce an audible click. So in practice we multiply each grain by a smooth window (a Hann window, say) to taper those edges. Here is a handful of 50 ms grains lifted from the running example and played back with a big gap between them, first with hard rectangular edges and then windowed:

Raw (rectangular) grains

Windowed (Hann) grains

The same grains, played with a rectangular window (note the click at each edge) and with a Hann window (smooth).

Manipulating grains

Individual grains are not very interesting on their own. The power of granular synthesis comes from manipulating them as units before reassembly. One of the simplest manipulations is to reorder them. We can shuffle grains across the whole signal, or shuffle them only within short segments:

Fig. 66 Two ways to randomize grain order: globally (top), which fully scrambles the sound, or within short segments (bottom), which keeps the large-scale structure while blurring the fine detail.

Reordering grains produces a striking effect. It preserves the overall texture of the sound while erasing its specifics, a kind of controlled blur:

Granular texture (grains shuffled within segments)

For contrast: the raw samples shuffled

Shuffling grains keeps the character of the sound. Shuffling the raw samples (bottom) destroys it entirely, leaving only noise.

That contrast is the whole point. Shuffling grains keeps the sound recognizable, but shuffling the underlying samples (not grains) yields nothing but noise. Working at the level of grains, rather than samples, is what makes the effect musical. Order is not the only property we can manipulate: we could also change the grains’ amplitude, duration, pitch, or density before reassembling. You can explore all of these by editing the manipulate function below:

# hide
import numpy as np
import pyquist as pq


def iter_frames(audio, N_H, N_F):
    x = np.asarray(audio.samples).reshape(-1)
    for start in range(0, len(x) - N_F + 1, N_H):
        yield x[start:start + N_F]


def overlap_add(grains, N_H, sample_rate):
    grains = list(grains)
    N_F = len(grains[0])
    out = np.zeros(N_H * (len(grains) - 1) + N_F)
    for k, g in enumerate(grains):
        if len(g) == N_F:
            out[k * N_H:k * N_H + N_F] += g
    return pq.Audio(out.astype(np.float32), sample_rate)


def hann(n):
    return 0.5 * (1 - np.cos(2 * np.pi * np.arange(n) / n))

# Granular synthesis: chop the sound into overlapping grains, MANIPULATE them,
# and glue them back. Edit `manipulate` to invent your own effect!
def manipulate(grains):
    grains = [g * hann(len(g)) for g in grains]     # smooth each grain's edges
    out = []                                        # shuffle order within blocks
    for i in range(0, len(grains), 100):
        block = grains[i:i + 100]
        np.random.shuffle(block)
        out += block
    return out


N_F = 2048                     # grain size in samples (~46 ms)
N_H = 1024                     # spacing when extracting grains
N_H_out = 1024                 # spacing when reassembling (change to time-stretch!)

audio = pq.Audio.from_file("./assets/audio-trio.wav")
grains = [g for g in iter_frames(audio, N_H, N_F) if len(g) == N_F]
grains = manipulate(grains)
pq.play(overlap_add(grains, N_H_out, audio.sample_rate))

Time stretching

Here is a particularly useful manipulation. What if we decouple the hop length at which we extract grains from the hop length at which we overlap them back together? Call the extraction hop \(N_H\) and the reassembly hop \(N_H'\). If \(N_H' = 2 N_H\), we spread the grains out to twice their original spacing, doubling the output’s duration. If \(N_H' = \tfrac{1}{2} N_H\), we pack them together, halving it:

Fig. 67 Time stretching by decoupling the hops. The grains are unchanged, but reassembling them at twice the spacing (\(N_H' = 2 N_H\)) makes the output twice as long, halving the playback speed.

Original

Half speed (grains spread out)

Double speed (grains packed together)

Granular time stretching. Changing the spacing at reassembly changes the duration, and therefore the playback speed, while the grains themselves are untouched.

We have achieved time stretching. Spreading or packing the grains changes the total duration, and hence the playback speed, without touching the contents of the grains themselves.

This is the second time we have changed playback speed. The first was resampling in Chapter 7. Listen to the same speed changes done by resampling instead:

Half speed via resampling

Double speed via resampling

Resampling also changes the speed, but notice that it changes the pitch too, exactly like slowing down or speeding up a record.

The difference is crucial. Resampling changes duration and pitch together (slower means lower, faster means higher), which was exactly what we wanted for wavetable synthesis. But granular time stretching changes duration while keeping the pitch constant. Having both techniques suggests something powerful: decoupled control over pitch and duration. We can first resample the grains to change their pitch, and then independently time stretch them by changing their spacing:

Fig. 68 Decoupled pitch and time. First resample each grain, which shortens it and raises its pitch (row 2). Then reassemble the grains at a wider hop, which stretches the result back out in time (row 3). Because the two steps are independent, the output can be both slower and higher-pitched than the input.

Half speed and 20% higher pitch (resample + stretch)

Combining resampling (to shift the pitch up 20%) with granular time stretching (to slow to half speed) lets us control the two independently.

In practice, getting a clean result from granular time stretching requires a generous amount of overlap between grains, so that the crossfades between them are smooth.