import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

10.6 Real-time processing

Frame-based processing has one more role to play, which we will return to in Chapter 17: it is how real-time audio works. Suppose we want to synthesize an endless stream of audio on the fly, computing each sample \(x[n] = x(\tfrac{n}{f_s})\) just in time to be played. We could call our synthesis function once per sample, but function calls have computational overhead, and at tens of thousands of samples per second that overhead adds up fast. Worse, it is overkill: we cannot physically turn knobs fast enough to need per-sample control anyway.

Instead, real-time systems compute audio in frames, usually called blocks in this context. We pick a block length \(B\), and at each moment \(\frac{k \cdot B}{f_s}\) the operating system asks our program for the next \(B\) samples. This is exactly frame-based processing with \(N_H = N_F = B\). As long as we can compute each block in less than \(\frac{B}{f_s}\) seconds, the audio never runs dry and we achieve a real-time stream. We will develop this idea properly when we study real-time, interactive audio.

This connects directly to the unit generators of Chapter 4. A unit generator like an oscillator runs continuously, and in a real-time system we run it one block at a time, updating its parameters in between blocks as control events arrive (a user turning a knob, a note starting, a slider moving). The example below drives a sine oscillator block by block, feeding it frequency changes from a pq.Score as if a performer were injecting them live. The key detail is that we carry the oscillator’s phase across block boundaries (as we learned to do in Chapter 6), so the blocks stitch together seamlessly with no clicks. Each iteration of the loop stands in for one call from the audio system: check for new control events, compute \(B\) samples, and hand them off. Real systems run this loop forever, but the structure is identical.

# hide
import numpy as np
import pyquist as pq

# Real-time block-based synthesis: a sine oscillator (a unit generator) run one
# block at a time, with control events arriving between blocks. We carry the
# oscillator's phase across blocks so the seams never click. Edit the Score or
# the block size B and listen!
f_s = 44100
B = 512                                    # block size, in samples

# Frequency-change events (a stand-in for a user turning a knob live).
score = pq.Score([
    (0.0, {"freq": 220.0}),
    (0.5, {"freq": 330.0}),
    (0.75, {"freq": 440.0}),
])

freq = 220.0                               # current oscillator frequency (Hz)
phase = 0.0                                # phase accumulator, carried across blocks
blocks = []
for k in range(100):                        # ~0.75 s of audio, one block at a time
    t0 = k * B / f_s                        # this block begins at time t0
    for time, event in score:               # apply any events that land in this block
        if t0 <= time < t0 + B / f_s:
            freq = event["freq"]
    n = np.arange(B)
    blocks.append(np.sin(phase + 2 * np.pi * freq * n / f_s))
    phase += 2 * np.pi * freq * B / f_s      # carry phase into the next block

pq.play(pq.Audio(np.concatenate(blocks), f_s))