import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

4.3 Envelopes

We’ve seen that we can combine scores with timbres to produce richer music. But there’s one problem. In a musical score, events are finite in duration: you pluck a string, and the sound decays away after some time. The synthesis techniques of Chapter 3, however, produce tones of theoretically infinite duration: a sum of sinusoids just keeps going. Envelopes bridge this gap, taking us from infinite sustained tones to finite sound events.

Fig. 11 A plucked guitar string (from Chapter 3). Top: the raw waveform, with a zoomed inset revealing its quasi-periodic oscillation. Bottom (upper half only): a smooth curve tracing the waveform’s peak amplitude, its envelope, alongside a piecewise-linear approximation of that envelope.

Consider the plucked string above. The zoomed inset reveals the quasi-periodic behavior we’d expect from the synthesis of Chapter 3. But zoomed out, the waveform has a distinct shape: its peak amplitude rises sharply, then decays. If we trace an outline around the waveform’s peak amplitude, we get a curve that “envelopes” the oscillation within. If we could synthesize such a curve and multiply it by an oscillator, we could turn an infinite tone into a finite event. As the bottom panel suggests, even a simple piecewise-linear shape captures the essence.

A formal view

If sound is a function \(x(t) : \mathbb{R} \to \mathbb{R}\) mapping time to amplitude, an envelope is a function

\[\text{Envelope}(t) : \mathbb{R} \to [0, 1]\]

specifying an amplitude attenuation factor at each point in time, where 0 means silence and 1 means no attenuation. Crucially, an envelope is zero outside some finite window \((a, b)\):

\[\begin{split} \text{Envelope}(t) \begin{cases} \in (0, 1] & \text{if } a < t < b, \\ = 0 & \text{otherwise.} \end{cases} \end{split}\]

We apply an envelope to a sound by simple multiplication: \(x(t) \cdot \text{Envelope}(t)\). Because the envelope is zero outside \((a, b)\), the product is also zero there, regardless of how \(x(t)\) behaves. This accomplishes our goal of turning a potentially infinite sound into a finite one.

Oscillator alone

Oscillator times envelope

A 220 Hz sine before and after applying an attack/decay envelope.

Fig. 12 Top: the oscillator \(x(t)\), a 220 Hz sine. Middle: an attack/decay envelope (\(a_\text{dur} = 0.1\) s, \(d_\text{dur} = 0.9\) s). Bottom: their product. The product’s amplitude is bounded by the envelope (dashed), and it fades to silence at both ends.

Piecewise-linear envelopes

Envelopes are often described by piecewise-linear functions, parameterized by a set of control points \((t_1, a_1), (t_2, a_2), \ldots, (t_P, a_P)\). Between consecutive control points, the envelope interpolates linearly. Outside the first and last control points, it typically is assumed to take on the edge values: \(a_1\) if \(t \leq a_1\), or \(a_P\) if \(t \geq t_P\). Accordingly, for most envelopes, \(a_1 = a_P = 0\).

The simplest useful envelope has two segments and a single interior control point: an attack that rises linearly from 0 to a peak, followed by a decay that falls back to 0. We can write it with control points \((0, 0)\), \((a_\text{dur}, 1)\), and \((a_\text{dur} + d_\text{dur}, 0)\):

\[\begin{split} \text{adenv}(t) = \begin{cases} \dfrac{t}{a_\text{dur}} & \text{if } 0 \le t < a_\text{dur}, \\[2mm] 1 - \dfrac{t - a_\text{dur}}{d_\text{dur}} & \text{if } a_\text{dur} \le t \le a_\text{dur} + d_\text{dur}, \\[2mm] 0 & \text{otherwise.} \end{cases} \end{split}\]

In code, we can express any piecewise-linear envelope compactly with np.interp, which handles the segment-by-segment interpolation for us:

def adenv(a_dur: float, d_dur: float, N: int, n: int = 0) -> np.ndarray:
    t = (n + np.arange(N)) / F_S
    env = np.interp(
        t, [0.0, a_dur, a_dur + d_dur], [0.0, 1.0, 0.0]
    )
    return env[:, np.newaxis]

Note that adenv returns an np.ndarray, not a pq.Audio. This is a deliberate choice: an envelope is an amplitude-shaping curve, not something we intend to listen to as audio.

The trailing [:, np.newaxis] reshapes the result to (N, 1) so that, recalling the (num_samples, num_channels) convention from Chapter 2, the envelope broadcasts cleanly across the channels of a pq.Audio when we multiply by adenv(...). Extending this to an arbitrary number of control points is left as an exercise to the reader.

Fig. 13 The output of adenv(0.1, 0.9, ...) over one second: a 0.1 s attack to the peak, then a 0.9 s decay. The three control points are marked.

A 220 Hz sine multiplied by adenv(0.1, 0.9, ...), producing a finite note. The full code is in code/envelope.py.

The widget below builds the same envelope from its two durations. Drag the attack and the decay, and listen to how the note starts and ends.

# hide
# no-output
from IPython.utils.capture import capture_output
with capture_output():
    %pip install -q plotly anywidget

import asyncio
import os
import numpy as np
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import ipywidgets as widgets
from IPython.display import Audio
import icm_plotly
from icm_plotly import RED, BLUE, GOLD, IRON, TEAL, STEEL

Drag the attack and decay. The top panel is the envelope with its three control points in gold, and the bottom panel is the 220 Hz tone multiplied by it. The audio card plays the current shape. Pull the attack all the way down and you will hear the click that a sudden start produces.

# hide
# autorun
A0, D0, F0 = 0.1, 0.9, 220.0        # the chapter's adenv(0.1, 0.9) settings

T_TOT = 2.0                         # two seconds on screen
t = np.linspace(0.0, T_TOT, 8000)
TONE = np.sin(2 * np.pi * F0 * t)
SR = 44100
T_PLAY = np.arange(int(T_TOT * SR)) / SR
TONE_P = np.sin(2 * np.pi * F0 * T_PLAY)

def adenv(a_dur, d_dur, tt):
    # the chapter's piecewise-linear attack/decay, via np.interp
    return np.interp(tt, [0.0, a_dur, a_dur + d_dur], [0.0, 1.0, 0.0])

def figure():
    fig = make_subplots(rows=2, cols=1, shared_xaxes=True,
                        row_heights=[0.42, 0.58], vertical_spacing=0.1)
    env = adenv(A0, D0, t)
    fig.add_scatter(x=t * 1000, y=env, mode="lines",
                    line=dict(color=RED, width=2.2), row=1, col=1)
    fig.add_scatter(x=[0, A0 * 1000, (A0 + D0) * 1000], y=[0, 1, 0],
                    mode="markers", marker=dict(color=GOLD, size=9),
                    row=1, col=1)
    fig.add_scatter(x=t * 1000, y=TONE * env, mode="lines",
                    line=dict(color=RED, width=1.0), row=2, col=1)
    for sign in (1.0, -1.0):      # the outline sits on top of the band
        fig.add_scatter(x=t * 1000, y=sign * env, mode="lines",
                        line=dict(color=IRON, width=1.6, dash="dash"),
                        row=2, col=1)
    fig.update_yaxes(range=[-0.08, 1.14], title_text="Envelope",
                     fixedrange=True, row=1, col=1)
    fig.update_yaxes(range=[-1.15, 1.15], title_text="Amplitude",
                     fixedrange=True, row=2, col=1)
    fig.update_xaxes(fixedrange=True, row=1, col=1)
    fig.update_xaxes(range=[0, T_TOT * 1000], title_text="Time (ms)",
                     fixedrange=True, row=2, col=1)
    return fig

def controls(fig):
    a = widgets.FloatSlider(description=r"Attack $a_\text{dur}$ (s)", min=0.005,
                            max=0.5, value=A0, step=0.005, readout_format=".3f")
    d = widgets.FloatSlider(description=r"Decay $d_\text{dur}$ (s)", min=0.05, max=1.5,
                            value=D0, step=0.05)
    readout = widgets.HTML()

    # the defaults snapshot the arrays; the page's notebooks share one kernel
    def update(a, d, t=t, TONE=TONE, adenv=adenv, readout=readout):
        env = adenv(a, d, t)
        with fig.batch_update():
            fig.data[0].y = env
            fig.data[1].x = [0, a * 1000, (a + d) * 1000]
            fig.data[2].y = TONE * env
            fig.data[3].y = env
            fig.data[4].y = -env
        readout.value = (f"<span style='font-size:0.9em'>control points "
                         f"(0, 0), ({a:.3f}, 1), ({a + d:.3f}, 0) "
                         f"&nbsp;·&nbsp; the note lasts <i>a</i><sub>dur</sub> + "
                         f"<i>d</i><sub>dur</sub> = {a + d:.3f} s</span>")

    widgets.interactive_output(update, {"a": a, "d": d})

    # the audio card under the controls: the previous clip stays in place
    # while you drag (so the layout never jumps) and is swapped for the new
    # one when the pointer releases (keyboard nudges settle on a timer). It is
    # written through the Output's synced `outputs` trait, which works
    # outside a kernel message, where display() output has no destination
    out = widgets.Output()
    gate = icm_plotly.release_gate()   # pointer state: is a slider mid-drag?
    pending = []
    dirty = []

    def render(T_PLAY=T_PLAY, TONE_P=TONE_P, SR=SR, adenv=adenv):
        x = 0.125 * TONE_P * adenv(a.value, d.value, T_PLAY)
        audio = Audio(x.astype(np.float32), rate=SR, normalize=False)
        data, metadata = get_ipython().display_formatter.format(audio)
        # one assignment swaps the old card for the new one in place, so
        # the page never shows an empty card and nothing shifts
        out.outputs = ({"output_type": "display_data",
                        "data": data, "metadata": metadata},)


    async def settle():
        await asyncio.sleep(0.25)
        pending.clear()
        if dirty and not gate.dragging:
            dirty.clear()
            render()

    def on_change(_):
        dirty.append(True)
        if pending:
            pending.pop().cancel()
        pending.append(asyncio.ensure_future(settle()))

    def on_release(change):
        if not change["new"] and dirty:
            if pending:
                pending.pop().cancel()
            dirty.clear()
            render()

    gate.observe(on_release, names="dragging")

    for s in (a, d):
        s.observe(on_change, names="value")
    if not os.environ.get("ICM_BOOK_BUILD"):   # the build bakes no card
        render()
    return widgets.VBox([a, d, readout, out, gate])

icm_plotly.show(figure, controls)

Attack \(a_\text{dur}\) (s)0.100

Decay \(d_\text{dur}\) (s)0.90

control points (0, 0), (0.100, 1), (1.000, 0)  ·  the note lasts a_dur + d_dur = 1.000 s