0.0 Why study computer music?

0.0 Why study computer music?#

Have you ever wondered how sound and music are stored on and processed by computers? Or how any song can be efficiently streamed to your phone over the network? Or how plugins in your digital audio workstation are working behind the scenes? If so, then you are already asking questions in the domain of computer music. This book is an invitation to take those questions seriously, and to develop the technical principles needed to answer them rigorously.

Music and computation are inextricably linked#

Computing has become ubiquitous within everyday music practice. When you stream a song on your phone, computers are compressing audio, buffering it across a network, and converting digital samples back into sound. When a producer mixes a track in a digital audio workstation (DAW), they are orchestrating thousands of computations per second to filter, equalize, and combine signals. When you attend a live show, digital mixing consoles route, equalize, and apply effects to every signal coming off the stage in real time, while in-ear monitor systems compute a personalized mix for each performer.

Even before the advent of digital computers, music and computation in the broader sense have been deeply entwined. The earliest theories of musical instruments were built on numerical relationships: Pythagorean ratios between string lengths, the mathematics of consonance and dissonance. Some of the deepest properties of music, such as pitch and rhythm, are fundamentally about periodicity: patterns that repeat in time at definable rates. To study music carefully is, almost unavoidably, to study computation and mathematics.

If you are interested in better understanding these relationships, then computer music is for you.

Technology is upstream of musical possibility#

Music and technology have always co-evolved. In the hands of musicians, new technologies expand the creative and cultural boundaries of what music can be. A pervasive theme throughout music history is that each major technological development opens up new creative opportunities for artists, opportunities that often could not even be articulated until the technology made them imaginable.

The pianoforte’s expressive dynamic range allowed Beethoven to compose his sonatas in a way that would have been impossible on the harpsichord. The Beatles wielded the entire studio as an instrument, using multitrack recording technology to make an album like Revolver conceivable. Amplification and electricity transformed the electric guitar into Jimi Hendrix’s voice. Digital sampling let Kate Bush build the sonic world of Hounds of Love from fragments of real-world sound.

If you are interested in building new computing tools that may expand the possibilities of music, then computer music is for you.

Inspiration: FM Synthesis#

To make this concrete, consider one of the most influential episodes in the history of computer music: John Chowning’s invention of frequency modulation (FM) synthesis [Cho73]. Chowning’s work is a kind of “full stack” example of what computer music can be, weaving together acoustics, mathematical theory, programming, instrument design, and ultimately musical culture.

Music acoustics: Real musical sounds are not pure tones. They contain rich mixtures of many time-varying periodic components that fade in, fade out, and shift in relative strength over the duration of a note. Synthesizing such sounds convincingly is challenging, especially with the limited compute available in the 1970s, because each component nominally requires its own oscillator. Consider, for example, the dense spectral fingerprint of a percussive chime instrument:

Orchestral chime. chimes_f#3_p_1.wav by sgossner, License: Attribution 4.0.

Mathematical theory: Working at the Stanford Artificial Intelligence Laboratory (SAIL), Chowning realized that the well-known method of frequency modulation, when applied in the audio range, produces infinitely complex spectra by chaining just two simple sinusoidal components together in a particular way. The basic FM equation is

\[x(t) = \sin\left(2 \pi f_c t + I \sin(2 \pi f_m t)\right).\]

If this doesn’t mean much to you now, no worries - you’ll learn more about this equation and its parameters when we study FM in detail later in this text. Focus for now on the high level: Chowning showed that, by carefully controlling these parameters over time, this single equation could imitate a striking range of natural musical sounds:

FM bell sound synthesized using Csound fmbell.

Efficient programming: The mathematical elegance of FM is only useful if it can be computed fast enough (tens of thousands of times per second) to produce a continuous audio stream. This requires careful, efficient implementations, bringing an algorithmic perspective to computer music. In Python, an efficient FM synthesizer might look something like:

def fm(f_c, f_m, I, f_s, T):
    audio = [0.0] * int(f_s * T)  # audio buffer
    p_c, p_m = 0.0, 0.0  # carrier/modulator phase in radians
    d_c, d_m = (2.0 * math.pi * f / f_s for f in (f_c, f_m))  # radians per sample
    for i in range(len(audio)):
        audio[i] = sin_fast(p_c + I * sin_fast(p_m))
        p_c, p_m = p_c + d_c, p_m + d_m
    return audio

Observe a few high-level changes from the formula above: (1) we’re using discrete computation instead of continuous math, (2) we’re pre-computing some operations outside of the for loop, and (3) we’re calling sin_fast which uses a pre-computed lookup table (see the full example here). These were essential optimizations in 1973, and remain useful today, e.g., for running many FM synthesizers in parallel in your DAW.

Instrument design: Yamaha licensed FM as the synthesis engine in the legendary DX7 synthesizer, turning a research result into a piece of hardware that could be played on stage and in the studio.

Fig. 1 The Yamaha DX7 (1983), the first commercially successful digital synthesizer, brought Chowning’s FM synthesis to musicians worldwide.#
Music culture: The DX7 was adopted by thousands of musicians and became, in many ways, the sound of the 1980s. Ironically, while FM had originally been explored as a way to imitate existing acoustic instruments, musicians ended up preferring the synthesizer’s ability to create entirely novel sounds that no acoustic instrument could produce. You can hear the DX7 unmistakably in tracks like A-ha — “Take On Me” and Whitney Houston — “Didn’t We Almost Have It All”.

A mathematical insight, inspired by acoustics, refined into an algorithm, embodied in a piece of hardware, became a defining aesthetic of an era. This is the kind of impact that computer music makes possible.

Try FM synthesis yourself below: the same equation, with sliders for its three parameters.

# hide
# no-output
from IPython.utils.capture import capture_output
with capture_output():
    %pip install -q plotly anywidget

import asyncio
import os
import numpy as np
import plotly.graph_objects as go
from plotly.subplots import make_subplots
import ipywidgets as widgets
from IPython.display import Audio
import icm_plotly
from icm_plotly import RED, BLUE

Drag the sliders: the waveform is on the left, and the frequencies it contains are on the right. The audio card underneath rings the result like a bell, always at the current settings.

# hide
# autorun
FC0, FM0, I0 = 220.0, 220.0, 2.0        # starting parameters

t_wave = np.linspace(0.0, 0.02, 1200)   # 20 ms of waveform
N = 4096                                # spectrum: ~93 ms, hann-windowed
sr = 44100
t_spec = np.arange(N) / sr
win = np.hanning(N)
freqs = np.fft.rfftfreq(N, 1 / sr)
mask = freqs <= 5000
t_bell = np.arange(2 * sr) / sr          # two seconds, for the ear
env = np.exp(-3 * t_bell)               # a bell's decay

def fm(fc, fm_, I, t):
    return np.sin(2 * np.pi * fc * t + I * np.sin(2 * np.pi * fm_ * t))

def spectrum(fc, fm_, I):
    X = np.abs(np.fft.rfft(fm(fc, fm_, I, t_spec) * win))
    return X[mask] / X.max()

def figure():
    fig = make_subplots(rows=1, cols=2, horizontal_spacing=0.12)
    fig.add_scatter(x=t_wave * 1000, y=fm(FC0, FM0, I0, t_wave),
                    mode="lines", line=dict(color=RED, width=1.8),
                    row=1, col=1)
    fig.add_scatter(x=freqs[mask], y=spectrum(FC0, FM0, I0),
                    mode="lines", line=dict(color=BLUE, width=1.5),
                    row=1, col=2)
    fig.update_xaxes(range=[0, 20], title_text="Time (ms)",
                     fixedrange=True, row=1, col=1)
    fig.update_yaxes(range=[-1.1, 1.1], title_text="Amplitude",
                     fixedrange=True, row=1, col=1)
    fig.update_xaxes(range=[0, 5000], title_text="Frequency (Hz)",
                     fixedrange=True, row=1, col=2)
    fig.update_yaxes(range=[0, 1.05], title_text="Magnitude",
                     fixedrange=True, row=1, col=2)
    return fig

def controls(fig):
    fc = widgets.FloatSlider(description="Carrier f_c (Hz)", min=55, max=880,
                             value=FC0, step=5)
    fmod = widgets.FloatSlider(description="Modulator f_m (Hz)", min=55,
                               max=880, value=FM0, step=5)
    idx = widgets.FloatSlider(description="Index I", min=0, max=10, value=I0,
                              step=0.1)

    # the defaults snapshot the arrays; the page's notebooks share one kernel
    def update(fc, fm_, I, t_wave=t_wave, fm=fm, spectrum=spectrum):
        with fig.batch_update():
            fig.data[0].y = fm(fc, fm_, I, t_wave)
            fig.data[1].y = spectrum(fc, fm_, I)

    widgets.interactive_output(update, {"fc": fc, "fm_": fmod, "I": idx})

    # the audio card under the controls: the previous clip stays in place
    # while you drag (so the layout never jumps) and is swapped for the new
    # one when the pointer releases (keyboard nudges settle on a timer). It is
    # written through the Output's synced `outputs` trait, which works
    # outside a kernel message, where display() output has no destination
    out = widgets.Output()
    gate = icm_plotly.release_gate()   # pointer state: is a slider mid-drag?
    pending = []
    dirty = []

    def render(t=t_bell, env=env, sr=sr):
        # a bell: index and amplitude decay together, which makes it ring
        x = env * np.sin(2 * np.pi * fc.value * t
                         + idx.value * env * np.sin(2 * np.pi * fmod.value * t))
        x *= 0.125 / np.abs(x).max()          # about -18 dBFS, a safe level
        audio = Audio(x.astype(np.float32), rate=sr, normalize=False)
        data, metadata = get_ipython().display_formatter.format(audio)
        # one assignment swaps the old card for the new one in place, so
        # the page never shows an empty card and nothing shifts
        out.outputs = ({"output_type": "display_data",
                        "data": data, "metadata": metadata},)

    async def settle():
        await asyncio.sleep(0.25)
        pending.clear()
        if dirty and not gate.dragging:
            dirty.clear()
            render()

    def on_change(_):
        dirty.append(True)
        if pending:
            pending.pop().cancel()
        pending.append(asyncio.ensure_future(settle()))

    def on_release(change):
        if not change["new"] and dirty:
            if pending:
                pending.pop().cancel()
            dirty.clear()
            render()

    gate.observe(on_release, names="dragging")

    for s in (fc, fmod, idx):
        s.observe(on_change, names="value")
    if not os.environ.get("ICM_BOOK_BUILD"):   # the build bakes no card
        render()
    return widgets.VBox([fc, fmod, idx, out, gate])

icm_plotly.show(figure, controls)

Computing is the frontier of music technology#

Computation is now a key component of music on stage, in the studio, and in your ears. From software synthesizers to streaming codecs to noise-cancelling headphones, computation is increasingly synonymous with music. It is, in most contexts, the mechanism through which music is made, distributed, and experienced.

This trend of music and technology co-evolving will almost certainly continue as we venture into new technologies such as artificial intelligence. Like past technological developments (recording, amplification, sampling) these newer technologies will likely reshape the economic landscape of music, but they will also present new creative opportunities for those who learn to use them thoughtfully. If you are interested in understanding how computers synthesize, manipulate, and ultimately reshape musical sound, then computer music is for you.