import matplotlib
if not hasattr(matplotlib.RcParams, "_get"):
    matplotlib.RcParams._get = dict.get

16.2 Music understanding#

So far, we have studied how AI can generate music. The opposite direction, recognizing high-level semantic information in music, is the goal of music information retrieval (MIR), also called music understanding. Most often, the input is a recording, but MIR also works with symbolic music, and even images of sheet music:

A diagram. A blue waveform labeled Recording points into a box labeled Music understanding, which has arrows to five outputs. Genre: jazz. Tempo and key: 120 BPM, C major. Score: a short staff of notes. Instrument stems: three small waveforms labeled piano, bass, and drums. Similar music: three small gray waveforms.

Fig. 138 Music understanding recognizes many kinds of information in a recording, such as its genre, its tempo and key, its score, the separate recordings of its instruments (stems), and other music that sounds similar.#

Music classification#

Say you were asked to write a program that recognizes the genre of a recording. For example, given these two recordings, it should output “punk” for the first, and “jazz” for the second:

Punk

Jazz

Excerpts of two clips from the FreeTZAN dataset. The Zombie Dandies - Red the Hunter, License: CC BY 4.0. Kevin MacLeod - I Knew a Guy, License: CC BY 3.0.

How would you do it? It’s not at all obvious how we would write down rules for this decision by hand. Perhaps the best that we can do is to learn to make it from data. This is the idea of supervised learning:

Definition 55 (Supervised learning)

Given training examples of inputs \(x\) paired with labels \(y\), supervised learning fits a model \(f_\theta\) to predict the label of each input, \(f_\theta(x) \approx y\), by minimizing a loss function that measures the difference between its predictions and the labels.

For example, \(x\) might be a recording and \(y\) its genre, such as “jazz” or “punk”. Notice the similarity to training a language model, where the “label” of each context is the next element. Once trained, the model can hopefully predict labels for new inputs that it has never seen.

Audio features#

What should we use as the input \(x\)? A thirty-second recording contains over a million samples, and individual samples are not very meaningful on their own. Instead, consider how you recognize genres. Recall from Chapter 15 that our inner ear decomposes sound into frequency bands, the critical bands. We perceive sound in the frequency domain, hearing the buzz of a distorted guitar or the shimmer of a cymbal, not individual samples. So it stands to reason that we should transform sound into similar features before classifying it.

MIR systems often use the spectrograms of Chapter 10 as features. A log-mel spectrogram goes a step further. Much like the critical bands, it groups the frequency bins of each frame into a few dozen bands that are evenly spaced on the mel scale, a perceptual scale of pitch. Then, much like our perception of loudness, it takes the logarithm of the energy in each band:

Three plots of the same four seconds of a jazz trio. Left, the waveform, labeled 176,400 samples. Middle, a spectrogram up to 11 kHz, labeled 513 bins per frame, showing thin horizontal lines for the harmonics of each note. Right, a log-mel spectrogram, labeled 64 bands per frame, which shows the same notes in coarser detail, with more resolution at low frequencies.

Fig. 139 The jazz trio from Chapter 10, as a waveform (left), a spectrogram (middle), and a log-mel spectrogram (right). Each is more compact, and closer to our perception, than the last.#

Example: genre classification. Let’s try this on FreeTZAN, a small, openly licensed dataset of thirty-second clips from the Free Music Archive [DBVB17], in \(10\) genres: ambient, classical, electronic, experimental, folk, hip-hop, jazz, pop, punk, and rock. We’ll use \(1000\) of its clips for training, and \(100\) others for evaluating accuracy, with no artist appearing in both. The labels are the genres that the artists chose for their own tracks, so they are noisy, and some clips are hard to classify even for a human listener.

The code below compares three kinds of features: (1) one second of the waveform, downsampled to \(1\) kHz, (2) the amplitude spectrogram, and (3) the log-mel spectrogram with \(64\) bands. To summarize each clip as a single vector, we average the spectrogram over time. The log-mel spectrogram is compact enough that we can afford two numbers per band: its average over time, and its standard deviation over time, which captures how much the band fluctuates (for example, with drum hits). (To fit the dataset into this book, these features were computed ahead of time.) For each kind of feature, the code trains a logistic regression classifier, one of the simplest models for classification:

# hide
from collections.abc import Callable

import numpy as np


def train_classifier(X: np.ndarray, y: np.ndarray, l2: float = 0.01, steps: int = 300,
                     learning_rate: float = 0.01) -> Callable[[np.ndarray], np.ndarray]:
    """Train a logistic regression classifier, one of the simplest models for
    classification, and return a function that predicts the labels of new inputs.

    The classifier first standardizes each feature (to a mean of 0 and a standard deviation
    of 1). Then it predicts the probability of each label as softmax(x W + b), where the
    parameters W and b are trained to minimize the average negative log-likelihood of the
    training labels, plus a small penalty (l2 / 2) ||W||^2 that discourages large weights.
    For training, it uses Adam, a variant of gradient descent that adapts the step size of
    each parameter."""
    mean, std = X.mean(axis=0), X.std(axis=0) + 1e-8
    X = (X - mean) / std
    Y = np.eye(y.max() + 1)[y]                                 # one-hot labels
    params = [np.zeros((X.shape[1], Y.shape[1])), np.zeros(Y.shape[1])]   # W and b
    m = [np.zeros_like(p) for p in params]                    # Adam's running averages
    v = [np.zeros_like(p) for p in params]
    for t in range(1, steps + 1):
        W, b = params
        z = X @ W + b
        P = np.exp(z - z.max(axis=1, keepdims=True))
        P /= P.sum(axis=1, keepdims=True)                     # softmax: P(label | x)
        G = (P - Y) / len(X)                                  # gradient of the loss w.r.t. z
        grads = [X.T @ G + l2 * W, G.sum(axis=0)]
        for p, g, m_p, v_p in zip(params, grads, m, v):
            m_p[:] = 0.9 * m_p + 0.1 * g
            v_p[:] = 0.999 * v_p + 0.001 * g ** 2
            p -= learning_rate * (m_p / (1 - 0.9 ** t)) / (np.sqrt(v_p / (1 - 0.999 ** t)) + 1e-8)
    W, b = params
    return lambda X_new: np.argmax(((X_new - mean) / std) @ W + b, axis=1)
data = np.load("./assets/freetzan-features.npz")   # features of FreeTZAN's clips
genre = data["genre"]                                # labels: 0 to 9, one per genre
train, test = data["split"] == "train", data["split"] == "test"

for name in ["waveform", "spectrogram", "log_mel"]:
    X = data[name].astype(np.float64)                # one row of features per clip
    predict = train_classifier(X[train], genre[train])   # learn from the 1000 training clips
    print(f"{name:11s} ({X.shape[1]:4d} features per clip): "
          f"train accuracy {np.mean(predict(X[train]) == genre[train]):4.0%}, "
          f"test accuracy {np.mean(predict(X[test]) == genre[test]):4.0%}")
waveform    (1000 features per clip): train accuracy  97%, test accuracy  15%
spectrogram ( 513 features per clip): train accuracy  67%, test accuracy  36%
log_mel     ( 128 features per clip): train accuracy  56%, test accuracy  41%

Since the \(10\) genres are equally common, guessing at random would be right \(10\%\) of the time. The waveform does barely better than chance, at \(15\%\). The spectrogram does more than twice as well, at \(36\%\). The log-mel spectrogram does better still, at \(41\%\), with a quarter as many features as the spectrogram.

Overfitting. Notice that the waveform classifier is \(97\%\) accurate on its training clips, but only \(15\%\) accurate on the other clips. This is called overfitting, and it shows up as a gap between a model’s performance on its training data and on held-out data, data that it never saw during training. This is why we always evaluate models on held-out data.

Music transcription#

In Chapter 10, we transcribed a simple sine melody by taking the loudest frequency of each frame. Can’t we do the same for “real” music? Unfortunately, transcription is much more complicated than that. Real instruments produce harmonics (Chapter 3), so the loudest frequency may be a harmonic rather than the fundamental. And real music is polyphonic, with many notes from many instruments sounding at once, whose harmonics overlap:

Left, a short score of four quarter notes: C4, G4, C4 and G4 together, and C4 and C5 together. Right, a spectrogram of the score synthesized with ten harmonics per note, from 0 to 2500 Hz, with one column per beat, labeled C4, G4, C4 + G4, and C4 + C5. Each note appears as a stack of evenly spaced horizontal lines. Cyan circles mark the harmonics shared by two notes: three of them for C4 + G4, and four of them for C4 + C5.

Fig. 140 A toy score (left) and the spectrogram of its additive synthesis (right), with \(10\) harmonics per note. Circles mark the harmonics shared by two notes. Every harmonic of C5 is also a harmonic of C4, so adding C5 to C4 adds no new frequencies at all.#

The code below tries two naive strategies on the jazz trio, neither of which should be expected to produce satisfactory results: (1) the loudest frequency of each frame, as in Chapter 10, and (2) treating every pitch whose energy is in the top \(K\%\) of a pitch-band spectrogram as a note:

# hide
from typing import Iterator
import numpy as np
import matplotlib.pyplot as plt
import pyquist as pq
from pyquist.helper import frequency_to_pitch, pitch_to_frequency

Hide setup

# collapse
def iter_frames(audio: pq.Audio, N_H: int, N_F: int) -> Iterator[np.ndarray]:
    for start in range(0, len(audio) - N_F + 1, N_H):
        yield audio.samples[start:start + N_F]


def stft(audio: pq.Audio, N_F: int, N_H: int) -> tuple[np.ndarray, np.ndarray]:
    """The magnitude spectrum of every frame, shape (num_frames, N_F + 1), and the bin frequencies.

    Frame k is centered on time k * N_H / sample_rate. Each windowed frame is zero-padded to 2 * N_F
    samples before the DFT, which samples its spectrum on a finer grid of frequencies.
    """
    padded = pq.Audio(np.pad(audio.samples, ((N_F // 2, N_F // 2), (0, 0))), audio.sample_rate)
    frames = np.array(list(iter_frames(padded, N_H, N_F)))[:, :, 0]
    S = np.abs(np.fft.rfft(frames * np.hanning(N_F), n=2 * N_F, axis=1))
    S[:, 0] = 0.0                                         # ignore 0 Hz, which has no pitch
    return S, np.fft.rfftfreq(2 * N_F, 1 / audio.sample_rate)


def mel_filterbank(freqs: np.ndarray, num_bands: int, f_min: float, f_max: float) -> tuple[np.ndarray, np.ndarray]:
    """Triangular filters evenly spaced on the mel scale, shape (num_bands, len(freqs)), and their centers in Hz."""
    mel_min, mel_max = 2595 * np.log10(1 + np.array([f_min, f_max]) / 700)
    edges = 700 * (10 ** (np.linspace(mel_min, mel_max, num_bands + 2) / 2595) - 1)   # mel back to Hz
    lower, center, upper = edges[:-2, None], edges[1:-1, None], edges[2:, None]
    rising, falling = (freqs - lower) / (center - lower), (upper - freqs) / (upper - center)
    return np.maximum(0.0, np.minimum(rising, falling)), edges[1:-1]


def pitch_spectrogram(S: np.ndarray, freqs: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """A mel spectrogram over the piano's range, with one row per pitch, shape (num_frames, num_pitches).

    Each mel band is labeled with the MIDI pitch nearest its center frequency. We use enough bands that
    nearly every piano key gets one, and where several bands share a pitch we keep the loudest.
    """
    filters, centers = mel_filterbank(freqs, num_bands=512, f_min=27.5, f_max=4186.0)
    mel = S @ filters.T
    band_pitches = np.round(frequency_to_pitch(centers))
    pitches = np.unique(band_pitches).astype(int)
    return np.stack([mel[:, band_pitches == p].max(axis=1) for p in pitches], axis=1), pitches


def notes_from_frames(on: np.ndarray, pitches: np.ndarray, seconds_per_frame: float) -> pq.Score:
    """Turn a (num_frames, num_pitches) on/off grid into a score: each run of "on" frames is one note."""
    events = []
    for j, pitch in enumerate(pitches):
        changes = np.diff(np.concatenate([[0], on[:, j].astype(int), [0]]))   # +1 at a note's start, -1 after it
        for start, end in zip(np.flatnonzero(changes == 1), np.flatnonzero(changes == -1)):
            duration = (end - start) * seconds_per_frame
            events.append((start * seconds_per_frame, {"pitch": int(pitch), "duration": duration}))
    return pq.Score(sorted(events, key=lambda event: event[0]))


def tone(pitch: int, duration: float, sample_rate: int = 44100, **kwargs) -> pq.Audio:
    """A soft tone with three harmonics and 10 ms fades, for listening to a transcription."""
    t = np.arange(int(duration * sample_rate)) / sample_rate
    f0 = pitch_to_frequency(pitch)
    x = sum(np.sin(2 * np.pi * h * f0 * t) / h for h in [1, 2, 3])
    fade = np.clip(np.minimum(t, duration - t) / 0.01, 0.0, 1.0)
    return pq.Audio(fade * x, sample_rate)


def sonify(score: pq.Score, original: pq.Audio) -> pq.Audio:
    """Render a transcription with tone(), at the same length and loudness (RMS) as the original."""
    out = pq.Audio.zeros(len(original), 1, original.sample_rate)
    if len(score) == 0:
        return out
    rendered = score.render(tone)[:len(original)]
    out[:len(rendered)] += rendered
    return out * np.sqrt(np.mean(original.samples ** 2) / np.mean(out.samples ** 2))


def plot_transcriptions(S: np.ndarray, freqs: np.ndarray, seconds_per_frame: float, scores: dict[str, pq.Score]) -> None:
    """One piano roll per score (red bars) over the pitch spectrogram (gray: darker is louder)."""
    P, pitches = pitch_spectrogram(S, freqs)
    t = np.arange(len(P)) * seconds_per_frame
    fig, axes = plt.subplots(len(scores), 1, figsize=(12, 3.8 * len(scores)), sharex=True, squeeze=False)
    for ax, (label, score) in zip(axes[:, 0], scores.items()):
        ax.pcolormesh(t, pitches, 20 * np.log10(P.T / P.max() + 1e-6), cmap="Greys", vmin=-50, vmax=10,
                      shading="nearest")
        ax.barh([e.kwargs["pitch"] for e in score], [e.kwargs["duration"] for e in score],
                left=[e.time for e in score], height=1.0, color="#C41230")
        ax.set_xlim(0, t[-1])
        ax.set_ylim(20.5, 108.5)
        ax.set_yticks(range(24, 109, 12))
        ax.set_ylabel("MIDI pitch", fontsize=12)
        if label:
            ax.text(0.01, 0.95, label, transform=ax.transAxes, fontsize=12, va="top",
                    bbox=dict(facecolor="white", edgecolor="none", alpha=0.8))
    axes[-1, 0].set_xlabel("Time (s)", fontsize=12)
    fig.tight_layout()
    plt.show()
# Strategy 1 (Chapter 10): the loudest frequency in each frame, rounded to the nearest pitch
audio = pq.Audio.from_file("./assets/audio-trio.wav")
pq.play(audio)                                            # what we are transcribing
N_F, N_H = 8192, 1024                                     # long frames: the bass notes are low
S, freqs = stft(audio, N_F, N_H)
loudest = np.round(frequency_to_pitch(freqs[np.argmax(S, axis=1)]))
loudest[S.max(axis=1) < 0.05 * S.max()] = -1             # too quiet: no pitch
pitches = np.arange(21, 109)                              # the 88 piano keys
on = loudest[:, None] == pitches                          # (num_frames, 88): at most one pitch on per frame
score = notes_from_frames(on, pitches, N_H / audio.sample_rate)   # a new note whenever the pitch changes
print(f"{len(score)} notes")
plot_transcriptions(S, freqs, N_H / audio.sample_rate, {"": score})
pq.play(sonify(score, audio))
18 notes
../../_images/61a85df05d5fb70838d93339045bb636414e963ec35a3136983c10fb320acdde.png
# Strategy 2: a pitch is on in a frame if its bin is among the loudest K% of all bins
P, pitches = pitch_spectrogram(S, freqs)                  # mel spectrogram, (num_frames, num_pitches)
scores = {}
for K in [1, 10]:
    on = P > np.percentile(P, 100 - K)                    # one global threshold for every bin
    score = notes_from_frames(on, pitches, N_H / audio.sample_rate)   # each run of "on" frames is a note
    print(f"K = {K}%: {len(score)} notes")
    scores[f"K = {K}%"] = score
plot_transcriptions(S, freqs, N_H / audio.sample_rate, scores)
for score in scores.values():
    pq.play(sonify(score, audio))
K = 1%: 18 notes
K = 10%: 132 notes
../../_images/9f05364e3fc400290cc82e5476121fe3abaa4ed2823eb474525ef82f7a7784d2.png

Neither strategy comes close to a faithful transcription. To transcribe real music, we need to train a model on recordings paired with the exact pitches and timings of their notes [HES+18]. The details are out of scope for this book, but the end of this chapter points to further reading.

Beyond classification and transcription#

We have only scratched the surface of music understanding. MIR studies many more tasks, such as tracking beats, recognizing chords, separating instruments, transcribing lyrics, and recommending music. But most of these tasks share a few common goals and techniques: (1) recognizing or predicting the underlying components of musical recordings, such as their notes, chords, beats, and instruments, (2) training neural networks on labeled data, and increasingly on large amounts of unlabeled data, and (3) evaluating and comparing methods on held-out data. The end of this chapter points to research on many of these tasks.