10.2 Windowing

In the previous section, we saw that overlapping frames can affect amplitude. The predictability of this suggests a potential mechanism to counteract it. Let’s work our way towards a solution.

Instead of extracting raw frames, we can multiply each frame by a window function \(w \in \mathbb{R}^{N_F}\) as we extract it:

Definition 30 (Windowed frame extraction)

Given a window \(w \in \mathbb{R}^{N_F}\), the windowed frame \(x'_k\) is the extracted frame multiplied sample-by-sample by the window:

\[x'_k[n] = w[n] \cdot x[k \cdot N_H + n] = w[n] \cdot x_k[n].\]

Note

When \(w\) is the rectangular window (all ones) from Chapter 8, windowed frame extraction reduces exactly to the plain frame extraction of Definition 28: multiplying every sample by one leaves the frame unchanged. Plain framing is just the special case where \(w[n] = 1\).

Overlap-add then reassembles the windowed frames, \(\hat{x}[n] = \sum_k x'_k[n - k \cdot N_H]\). When does this still give perfect reconstruction? The condition is that the overlapping windows add up to the same non-zero constant at every sample:

Definition 31 (Constant overlap-add)

A window \(w\) and hop length \(N_H\) satisfy the constant overlap-add (COLA) property if the shifted windows sum to a non-zero constant \(c\) at every sample \(n\):

\[\sum_{k} w[n - k \cdot N_H] = c.\]

When they do, overlap-add reconstructs the original signal up to that constant factor, so dividing it out recovers \(x\) exactly:

\[x[n] = \frac{1}{c}\, \hat{x}[n] = \frac{1}{c} \sum_{k} x'_k[n - k \cdot N_H].\]

Note

Strictly, COLA holds only if we imagine the windows continuing infinitely in both directions. At the very edges of a finite signal (near \(0\) and \(T\) seconds) fewer windows overlap, so their sum falls short of \(c\). But as long as the sum is constant in the steady state away from the edges, reconstruction is perfect there, and the affected fraction of the signal shrinks as the signal grows longer.

Many combinations of window, frame length, and hop length satisfy COLA. The simplest is the rectangular window (all ones) at 0% overlap, which is exactly the perfect-reconstruction case we already saw (\(c = 1\)). A more useful one is the Hann window,

\[w[n] = \frac{1}{2}\left(1 - \cos\!\left(\frac{2\pi n}{N_F}\right)\right),\]

a raised cosine bump that tapers smoothly to zero at both ends, used at 50% overlap (where the overlapping windows again sum to a constant):

Fig. 64 Hann windows at 50% overlap satisfy constant overlap-add: although each window rises and falls, the overlapping windows always sum to the same constant (bold line), so overlap-add reconstructs the signal exactly.

Why would we ever prefer a tapered window to a plain rectangle, if both reconstruct perfectly? The reason is spectral leakage, which we met in Chapter 8: because a smooth window has a cleaner spectrum than a rectangle, it smears far less energy across frequencies. This will matter later in this chapter, when we use frame-based processing to both manipulate individual frames (in granular synthesis) and decompose a sound into both time and frequency at once (the short-time Fourier transform).

Boundary conditions

In addition to the COLA edge cases, an eagle-eyed reader may have noticed we glossed over some other edge cases.

Firstly, what do we do with the fractional frame at the end of a signal, where a frame starts inside the signal (\(k \cdot N_H < N\)) but runs off the end (\(k \cdot N_H + N_F \geq N\))? Two conventions are common: we can zero-pad, filling the missing tail of the frame with zeros, or we can simply truncate, discarding any frame that does not fit completely. Both are widely used.

Secondly, where should we anchor a frame relative to its timestamp? A frame canonically describes time at \(t_k = k \cdot N_H\) samples. We have defined this sample as the first of the corresponding frame, i.e., \({x_k[0] = k \cdot N_H}\). But it may be more intuitive in some cases to center the frame around this timestep, i.e., \({x_k[\frac{N_F}{2}] = k \cdot N_H}\).

These two choices, alignment and padding, are independent, giving four combinations in all:

Fig. 65 The four boundary conventions: {left-aligned, centered} \(\times\) {truncate, zero-pad}, shown with no overlap. Hatched regions are zero-padding beyond the signal; the dashed line marks the signal’s end. You will encounter these in practice as arguments like pad=True or center=False.

Unless otherwise noted, assume we are referring to the left-aligned, truncate standard henceforth. In any case, these are just boundary conditions, affecting a smaller and smaller fraction of frames as the signal grows longer, so we will mostly ignore them from here on.