10.0 Extracting frames

We begin with the most basic operation: chopping a signal into frames.

Definition 27 (Frame extraction)

To extract frames of frame length \(N_F\) from a signal \(x\), the \(n\)-th sample of the \(k\)-th frame \(x_k\) is

\[\begin{split}x_k[n] = \begin{cases} x[k \cdot N_H + n] & \text{for } n \in \{0, 1, \ldots, N_F - 1\}, \\ 0 & \text{otherwise,}\end{cases}\end{split}\]

where \(N_H\) is the hop length, the spacing in samples between the start of one frame and the start of the next.

That is all there is to it: we extract segments of \(N_F\) samples along the signal in increments of \(N_H\) samples, and each stop is a frame. The simplest case takes \(N_H = N_F\), so the frames tile the signal end to end:

Fig. 60 Extracting frames with \(N_H = N_F\): the frames tile the signal one after another with no overlap.

If the signal is sampled at \(f_s\), this produces frames at a frame rate of

\[f_k \left[\frac{\text{frames}}{\text{second}}\right] = f_s \left[\frac{\text{samples}}{\text{second}}\right] \cdot \frac{1}{N_H} \left[\frac{\text{frames}}{\text{sample}}\right].\]

Frames give us a new unit of time, complementing the seconds and samples we already know. The offset of frame \(k\) is \(k \cdot N_H\) samples, so its natural timestamp \(t_k\) is \(\frac{k \cdot N_H}{f_s}\) seconds. For example, at \(f_s = 44{,}100\) Hz with \(N_H = 1024\), frame \(10\) represents the moment \(t_{10} = \frac{10 \cdot 1024}{44100} \approx 232\) ms. Conversely, a recording of duration \(T\) spans \(\frac{T \cdot f_s}{N_H}\) frames, so a ten-second file at these settings is about \(\frac{10 \cdot 44100}{1024} \approx 430.7\) frames. (We will deal with that fractional frame shortly.)

The relationship between \(N_F\) and \(N_H\) controls how much consecutive frames overlap. When \(N_H < N_F\), each frame shares some samples with its neighbors. We quantify this as the overlap, expressed as a fraction of the frame length:

\[\text{overlap} = \frac{N_F - N_H}{N_F}.\]

At \(N_H = N_F\) there is no overlap (0%); at \(N_H = N_F/2\) the frames overlap by half (50%). The animation below shows a single frame advancing across a signal at three overlap settings:

Fig. 61 The same frame length \(N_F\) at three overlaps. Lowering the hop \(N_H\) increases the overlap, packing the frames more densely (thin gray lines mark each frame offset \(t_k\)).