10.0 Extracting frames

We begin with the most basic operation: chopping a signal into frames.

Definition 28 (Frame extraction)

To extract frames of frame length \(N_F\) from a signal \(x\), the \(n\)-th sample of the \(k\)-th frame \(x_k\) is

\[\begin{split}x_k[n] = \begin{cases} x[k \cdot N_H + n] & \text{for } n \in \{0, 1, \ldots, N_F - 1\}, \\ 0 & \text{otherwise,}\end{cases}\end{split}\]

where \(N_H\) is the hop length, the spacing in samples between the start of one frame and the start of the next.

That is all there is to it: we extract segments (frames) of \(N_F\) samples, moving through the signal by increments of \(N_H\) samples. The simplest case takes \(N_H = N_F\), so the frames tile the signal end to end:

Fig. 61 Extracting frames with \(N_H = N_F\): the frames tile the signal one after another with no overlap.

If the signal is sampled at \(f_s\), this produces frames at a frame rate of

\[f_k \left[\frac{\text{frames}}{\text{second}}\right] = f_s \left[\frac{\text{samples}}{\text{second}}\right] \cdot \frac{1}{N_H} \left[\frac{\text{frames}}{\text{sample}}\right].\]

Frames give us a new unit of time, complementing the seconds and samples we already know. The offset of frame \(k\) is \(k \cdot N_H\) samples, so it is associated with the timestamp \(t_k \coloneq \frac{k \cdot N_H}{f_s}\) seconds.

For example, at \(f_s = 44{,}100\) Hz with \(N_H = 1024\), the frame with index \(10\) represents the moment \(t_{10} = \frac{10 \cdot 1024}{44100} \approx 232\) ms. Conversely, a recording of duration \(T\) spans \(\frac{T \cdot f_s}{N_H}\) frames, so a ten-second file at these settings is about \(\frac{10 \cdot 44100}{1024} \approx 430.7\) frames. (We will deal with that fractional frame shortly.)

The relationship between \(N_F\) and \(N_H\) controls how much consecutive frames overlap. When \(N_H < N_F\), each frame shares some samples with its neighbors. We quantify this as the overlap, expressed as a fraction of the frame length:

\[\text{overlap} = \frac{N_F - N_H}{N_F}.\]

At \(N_H = N_F\) there is no overlap (0%); at \(N_H = N_F/2\) the frames overlap by half (50%). The animation below shows a single frame advancing across a signal at three overlap settings:

Fig. 62 The same frame length \(N_F\) at three overlaps. Lowering the hop \(N_H\) increases the overlap, packing the frames more densely (thin gray lines mark each frame offset \(t_k\)).

Note

Nothing stops us from choosing \(N_H > N_F\), which makes the overlap negative. This leaves gaps between consecutive frames, so some samples are not included in any frame at all. This is rarely what we want, since it discards information, so in practice we keep \(N_H \le N_F\).