16.4 Summary

16.4 Summary#

  • A language model predicts each element of a sequence from the elements before it. \(n\)-gram models suffer from the curse of dimensionality. A neural language model uses a neural network \(f_\theta\) to condition on far more context, up to the entire history. Its parameters are far fewer than the possible contexts, so it must share information between similar contexts.

  • Neural networks are trained by minimizing a loss function, such as the negative log-likelihood, with gradient descent. The negative log-likelihood is essentially the entropy of the training data, as measured by the model.

  • Modern music language models use more expressive alphabets, more context, and far more data. But like \(n\)-gram models, they predict \(P(\text{next} \mid \text{context})\) and generate one element at a time.

  • WaveNet applies language modeling directly to audio samples, but waveforms are very long. A codec language model first learns to compress audio into discrete tokens, then models the tokens with a language model. Putting control information in the context makes generation controllable.

  • Music information retrieval recognizes high-level information in music, often with supervised learning. Input features like the log-mel spectrogram matter. Models are evaluated on held-out data, which reveals overfitting.

  • Transcribing real music is hard because of harmonics and polyphony, and requires learned models.