16.4 Summary#
A language model predicts each element of a sequence from the elements before it. \(n\)-gram models suffer from the curse of dimensionality. A neural language model uses a neural network \(f_\theta\) to condition on far more context, up to the entire history. Its parameters are far fewer than the possible contexts, so it must share information between similar contexts.
Neural networks are trained by minimizing a loss function, such as the negative log-likelihood, with gradient descent. The negative log-likelihood is essentially the entropy of the training data, as measured by the model.
Modern music language models use more expressive alphabets, more context, and far more data. But like \(n\)-gram models, they predict \(P(\text{next} \mid \text{context})\) and generate one element at a time.
WaveNet applies language modeling directly to audio samples, but waveforms are very long. A codec language model first learns to compress audio into discrete tokens, then models the tokens with a language model. Putting control information in the context makes generation controllable.
Music information retrieval recognizes high-level information in music, often with supervised learning. Input features like the log-mel spectrogram matter. Models are evaluated on held-out data, which reveals overfitting.
Transcribing real music is hard because of harmonics and polyphony, and requires learned models.