16.3 Music AI frontiers#

Music AI is a fast-moving field. In this section, we briefly survey some of its active research directions. For each topic, we give just a sentece or two of context and provide pointers to key papers and datasets. Treat this as a starting point for further reading.

Music information retrieval#

Music information retrieval (MIR) aims to extract meaningful information from music, most often from audio. A recurring theme in modern MIR is pre-training. A single large model learns representations from lots of unlabeled music, and these representations are then reused for many downstream tasks [CDL21, LYZ+24].

  • Tagging. Tagging systems predict descriptive labels such as genre, mood, or instrumentation for a recording. Convolutional neural networks applied to spectrograms or raw waveforms are standard approaches [CFS16, PNP+18]. Well-known datasets include GTZAN [TC02], whose faults are documented in [Stu13], as well as MagnaTagATune [LWM+09], the Million Song Dataset [BMEWL11], FMA [DBVB17], and MTG-Jamendo [BWT+19].

  • Transcription. Transcription converts audio into a symbolic representation such as MIDI. Onsets and Frames [HES+18] substantially improved piano transcription, and the MAESTRO dataset of aligned piano audio and MIDI [HSR+19] enabled further progress. MT3 transcribes many instruments at once [GSM+22], and MusicNet [THK17] is a dataset of labeled classical recordings. Related tasks include pitch tracking [KSLB18], lead sheet transcription [DTL22], lyrics transcription [RKX+23, ZYP+23], and beat tracking [BKW16, Ell07, FSW24].

  • Source separation. Source separation splits a mix into its individual parts, often called stems, such as vocals, drums, and bass. The Demucs family of models is widely used [Def21, DUBB19, RMD23], and MUSDB18 [RLS+17] is the standard benchmark dataset.

  • Retrieval. Retrieval systems find recordings that match a query. Audio fingerprinting, popularized by Shazam [Wan03], identifies a recording from a few seconds of noisy audio. Joint embeddings of music and text allow search by natural-language description [HJL+22, WCZ+23], and CLaMP 3 [WGY+25] extends this to sheet music, MIDI, and audio.

  • Recommendation. Recommender systems suggest music to listeners, often by collaborative filtering of listening data across many users [HKV08]. Content-based methods analyze the music itself, which helps recommend new or unpopular songs that have little listening history [MBL12, vdODS13].

Generation#

Modern music generation systems mostly combine the ideas from earlier in this chapter. They compress audio into a compact representation, then model that representation with a large generative model.

  • Codec language models. A neural audio codec compresses audio into a sequence of discrete tokens [DCSA23, KSL+23, ZLO+22]. A language model is then trained on these tokens, much like the language models we trained on symbolic music. Examples include AudioLM [BMV+23], MusicLM [ADB+23], MusicGen [CKG+23], and the open song generation model YuE [YLG+26].

  • Latent diffusion models. Latent diffusion models, originally developed for images [RBL+22], compress audio into a continuous latent representation with an autoencoder, then generate in that latent space with diffusion [HJA20]. Examples include AudioLDM [LCY+23] and the Stable Audio family [ECT+24, EPC+24, EPC+25]. Diffusion models can also follow precise time-varying controls such as melody, dynamics, and rhythm [WDWB24]. Collections of Creative Commons-licensed music, like FMA [DBVB17] and MTG-Jamendo [BWT+19], offer an alternative to commercial catalogs for training. For example, Stable Audio Open was trained only on Creative Commons recordings from Freesound and FMA [EPC+25].

  • Cross-modal understanding. Text control requires models that relate music to language. Contrastive models learn a shared embedding space for audio and text [EDAIW23, HJL+22, WCZ+23], following CLIP for images and text [RKH+21]. For instance, MusicLM is conditioned on MuLan embeddings, and AudioLDM on LAION-CLAP embeddings [WCZ+23]. MusicLM also released MusicCaps, a dataset of music clips with captions written by expert musicians [ADB+23]. Large language models are also being extended to describe and answer questions about music [DML+24, GDSB24, GGK+26, YLW+24].

  • Evaluation and post-training. Measuring the quality of generated music remains difficult. The Fréchet Audio Distance [KZRS19] is widely used, but it correlates only weakly with human preferences [HNS+25]. Music Arena collects live human preferences to compare text-to-music systems [KCA+25], and human preferences can also be used to fine-tune generators [CGV+24].

Music AI interaction#

  • Co-creation. Much research asks how AI can support, rather than replace, human musicians. Studies examine how novices steer generative models [LCH+20], how teams wrote songs with AI in the AI Song Contest [HKNR+20], and what practising musicians need from co-creative tools [KLRLP25]. Systems like Amuse [KLD25] offer songwriting suggestions from multimodal inspirations. Others generate accompaniment for a user’s singing or tracks [DCR+23, NPA+24], or fill in music around notes a user has written [THDL24].

  • Real-time. Live performance requires models that respond with low latency. Piano Genie [DSD19] lets anyone improvise on the 88-key piano with just eight buttons. ReaLchords [WCK+24] and ReaLJam [SWS+25] improvise chord accompaniment to a live melody. Live music models such as Magenta RealTime [LyriaTeamCM+25] generate a continuous stream of music that users steer with text or audio prompts. Fast models also enable real-time synthesis [CE21] and low-latency text-to-audio generation [NEZ+25]. For a survey of 184 live music agent systems, see [KBW+26].

Broader considerations#

  • Ethical and cultural considerations. A systematic review of 884 generative audio papers found that fewer than 10% discuss potential negative impacts, such as fraud, deepfakes, and copyright infringement [Bar23]. Earlier work examined the ethics of MIR technology [HSC18] and of AI music creation [Mor21]. Other work brings music AI to traditions that it rarely serves, such as 15th-century Korean court music [HGK+24]. Surveys of how artists actually use AI offer another perspective [PZP+25].

  • Data and economic considerations. Generative models are trained on large catalogs of recordings, which raises open questions of copyright law [SIBT+19]. Training data attribution estimates which training examples influenced a given output [BFGP24, CKC+25]. These estimates may be used to help quantify royalty payments owed to data creators [DJZ+23, ZJD+26].

  • Watermarking and detection. Watermarking embeds an imperceptible signal in generated audio so that it can later be identified [CWL+23, SRFE+24]. Detection instead classifies whether audio is AI-generated without relying on a watermark. Detectors can be highly accurate on known generators, but may fail on audio manipulations or unseen models [AMBH24, AMBH25]. SONICS provides a dataset of fully AI-generated songs for detection research [RHS+25].