MultiVerse Keeps Creative Intent Alive

While working on his master's degree in music, Carnegie Mellon University's Alex Wang began considering how music production and technology could create new experiences for consumers. The idea started with video games, because Wang said that, by design, they offer different experiences for individual players.

"I was really inspired by how dynamic video games can be," Wang said. "Game creators don't design a game to be experienced in exactly the same way by every player. They design a space of possibilities, where each player can have a unique experience while still staying within the creator's vision. I started wondering why digital music couldn't work the same way. I always wanted the opportunity to write songs that could evolve and be recontextualized as time went on, even after they were written."

Now, as a musician and game developer turned doctoral student in the Kenneth C. Griffin School of Computer Science's Human Computer Interaction Institute (HCII), he's creating the tools that make the music experience as dynamic as he always imagined.

Wang is part of a research team that developed MultiVerse, a tool that allows music creators to automatically change lyrics to personalize the experience for listeners. Researchers call this technology adaptive lyrics, which means the person who made the song can specify what aspect of a lyric can be modified. For example, if an artist wrote a song with a lyric about taking the train to New York City, they could use MultiVerse to create versions of the song that update the train's destination based on where the listener is going. (Instead of New York City, for example, the destination could be Pittsburgh.)

While current generative AI tools can make entire songs or write a musician's song to their personal preferences, these tools neglect the original artists' intent, style and preference. MultiVerse fills this gap.

"We can now use AI to generate custom songs on any topic, for example about my favorite food, sports team or a camping trip with friends," said David Lindlbauer, an assistant professor in the HCII. "With a simple prompt, AI will hallucinate the lyrics and melody, and produce a song that will likely be nearly indistinguishable from one written by a human creator. But AI takes the artist and their creative process fully out of the loop, which is not necessarily desirable. We are interested in the general area of dynamic music, and see music creation as a spectrum from fully manual to fully AI-generated. We are exploring this spectrum, and how to bring the benefits of both worlds together."

MultiVerse takes a creator's lyrics and allows them to adjust specific guardrails before working with the technology to adapt the lyrics. For example, Wang wrote a song about returning to the Gates Hillman Center, and the opening lyrics are: "Back in Gates, winding up the helix bend/Pausch lights flicker as the daylight ends."

MultiVerse offers three main categories of steering controls: Intent, which are feelings or themes; structure, the rhythm or syllable count; and context, what audience information, such as location or mood, the lyrics adapt to. With Wang's song, the parts that one could change include where the song is based: "Back in [Familiar Place], [Familiar Routine, rhyme with end]/[Familiar Sight] as the daylight ends." The system generates multiple versions of the lyrics and every version presented to the creator is computationally verified to ensure that the artist's constraints are met. If a lyric's rhyme scheme is ABAB and MultiVerse generates an AABB scheme, that option is rejected.

"Music streaming services are increasingly partnering with artists and record labels to explore adaptive media in the form of AI song remixes," said Computer Science Department Assistant Professor Chris Donahue. "But how can we build tools that let artists constrain the types of modifications these features can make to their music?"

The research team tested the system with 10 songwriters. Their feedback highlighted that MultiVerse made them feel like they had more control compared to more hands-off generative AI tools.

This team will present their research at the Association for Computing Machinery's Symposium on User Interface Software and Technology (UIST) conference next month in Detroit.