15-322 Introduction to Computer Music: Term Projects

Every student is required to complete two extended projects in the second half of the semester:

A Term Implementation Project
A Term Composition

Different phases of these are P6, P8, P9, P10, and P11. These projects combines the development of some software with the creation of music, ideally using that software. Students may form teams, but it is essential that team members each take responsibility and credit for some aspect of the project. Team members may chose to create separate compositions and only collaborate on software. Alternatively, a "composition team" may form from different technical projects to combine multiple techniques in a single composition.

An Example Project

An example is the best way I can describe the features of a good project, so here it is: The latest Nyquist IDE (based on a project from a previous class) is great for editing text and displaying sounds (at least short ones), but there is no support for entering functions of time. This project adds a new view to the IDE allowing the user to edit piece-wise linear functions of time. Each function has a name, and multiple functions can be stored in one file. The functions are implemented by defining a Lisp function containing a call to PWL. If the user creates a time function named foo in the IDE, then the user can write foo() in Nyquist code to obtain the function. When the user opens one of these files in the IDE, the IDE parses the code and can then display functions graphically. The user can add, move, and delete breakpoints, change function names, copy functions to a new name, etc. It would be great to be able to display multiple functions in different colors so they can be edited in concert. Also, it should be possible to snap to user-controlled grid lines, zoom in or out, change the vertical axis, etc.

Having convenient direct-manipulation access to functions of time opens up a new way to control synthesis. Complex sounds with several parameters can be made to evolve according to time-varying controls that are edited graphically. These parameters in the editor become a sort of musical score notation. The composition associated with this project uses this approach to composition, resulting in music with slowly-varying, but very carefully controlled and refined textures.

Why is this good? It has the following features that I am looking for:

It’s Computer Music: there’s computation, and there’s music.
It requires some thought about music representation and music processing.
It extends Nyquist (or Audacity). Notice that the project is not even a complete stand-alone system, only the extension of one, so the student does less work and leverages what’s already there.
The project isn’t too easy, but it’s not impossibly hard.
Students will be able to use this in the future. (In fact, most of this has been done -- try it!)
The music is integral to the project. It’s almost obvious what the music has to be and how the implementation is essential to the music creation. The music and the technology guide each other.
The music is not conventional, popular music that might be better played on a piano. Instead, the music could only be created by computer.

Your project will probably not have to have all these features, but I hope this give you a good idea of what I am looking for. Review the list above with respect to your project.

The Term Projects in Stages

The project has several parts:

Project 6: Proposal (due Tuesday, Feb 22)

A proposal is required. Your proposal should state clearly:

the general nature of what you will implement,
what code, journal articles, etc. you will rely on, port, implement, modify, etc.,
a significant milestone that you will report/deliver as the interim report
a specific list of deliverables for your final project submission
how will you measure your success? E.g. what functionality will you demonstrate?

All projects must be approved by the instructor, so this really is a proposal. I will usually suggest some changes, and if the project really seems out of line, I’ll let you know early and without penalty. (A clear proposal for a bad project gets a good grade and a request for another submission, a vague proposal for a good project gets a bad grade and an OK to proceed).

Copy your proposal (as a PDF) to your afs project folder p06 using the filename USERID-p06.pdf, as usual. The deadline is 11:59pm, Tues, Feb 22nd.

Project 8: Interim Report (due Thursday, March 24)

An interim report is required. Ideally, you will have completed the milestone described in your project proposal. If you fail, you should describe clearly what you have done, what problems you encountered, whether this will affect your ability to finish the project as planned, and any modifications (subject to instructor approval) you feel you should make to the project. You should submit interim code, sound results, and other data to substantiate your report. Put everything in p08 in your afs hand-in folder.

In addition to your interim report, you should answer the following questions:

Suppose you compute a 1024-point FFT on a sound sampled at 44.1kHz using Nyquist. What is the size of the resulting array containing the spectrum data?
Most of the array consists of real and imaginary coefficients corresponding to the amplitudes of different frequencies. What frequency corresponds to array locations 1 and 2?
What frequency corresponds to array locations 5 and 6?
Suppose you want to write a pattern that consists of: random choices from a list of values, added to a time-varying value determined by a piece-wise linear envelope. You write a function to evaluate the envelope at the current time:
```
set contour = pwl(10, 4, 20, -3, 25)
function get-contour-value()
  return sref(contour, 0)
```
The pattern will be evaluated inside a seqrep expression using the expression next(pitch-pat) as part of the computation of each sound in a long sequence, and pitch-pat looks like:
```
set pitch-pat = make-sum(make-random(list(c4, f4, g4, bf4)),
                                                  )
```
What expression goes into the blank (you need a pattern that will fetch the current value of contour)?

Project 9: The Composition (due Thursday, April 21)

Ideally, your composition project will demonstrate what you have learned in the class. Aim for a length of about 5 minutes. Longer works are acceptable, but length will be considered when we program the final concert. You should have some interesting ideas from the in-class listening sessions, and your project implementation will also guide you. It is probably better to make a piece in the tradition of contemporary art music than to create a rap tune, but the instructor will do his best to stay open-minded. Regardless of the musical genre, the music should reflect careful thought and imagination. On the technical side, the music should show some mastery of the techniques taught in class. E.g. a piece that is acoustically dry, monophonic, exhibits clipping, shows no variation in timbre or articulation, and which uses a completely flat tempo will not be acceptable. A piece should include appropriate reverberation, stereo placement, effective changes in dynamics (without clipping), timbres that evolve in interesting ways. and sounds shaped into flowing phrases. These are not easy requirements -- it takes time and effort to produce musical results from computers. Grading will be based on how well the piece communicates a musical idea, i.e. the intentions of the composer, and the extent to which the piece was carefully constructed using techniques learned in class and developed in the project. To a large extent, the grade will be based on effort. It is usually clear whether a piece was thrown together quickly or whether many details were carefully adjusted. If you feel you have worked very hard and still are not satisfied with the result or think that your effort is not apparent in the composition, you should submit early drafts and explain what you have done to refine the work.

Every year, more than one student hands in a piece with unacceptable technical problems, often pops where oscillators turn on or splices occur without an smoothing envelope. You must listen carefully and critically with headphones (not laptop speakers) while developing your piece and pay attention to clicks and clipping.

What to turn in:

Copy the final piece to your p09 folder in your afs area. The preferred format is 16-bit, stereo, PCM at 44100. If you have more or less than 2 channels, you might want to make a stereo mix for inclusion on an mp3 collection we plan to create for the class. You can then submit both your concert version and the stereo mix.
Program notes: a short paragraph about your piece, suitable for inclusion in a concert program.
Short bio: a short paragraph about you, suitable for inclusion in a concert program.
Code: source code you used to create your piece. We realize you may have used other non-programming techniques but would like to see whatever code you wrote.
Technical notes: anything else you want to tell us about your piece and how it was created (not for inclusion in program notes).

Example program notes and bio:

The Journey of the Fly
by Charles-Christopher Onyeama

My piece was designed to outline the journey of one fly, from getting lost from his home, to finding a new resting place where he could be happy. It involves heavy sampling of various noises from everyday life, put together in what I think is a very nice tale. I used samples that I found from all over the Internet, combined with techniques such as panning, envelopes, granular synthesis, sound modulation, etc. This involved heavy use of Audacity.
I am an ECE major, minoring in CS. I consider myself an analytical/inquisitive person about people and the world around me, and as such, I like listening to modern electro-acoustic music to try to examine what the composer was trying to convey. My interests include manga, video games, etc., and I am from Los Angeles, CA.

Project 10: The Written Project Submission (due Friday, Apr 29)

Your goal is to make a complete package of software, documentation, example code, and sound examples. For example, if your project is to create a phase vocoder, you should submit the software implementation, document how to apply the vocoder and use all the parameters, provide code examples that apply the vocoder to a test sound, and one or more sounds that illustrate the effect of the vocoder. Another student should be able to use the vocoder given your final submission. Put everything in your p10 folder.

The Concert (Sunday, May 1, Concert I at 4:30PM-6:30PM, Concert II at 7:30PM-9:30PM, location GHC 4401, Rashid Auditorium)

Your friends are invited. Attendance is required. This is a large class, so it will not be possible to perform every piece. The instructor and teaching assistants will select pieces if necessary to keep the program length reasonable. There will be a sound check at 4PM for anyone concerned about setting playback levels in advance. We'll try to get a generally good level for everyone, and levels can be controlled during each performance.

Project 11: The Class Presentation

Talks: in the Exam period
Slides: due April 29 to allow staff to prepare

You will present your project to the class. Plan for 4 minutes maximum. This is a very short presentation. You should present your work in three parts. Use slides (preferably PowerPoint) to supplement your talk.

Motivate your work. What purpose (outside of your education) is served by your project? What existing problem does your work solve? (1 slide)
State your approach and goals. How does your project solve the problem just stated? Be sure to state clearly and with some detail exactly what you did. (1 or 2 slides)
What is the state of your project? Tell what is complete and working as well as what would be the next step if you (or someone else) were to continue working. (1 slide)

Team projects will get 1 extra minute per person (3 + n minutes for n people), and each person in the team should talk about their individual contributions or the aspect of the project they know the best.

You must submit PowerPoint or PDF slides in advance to your hand-in directory: p11 in your afs hand-in folder..

These will be organized into one monster file so that we have zero setup time between class presentations.

If you have sound file examples (recommended):

collect them all in a directory named your-andrew-id
at the appropriate points on your slides, put something like "play: filename"
submit a zip file of the whole directory to p11 in your afs hand-in folder.
we will unzip the file and it will be available for your class presentation
do not assume that links from PowerPoint to soundfiles will be maintained when your slides are merged into one big file.

Assuming a 4-minute talk, you can rehearse it 5 times in 20 minutes. Do it! (If this seems unnecessary and you are not a musician, do it just to learn what musicians know that you don’t -- seriously.)

A List of Project Ideas

Feel free to ask the instructor for more details on any of these.

Nyquist-related
- Extending Nyquist by writing functions in Nyquist:
  - Port instruments from articles or other systems
  - Develop library of effects with examples and documentation
  - Spectral features as control parameters for synthesis. What does this mean? Many interesting forms of synthesis are based on the analysis of one sound to obtain data to control another. In this case, spectral features like brightness (the average frequency weighted by amplitude), flux (derivative of spectral amplitude), and frequencies of peaks (also known as peak-tracking) can be used to obtain control information. A project would consist of some analysis software to extract spectral features and some compelling examples of sound synthesis using those features.
  - Implementation or ports of spectral manipulation software (e.g. from Eric Lyon and Chris Penrose)
  - Rhythm guitar synthesizer that takes a strumming pattern, a chord progression, and a guitar string synthesizer function and computes a sound. This should make it possible to create a range of guitar performances without having to notate the time and pitch of each note.
  - There "sample packs" at freesound.org. Find an interesting sample pack and build an interface via Nyquist function calls, e.g. bell-init(), and bell-note(...).
  - "Autotune": Automatic "pitch correction" software made its way from studio productions such as Cher’s "Believe" into popular culture when The Gregory Brothers autotuned news broadcasts and polital speeches to music tracks. Project suggestion: Import a speech track into Audacity and label word boundaries. Export the labels. Write Nyquist code to import Audacity labels and split the speech track into separate units. Read a Nyquist score structure indicating a sequence of target pitches and durations. Use a vocoder to synthesize speech at the indicated pitches and durations. This could be a 2-person project with one person implementing the vocoder and another processing the melody and speech data to determine input data for the vocoder and to reassemble vocoder output.
  - Linux has some nice open source software synthesizers. Reimplement an entire synthesizer or just one preset in Nyquist and try to get the same sound.
  - Fundamental frequency estimation: The YIN algorithm is implemented in Nyquist for fundamental frequency (pitch) estimation, but for steady tones, you can probably obtain more accuracy using autocorrelation or enhanced autocorrelation over multiple periods. The idea is to shift a signal by a few periods and compute the correlation between the shifted and orignal signals (this is autocorrelation). The autocorrelation peaks where the signal is shifted an even number of periods. By spanning multiple periods and perhaps by doing some interpolation, one can hope to get sub-sample period estimation which is necessary for tuning and other applications. Hint: The autocorrelation can be computed quickly using the FFT.
  - Students sometimes ask about obtaining parameters from recorded sounds for use in synthesis. Write a Nyquist function that takes a short sound as input, performs a fundamental frequency (pitch) analysis to determine the length of periods. Resample the signal so that the period becomes a power of two. Extract an exact period from the audio and perform an FFT. The FFT bins represent the harmonics of the signal. Convert these complex values to magnitudes. Synthesize a waveform by summing sinusoids and listen to and compare a synthesized tone to the original sample. Group Project Idea: someone else in the group can take series of extracted waveforms and use spectral interpolation synthesis (see siosc in the Nyquist manual) to create tones with a time-varying spectrum. This project could also be combined with the previous idea for fundamental frequency estimation.
  - David Wessel uses a very interesting synthesis technique that takes a very large FFT, estimates the peak frequencies and amplitudes, and then builds an array of amplitudes indexed by 0.01 semitone units (called "cents"). Then a few hundred sines are generated where their frequency is picked randomly according to the probability density function represented by the array. Sines play for a short time and fade out only to pick a new random frequency from the table. The result is a shimmering texture that approximates the input sound spectrum, which can be something very rich like an orchestra. I think this could be done without much difficulty in Nyquist.
- Extending Nyquist by writing functions in C
  - Phase Vocoder support
  - Physical models from Synthesis Tool Kit (STK) (partially implemented)
  - MQ analysis/synthesis (or the more advanced Loris system or Juan Pampin’s system)
  - Max Mathews uses an interesting synthesis method based on a vector (phasor) that rotates and decays incrementally on every step (sample). This is essentially a decaying partial. He leaves many of these running and simulates hitting, strumming, or bowing by adding offsets. This would be an interesting new unit generator for Nyquist.
  - Support VST plugins in Nyquist.
  - Allow Nyquist to import csound unit generators
  - Support SDIF I/O (see SDIF library from UCB)
  - Add support for Allegro score representation using the allegro library (written in C++).
  - I have some code from a previous project to read Gravis Ultrasound Sound Font files. The goal is to be able to import sounds, envelope data, etc. from these files to construct a Nyquist function that can play instruments described by these files. This would instantly expand Nyquist’s library of conventional instrument sounds.
- Extending the Nyquist IDE by writing functions in Java
  - Add GUI controls to Nyquist. Nyquist already has an array of floats that can be accessed at run time to get scalar (float) values or time-varying signals. These "sliders" can be set by sending special strings from the IDE or by sending OSC messaages over the network. What is lacking is some Java IDE code to create sliders, save configurations in the workspace.lsp file, etc. Probably slider creation should be done by graphical interaction, but the configuration should be saved in the workspace (see the envelope editor and EQ editors for examples).
  - The Nyquist IDE has plenty of rough edges. I don’t like to encourage "cosmetic coding" as a class project, but if you are an expert Java GUI programmer, there are a number of problems that could use your skills: The window layout is different on every system, but there must be a way to do a good job of window sizing and placement. Fonts are terrible on Linux. The sliders used in the Browser and other places are not compact -- it would be nice to have sliders that look like those in professional audio applications. Parenthesis balancing has come a long way from the original implementation, but it could still use some improvement. In particular, when the cursor is positioned after a close parenthesis, the corresponding open parenthesis should be highlighted. Graphical object event handling in the code is inconsistent. A "design pattern" for creating and handling graphical objects to guide future implementation and rewrites would be nice.
  - The EQ editor is preliminary -- it should allow the user to specify the range of frequencies, the number of frequency channels, and support multiple audio channels, either locked together or independent.
  - The envelope editor is awkward, lacks labels, and editing features, but it’s a good start. A good project would propose an improved design and implement a replacement.
  - The IDE’s "piano roll" score editor is a rough start, and the user interface needs a lot of work, including scroll bars, labels, zoom, selection, access to attributes other than pitch and time, the ability to quantize or snap to grids, etc. A project in this area needs to make a specific interface design and implementation plan.
  - Spatial positioning of sound is difficult to do in Lisp. A graphical editor might help to position sound sources in stereo or 5.1 or whatever. Especiallly interesting would be the ability to specify moving sound sources and to edit their paths.
  - There have been several attempts to create a graphical score entry system. The idea is that you represent different Nyquist behaviors with different shapes and/or colors on a pitch vs. time graph. This is similar to a piano-roll editor except the shapes or colors indicate different behaviors (Nyquist functions). There may be other parameters encoded in the height, color, texture, or shape of each graphical object in the score. Each object can be edited using a property sheet, and all the properties are passed to Nyquist as parameters to the behavior, and behaviors are organized in terms of time and duration according to their (editable) graphical layout. The main idea is to provide some graphical score editing but to avoid strong ties to traditional music notation or assumptions that music is best represented in terms of pitch and time (only). Maybe this should be a superclass of the piano roll editor or the two should share an abstract superclass.
  - Nyquist has some nice drum samples, but the interface for creating drum patterns is very primitive. On the other hand, it’s not easy to create a good graphical interface to edit drum patterns. It is especially important to support unconventional patterns, odd meters, etc., but not obvious how to do this in a user-friendly way.
  - Speaking of drum patterns, Dan Trueman demonstrated a fascinating interface for rhythmic patterns called the Cyclotron (ICMC 2008). Although his system was real-time and interactive, his original work was a non-real-time system. Given Nyquist’s quick computation, I think it would interesting to create a Cyclotron-like interface in Java that outputs Nyquist scores.
  - It would be nice to have a way to quickly examine audio files in the plot window. This would require Java to read an audio file and generate a plot, possibly allowing the user to zoom and scroll. Maybe there is a nice plot object already available. Other ideas: reading big audio files can be very slow. Probably a thread could construct an array of points to plot and hand it off to the GUI so that the GUI would not lock up while reading audio. Maybe audio could be plotted automatically whenever Nyquist closes an audio file (an option set in Preferences.)
Audacity-related -- In general, students have had a difficult time getting up to speed and contributing to Audacity, but at the same time, there is much room for improvement. Note that Audacity includes Nyquist as a scripting language.
- The Audacity/Nyquist interface is not quite right -- it does not allow access to all of Audacity, making it hard or impossible to write plug-ins that utilize more than one track. Also, the selection can only be read once in Nyquist, making it impossible to do things like normalization without storing the entire audio selection in Nyquist memory.
- Scales to display time, beats, samples, etc.
- Add breakpoint envelope editor to Audacity
- Ability to edit a tempo track and display beat locations
- Add MIDI editing functions.
- Audacity editing interface for editing live recordings (consult with Riccardo Schultz)
- cross-fade editor dialog
- play list interface
- play list engine to compute and play segments and crossfades
Other
- Develop open-source beat-tracking software.
- Module to send digital audio over networks
- Add support for SDIF I/O to Audio File Library
- Build system to segment a performance into individual notes
- Play synchronized midi and audio using PortMidi and PortAudio
- Develop/evaluate software to track beats, making progress toward a system for beat-synchronous digital audio effects for pop/rock bands. In general, robust tempo and beat tracking does not exist, but there are some good techniques for extracting beat features and a lot of work has been done. (By the way, amplitude peaks are an obvious feature, but do not work well in general.) For a specialized application such as tracking beats in a rock band, there are many "tricks" such as putting a mic on the bass drum or bass guitar, looking for drum patterns, and even hooking up to MIDI output from a keyboard (if any). Trying to build a good beat tracking system from scratch is not a good idea, but evaluating a specialized approach to a narrowly defined problem, building upon known techniques would be very interesting.
- Work on sound file support in Nyquist. I recently adopted libsndfile, but then found that FLAC support is apparently not up-to-date (there are other libraries for FLAC) and as mentioned above, there is no support for SDIF files or MP3 files. I think the ability to read loop data out of AIFF sample files has been lost in the conversion to libsndfile, and I’d like to put it back -- ideally offering improvements to libsndfile. Finally, libsndfile does not really support compilation to universal binaries on the Mac or compilation under Win32. I hacked the sources to make this work, but now libsndfile (in nyquist/nylsf) is dependent on a nyquist configuration file. This whole thing needs some thought -- how to incorporate various libraries in a cross-platform way without having to install gnu tools and run config for every library on every platform. Ideally, the result would be a new "meta" library that includes libsndfile, flac support, AIFF sample files, and MP3 support, and compiles on multiple platforms.

© 2010 - Roger B. Dannenberg