Developing and Debugging an Audacity Plug-In for Normalization Using Nyquist

Roger B. Dannenberg

2 July 2010

Introduction

Normalization or any operation that scans an entire sound runs the risk of allocating lots of sample memory in Nyquist. Here is how to avoid that problem. For just the results, you can skip ahead to here.

It might seem that if anyone knows how to make Audacity plug-ins using Nyquist, it's me, but in reality I work almost entirely in Nyquist, and Audacity is almost entirely managed and maintained by other people now. I ran into lots of little problems, so I thought I would describe the process here for my own memory and perhaps to help others.

Getting Started

I downloaded the latest released version of Audacity (1.3.12) and tried out something simple: I tried to type (peak s) into the Nyquist prompt.

Lesson 1: You can't run an effect without some selected audio.

OK, that's easy: I run Generate:Noise... to create noise, then try again. Typing (peak s) into Generate:Nyquist Prompt... returns "Nyquist did not return audio." Well, of course, it's supposed to return a number. The problem is that (peak s) is missing a parameter. This is verified by clicking the Debug button instead of the OK button when running Generate:Nyquist Prompt....

Lesson 2: Use the Debug button to run a Nyquist expression, not the OK button.

I changed the expression to (peak s ny:all) and things worked as expected.

Editing Plug-Ins

I fired up Emacs to make a plug-in. After a little searching, I found plug-ins in the Audacity Plug-In folder. Rather than type a plug-in, I copied one and edited it. I tried to save it back to the same folder with a new name, and that seemed to work, but the plug-in did not appear in the Audacity menu.

Lesson 3: Restart Audacity to rescan the plug-in folder and pick up new plug-ins.

Unfortunately, things were not all that simple...

Lesson 4: Windows Vista sometimes pretends to write to system directories.

From Emacs, it seemed as if I was reading and writing to the Audacity Plug-In directory, but in reality, the file was going into another directory. I assume Vista tries to create the illusion of writing for compatibility with earlier Windows software, but increases security by preventing the writes. I found the actual location by searching in Explorer, then copied the file by hand to the Plug-Ins directory. After that, I was able to edit and modify the file directly from the editor.

Next, I wanted to display some debugging information. After a bit of search, I discovered you can add a line to the plug-in to enable printing:
;debugflags trace
With this in place, I could finally do what I set out to do: print some heap info, run a plug-in, and look at the heap again.

Lesson 5: Use ";debugflags trace" to get debug info from a plug-in.

Here's the plug-in:
;nyquist plug-in
;version 1
;type analyze
;categories "http://audacityteam.org/namespace#OnsetDetector"
;name "rbd test"
;action "Performing rbd test ..."
;info "by Dannenberg for testing"
;debugflags trace

(print "before run")
(info)
(defun rbd-test ()
  (format nil "this is a test: ~A" (peak s ny:all)))
(setf result (rbd-test))
(print "after run")
(info)
result
Note that I defined the action as a function named rbd-test and called it. I could have just evaluated the expression directly. Note also that I assigned the result to result so that I could print more information and then return the value. Finally, I returned a string as an experiment. You can return either a string or a number.

Lesson 6: You can return and print a value by returning either a string or a number.

Oddly, the debugflags option seems to work only after writing a new plug-in. After running the plug-in once, the debug output window does not appear. When I rewrite the file with the editor and run the plug-in, the debug output appears again. Running the plug-in again without rewriting the file generates a result, but no debug output! This calls for some Audacity debugging...

Debugging Audacity

I have a version of Audacity sources that I'm currently working on, so that should be good for testing and debugging the "debugflags" option. I started Visual C++ Express Edition and opened my audacity.sln file. Then, I used F5 (Debug:Start Debugging) to start Audacity.

Of course, now the rbd-test.ny plug-in is missing because I'm running a different Audacity. After copying rbd-test.ny to audacity-src/plug-ins, I still do not see my "effect" in the Analyze menu. Not surprisingly, I found another directory:
C:\Users\rbd\audacity\audacity-src\win\Debug\plug-ins
and after moving rbd-test.ny here, the plug-in works. I still get the behavior where the plug-in text output from Nyquist is not displayed the second time I run the plug-in. Debugging can begin.

Nyquist effects are run in src/effects/nyquist/Nyquist.cpp. Each effect has an mDebug flag that enables the collection of text output from Nyquist and displays it in a dialog box after the effect runs. For the Nyquist Prompt... effect, mDebug is set when the user clicks the Debug button, and mDebug is cleared after the effect runs. The statement that clears mDebug also clears mDebug set by debugflags, which is why debugflags only takes effect the first time the effect is executed. (I fixed this bug so that debugflags is permanent -- maybe we'll see it in a future release.)

Memory Usage

Now for the real problem: how can we determine the peak value of a signal without reading in the entire signal? To determine how much signal memory is in use, run a garbage collection and then call info to print memory information. Alternatively, you can set *gc-flag* and the garbage collector will print some information:
(gc) (info)
or
(setf *gc-flag* t) (gc)
The rbd-test.ny output shows that sample memory is allocated for the entire signal that is analyzed. The output after a couple of runs is:
"before run"

[ Free: 22627, GC calls: 26, Total: 1297906; samples 45KB, 43KB free]
"after run"

[ Free: 22613, GC calls: 27, Total: 1298013; samples 45KB, 0KB free]
The "samples" number is how much sample memory has been allocated. The last line shows that after running the effect, but before returning the result, there are 45KB of samples, with 0KB free. I would have expected 4 bytes/sample times 44.1K samples (I analyzed 1s of audio), or about 176KB of samples. Maybe Nyquist is counting samples instead of bytes -- another potential bug.

In any case, the problem must be that the variable s is bound to the signal being analyzed. We should unbind s or use a destructive method to compute the peak. As an experiment, I wrote a function to copy s to a local, set s to nil, and return the local. If I compute peak on this function, it appears that about 100KB of sample memory is used, even on 10 minutes of audio. The exact amount of memory depends upon when garbage collection is run. To minimize memory, one should call for garbage collection immediately after setting s to nil, but Nyquist will call the garbage collector automatically before memory allocation becomes a problem.

Normalize

It should now be possible to write a normalize function in Nyquist that is well-behaved with respect to memory. The function will have to run in two parts: part 1 will compute the peak value and save it. Part 2 will scale the signal. Two passes are required to avoid saving the samples in Nyquist memory.

I wrote normalize-part1 to do the analysis and normalize-part2 to alter the signal. I save the peak value as a property of the symbol *scratch* with the property symbol normalize-part1. Unfortunately, this did not work. After some experimentation, it seems that while the *scratch* symbol may be retained from one effect to the next, the symbol normalize-part1 is being removed in the post-effect cleanup. This is a bug we will have to fix.

Breakpoints and Debug Printing

While debugging changes to the Audacity/Nyquist interface in nyx.c (part of libnyquist), I tried to set breakpoints in nyx.c but got messages saying "the source code is different from the original version." I ended up using some print statements to debug (very annoying). But printf() does not work unless you have a console application, so I ended up using OutputDebugString(str). To print a variable in a string, I used strcpy() and strcat() to build up messages -- what a pain!

Somehow in this process, I must have done more than just add output statements, because further experimentation showed that changing nyx.c and running Audacity did not actually compile and execute the changed code.

Lesson 7: If you make a change to nyx.c and rerun Audacity from the debugger, it will prompt you to rebuild libnyquist, but the code changes do not actually affect the application when it runs. (Perhaps other libraries have this problem too.)

OK, so this indicates some dependencies are not right. I decided to try increasingly costly builds to see what is required after code changes. First, I built the Audacity project. Visual C++ said it was already up-to-date, so I didn't even try running it again. Next, I tried forcing a link (right click on Audacity in the "Solution Explorer" panel, and under "Project Only ..." select "Link Only Audacity". That did the trick.

Lesson 8: After you change nyx.c (and perhaps any source file in lib-src), you need to relink Audacity manually for the changes to take effect.

After I relinked Audacity, I found that breakpoints work again as expected, so the real problem with breakpoints is that I did not have a consistent application until I manually relinked Audacity.

Further work showed that the Build Solution menu item has the same problem of not building a consistent application, but that's not too surprising given that running the debugger does not do a proper build.

Finally, there is a bug in the garbage collector output messages: it says KB, but the number is the number of samples divided by 1024. The true memory usage in bytes is approximately four times greater becaues there are 4 bytes per sample.

Working Normalize Plug-Ins

After all this, I have two plug-ins: an Analysis plug-in that detects the peak and an Effect plug-in that scales the audio based on the peak. These effects depend on a code fix to Audacity that allows properties to be passed on the *scratch* symbol. The effects are as follows:

normalize-part1.ny

;nyquist plug-in
;version 3
;type analyze
;name "Normalize Part 1"
;categories "http://lv2plug.in/ns/lv2core#AnalyserPlugin"
;action "Finding peak amplitude ..."
;info "After this analysis step, run normalize-part2 to normalize the selection."


;; For debugging, we get memory status before and after running the effect.
(setf *gc-flag* t)
(gc)

;; Get the input signal s, but set s to nil to avoid retaining samples.
(defun get-signal ()
  (let ((local-s s))
     (setf s nil)
     local-s))

;; Compute the peak value of s
(setf peak-value (peak (get-signal) ny:all))

(gc)

;; Save the peak-value on *scratch* for later use.
;;    Assume that the effect name (normalize-part1) can be safely used
;;    as a property symbol without any conflicts.
(putprop '*scratch* peak-value 'normalize-part1)

;; Return a string:
(format nil "The peak value is: ~A" peak-value)

normalize-part2.ny

;nyquist plug-in
;version 3
;type process
;name "Normalize Part 2 ..."
;categories "http://lv2plug.in/ns/lv2core#AmplifierPlugin"
;action "Normalizing ..."
;control db "Normalize maximum amplitude to ..." real "dB" 0.0 -24.0 0.0
;info "You must run Analyze: Normalize Part 1 before this plug-in."

;; For debugging, we get memory status before and after running the effect.
(setf *gc-flag* t)

(format t "db is ~A~%" db)

(setf peak-value (get '*scratch* 'normalize-part1))
(format t "peak-value is ~A~%" peak-value)

(cond ((floatp peak-value)
       (mult s (/ (db-to-linear db) peak-value)))
      (t
       "No peak-value found -- run normalize-part1 to compute the peak value"))