From Theory to Practice
Scheduling and Graphical User Interfaces
Computer Music Systems and Information Processing

Roger B. Dannenberg
Jan 2018

Introduction

We've seen a lot of concepts and pseudo-code:

All of these of course must be implemented in order to use them (e.g. for Project 2).

We're going to look at implementations and libraries as follows:

A scheduler is implemented in lib/sched.srp, and this can be used with both serpent64 and wxserpent64, but the details are a little different.

Recently, Serpent has been extended with co-routine-like threads. All of the scheduling examples in this document have been implemented using Serpent threads, and these examples are described on this page on using threads.

Scheduling in serpent64

Simple Example

(
Code available here.) Let's make a simple program to print a message every second.
require "debug"
require "sched"

def activity(i):
    display "activity", i, sched_rtime, time_get()
    sched_cause(1, nil, 'activity', i + 1)

sched_init()
sched_cause(1, nil, 'activity', 101)
sched_run()
Notes:
Here's what some output looks like:
activity: i = 101, sched_rtime = 1, time_get() = 1
activity: i = 102, sched_rtime = 2, time_get() = 2.002
activity: i = 103, sched_rtime = 3, time_get() = 3.001
activity: i = 104, sched_rtime = 4, time_get() = 4.001
activity: i = 105, sched_rtime = 5, time_get() = 5
activity: i = 106, sched_rtime = 6, time_get() = 6
activity: i = 107, sched_rtime = 7, time_get() = 7.001
activity: i = 108, sched_rtime = 8, time_get() = 8
activity: i = 109, sched_rtime = 9, time_get() = 9.001
activity: i = 110, sched_rtime = 10, time_get() = 10.001
^C
Notice that the ideal event times, accessed as sched_rtime are precisely 1 second apart, as if the computer is infinitely fast and time is not quantized. The actual time, accessed as time_get() is close to the ideal time, but sometimes a millisecond or two late.

Two Activities

(
Code available here.) Scheduling is most useful when there are many "threads" whose events are interleaved in time.

We use temporal recursion, i.e. functions (re)scheduling themselves instead of true threads or processes.

The next example shows two activities. activity_1 runs every second; activity_2 runs every 3 seconds.
require "debug"
require "sched"


def activity_1(i):
    display "act1    ", i, sched_rtime, time_get()
    sched_cause(1, nil, 'activity_1', i + 1)


def activity_2(i):
    display "    act2", i, sched_rtime, time_get()
    sched_cause(3, nil, 'activity_2', i + 1)


sched_init()
sched_cause(1, nil, 'activity_1', 101)
sched_cause(1, nil, 'activity_2', 201)
sched_run()
Here is some output generated by the program. Notice that when 2 events are scheduled for the same time, they run sequentially, but it is not obvious which will run first. (In this case, ordering is deterministic, but order depends on the implementation of the priority queue in the scheduler.)

Notice also that activity 1 ("act1") runs every second, while activity 2 ("act2") runs every 3 seconds.
    act2: i = 201, the_sched.time = 1, time_get() = 1.001
act1    : i = 101, sched_rtime = 1, time_get() = 1.001
act1    : i = 102, sched_rtime = 2, time_get() = 2
act1    : i = 103, sched_rtime = 3, time_get() = 3.001
act1    : i = 104, sched_rtime = 4, time_get() = 4.001
    act2: i = 202, sched_rtime = 4, time_get() = 4.002
act1    : i = 105, sched_rtime = 5, time_get() = 5
act1    : i = 106, sched_rtime = 6, time_get() = 6
act1    : i = 107, sched_rtime = 7, time_get() = 7
    act2: i = 203, sched_rtime = 7, time_get() = 7
act1    : i = 108, sched_rtime = 8, time_get() = 8.001
act1    : i = 109, sched_rtime = 9, time_get() = 9
act1    : i = 110, sched_rtime = 10, time_get() = 10
    act2: i = 204, sched_rtime = 10, time_get() = 10
act1    : i = 111, sched_rtime = 11, time_get() = 11.001
act1    : i = 112, sched_rtime = 12, time_get() = 12.002
^C

Multiple ”Threads“ Using One Function

(
Code available here.) Let's go back to the first example and initialize the schedule with two calls to activity.
require "debug"
require "sched"


def activity(i):
    display "activity", i, sched_rtime, time_get()
    sched_cause(1, nil, 'activity', i + 1)


sched_init()
sched_cause(1, nil, 'activity', 101)
sched_cause(1.1, nil, 'activity', 201) 
sched_run()
Here is some output generated by the program. Notice that there are two independent "threads" or sequences of calls to activity, even though there in only one function. The events are separated by 1 second, with one set of events starting at 1 and the other starting at 1.1.
activity: i = 101, sched_rtime = 1, time_get() = 1.002
activity: i = 201, sched_rtime = 1.1, time_get() = 1.101
activity: i = 102, sched_rtime = 2, time_get() = 2.001
activity: i = 202, sched_rtime = 2.1, time_get() = 2.102
activity: i = 103, sched_rtime = 3, time_get() = 3
activity: i = 203, sched_rtime = 3.1, time_get() = 3.1
activity: i = 104, sched_rtime = 4, time_get() = 4
activity: i = 204, sched_rtime = 4.1, time_get() = 4.1
activity: i = 105, sched_rtime = 5, time_get() = 5.001
activity: i = 205, sched_rtime = 5.1, time_get() = 5.101
^C

The Stopping a “Thread” Pattern

(
Code available here.) How can we stop a function that keeps rescheduling itself? The simplest way is to set a flag that causes the function to return immediately. This is not a good solution. To restart the sequence of calls, you have to unset the flag. If you unset before a scheduled event wakes up and notices the flag, then the flag will have no effect, and you end up with two "threads" as in the previous example.

A good solution uses a counter rather than a flag, and every sequence of calls (every "thread") has an identifier that must match the counter.

require "debug"
require "sched"

activity_id = 0 // used to kill "threads"
 
def activity(id, i):
    if id != activity_id:
        display "activity terminates", id, sched_rtime, time_get()
        return
    display "activity", id, i, sched_rtime, time_get()
    sched_cause(1, nil, 'activity', id, i + 1)

def activity_start(i):
    // kill any old running activity 
    activity_id = activity_id + 1
    activity(activity_id, i) // start a new activity "thread"

sched_init()
sched_cause(1, nil, 'activity_start', 101)
sched_cause(4.5, nil, 'activity_start', 201)
sched_run()
Here is some output generated by the program. Notice that the first "thread" is effectivly killed at time 4.5 when activity_id is incremented, but the thread does not "know" this until it wakes up at time 5, "sees" activity_id has changed, and terminates (returns without performing any action or scheduling any future calls).
activity: id = 1, i = 101, sched_rtime = 1, time_get() = 1.001
activity: id = 1, i = 102, sched_rtime = 2, time_get() = 2.002 activity: id = 1, i = 103, sched_rtime = 3, time_get() = 3.001 activity: id = 1, i = 104, sched_rtime = 4, time_get() = 4.001 activity: id = 2, i = 201, sched_rtime = 4.5, time_get() = 4.5 activity terminates: id = 1, sched_rtime = 5, time_get() = 5.001 activity: id = 2, i = 202, sched_rtime = 5.5, time_get() = 5.501 activity: id = 2, i = 203, sched_rtime = 6.5, time_get() = 6.501 activity: id = 2, i = 204, sched_rtime = 7.5, time_get() = 7.501 ^C

Spawning New Activity

(
Code available here.) A "thread" can be spawned at any time. Here's a "thread" that runs every second. Each second, it spawns another thread that runs every 0.1 seconds, but only 3 times.
require "debug"
require "sched"

def activity(i):
    display "activity", i, sched_rtime, time_get()
    sched_cause(0.1, nil, 'subactivity', 0)
    sched_cause(1, nil, 'activity', i + 1)

def subactivity(j):
    display "     sub", j, sched_rtime, time_get()
    if j < 2:
        sched_cause(0.1, nil, 'subactivity', j + 1)
Here is some output generated by the program.
activity: i = 101, sched_rtime = 1, time_get() = 1.001
     sub: j = 0, sched_rtime = 1.1, time_get() = 1.102
     sub: j = 1, sched_rtime = 1.2, time_get() = 1.202
     sub: j = 2, sched_rtime = 1.3, time_get() = 1.3
activity: i = 102, sched_rtime = 2, time_get() = 2.002
     sub: j = 0, sched_rtime = 2.1, time_get() = 2.101
     sub: j = 1, sched_rtime = 2.2, time_get() = 2.2
     sub: j = 2, sched_rtime = 2.3, time_get() = 2.3
activity: i = 103, sched_rtime = 3, time_get() = 3.001
     sub: j = 0, sched_rtime = 3.1, time_get() = 3.101
     sub: j = 1, sched_rtime = 3.2, time_get() = 3.201
     sub: j = 2, sched_rtime = 3.3, time_get() = 3.302
^C

Virtual Time, Fixed Tempo

(
Code available here.) Let's run activity under a virtual time scheduler, at 2x speed, so the events are every 0.5 seconds instead of every 1 second. We use the already-created vtsched, an instance of Vscheduler.
require "debug"
require "sched"

def activity(i):
    display "activity", i, sched_rtime, time_get(), sched_vtime
    sched_cause(1, nil, 'activity', i + 1)

sched_init()
sched_select(vtsched)
sched_set_bps(2) // 2 beats per second
// sched_trace = t
sched_cause(1, nil, 'activity', 101)
sched_run()

Here's the output. Notice the_sched (which is vtsched) time is not even close to real time because this is virtual time (or beats, if you prefer).

activity: i = 101, sched_rtime = 0.5, time_get() = 0.5, sched_vtime = 1
activity: i = 102, sched_rtime = 1, time_get() = 1.002, sched_vtime = 2
activity: i = 103, sched_rtime = 1.5, time_get() = 1.502, sched_vtime = 3
activity: i = 104, sched_rtime = 2, time_get() = 2.001, sched_vtime = 4
activity: i = 105, sched_rtime = 2.5, time_get() = 2.502, sched_vtime = 5
^C

Virtual Time, Variable Tempo

(
Code available here.) The speed or tempo can be changed. Here, we set the tempo to 0.5 beats per second on beat 4 in the next-to-last line of code. You could also call set_bps() within activity, or elsewhere for that matter.
require "debug"
require "sched"


def activity(i):
    display "activity", i, sched_rtime, time_get(), sched_vtime
    sched_cause(1, nil, 'activity', i + 1)


sched_init()
sched_select(vtsched)
sched_set_bps(2) // 2 beats per second
// equivalent to: sched_set_period(0.5)  
sched_cause(1, nil, 'activity', 101)
sched_cause(4, nil, 'sched_set_bps', 0.5)
sched_run()

Here's the output. Notice the real times (time_get()) are every 0.5s until beat 4, then every 2s.

activity: i = 101, sched_rtime = 1, time_get() = 0.5
activity: i = 101, sched_rtime = 0.5, time_get() = 0.5, sched_vtime = 1
activity: i = 102, sched_rtime = 1, time_get() = 1.002, sched_vtime = 2
activity: i = 103, sched_rtime = 1.5, time_get() = 1.501, sched_vtime = 3
activity: i = 104, sched_rtime = 2, time_get() = 2, sched_vtime = 4
activity: i = 105, sched_rtime = 4, time_get() = 4.002, sched_vtime = 5
activity: i = 106, sched_rtime = 6, time_get() = 6.002, sched_vtime = 6
^C

Graphical User Interfaces

Basics

wxserpent automatically creates default_window (and a text output window if there is any text output).

Graphical objects -- buttons, windows, menus, canvases to paint on -- are all represented by Serpent objects.

Simple Example -- Button

(Code available here.) Create a button that prints "hello world" when pressed.

require "debug"
require "wxserpent"


button = Button(0, "Hello", 5, 5, 75, 20)
button.method = 'button_pressed'


def button_pressed(obj, event, x, y):
    print "Hello World

Notes:

Button as it appears on OS X

Menus

(Code available here.) Menus are objects; menu items are not.

require "debug"
require "wxserpent"


file_menu = default_window.get_menu("File")
file_menu.item("Hello", "Print Hello World", false, 
               nil, 'file_menu_handler')


def file_menu_handler(obj, event, x, y):
    display "file_menu_handler", obj, event, x, y
    print "Hello World"

Notes:

Image of a Menu in OS X

Labeled Sliders

(Code available here.) The Slider class makes a simple linear slider. A more "application ready" slider is implemented by the Labeled_slider class, which displays a label, the text value corresponding to the slider, uses floating point numbers, and can give you a linear or dB scale.

require "debug"
require "wxserpent"
require "slider"


slider = Labeled_slider(0, "Volume", 5, 5, 250, 20, 70, 0, 1, 0.5, 'linear')
slider.method = 'slider_handler'


def slider_handler(obj, x):
    display "slider_handler", obj, x
    print "Slider changed to:", x

Notes:

Labeled slider

Scheduling in wxserpent64

Simple Example

(Code available here.) This example is equivalent to our first example, only adapted to wxserpent64.

require "debug"
require "sched"

def activity(i):
    display "activity", i, sched_rtime, time_get()
    sched_cause(1, nil, 'activity', i + 1)

sched_init()
sched_cause(1, nil, 'activity', 101)

Notes:

Play Some MIDI Example

(Code available here.) In this example, we open a MIDI output device change activity to play a MIDI note.

require "debug"
require "wxserpent"
require "sched"
require "midi-io"
require "prefs"
require "mididevice"

def activity(p):
    display "activity", p, sched_rtime, time_get()
    midi_out.note_on(0, 60 + p % 12, 0)
    time_sleep(0.1)
    p = p + 1
    midi_out.note_on(0, 60 + p % 12, 100)
    sched_cause(0.2, nil, 'activity', p)

sched_init()
prefs = Prefs("./prefs.txt")
midi_devices = Midi_devices(prefs, open_later = true)
success = midi_devices.open_midi(latency = 10, device = 'midi_out_device')
if not success
    print "PLEASE SELECT A VALID OUTPUT DEVICE AND RESTART THIS PROGRAM"
else
    sched_cause(1, nil, 'activity', 0)

Notes:

Slider Volume Control Example

(Code available here.) Let's add some controls. This example uses the “stopping a thread” pattern and adds a Stop button. There are also sliders to control the period of time between notes and the MIDI velocity. We could use the first parameter of sched_cause() to control the period between notes, in which case the period change would take effect on the next note. Instead, we think of “period between notes” as a continuous indicator of tempo that can change at any time. We use sched_set_period() to change the tempo immediately when the slider is moved.

wxserpent window with midi period and
        volume controls

require "debug"
require "wxserpent"
require "sched"
require "midi-io"
require "prefs"
require "mididevice"
require "slider"

player_id = 0 // use "stopping a thread" pattern

def activity(id, p):
    display "activity", p, sched_rtime, time_get()
    midi_out.note_on(0, 60 + p % 12, 0)
    if id != player_id: // stop AFTER note-off
        return
    p = p + 1
    var vel = int(velocity.value())
    midi_out.note_on(0, 60 + p % 12, vel)
    sched_cause(1, nil, 'activity', id, p)

def stop(rest ignore)
    player_id = player_id + 1

Button(0, "Stop", 5, 5, 50, 20).method = 'stop'

def set_period(obj, x):
    display "set_period", obj, x
    sched_select(vtsched)
    sched_set_period(x)

period = Labeled_slider(0, "Period", 5, 30, 250, 20,
                        70, 0.05, 5, 0.2, 'exponential')
period.method = 'set_period'

velocity = Labeled_slider(0, "Velocity", 5, 55, 250, 20,
                          70, 1, 127, 100, 'linear')

sched_init() // creates vtsched and rtsched
prefs = Prefs("./prefs.txt")
midi_devices = Midi_devices(prefs, open_later = true) 
success = midi_devices.open_midi(latency = 10, device = 'midi_out_device') 
if not success 
    print "PLEASE SELECT A VALID OUTPUT DEVICE AND RESTART THIS PROGRAM"
else 
    sched_select(vtsched)
    sched_set_period(0.2)
    sched_cause(real_delay(5), nil, 'activity', player_id, 0)

Notes:

Forward synchronous scheduling

Timestamps in PortMidi

The previous examples do some behind-the-scenes work in schedulers and in midi-io to pass logical times as timestamps to PortMidi, the MIDI interface library used by Serpent. This section aims to reveal how this works.

PortMidi supports forward synchronous scheduling. In brief, everything is timestamped. The goal is to compute everything slightly ahead of real time, pass timestamped messages to the device driver, and let the device driver send the messages according to the timestamp to obtain very precise timing.

Forward synchronous scheduling is supported by the midi-io library which simplifies the process. You should use it to get better timing, especially when using wxserpent (because graphical redisplays can be slow, and the timer callback will not be issued while redrawing the display), but only if you can tolerate some latency.

Make an Example Without Timestamps

(Code available here.) To explore forward synchronous scheduling, let's first make a counter-example that has noticeable timing problems. Our simple examples on modern machines are so fast you might not notice any problems, even without careful timestamping, so we'll make an artificial situation where, at the beginning of our function to play MIDI, we delay a random amount whose distribution is controlled by a slider.

require "debug"
require "wxserpent"
require "sched"
require "midi-io"
require "prefs"
require "mididevice"
require "slider"


player_id = 0 // use "stopping a thread" pattern


def activity(id, p):
    time_sleep(random() * jitter.value())
    midi_out.note_on(0, 60 + p % 24, 0)
    if id != player_id: // stop AFTER note-off
        return
    p = p + 1
    midi_out.note_on(0, 60 + p % 24, 100)
    sched_cause(0.08, nil, 'activity', id, p)


def stop(rest ignore)
    player_id = player_id + 1

Button(0, "Stop", 5, 5, 50, 20).method = 'stop'


jitter = Labeled_slider(0, "Jitter", 5, 55, 250, 20,
                          70, 0, 0.05, 0, 'linear')


sched_init() // creates vtsched and rtsched
prefs = Prefs("./prefs.txt")
midi_devices = Midi_devices(prefs, open_later = true)
success = midi_devices.open_midi(latency = 0, device = 'midi_out_device')
if not success
    print "PLEASE SELECT A VALID OUTPUT DEVICE AND RESTART THIS PROGRAM"
else
    sched_cause(1, nil, 'activity', player_id, 0)

Notice that in the call to open_midi the latency is explicitly set to zero, which means timestamps are not used by PortMidi, and messages are delivered immediately. This still works pretty well, but if you crank the jitter slider up to 25 ms, you should notice timing irregularities.

Using Forward Synchronous Scheduling

To get forward synchronous behavior in the previous program, just change latency = 0 to latency = 25.

Now, if you crank the jitter slider up to 25ms, the program will run with same timing irregularities as before, but the MIDI output timing will be close to perfect!

Let's look under the hood and see how this happens.

Animation Example

(Code available here.) In this example, we revisit “Scheduling in wxserpent64,” adding some simple commands to draw shapes. This program uses the scheduler to schedule a refresh() operation that redraws a Canvas object at 30 frames per second.

require "debug"
require "wxserpent"
require "sched"


def activity(i):
    display "activity", i, sched_rtime, time_get()
    // 30 frames per second
    sched_rcause(1/30, nil, 'activity', i + 1)
    if random() < 0.03 // 3% of the time move rectangle
        animate.change_square()
    animate.refresh(t) // t means true, redraw everything


def timer_callback()
    rtsched.poll(time_get())


class Animate (Canvas)
    var ex, ey  // location of ellipse
    var sx, sy  // location of square
    var sr, sg, sb // square color (red, green, blue)

    def init(parent, x, y, w, h)
        super.init(parent, x, y, w, h) // calls Canvas's init()
        ex = 20 // initialize this subclass of Canvas
        ey = 20
        sx = 100
        sy = 100
        sr = 200
        sg = 200
        sb = 200

    def paint(x) // x == true means redraw everything
        // this code always redraws everything
        //display "Animate::paint", x
        set_brush_color("GRAY")
        draw_ellipse(ex, ey, 50, 30)
        ex = ex + 3
        if ex > get_width() - 50
            ex = 20
        set_brush_rgb(sr, sg, sb)
        draw_rectangle(sx, sy, 40, 40)

    def change_square()
        sx = random() * animate.get_width()
        sy = random() * animate.get_height()
        sr = int(random() * 256)
        sg = int(random() * 256)
        sb = int(random() * 256)


def main()
    animate = Animate(WXS_DEFAULT_WINDOW, 0, 0, 300, 300)
    sched_init()
    rtsched.cause(1, nil, 'activity', 101)
    rtsched.time_offset = time_get()
    wxs_timer_start(2, 'timer_callback')


main()

Notes: