Ch. 8 – Interactive Musical Instruments

Topics:   Computer musical instruments, graphical user interfaces, graphics objects and widgets, event-driven programming, callback functions, Play class, continuous pitch, audio samples, MIDI sequences, paper prototyping, iterative refinement, keyboard events, mouse events, virtual piano, parallel lists, scheduling future events.

This chapter explores graphical user interfaces and the development of interactive musical instruments.

While it takes years to master playing a guitar or violin, it is much easier for beginners to play a computer-based musical instrument, especially if they already have experience with computer games and other graphical applications. The goal here is not to replace traditional instruments or to undermine the years of experience required to achieve expertise in performing with them, but to introduce and engage more people in musical performance and composition.

Interactive computer-based musical instruments offer considerable versatility. They can be used by a single performer or by multiple performers in ensembles, like Laptop Orchestras. It is also possible to have an ensemble that includes both traditional instruments and computer-based instruments.

Creating a Display

To build programs with GUIs, you need the following statement:

As with the music library, the GUI library follows Alan Kay’s maxim that “simple things should be simple, and complex things should be possible”.

A program’s GUI exists inside a display (window).  Displays contain other GUI components (graphics objects and widgets).

For example, this:

creates a display with the given title, width, and height (as shown below):

A sample display (Mac).

A sample display (Mac).

A sample display (Windows).

A sample display (Windows).

Once a display has been created, you can add GUI components as follows:

where object is a GUI widget or graphics object.

A display’s origin – (0, 0) – is at the top-left corner.  The coordinates x, y above specify where to place the object in the display.

For example, the following code:

draws the following shape:

A sample shape drawn with graphics primitives.

A sample shape drawn with graphics primitives.

Displays may contain any number of GUI components, but they cannot contain another display.

A program may have several displays open.  Also, a program can specify where a display is placed on the screen.

Random circles on a Display

This code sample (Ch. 8, p. 246) demonstrates how to create a Display and draw random filled Circles on it.  It combines some of the programming building blocks we have learned so far (namely randomness, loops, and GUI functions).

Every time you run this program, it generates 1000 random circles and places them on the created display (as seen below):

1000 random circles on a display.

1000 random circles on a display.


A simple musical instrument

This code sample (Ch. 8, p. 251) demonstrates event-driven programming. It creates a GUI control surface consisting of two Button widgets. The first starts a note. The second stops the note. Each button utilizes its own callback function, which performs the desired functionality, when (and if) the button is pressed.

This code also demonstrates Play functions used for generating music interactively.

The program creates the following GUI.  Pressing the top Button starts a note, while pressing the bottom one stops it.

A single-note GUI control surface.

A single-note GUI control surface.

Soon, we will see how to generalize this idea to create more versatile interactive musical instruments.


Changing the background color interactively

This code sample demonstrates how to use sliders to update values in real time. It creates a GUI control surface consisting of three Slider and several Label widgets. The sliders control the color of the Display by updating its RGB values.  Similar code can be written to control any type of useful parameters.

(This example was contributed by Mallory Rourk.)

The program creates the following GUI.  Moving the RGB sliders updates the display color.

RGB sliders update display color.

A display with three sliders that update its color.


An audio instrument for continuous pitch

This code sample (Ch. 8, p. 256) demonstrates how to use GUI functions to create a simple instrument for changing the volume and frequency of an audio loop in real time.  It introduces the AudioSample class.  This class is very powerful for constructing arbitrary musical instruments for live performance.  The program, through this class, uses a loop from Moondog’s Lament I, “Bird’s Lament”.

NOTE: You should save this audio filemoondog.Bird_sLament.wav, in your jythonMusic folder prior to running this program:

The program creates the GUI control surface shown below.

A continuous pitch GUI instrument.

A continuous pitch GUI instrument.


Drawing musical circles

This code sample (Ch. 8, p. 268) demonstrates how to use event handling to build an interactive musical instrument. In this simple example, the user plays notes by drawing circles.

Screen snapshot of musical performance using a simpleCircleInstrument.

Screen snapshot of musical performance using a simpleCircleInstrument.

The diameter of the circle determines the pitch of the note (within the major scale). To draw a circle, the user presses the mouse on the display and starts moving the mouse. The point where the mouse is released determines the size of the circle (also the pitch of the note).

The mathematics used to map the diameter of the circle is relatively simple, but generates a musical effect that is interesting. Although the user might expect that doubling of the diameter will double the pitch of the note, that’s not the case. It takes a little while to figure out how mouse distance corresponds to pitch, and it is precisely this challenge that makes this “game” interesting – not unlike regular video games.

In this case, we synthesize the functionality of the GUI control surface by associating callback functions with mouse events.

The sample figure (above) shows the image generated from one musical performance with this instrument. Part of the performance is lost, namely the pitch and timing of the notes. What remains is still artistically interesting in its own right. This also demonstrates the creative possibilities of combining the GUI and music libraries to build interactive musical instruments.


Creating a virtual piano

This code sample (Ch. 8, p. 274) demonstrates how to create an interactive musical instrument that incorporates images.  The following program combines GUI elements to create a realistic piano which can be played through the computer keyboard.

It associates the keys “Z”, “S”, and “X”, on your computer keyboard, with the first three GUI piano keys, respectively.  In other words, you play the GUI piano via your computer keyboard (seeing which keys are pressed).

The program loads an image of a complete piano octave, namely iPianoOctave.png, to display a piano keyboard with 12 keys unpressed.  Then, to generate the illusion of piano keys being pressed, it selectively adds the following images to the display:

Note: The above images have to be saved in your jythonMusic folder, prior to running this program.

Here is the GUI surface this program generates:

The iPiano GUI with keys C, D sharp, and F being pressed.

The iPiano GUI with keys C, D sharp, and F being pressed.

Here is the program:

Note: To improve typing accuracy, some computer keyboards do not allow certain keys to be typed together, such as Z, S, and X. Therefore, you may notice that your virtual piano has the same limitation.


Creating a virtual piano – a variation

This code sample (Ch. 8, p. 279) demonstrates how to perform the same (above) task using parallel lists, for economy.  Specifically, as you expand the above program, you will soon realize the repetitive nature of the task.  For each of the computer keys you wish to associate with a virtual piano key, you have to add a special elif case in each of the two callback functions (in addition to loading the corresponding key-pressed icon).  This can take some time to implement.

This program introduces parallel lists to hold related information:

  • downKeyIcons holds the icons corresponding to “pressed” piano keys,
  • virtualKeys holds the virtual keys (e.g., VK_Z, etc.) corresponding to the above piano keys (pressing these keys on the computer keyboard “presses” the corresponding piano keys),
  • pitches holds the note pitches corresponding with the above (piano) keys, and
  • iconLeftXCoordinates holds the X coordinate of each “pressed” piano key icon, so it perfectly aligns with the underlying “unpressed” piano octave icon.

These lists can be extended to support more keys (currently, only 6 keys are functional).

Here is the new program:


Using Timers to schedule future events

So far we have seen how to capture user input through graphical user interfaces to drive our program’s behavior, i.e., event-driven programming.

Sometimes it is also useful to schedule computer-initiated (as opposed to user-initiated) events to happen sometime in the future. This allows us, among other things, to build buttons that temporarily change color (or shape) when pressed, etc. To do so, we simply change the color (or shape) of a graphical object when it is clicked on, and then schedule another change in color (or shape) to occur a fraction of a second later.

This is accomplished using Timer objects. Timer objects are given a function to call, and a delay specifying after how much time to wait to call it.

This code sample (Ch. 8, p. 283) presents a generative music application consisting of circles drawn randomly onto a display.

  • Every circle is connected to a note—as the circle is drawn, a pitch is sounded.
  • The color of the circle determines the pitch of the note—the redder the color, the lower the note.
  • The circle’s size (radius) determines the volume of the note—the bigger the radius, the louder the note.

To make the music aesthetically pleasing, pitches are selected from the major scale.  This concept is inspired by Brian Eno’s “Bloom” musical app for smartphones.

Graphical user interface of “Random Circles with Timer” program.

Graphical user interface of “Random Circles with Timer” program.

Circles are initially drawn once per 0.5 seconds (500 milliseconds), or at a rate of 2 circles per second.  This rate works quite well with adding circles to a display, possibly reminding the viewer of drops of rain.  Additionally, the program presents a secondary display (see figure above), which allows the user to control the delay between successive drops (i.e., circle-notes).

This program also demonstrates how to use a secondary display, in this case with a Slider, to control actions on the primary display.  Additional widgets could be easily added, to control other aspects of the application.

Here is the program:

As seen above, a Timer is provided with a delay value (in milliseconds) and a function to be called. Once started, the Timer waits the specified amount of time, then calls the function, perhaps only once, or perhaps indefinitely, with delays between all the calls.

As we will see again in Ch. 10, Timers are great for animation.