Ch. 9 – Making Connections

Topics:   Connecting to MIDI devices (pianos, guitars, etc.), the Python MIDI library, MIDI programming, Open Sound Control (OSC) protocol, connecting to OSC devices (smartphones, tablets, etc.), client-server programming via OSC messages, the Python OSC library, creating hybrid (traditional + computer) musical instruments.

In the previous chapter, we begun designing interactive musical instruments for live performance, employing the GUI library. Here, we extend this by exploring connections between a computer (e.g., a laptop) and external devices (e.g., MIDI instruments, synthesizers, various control surfaces, and smartphones). This is accomplished through the MIDI and OSC (Open Sound Control) libraries.

Now, in addition to developing GUIs that make your computer into an arbitrary musical instrument, we can construct hybrid musical instruments, combining traditional musical instruments (e.g., piano or guitar) with your computer.

For example, here is such a hybrid instrument, called Monterey Mirror:

In order for such an instrument to be connected to a Python program, it needs to be MIDI or OSC enabled.  The Python program can then alter the traditional “functionality” of the instrument to anything that you (the designer, or “digital luthier”) – decide.  This can produce futuristic instruments, combining guitars, pianos, percussion, smartphones, tablets, and other MIDI or OSC-enabled surfaces.  This newfound capability allows you to literally shape the musical experience of others.

MIDI Devices – Connecting to Pianos, Guitars, etc.

Here we explore how to connect Python programs to external MIDI devices.  This is accomplished by utilizing the MIDI library.  There are two types of possible connections, input and output:

  • Using MIDI input (one or more), we allow our program to receive incoming MIDI data from external devices, such as MIDI-enabled instruments (e.g., guitars, pianos, violins, saxophones, etc.).
  • Using MIDI output, we enable our programs to drive external synthesizers.

Patch messages from a MIDI input device to a MIDI output device

This is a simple program to demonstrate how to open connections to input and output MIDI devices. The program receives the input messages and sends them directly to the output device.  Also, it prints out the messages to the screen.

Use this program to test your MIDI connections.  Also, use it to explore what types of messages are produced by different buttons on a MIDI device.  Then, you can map these buttons to any functionality you wish.

 

This creates a MidiIn object, which allows us to establish communication with a MIDI instrument connected on our computer.  When line 10 above is executed, the following GUI is shown:

Display opened when creating a MidiIn object.   It contains all available input MIDI devices.

Display opened when creating a MidiIn object. It contains all available input MIDI devices.

This GUI contains a drop-down list populated with all available input MIDI devices.  When we select one of these devices, the MidiIn object is connected to that device (and the GUI disappears).

 

The program above also creates a connection to an output MIDI device (e.g., a synthesizer) via the MidiOut object. When line 13 is executed, the following GUI is shown:

 Display opened when creating a MidiOut object. It contains all available output MIDI devices.

Display opened when creating a MidiOut object.  It contains all available output MIDI devices.

This GUI contains a drop-down list populated with all available output MIDI devices.  When we select one of these devices, the MidiOut object is connected to that device (and the GUI disappears).

Once such an object has been created, functions similar to those provided by the Play class are available.  (The main difference is that the MidiOut object connects to an external synthesizer, whereas Play is connected to the internal Java synthesizer.)

A program may have several MidiIn and MidiOut objects, one for each desired input and output MIDI device. This opens various possibilities for communication and processing of incoming and outgoing MIDI data.

 


Drawing circles through MIDI input

This code sample (Ch. 9, p. 291) is a variation of the last example in chapter 8 (“Random Circles with Timer”).  The program now receives input from an external MIDI instrument, such as a MIDI guitar, piano, or saxophone (among others). It eliminates the Timer object, since timing information will now be provided by the user through playing notes on a physical musical instrument.

 


OSC Devices – Connecting to Smartphones, Tablets, etc.

This section describes how to connect various everyday devices, such as smartphones, to your music-making programs, through the Open Sound Control (OSC) protocol.  This can be done through existing OSC applications and/or the OSC library.

OSC is similar to MIDI in that it allows different devices to connect to and communicate with each other.  It is different from MIDI in that it does not require the devices to be connected via special cables.  OSC uses the Internet to send and receive messages, and as long as the devices you wish to connect to are on the network they can “talk” to each other.

OSC Messages

OSC messages look similar to the URLs we type in browsers.  They usually consist of an address pattern and optional arguments:

  • OSC address patterns form a hierarchical name space, such as “/hello/world/”.  It is up to the OSC client and server to decide what addresses will be used to communicate.
  • Arguments mainly are integers, floats, strings, and booleans.  OSC messages may include zero or more arguments, as agreed between OSC clients and servers.

The advantages of OSC over MIDI are primarily connectivity over the Internet (no dedicated wires!), flexibility of setting up message types (OSC addresses), and use of standard data types (e.g., integer, float, string, boolean).

We will now explore how to write simple OSC programs in Python.  These programs can be combined with existing OSC applications (clients or servers) running on other devices (or the same computer).

 


Receiving OSC messages

This code sample (Ch. 9, p. 299) demonstrates how to create an OSC server.  The OSC server (or OSC receiver, or OSC input) receives data from another OSC client (see next section).

Actually, this and the next code samples are made to communicate with each other. Again, the OSC client device will control the OSC server device. You should use two different computers. But, for simplicity, you could use a single computer (just run the following two programs on separate windows). Even if you use a single computer to execute both programs, the messages will be delivered via the Internet.

Here is the program to receive the OSC messages:

The above program creates an OscIn object, which sets up the computer for incoming OSC messages. This specifies which port is being used.

When you run this program, it will output the following:

where “xxx.xxx.xxx.xxx” is the IP address of the receiving computer (e.g., “192.168.1.223”).

The OSC client(s) wishing to communicate with this server need to use the precise IP address and the port number. (If you are behind a firewall, you need to ensure this port (e.g., “57110”) is open for incoming traffic.

Notice the special OSC address “/.*” – this matches for all incoming addresses.  Actually, the OSC library allows the use of regular expressions to specify OSC addresses used with function onInput().   However, you should be able to handle most of your needs with simple OSC addresses, similar to “/gyro” and “/accelerometer”, as shown in the last code sample.

 


Sending OSC messages

This code sample (Ch. 9, p. 301) demonstrates how to create an OSC client.  The OSC client (or OSC sender, or OSC output) sends data to an OSC server (see above).

Actually, this and the previous example are made to communicate with each other.  (See above for more instructions.)

You should “xxx.xxx.xxx.xxx” with the IP address of a receiving computer (e.g., “192.168.1.223”).

Alternatively, to send messages to an OSC server you may use another OSC client, such as TouchOSC or OSC Control.  Using existing OSC client apps allows Python programs to be driven by external devices, such as smartphones, tablets, and other computers.  This way you may build sophisticated musical instruments and artistic installations.

 


Making music with your smartphone

In this code sample (Ch 9. p. 307) we modify the “Random Circles Through Midi Input” program (see above) to receive input from a smartphone, using the OSC protocol.

To send OSC data from the smartphone, we used the free app, OSC Control:

Using a smartphone to create music on a laptop (via OSC messages).

Using a smartphone to create music on a laptop (via OSC messages).

We selected OSC Control because it is free and available on both iPhone and Android devices. Of course, there are many other OSC apps out there.

Performance Instructions

This program creates a musical instrument out of your smartphone.  It has been specifically designed to allow the following performance gestures:

  • Ready Position:  Hold your smartphone in the palm of your hand, flat and facing up, as if you are reading the screen.  Make sure it is parallel with the floor.  Think of an airplane resting on top of your device’s screen, its nose pointing away from you, and its wings flat across the screen (left wing superimposed with the left side of your screen, and right wing with the right side of your screen).
  • Controlling Pitch:  The pitch of the airplane (the angle of its nose – pointing up or down) corresponds to musical pitch.  The higher the nose, the higher the pitch.
  • Controlling Rate:  The roll of the airplane (the banking of its wings to the left or to the right) triggers note generation.  You could visualize notes falling off the device’s screen, so when you roll/bank the device, notes escape (roll off).
  • Controlling Volume:   Device shake corresponds with loudness of notes. The more intensely you shake or vibrate the device as notes are generated, the louder the notes are.

To summarize, the smartphone’s orientation (pointing from zenith to nadir) corresponds to pitch (high to low). Shaking the device plays a note—the stronger, the louder. Tilting the device clockwise produces more notes.

On the server side, i.e., the program you are controlling with your smartphone:

  • Note pitch is mapped to color of circles (lower means darker/browner, higher means brighter/redder/bluer).
  • Shake strength is mapped to circle size (radius).
  • Finally, the position of the circle on the display is random.

All these settings could easily be changed.  We leave that as an exercise.

Here is the program:

 

Using OSC allows you to use Internet-enabled mobile computing devices (like smartphones and tablets), to connect and control your programs. This facilitates a new way of thinking about computer programming and instrument design.