# Clive

Clive is my system for performing live music by coding in the C programming language, to manipulate audio processing algorithms while they are running.

# 1 Listen

# 1.1 Tracks

# 1.2 Live Sessions

Web pages with code diffs synchronized to audio playback.

# 1.3 Tips and Tricks

# 1.3.1 Clive in Bela IDE

Proof of concept:

1920x1080p30 MP4 (7MB), 3m00s, sound

Bela is an embedded platform for audio processing. It has a web-based IDE hosted on the board, so you can code in the browser. Normally in a Bela C++ project, when making changes you need to stop, recompile, run again, which loses all state, and takes a long time. I ported clive-server’s fileystem watcher and dlopen thing to Bela, so that the “real” audio render callback is in a shared library, so you can change it at runtime.

Bela’s web IDE saves the file to the device filesystem on each change, which triggers clive’s rebuild and reload and hotswap. It still takes a long (unpredictable) amount of time even with simple code, but the old code keeps running until the new code is ready, without silence or pops from restarting the audio codec, and the state is preserved across reloads (size hardcoded to 64MB).

Currently it’s somewhat of a hack: the code assumes its project name is clive stored in /root/Bela/projects/clive. Build commands are a bit copy/pasted from the Bela Makefile without really knowing what I’m doing. But it’s more promising (especially easier to reproduce) than my previous attempts with cross-compiling from a connected computer (in Feb 2019). I need to investigate select() to timeout the blocking inotify read calls so I can stop it cleanly.

# 1.3.2 Declicking

Fade on reload:

1024x576p10 MP4 (3MB), 3m43s, sound

Naive editing of DSP code while livecoding leads to clicks which sound really bad. Using Clive’s reload notification, it’s possible to eliminate these clicks by fading over a number of beats (start time quantized to a multiple of the beat).

It’s a bit more editing work, but sounds much better.

There are some gotchas though: interrupting a fade is not possible without clicks, forgetting to remove or edit a fade() that has finished will lead to the fade restarting over.

The only fade currently implemented is linear, shouldn’t be too hard to add equal-power curves for cross-fades, or raised cosine half-period for fades in and out.

typedef double sample;
#define SR 48000
#include \"dsp.h\"

typedef struct {
  int reloaded;

  double clock;
  double fadeclock;
  int fading;

} S;

int go(S *s, int inchannels, const float *in, int outchannels, float *out)
{
  if (s->reloaded) s->fading = 0;

  sample quant = 4; // this sets the quantisation boundary in beats
  sample bpm = 123.456;
  sample hz = 50;

  sample clock0 = s->clock;
  sample clock = s->clock += bpm / 60 / SR;
  if (wrap(clock / quant) < wrap(clock0 / quant) && ! s->fading)
  {
    s->fadeclock = s->clock;
    s->fading = 1;
  }
#define fade(n,out,in) \\
  mix(out, in, clamp(s->fading * (s->clock - s->fadeclock) / n, 0, 1))

  sample a = twopi * wrap(hz * clock / (bpm / 60));
  sample t = twopi * clock;

  sample wave[2];
  for (int c = 0; c < 2; ++c)
  {

    // sound
    wave[c] = 0;

  }

  for (int c = 0; c < outchannels; ++c)
  {
    out[c] = 0;
    if (c < 2)
    {
      out[c] = wave[c];
    }
  }
  s->reloaded = 0;
  return 0;
}

# 2 Code

# 2.1 Core

Engine code only (small) code.mathr.co.uk/clive-core

git clone https://code.mathr.co.uk/clive-core.git

# 2.2 Main

Engine code plus mathr performances (large) code.mathr.co.uk/clive

git clone https://code.mathr.co.uk/clive.git

# 3 Getting Started

(These notes from 2018 may be out of date.)

# 3.1 Introduction

  • Linux only (sorry)

  • install dependencies, configure sudo and JACK

  • clone the repository

  • launching, exiting

  • an example

# 3.2 Dependencies

Debian

# apt install \
    sudo git ca-certificates \
    build-essential pkg-config \
    libjack-jackd2-dev qjackctl \
    cpufrequtils ecasound \
    xterm htop geany \
    python-pygments

# 3.3 Configuring sudo for cpufrequtils

Replace claude with your username and latte with your machine name:

$ sudo visudo /etc/sudoers.d/cpufreq-set
claude latte = (root) NOPASSWD: \
  /usr/bin/cpufreq-set

The file contents should be one line without \, just split to fit slides!

This allows to change CPU frequency governor without password.

# 3.4 Configuring JACK for realtime

Provided by JACK packaging on Debian:

$ cat /etc/security/limits.d/audio.conf
@audio   -  rtprio     95
@audio   -  memlock    unlimited

To check that you are in the audio group:

$ groups

# 3.5 Download clive repository

git clone https://code.mathr.co.uk/clive-core.git
cd clive-core
git config user.email "you@example.com"
git config user.name "Your Name"

# 3.6 Launching

Launch qjackctl and configure your sound card.

Launch clive:

cd clive-core/src
./start.sh

After a short delay, there should be 2 terminal windows on the left and the geany text editor which can be resized to fit on the right.

Edit go.c, press Ctrl-S to save, which will recompile and reload the code.

# 3.7 Exiting

  • Ctrl-C in the terminal running ./start.sh

  • Stop and rewind JACK transport

  • git checkout main

# 3.8 Example: a metronome

git checkout origin/metronome
git checkout -b metronome

#define SR 48000
#include "dsp.h"

typedef struct {
  int reloaded;
  PHASOR clock, osc;
} S;

int go(S *s, int channels, const float *in, float *out) {
  if (s->reloaded) { s->reloaded = 0; }
  double env = phasor(&s->clock, 125/60.0) < 0.25;
  double osc = sin(twopi * phasor(&s->osc, 440));
  double o = env * osc;
  for (int c = 0; c < channels; ++c) { out[c] = o; }
  return 0;
}

# 4 Workshops and Talks

  • slides - getting started with clive, at TOPLAP Moot 2018

  • slides - live coding audio in C with clive, Zagreb 2019

  • video - slides - technical talk about how clive works, at THSF#9 2018

  • slides - another technical talk, at TOPLAP Moot 2018

# 5 Lightning Talk

(This is the short story about how Clive works.)

# 5.1 JACK Audio

  • Process callback delegates to a function pointer

  • Passes a pointer to a preallocated memory area

  • Main thread can change the function pointer

# 5.2 Dynamic Reloading

  • dlopen(), dlsym(), -ldl

  • Race condition: don’t unload running code

  • Double buffering: dl caches aggressively

# 5.3 Detecting File Changes

  • inotify

  • if go.c changes: compile it (by calling make)

  • if go.so changes: make a copy and load it

# 6 Live-coding Audio in C

(This was a paper I wrote in 2017-2018, but didn’t publish anywhere.)

# 6.1 Abstract

An audio live-coding skeleton is presented, implemented in C and adaptable to languages that expose the C ABI from shared libraries. The aim is to support a two-phase edit-commit coding cycle allowing long-lived signal processing graphs to be modified.

The skeleton watches a directory for changes, recompiling changed sources and reloading changed shared objects. The shared object defines an audio processing callback. The skeleton maintains a memory region between reloads, allowing callback state to be preserved so that audio processing can continue uninterrupted.

# 6.2 Motivation

(At least) two popular music software packages allow live-coding. Pure-data allows the manipulation of audio processing graphs at runtime, but live-coding requires careful preparation as there is no edit/commit cycle: every change is live as soon as it is made. Moreover each edit recompiles the whole DSP chain, which is not a realtime-safe operation and can lead to audio dropouts.

Supercollider takes a different approach, running selected code only when a certain key combination is pressed. However, Supercollider DSP code runs on a server distinct from its language, and synths (compiled by a client such as sclang and sent to the scsynth server) can’t be modified once they are running, beyond setting control variables or re-ordering synths within a group – the code within each synth is immutable.

The main goal of Clive is to be able to edit digital synthesis algorithms while they are running, with a two phase edit/commit cycle: code can be edited freely, until it is saved to disk, at which point it is re-compiled and (if successful) re-loaded into an active audio processing engine. The reasons for using the C programming language are familiarity to the author, precise control of memory layout of data structures allowing code reload with state preservation, speed of compilation (relative to higher-level languages like C++), and predictable runtime performance.

Performance with Clive typically involves pre-preparation, at least of common unit generators like oscillators and filters, perhaps higher-level effects like chorus or dynamic range compression, extending to more-complete compositions with complex data flow. The live-coding aspect involves editing a file in the performer’s favourite text editor, with the act of saving with Ctrl-S or other shortcut being timed to allow the new code to start executing after the latency of compilation.

# 6.3 Implementation

# 6.3.1 The skeleton

The live engine starts a JACK client, whose audio processing function is a thin wrapper around a callback function pointer. The function pointer is initially set to a function that fills its output buffers with silence. The main program watches the current directory for changes, recompiling changed sources and reloading a shared library (named go.so) on changes. The shared library defines a symbol go matching the callback_t interface. The callback is set to the address of this function. The callback is passed a pointer to a memory area that is preserved between code reloads.

typedef void (*callback_t)(void *, float *, int, float *, int);
volatile callback_t callback = silence_callback;
void *data;
int process(jack_nframes_t n, void *arg) {
 float in[nin];
 float out[nout];
 // ... get JACK buffers
 for (int i = 0; i < n; ++i) {
  // ... copy buffers to in
  callback(data, in, nin, out, nout);
  // ... copy out to buffers
 }
 return 0;
}
int main() {
 data = calloc(1, bytes);
 load_library("go.so");
 // ... launch JACK client
 // the following is pseudo-code
 watch w = watch_directory(".");
 event e;
 while (( e = wait_event(w) )) {
  if (has_changed(e, "go.so"))
   load_library("go.so");
  if (has_changed(e, "go.c"))
   recompile("go.c");
 }
 return 0;
}

# 6.3.2 Dynamically reload libraries

The POSIX.1-2001 dl interface is used to load shared libraries at runtime. The library is loaded with dlopen, the callback function symbol address is found with dlsym, and when no longer needed the code can be unloaded with dlclose. The type cast gymnastics on line 9 are to avoid a warning in strict ISO C99 mode.

void *old_library = 0;
void *new_library = 0;
void load_library(const char *name) {
 if (! name)
  return;
 if (! (new_library = dlopen(name, RTLD_NOW)))
  return;
 callback_t *new_callback;
 *(void **) (&new_callback) = dlsym(new_library, "go");
 if (new_callback) {
  set_callback(new_callback);
  // unload old library
  if (old_library)
   dlclose(old_library);
  old_library = new_library;
  new_library = 0;
 } else {
  dlclose(new_library);
  new_library = 0;
 }
}

# 6.3.3 Mitigate races

There is a race condition, especially problematic on multi-core architectures: the old code must not be unloaded while the JACK callback is still running it, otherwise the process will crash. To mitigate this risk (though not prevent it entirely), a flag is added that makes the main process spin until the current DSP block’s JACK processing is finished. The old library is probably safe to unload after set_callback has returned. To avoid a total stop to the live performace in the unlikely event of this crash, a shell wrapper script relaunches the engine automatically.

volatile bool processing = false;
int process(jack_nframes_t n, void *arg) {
 processing = true;
 // ...
 processing = false;
 return 0;
}
void set_callback(callback_t new_callback) {
 while (processing)
  ;
 callback = new_callback;
}

# 6.3.4 Defeat caches

Unfortunately libdl performs rather aggressive caching, meaning that a library of the same name is not reloaded if it is already open, even if its contents have changed.

A first try was to unload the old code and load the new code, hoping that it would be quick enough to avoid audio dropouts (the callback function pointer was set to silence before unloading to avoid a crash). The dropouts were too severe to make this viable in a performance situation.

A second attempt was to use two symlinks for double-buffering, but even this does not defeat the cache. A real file copy appears to be necessary, and finally the current version copies alternately to two separate files in a double-buffering scheme.

const char *copy[2] =
 { "cp -f ./go.so ./go.a.so"
 , "cp -f ./go.so ./go.b.so" };
const char *library[2] =
 { "./go.a.so", "./go.b.so" };
int which = 0;
void copy_library() {
 if (! system(copy[which]))
  return library[which];
 else
  return 0;
}
void load_library(const char *name) {
 // ...
 if (new_callback) {
  which = 1 - which;
  // ...
 }
}
int main() {
 // ...
  if (has_changed(e, "go.so"))
   load_library(copy_library());
}

# 6.3.5 Notify reloads

It can be useful when live programming audio to perform initialization once only. To that end, clive mandates that the first word of the memory area is an int, that is set to 1 when the code is reloaded. The callback code is responsible for clearing it, necessary to detect subsequent reloads.

volatile bool reloaded = false;
int process(jack_nframes_t n, void *arg) {
 if (reloaded) {
  int *p = data;
  *p = 1;
  reloaded = 0;
 }
 // ...
 return 0;
}
void set_callback(callback_t new_callback) {
 // ...
 reloaded = true;
}

# 6.3.6 Watch for changes

On Linux, the filesystem can be monitored for changes using inotify. The function inotify_add_watch is used to check for file close events, of files that were opened for writing. The inotify_event structure contains the name of the file that changed, which is compared to two cases: if the source go.c has changed, recompile it; if the library go.so has changed, reload it. Hopefully the first case triggers the second case, otherwise the live coder should check the terminal for compilation error messages.

It is important to start watching before the main loop, rather than stopping and starting repeatedly within the loop, lest events are missed between loop iterations. To support languages other than C, it is necessary to modify line 27 to reference the new source file, and add the approriate compilation commands to the Makefile.

int main() {
 // ...
 int ino = inotify_init();
 if (ino == -1) return 1;
 int wd = inotify_add_watch(ino, ".", IN_CLOSE_WRITE);
 if (wd == -1) return 1;
 ssize_t buf_bytes = sizeof(struct inotify_event) + NAME_MAX + 1;
 char *buf = malloc(buf_bytes);
 while (true) {
  load_library(copy_library());
  // read events (blocking)
  bool library_changed = false;
  while (! library_changed) {
   memset(buf, 0, buf_bytes);
   ssize_t r = read(ino, buf, buf_bytes);
   if (r == -1)
    sleep(1);
   else {
    char *bufp = buf;
    while (bufp < buf + r) {
     struct inotify_event *ev = (struct inotify_event *) bufp;
     bufp += sizeof(struct inotify_event) + ev->len;
     if (ev->mask & IN_CLOSE_WRITE) {
      if (0 == strcmp("go.so", ev->name))
       // reload
       library_changed = true;
      if (0 == strcmp("go.c", ev->name))
       // recompile
       system("make --quiet");
     }
    }
   }
  }
 }
 close(ino);
 return 0;
}

# 6.3.7 Squash denormals

A common problem in audio processing on x86_64 architectures is the large increased cost of processing denormal floating point numbers when recursive algorithms decay to zero. This is achieved in clive by setting the DAZ (denormals-are-zero) and FTZ (flush-to-zero) CPU flags, and ensuring that the shared library code is compiled with the gcc flag -mfpmath=sse which makes calculations use the SSE unit.

int process(jack_nframes_t n, void *arg) {
#ifdef __x86_64__
 fenv_t fe;
 fegetenv(&fe);
 unsigned int old_mxcsr = fe.__mxcsr;
 fe.__mxcsr |= 0x8040;
 fesetenv(&fe);
#endif
 // ...
#ifdef __x86_64__
 fe.__mxcsr = old_mxcsr;
 fesetenv(&fe);
#endif
 return 0;
}

# 6.4 Example

Thus far in this paper all the code presented has been from the live engine. To remedy this, here is a short callback example, presenting a simple metronome tone:

#include <math.h>
#define SR 48000
#define PI 3.141592653589793
double wrap(double x) {
  return x - floor(x);
}
typedef struct { double phase; } PHASOR;
double phasor(PHASOR *s, double hz) {
  return s->phase = wrap(s->phase + hz / SR);
}
typedef struct {
  int reloaded;
  PHASOR beat;
  PHASOR tone;
} S;
extern void go(S *s, float *in, int nin, float *out, int nout) {
  double beat = phasor(&s->beat, 120 / 60.0) < 0.25;
  double tone = sin(2 * PI * phasor(&s->tone, 1000));
  double metronome = beat * tone;
  for (int c = 0; c < nout; ++c)
    out[c] = metronome;
}

# 6.5 Evaluation and future work

Clive’s code swapping is quantized to JACK process callback block boundaries, which is not musically useful. A workaround can be to scatter the callback code with sample-and-hold unit generators so that the previous values are kept until some future trigger (from an already-running clock oscillator perhaps) synchronizes them with the desired sound. The need in Clive to specify unit generator storage separately from the invocation of the generator makes this doubly painful.

Clive processes one sample at a time, which is not necessarily the most efficient way to process audio. It may be useful to add unit generators to support reblocking, alongside vector operations, to Clive’s library. This reblocking would be necessary if FFT operations were to be supported, which would also benefit from reblocking with overlap.

The Bela platform is interesting for low-latency audio processing. Clive could be adapted to cross-compile into a device directory mounted on the host, with the engine split into two parts (device to reload library code, and host to recompile source code). Bela also supports electronic connections (both analogue and digital pins), which would be nice to expose to Clive callbacks.

The OpenGL graphics library is interesting for programmable shaders. Clive could be adapted to load and compile GLSL source code for live-coding visuals, perhaps focusing on fragment shaders similar to Fragmentarium. A way for Clive’s audio processing callbacks to send control data to OpenGL visualisations would allow tight audio-visual synchronisation.

# 6.6 Conclusion

A live coding system for audio processing in C has been presented. The author has performed with it many times since its first version in 2011, including both seated events and club-style Algorave situations, as well as sessions at home.

# 6.7 References

# 7 Bootable USB

(This is some years old, but might still be useful.)

A bootable Debian Live for live-coding audio in C using Clive.

# 7.1 USB Features

  • non-free firmware for wide hardware support

  • persistence partition to save data between sessions

  • XFCE4 desktop

  • Pipewire audio system

  • prerequisites for Clive

# 7.2 USB Requirements

  • amd64 CPU

  • 1GB RAM

# 7.3 USB Download

Download via BitTorrent:

# 7.4 USB Usage

  • Insert portable USB flash drive into computer.

  • Turn on computer.

  • You may need to change boot device order in BIOS or EFI settings.

  • At USB’s boot menu just press return.

  • Wait until the desktop loads; login is automatic.

  • If screen-saver locks desktop, username is user, password is live.

# 7.5 USB Creation

To write the ISO to a portable USB flash drive (minimum 2GB):

git clone https://code.mathr.co.uk/clive-usb.git
cd clive-usb
sudo ./create-usb.sh /path/to/live-image-amd64.hybrid.iso /dev/sdX

MAKE SURE TO USE THE CORRECT DEVICE INSTEAD OF /dev/sdX. PICKING THE WRONG DRIVE CAN WIPE OUT YOUR DATA

# 7.6 USB Building

On Debian Bullseye with live-build and cryptsetup installed:

git clone https://code.mathr.co.uk/clive-usb.git
cd clive-usb
./rebuild.sh

Building needs about 10GB of space.

# 7.7 USB Testing

Create a 4GB USB key image and an 8GB hard drive image.

qemu-img create -f raw hda.img 8G
sudo ./create-usb.sh live-image-amd64.hybrid.iso
sudo chown myuser:mygroup clive-workshop.img
./emulate.sh

If the screen is blank in XFCE4, try switching to a virtual terminal in the QEMU monitor:

sendkey ctrl-alt-f1

and run

xfconf-query -c xfwm4 -p /general/vblank_mode -s xpresent
sudo service lightdm restart