JACK Design Documentation

Basic constraints that affect audio application design come from the audio hardware level. At some point in time, the soundcard is started. After this, the audio application must transfer audio data to the device at a constant speed in order to avoid buffer underruns and overflows. This behaviour is sometimes described as running at real-time speed. Some headroom is available, as all soundcards have room for at least a few audio blocks. Here the term block refers to the amount of data processed per one hardware interrupt. The soundcard’s driver software usually implements another layer of buffering. But every time we add buffering, independently from the implementation level, we are also adding delays to the signal path.

But this headroom is not necessarily enough. In modern, pre-emptive operating systems, a normal user application doesn’t have exclusive access to hardware resources. This means that the application gets to run at certain, not necessarily even, intervals. From audio application’s point of view, it’s critical that it gets a chance to communicate with the audio device often enough. Like all applications with real-time restrictions, audio applications can’t do much transfer in advance, and they can’t lag much behind.

Most operating systems that implement the POSIX [posix1003.1] specification provide means to run processes with so-called real-time scheduling. In practice this means that once a process gets to run, it can keep the processor until either it is blocked by an I/O request, it loses the processor to a higher priority real-time process, a hardware interrupt occurs, or it willingly gives the processor up.

One very efficient way of implementing audio applications is having one real-time scheduled thread or a process that handles all soundcard I/O. But even with this approach, there are a number of pitfalls. Allocating memory with standard system services is not a deterministic operation. In other words it’s not predictable how much one allocation takes time as it’s not a direct function of the amount of requested memory.

The same applies to many system services such as accessing files, network devices, and so on. Implementing the audio thread needs very careful planning. When the audio thread needs to communicate with other software modules, various nonblocking techniques like read-writer locks [readwriterlocks] are required. Most of these rely on atomic operations and shared memory.

Modularization is a central concept in computer science. It’s easier to handle complex problems by dividing them first into smaller components. Small components have also proved to be powerful building blocks. Good examples of this are the standard UNIX text processing tools. They are simple, yet powerful.

The same approach should also be useful for processing audio data. In practice, however, this has proved to be very difficult to implement. There are numerous audio servers and application frameworks available, but nearly all them have serious problems when it comes to latency and bandwidth considerations. The basic problem is that all common mechanisms for process-to-process communication (IPC) involve possibly blocking system calls. This is especially true for network operations. These problems can be avoided to some degree by increasing the buffersizes. But this will also increase the signal path delay. And the more bandwidth is needed, the more restricting this limitation becomes.

For instance, if some client application offers its user a possiblity to interact in real-time with the audio generation process, the delay in audio output must not reach 5ms. After this limit, some people start noticing a delay between actions and the resulting audio. If the processed signal is to be combined with the original signal, a comb filter effect will begin to be heard at latencies above about 3 ms. In summary, the scheduling requirements for a low-latency system are quite tight, which requires a solid design from the beginning.

Now if we have multiple processes (programs) producing audio data, and one process that is handling all communication with the audio hardware, then we need some form of inter-process communication (IPC). There are however two possible problems with this approach: process switching overhead and performance problems in IPC mechanisms.

Let’s say that we have 5 processes producing audio, and one of them handles the audio hardware I/O. To avoid buffer underruns, during one hardware interrupt cycle, all 5 processes must have enough processor time to produce the next block of audio data. The more processes are involved, the higher the process switching overhead becomes. It also becomes increasingly difficult to adjust the relative priorities of different processes. If only one process fails to produce its audio fragment, a buffer underrun can occur.

In the above it is assumed that transferring data between processes doesn’t have any extra cost. This is of course not true. All IPC mechanisms cause some overhead, although some of them (especially shared memory) are very efficient. Here the cost comes primarily from process synchronation.

One way to avoid the possible IPC troubles is to locate all audio producing code into the audio engine process (audio engine refers to the process, or more specifically the thread, responsible for audio hardware i/o). Clients could be loaded as plugins into the engine. However, this approach has its own problems.

If we choose to go with all the multi-process approach, where all clients are out-of-process, we have another problem to contend with. Each client will be told by the engine when it is time to operate on its data. If the client then “returns” to the engine after every process cycle, many context switches will occur, causing increased latency or possible dropouts. Clients will have to be notified of shared memory locations, but this is not a big problem. The largest problem is how to communicate between the server and the client. There are two choices on a POSIX system, signals and polling on UNIX sockets.

The biggest problem of this approach is the increased client side complexity. Client applications must be divided into realtime critical (audio producing/consuming) and non-realtime parts (user interfaces, network and file i/o, etc). The critical parts are loaded to the server process as plugins, while the non-critical part run in a separate lower priority process. Some kind of IPC is also needed between the realtime and non-realtime parts of the client. To make things even more difficult, care must be taken that this communication never blocks the realtime critical process.

Another interesting question is how different types of applications can take advantage of this plugin-based approach.

jackd.c

driver.h

The jack_process() callback provided by each client is required to be real-time safe. In other words its code must be deterministic and not involve functions that might block for a long time. These functions are for example malloc, free, printf, pthread_mutex_lock, sleep, wait, poll, select, pthread_join, and pthread_cond_wait.

As the jack_process() must not contain any blocking function calls or other non-deterministic code, you have to be extra careful when implementing communication between your jack_process() callback and other application threads.

For a practical example, see the the source file capture_client.c that is distributed along with the JACK package. It contains one example of non-blocking communication.