Ollin

Audio

This library lets a sketch react to sound, and it can also produce a small sound of its own. Audio lives in a separate library so the drawing core stays free of AVFoundation. Add import OllinAudio beside import Ollin to use it.

There are two sides, analysis and generation. Analysis turns a stream of audio into a few values you read in draw(). The stream can be the microphone, a file, or a generated tone. The values are an overall amplitude, a frequency spectrum, the band queries bass/mid/treble, beats, and the note being sung or played (pitch, nearestPitch, chroma). Generation is a small oscillator, Tone, for audible feedback and self-contained demos. Both sides are a convenience layer over one typed core, the AudioAnalyzer.

The usual pattern is to create a source in setup(), keep it in a property, and read its values in draw().

import Ollin
import OllinAudio

final class Pulse: Sketch {
    let mic = AudioInput()

    override func setup() { try? mic.start() }

    override func draw() {
        background(.black)
        let r = (60 + Double(mic.amplitude) * 700) * scale
        fill(.white)
        drawCircle(width / 2, height / 2, r)
    }
}

Contents

AudioInput

AudioInput(fftSize: Int = 1024, smoothing: Float = 0.8)
func start() throws
func stop()
var isRunning: Bool        // true while the mic is capturing

AudioInput captures live audio from the system's default input. Call start() once, usually in setup(), then read the audio values in draw().

let mic = AudioInput()
override func setup() { try? mic.start() }

Capturing the microphone needs the user's permission, and start() requests it the first time it runs. Until the user grants it, the level reads as silence. start() throws for what it can see, an audio engine that will not start once permission is granted; a microphone that is refused, denied or restricted, or an engine that fails after the prompt has returned, is written to unavailableReason instead, a sentence worth drawing that is nil while nothing has gone wrong. Tone.play(), AudioPlayer.play() and Synth.start() carry the same property for an engine that will not start, since the first note calls the synth's start for you and has nobody to throw to; each is cleared by the next start that succeeds. A packaged app must include a microphone-usage description. Under swift run, the system prompts on first use instead.

AudioPlayer

AudioPlayer(path: String, fftSize: Int = 1024, smoothing: Float = 0.8) throws   // AudioError.resourceNotFound / .couldNotDecode
AudioPlayer(url: URL, …) throws
AudioPlayer(resource: String, withExtension: String, in: Bundle, …) throws
func play()
func pause()
func stop()
var loops: Bool
var isPlaying: Bool

AudioPlayer plays an audio file and analyzes it as it sounds. That lets you react to recorded music or a field recording the same way you react to the microphone. It decodes the formats AVFoundation reads (.m4a/AAC, .mp3, .wav, .aiff, .caf, …). The file is read a stretch at a time until it ends. A header that promises more than the file holds costs only what the file holds. A file whose sample rate is not one sound plays at (under 1 Hz, past 1,536 kHz, or not a number) throws .couldNotDecode, and so does a file that declares more than 1,024 channels.

let song = try AudioPlayer(path: "/path/to/track.m4a")
override func setup() { song.loops = true; song.play() }
override func draw() {
    for (i, m) in song.spectrum.prefix(64).enumerated() {
        let h = Double(m) * 4000 * scale
        drawRect(Double(i) * 16 * scale, height - h, 14 * scale, h)
    }
}

A bundled audio file follows the same provenance rules as any asset, so use one whose license permits redistribution, and credit it. Loading from a resource cannot default the bundle to .module, because that would resolve to Ollin's bundle rather than yours. Pass your bundle explicitly.

Under a headless export (Export) nothing audibly plays, so the player follows the export clock instead of the live engine. Each exported frame advances a sample playhead through the decoded file and feeds that slice to the analyzer. Exported frame k reads the file's analysis at k / fps seconds after play(), wrapped by loops. The result is identical on every run, so two exports of an audio-reactive sketch produce the same video. Create the player by the end of setup(), usually as a stored property, or the per-frame advance never finds it.

Tone

Tone(frequency: Double = 440, amplitude: Double = 0.3,
     waveform: Waveform = .sine, fftSize: Int = 1024, smoothing: Float = 0.5)
func play()
func stop()
var frequency: Double      // Hz, settable live
var amplitude: Double      // 0...1, settable live
var waveform: Waveform     // .sine, .triangle, .sawtooth, .square
var isPlaying: Bool        // true while the tone is sounding

Tone is a single oscillator that synthesizes a tone you can hear. It feeds the same analyzer, so you can also read the tone back. It is enough for audible feedback and self-contained audio-reactive demos, but it is not a full synthesizer. frequency, amplitude, and waveform are all settable from draw(). Its smoothing defaults to 0.5 rather than the input sources' 0.8, because a generated tone carries no room noise to damp. The lighter setting costs nothing in steadiness. It lets the readings follow the sketch's own level changes in about 90 ms instead of about 250 ms.

let tone = Tone(frequency: 220, waveform: .sine)
override func setup() { tone.play() }
override func draw() {
    tone.frequency = 110 + 440 * (mouseX / width)   // pitch follows the cursor
}

Soundtrack

Soundtrack(of: any AudioTapSource, fftSize: Int = 1024, smoothing: Float = 0.8)
func detach()

Soundtrack analyzes the sound of any tappable source. Today that means a playing video, because VideoPlayer (from OllinVideo) conforms to the core AudioTapSource seam. A sketch can therefore react to the soundtrack of the footage it is drawing:

let player = try VideoPlayer(path: "/path/to/clip.mp4")
lazy var sound = Soundtrack(of: player)
override func setup() { player.play() }
override func draw() {
    drawFrame(player)
    for (i, level) in sound.bands(32).enumerated() { … }   // bars over the footage
}

The analysis hears the source's own sound, before volume shaping, so volume = 0 keeps it reacting in silence. Two caveats apply. First, a hard isMuted = true stops the source's audio processing altogether, so prefer volume = 0. Second, a headless export reads as silence, because nothing audibly plays there. When you need an analysis that stays deterministic under export, use AudioPlayer, which decodes its own file. There is one consumer per source, so creating a second Soundtrack of the same player replaces the first. Call detach() to release the slot.

Reading audio

Every source is an AudioSource, and they all expose the same read surface, which forwards to its AudioAnalyzer:

var amplitude: Float                       // smoothed overall loudness, ~0...1
var spectrum: [Float]                       // per-bin magnitudes, low frequency first
var waveform: [Float]                       // the last fftSize samples, oldest first, -1...1
var bass: Float                             // energy in 20–250 Hz
var mid: Float                              // energy in 250–2000 Hz
var treble: Float                           // energy in 2000–8000 Hz
func magnitude(in range: ClosedRange<Double>) -> Float   // average over a Hz range
var smoothing: Float                        // response damping, 0...1
var beatSensitivity: Float                  // how far the flux must rise to count as a beat (1.5)
let binCount: Int                           // frequency bins in `spectrum`: half the FFT size

spectrum has fftSize / 2 bins. Each bin spans sampleRate / fftSize Hz, from 0 up toward the Nyquist frequency. The magnitudes are smoothed but not normalized, so scale them as you like for drawing. You can also use bands below, which does that shaping for you. waveform is a rolling window of the most recent fftSize samples, oldest first. That means a scope trace drawn from it stays continuous, no matter how the audio arrives in chunks. smoothing trades responsiveness for steadiness: 0 is raw and jittery, and a value near 1 is heavily damped. You can change it live.

let lows = tone.bass
let kick = tone.magnitude(in: 40...120)     // a tighter band
Three stacked panels from one analyzed instant: the raw waveform wiggle, the spectrum with spikes marked at the kick, bass, and melody frequencies, and 24 normalized band bars

bands & beats

Two higher-level reads sit on top of the raw spectrum, and they are the ones you usually want for audio-reactive visuals.

func bands(_ count: Int) -> [Float]   // normalized, log-spaced frequency bars (each ~0...1)
var beatCount: Int                     // beats detected so far
var timeSinceBeat: Double              // seconds of audio since the last beat
var beat: Float                        // a 0...1 pulse that hits 1 on a beat and decays
var beatSensitivity: Float             // onset threshold; higher = fewer beats (default 1.5)

bands(count) is the spectrum shaped for drawing. It gives count bars spread logarithmically, which is the way pitch is heard, so low notes are not crammed into a few bins. Each bar is normalized to roughly 0...1 by an adaptive gain and smoothed with a fast-attack / slow-release envelope. Turning a level into a bar height is then only a scale to pixels, with no hand-tuned gain. Call it once per frame with a fixed count.

for (i, level) in source.bands(48).enumerated() {
    let h = Double(level) * 300 * scale          // level is already 0...1
    drawRect(Double(i) * w, height - h, w * 0.8, h)
}

Beats come from spectral-flux onset detection. An onset is a sudden broadband rise, like a drum hit or a plucked string. The easiest use is beat, a ready-made pulse:

drawCircle(width / 2, height / 2, (40 + Double(source.beat) * 200) * scale)   // throbs on the beat

You can also fire something exactly once per beat by watching beatCount:

if source.beatCount > lastBeat { lastBeat = source.beatCount; spawnRipple() }

Raise beatSensitivity if it triggers too often, and lower it if it misses beats. The threshold is an absolute margin over the flux's own recent average, on a loudness-invariant scale. Because of that, quiet and loud material behave alike, and steady material (a held chord, a drone) does not produce false triggers. Beats are gated by a short refractory period, so a single hit does not fire twice.

The whole beat surface runs on the sample clock, so positions are counted in samples of audio. That means the same recording always beats at the same places, timeSinceBeat does not advance while no audio arrives, and detection is testable without hardware.

A six-second timeline in three strips: the loudness curve with regular peaks, the beat pulse snapping to one and decaying at each detection, and tick marks where beatCount incremented

Pitch and pitch classes

var pitch: DetectedPitch?      // the note heard, or nil when there is none
var nearestPitch: Pitch?      // its nearest note, the short form of pitch?.nearestPitch
var chroma: [Float]            // twelve pitch classes, C first, each 0...1, octaves folded

pitch follows the note being sung or played. It is nil when the window is silent or nothing in it repeats: noise, a room, a chord with no common period. That is why it reads with if let. When a note is heard, the DetectedPitch carries five values. frequency is the fundamental in Hz. confidence runs from 0 to 1: a pure tone reads 1, a sung or bowed note above 0.9, and noise never reaches 0.5, which is where reporting stops. nearestPitch is the nearest equal-tempered note, with A4 at 440 Hz. cents is the offset from that note, -50...50. midi is nearestPitch.midi + cents / 100. Map that onto a position, since equal steps of it are equal steps of pitch. The detector is YIN, the 2002 method of de Cheveigné and Kawahara. It finds the shortest delay after which the waveform repeats. A note with harmonics therefore reads at its fundamental even when the fundamental is the quiet part, or missing. It follows one note at a time, from about 40 Hz to 5 kHz. The window is about 50 ms, so a new note is heard that much after it starts. The reading is not smoothed: a held note holds steady on its own, and a sketch that wants a slow needle eases toward midi itself.

if let heard = mic.pitch {
    drawText("\(heard.nearestPitch)", width / 2, 80 * scale)         // "A4"
    let y = map(heard.midi, 48, 84, height, 0)               // pitch as a height
    drawCircle(width / 2, y, 30 * scale)
}

chroma is the twelve pitch classes of the window, C first and B last, each 0...1 with the strongest at 1. Every octave is folded onto the same twelve. A chord shows its notes whatever octave they sound in, which is what pitch cannot do. A silent window reads all zeros. It follows the pitch class profile Fujishima described in 1999, read off the spectrum's peaks the way Gómez's 2006 harmonic profile does. A note therefore lands in one class rather than leaking into its neighbors. It is not smoothed either, so ease toward it in the sketch for a steady bar.

for (i, level) in source.chroma.enumerated() {          // C first
    let h = Double(level) * 200 * scale
    drawRect(Double(i) * 40 * scale, height - h, 36 * scale, h)
}

The pitch side costs about a million multiplications a window. The analyzer therefore works it out only when a sketch reads one of the three, once per window. That happens on the thread that reads, never on the audio thread. Reading all three in one frame costs one pass. The Tuner example (Examples/Audio/Tuner) is all three on the bundled violin, with --mic to sing at it, and GuitarTuner (Examples/Audio/GuitarTuner) reads midi against the six open strings, the nearest lit and the needle on that string's own pitch.

Three panels from four seconds of a violin recording: the pitch as dots on a strip of semitones, the twelve pitch classes as rows over the same seconds with the melody drawn through the note names, and a tuner face reading the note and cents at one marked instant

AudioAnalyzer

AudioAnalyzer is the DSP behind every source. It windows incoming samples, runs a real FFT (Accelerate / vDSP), and publishes the values above. You rarely construct one directly, because each source owns its own and exposes it as .analyzer. It is public for two reasons. The read surface and smoothing stay reachable, and you can feed it samples from a source of your own:

let analyzer = AudioAnalyzer(fftSize: 2048, sampleRate: 48000)
analyzer.analyze(samples: ptr, count: n)    // or analyze(_ buffer: AVAudioPCMBuffer)

Samples arrive on the audio thread while a sketch reads on the main thread. The analyzer is internally locked, so the reads are safe from anywhere.

See the Spectrum example (Examples/Audio/Spectrum) for a self-contained sketch that generates a sound and then analyzes it. There a Tone glides in pitch, and its harmonics drive a ring of spectrum bars.

↑↓ move ↩ open