#### [Ollin](../../index.md) → [Documentation](../index.md) → [Helpers](index.md) → `Audio`
---
## Audio
This library lets a sketch react to sound, and it can also produce a small sound of its own. Audio lives in a separate library so the drawing core stays free of `AVFoundation`. Add `import OllinAudio` beside `import Ollin` to use it.
There are two sides, **analysis** and **generation**. Analysis turns a stream of audio into a few values you read in `draw()`. The stream can be the microphone, a file, or a generated tone. The values are an overall `amplitude`, a frequency `spectrum`, the band queries `bass`/`mid`/`treble`, beats, and the note being sung or played (`pitch`, `nearestPitch`, `chroma`). Generation is a small oscillator, `Tone`, for audible feedback and self-contained demos. Both sides are a convenience layer over one typed core, the [`AudioAnalyzer`](#audioanalyzer).
The usual pattern is to create a source in `setup()`, keep it in a property, and read its values in `draw()`.
```swift
import Ollin
import OllinAudio
final class Pulse: Sketch {
let mic = AudioInput()
override func setup() { try? mic.start() }
override func draw() {
background(.black)
let r = (60 + Double(mic.amplitude) * 700) * scale
fill(.white)
drawCircle(width / 2, height / 2, r)
}
}
```
### Contents
- [AudioInput](#audioinput) - analyze the live microphone
- [AudioPlayer](#audioplayer) - play and analyze an audio file
- [Tone](#tone) - generate (and analyze) an oscillator
- [Soundtrack](#soundtrack) - analyze the sound of a tappable source (a playing video)
- [Reading audio](#reading-audio) - `amplitude`, `spectrum`, `waveform`, and band queries
- [bands & beats](#bands-and-beats) - the ready-to-draw spectrum, and onset detection
- [Pitch and pitch classes](#pitch) - the note heard (`pitch`, `nearestPitch`) and the twelve classes (`chroma`)
- [AudioAnalyzer](#audioanalyzer) - the typed DSP core every source feeds
### AudioInput
```swift
AudioInput(fftSize: Int = 1024, smoothing: Float = 0.8)
func start() throws
func stop()
var isRunning: Bool // true while the mic is capturing
```
`AudioInput` captures live audio from the system's default input. Call `start()` once, usually in `setup()`, then read the [audio values](#reading-audio) in `draw()`.
```swift
let mic = AudioInput()
override func setup() { try? mic.start() }
```
Capturing the microphone needs the user's permission, and `start()` requests it the first time it runs. Until the user grants it, the level reads as silence. `start()` throws for what it can see, an audio engine that will not start once permission is granted; a microphone that is refused, denied or restricted, or an engine that fails after the prompt has returned, is written to `unavailableReason` instead, a sentence worth drawing that is `nil` while nothing has gone wrong. `Tone.play()`, `AudioPlayer.play()` and `Synth.start()` carry the same property for an engine that will not start, since the first note calls the synth's start for you and has nobody to throw to; each is cleared by the next start that succeeds. A packaged app must include a microphone-usage description. Under `swift run`, the system prompts on first use instead.
### AudioPlayer
```swift
AudioPlayer(path: String, fftSize: Int = 1024, smoothing: Float = 0.8) throws // AudioError.resourceNotFound / .couldNotDecode
AudioPlayer(url: URL, …) throws
AudioPlayer(resource: String, withExtension: String, in: Bundle, …) throws
func play()
func pause()
func stop()
var loops: Bool
var isPlaying: Bool
```
`AudioPlayer` plays an audio file and analyzes it as it sounds. That lets you react to recorded music or a field recording the same way you react to the microphone. It decodes the formats AVFoundation reads (`.m4a`/AAC, `.mp3`, `.wav`, `.aiff`, `.caf`, …). The file is read a stretch at a time until it ends. A header that promises more than the file holds costs only what the file holds. A file whose sample rate is not one sound plays at (under 1 Hz, past 1,536 kHz, or not a number) throws `.couldNotDecode`, and so does a file that declares more than 1,024 channels.
```swift
let song = try AudioPlayer(path: "/path/to/track.m4a")
override func setup() { song.loops = true; song.play() }
override func draw() {
for (i, m) in song.spectrum.prefix(64).enumerated() {
let h = Double(m) * 4000 * scale
drawRect(Double(i) * 16 * scale, height - h, 14 * scale, h)
}
}
```
A bundled audio file follows the same provenance rules as any asset, so use one whose license permits redistribution, and credit it. Loading from a resource cannot default the bundle to `.module`, because that would resolve to Ollin's bundle rather than yours. Pass your bundle explicitly.
Under a headless export ([Export](../output/export.md)) nothing audibly plays, so the player follows the export clock instead of the live engine. Each exported frame advances a sample playhead through the decoded file and feeds that slice to the analyzer. Exported frame `k` reads the file's analysis at `k / fps` seconds after `play()`, wrapped by `loops`. The result is identical on every run, so two exports of an audio-reactive sketch produce the same video. Create the player by the end of `setup()`, usually as a stored property, or the per-frame advance never finds it.
### Tone
```swift
Tone(frequency: Double = 440, amplitude: Double = 0.3,
waveform: Waveform = .sine, fftSize: Int = 1024, smoothing: Float = 0.5)
func play()
func stop()
var frequency: Double // Hz, settable live
var amplitude: Double // 0...1, settable live
var waveform: Waveform // .sine, .triangle, .sawtooth, .square
var isPlaying: Bool // true while the tone is sounding
```
`Tone` is a single oscillator that synthesizes a tone you can hear. It feeds the same analyzer, so you can also read the tone back. It is enough for audible feedback and self-contained audio-reactive demos, but it is not a full synthesizer. `frequency`, `amplitude`, and `waveform` are all settable from `draw()`. Its `smoothing` defaults to 0.5 rather than the input sources' 0.8, because a generated tone carries no room noise to damp. The lighter setting costs nothing in steadiness. It lets the readings follow the sketch's own level changes in about 90 ms instead of about 250 ms.
```swift
let tone = Tone(frequency: 220, waveform: .sine)
override func setup() { tone.play() }
override func draw() {
tone.frequency = 110 + 440 * (mouseX / width) // pitch follows the cursor
}
```
### Soundtrack
```swift
Soundtrack(of: any AudioTapSource, fftSize: Int = 1024, smoothing: Float = 0.8)
func detach()
```
`Soundtrack` analyzes the sound of any *tappable* source. Today that means a playing video, because `VideoPlayer` (from [`OllinVideo`](../video/video.md)) conforms to the core `AudioTapSource` seam. A sketch can therefore react to the soundtrack of the footage it is drawing:
```swift
let player = try VideoPlayer(path: "/path/to/clip.mp4")
lazy var sound = Soundtrack(of: player)
override func setup() { player.play() }
override func draw() {
drawFrame(player)
for (i, level) in sound.bands(32).enumerated() { … } // bars over the footage
}
```
The analysis hears the source's own sound, before volume shaping, so `volume = 0` keeps it reacting in silence. Two caveats apply. First, a hard `isMuted = true` stops the source's audio processing altogether, so prefer `volume = 0`. Second, a headless export reads as silence, because nothing audibly plays there. When you need an analysis that stays deterministic under export, use [`AudioPlayer`](#audioplayer), which decodes its own file. There is one consumer per source, so creating a second `Soundtrack` of the same player replaces the first. Call `detach()` to release the slot.
### Reading audio
Every source is an `AudioSource`, and they all expose the same read surface, which forwards to its [`AudioAnalyzer`](#audioanalyzer):
```swift
var amplitude: Float // smoothed overall loudness, ~0...1
var spectrum: [Float] // per-bin magnitudes, low frequency first
var waveform: [Float] // the last fftSize samples, oldest first, -1...1
var bass: Float // energy in 20–250 Hz
var mid: Float // energy in 250–2000 Hz
var treble: Float // energy in 2000–8000 Hz
func magnitude(in range: ClosedRange) -> Float // average over a Hz range
var smoothing: Float // response damping, 0...1
var beatSensitivity: Float // how far the flux must rise to count as a beat (1.5)
let binCount: Int // frequency bins in `spectrum`: half the FFT size
```
`spectrum` has `fftSize / 2` bins. Each bin spans `sampleRate / fftSize` Hz, from 0 up toward the Nyquist frequency. The magnitudes are smoothed but not normalized, so scale them as you like for drawing. You can also use [`bands`](#bands-and-beats) below, which does that shaping for you. `waveform` is a rolling window of the most recent `fftSize` samples, oldest first. That means a scope trace drawn from it stays continuous, no matter how the audio arrives in chunks. `smoothing` trades responsiveness for steadiness: 0 is raw and jittery, and a value near 1 is heavily damped. You can change it live.
```swift
let lows = tone.bass
let kick = tone.magnitude(in: 40...120) // a tighter band
```
### bands & beats
Two higher-level reads sit on top of the raw spectrum, and they are the ones you usually want for audio-reactive visuals.
```swift
func bands(_ count: Int) -> [Float] // normalized, log-spaced frequency bars (each ~0...1)
var beatCount: Int // beats detected so far
var timeSinceBeat: Double // seconds of audio since the last beat
var beat: Float // a 0...1 pulse that hits 1 on a beat and decays
var beatSensitivity: Float // onset threshold; higher = fewer beats (default 1.5)
```
**`bands(count)`** is the spectrum shaped for drawing. It gives `count` bars spread *logarithmically*, which is the way pitch is heard, so low notes are not crammed into a few bins. Each bar is normalized to roughly `0...1` by an adaptive gain and smoothed with a fast-attack / slow-release envelope. Turning a level into a bar height is then only a scale to pixels, with no hand-tuned gain. Call it once per frame with a fixed `count`.
```swift
for (i, level) in source.bands(48).enumerated() {
let h = Double(level) * 300 * scale // level is already 0...1
drawRect(Double(i) * w, height - h, w * 0.8, h)
}
```
**Beats** come from spectral-flux onset detection. An onset is a sudden broadband rise, like a drum hit or a plucked string. The easiest use is `beat`, a ready-made pulse:
```swift
drawCircle(width / 2, height / 2, (40 + Double(source.beat) * 200) * scale) // throbs on the beat
```
You can also fire something exactly once per beat by watching `beatCount`:
```swift
if source.beatCount > lastBeat { lastBeat = source.beatCount; spawnRipple() }
```
Raise `beatSensitivity` if it triggers too often, and lower it if it misses beats. The threshold is an absolute margin over the flux's own recent average, on a loudness-invariant scale. Because of that, quiet and loud material behave alike, and steady material (a held chord, a drone) does not produce false triggers. Beats are gated by a short refractory period, so a single hit does not fire twice.
The whole beat surface runs on the *sample clock*, so positions are counted in samples of audio. That means the same recording always beats at the same places, `timeSinceBeat` does not advance while no audio arrives, and detection is testable without hardware.
### Pitch and pitch classes
```swift
var pitch: DetectedPitch? // the note heard, or nil when there is none
var nearestPitch: Pitch? // its nearest note, the short form of pitch?.nearestPitch
var chroma: [Float] // twelve pitch classes, C first, each 0...1, octaves folded
```
**`pitch`** follows the note being sung or played. It is nil when the window is silent or nothing in it repeats: noise, a room, a chord with no common period. That is why it reads with `if let`. When a note is heard, the `DetectedPitch` carries five values. `frequency` is the fundamental in Hz. `confidence` runs from 0 to 1: a pure tone reads 1, a sung or bowed note above 0.9, and noise never reaches 0.5, which is where reporting stops. `nearestPitch` is the nearest equal-tempered note, with A4 at 440 Hz. `cents` is the offset from that note, `-50...50`. `midi` is `nearestPitch.midi + cents / 100`. Map that onto a position, since equal steps of it are equal steps of pitch. The detector is YIN, the 2002 method of de Cheveigné and Kawahara. It finds the shortest delay after which the waveform repeats. A note with harmonics therefore reads at its fundamental even when the fundamental is the quiet part, or missing. It follows one note at a time, from about 40 Hz to 5 kHz. The window is about 50 ms, so a new note is heard that much after it starts. The reading is not smoothed: a held note holds steady on its own, and a sketch that wants a slow needle eases toward `midi` itself.
```swift
if let heard = mic.pitch {
drawText("\(heard.nearestPitch)", width / 2, 80 * scale) // "A4"
let y = map(heard.midi, 48, 84, height, 0) // pitch as a height
drawCircle(width / 2, y, 30 * scale)
}
```
**`chroma`** is the twelve pitch classes of the window, C first and B last, each `0...1` with the strongest at 1. Every octave is folded onto the same twelve. A chord shows its notes whatever octave they sound in, which is what `pitch` cannot do. A silent window reads all zeros. It follows the pitch class profile Fujishima described in 1999, read off the spectrum's peaks the way Gómez's 2006 harmonic profile does. A note therefore lands in one class rather than leaking into its neighbors. It is not smoothed either, so ease toward it in the sketch for a steady bar.
```swift
for (i, level) in source.chroma.enumerated() { // C first
let h = Double(level) * 200 * scale
drawRect(Double(i) * 40 * scale, height - h, 36 * scale, h)
}
```
The pitch side costs about a million multiplications a window. The analyzer therefore works it out only when a sketch reads one of the three, once per window. That happens on the thread that reads, never on the audio thread. Reading all three in one frame costs one pass. The **Tuner** example (`Examples/Audio/Tuner`) is all three on the bundled violin, with `--mic` to sing at it, and **GuitarTuner** (`Examples/Audio/GuitarTuner`) reads `midi` against the six open strings, the nearest lit and the needle on that string's own pitch.
### AudioAnalyzer
`AudioAnalyzer` is the DSP behind every source. It windows incoming samples, runs a real FFT (Accelerate / vDSP), and publishes the values above. You rarely construct one directly, because each source owns its own and exposes it as `.analyzer`. It is public for two reasons. The read surface and `smoothing` stay reachable, and you can feed it samples from a source of your own:
```swift
let analyzer = AudioAnalyzer(fftSize: 2048, sampleRate: 48000)
analyzer.analyze(samples: ptr, count: n) // or analyze(_ buffer: AVAudioPCMBuffer)
```
Samples arrive on the audio thread while a sketch reads on the main thread. The analyzer is internally locked, so the reads are safe from anywhere.
---
See the **Spectrum** example (`Examples/Audio/Spectrum`) for a self-contained sketch that generates a sound and then analyzes it. There a `Tone` glides in pitch, and its harmonics drive a ring of spectrum bars.
