Synthesis

This page is about making a sketch play. Audio is the listening side, so it turns sound into values you read in draw(). This page is the other direction, where the sketch makes the sound. Add import OllinAudio alongside import Ollin to reach it.

The whole surface is one instrument. You hold it as a property and ask it for notes.

import Ollin
import OllinAudio

final class Chime: Sketch {
    let synth = Synth(.bell)

    override func mousePressed() {
        synth.play("C5", for: 1.5)
    }

    override func draw() {
        background(.black)
        fill(.white)
        drawCircle(width / 2, height / 2, 100 + Double(synth.amplitude) * 900)
    }
}

Two things in that sketch are worth noticing. Nothing starts the audio, because the first note starts the engine for you. The circle is then sized from synth.amplitude, which works because a Synth is an AudioSource in the way the microphone is. A sketch can read back whatever it is playing.

Contents

• Synth - the instrument, and how you ask for notes
• Expression - one note bent, pressed, or slid on its own
• Pitch - names, numbers, and the pitches between them
• Voice - what a note is made of
• Physical models - a plucked string, a struck shape, a bowed string, and a blown tube
• Patch - an instrument you build rather than pick
• Sampled instruments - an instrument made of recordings, and where to find more
• Wavetables - a row of cycles a note reads by position and moves through
• Grains - a sound cut into pieces and piled back up, with time and pitch apart
• Placing a sound - where it comes from in the 3D scene
• Sound in an export - carrying the music into the exported file
• Envelope - how a note arrives and how it goes
• Voice.Filter - what is taken out of it, and how that moves
• Effects - the chain the sound passes through on its way out
• What is not here yet

Synth

Synth(_ voice: Voice = .pluck, polyphony: Int = 16, fftSize: Int = 1024)

That makes an instrument. polyphony is how many notes may sound at once, tails included.

You ask for a note in one of two ways. If you give it a length, it lets go by itself:

synth.play("C4", for: 0.4)                  // a note that ends on its own
synth.play(60, velocity: 0.4, for: 0.4)     // quieter, by MIDI number
synth.play(chord: [60, 64, 67], for: 1.2)   // three at once

Without a length, it sounds until you let it go, which is what a held key needs:

synth.noteOn("C4")
synth.noteOff("C4")
synth.allNotesOff()     // let go of everything; the tails still sound

For held notes you usually poll rather than wait for an event, the same shape as drag-to-paint:

override func draw() {
    let down = isKeyDown("a")
    if down && !wasDown { synth.noteOn("C4") }
    if !down && wasDown { synth.noteOff("C4") }
    wasDown = down
}

The rest of the surface:

When voices run out, the next note takes one of them. A note that is already fading is taken before a held one. So a melody played over a held chord takes its voices from its own earlier notes rather than from the chord. A voice that is taken gets a few milliseconds to get out of the way, so it does not click.

Notes start on the next block of audio, a few milliseconds after you ask. That comes from the hardware, not from a queue. The wait is shorter than a frame, so a note asked for in draw() lands with the frame that asked for it.

A note can wait. play("C4", for: 0.4, after: 0.03) starts the note 30 milliseconds from now, on its own sample rather than on the next block. Notes asked for together can then be closer than a frame apart. A strum is three notes and three waits, and a sequencer's swing and ratchets are waits worked out from the beat. Zero, the default, lands the note on the next block. In an export the wait becomes the note's place on the clock. Letting everything go with allNotesOff() drops a note still waiting to start.

Expression

@discardableResult func noteOn(_ pitch: Pitch, velocity: Double = 0.8) -> PlayingNote
@discardableResult func play(_ pitch: Pitch, velocity: Double = 0.8, for duration: Double? = nil, after delay: Double = 0) -> PlayingNote
func noteOff(_ note: PlayingNote)

func bend(_ note: PlayingNote, semitones: Double)   // this note's pitch, on top of pitchBend
func press(_ note: PlayingNote, _ pressure: Double)  // 0...1: the drive of a bowed or blown note, the level of any other
func slide(_ note: PlayingNote, _ position: Double)  // 0...1, half way at rest: the filter's cutoff
var pitchBend: Double                                // every note at once, in semitones

A note is told three things while it sounds, and each reaches that note alone. noteOn hands back the note for the purpose. Hold it, and the note can be bent, pressed, slid, and let go, whatever it has been bent to since.

let note = synth.noteOn("C4")
synth.bend(note, semitones: 1.5)     // a semitone and a half up
synth.press(note, 0.7)
synth.slide(note, 0.9)
synth.noteOff(note)

Each value glides over a few milliseconds, so calling these every frame with a moving value moves the note smoothly. A value repeated is sent once, so a sketch that hands every held note its three values each frame costs the render thread nothing.

pitchBend on the instrument bends every note at once, the way a keyboard's wheel does. It adds to whatever each note is bent by on its own.

A note that is never bent, pressed, or slid sounds exactly as it did before, so nothing here changes an instrument that is not asked. noteOff(_ pitch:) still finds a bent note by the pitch it started on.

This is what a polyphonic-expression controller sends. Each note is on a channel of its own, with its bend, its pressure, and where the finger sits along the key. MIDIInput.heldNotes reads those, and the wiring from there to these three calls is a dozen lines, shown on that page. The Expression example (Examples/Audio/Expression) is a surface the mouse plays the same way.

Pitch

You write a pitch however you already think of one:

synth.play(60)                     // MIDI number
synth.play("C4")                   // note name, sharps or flats, any case
synth.play(60.5)                   // a quarter tone above middle C
synth.play(Pitch(frequency: 440))

Pitch.midi is a Double, so a pitch can sit between the keys. Bends, glides, and tunings outside the twelve equal steps all need that. frequency is derived from midi, so transposed(by: 12) always doubles the frequency, whatever it started from.

"C4" is middle C, and a name without an octave lands in that octave. A literal that is not a note name lands on middle C instead. It says so once rather than stopping the sketch. Use Pitch(name:), which returns nil, when you want to be told in code.

Voice

A Voice is what a note is made of. It is a value, so you can build one, copy it, adjust it, and hand it over at any time.

let synth = Synth(.pluck)

var glass = Voice.bell          // start from a preset
glass.envelope.release = 3
glass.detune = 0.5
synth.voice = glass             // notes already sounding are undisturbed

detune is smaller than it looks. A few hundredths of a semitone is the useful range. Two oscillators slightly apart drift in and out of phase, and that beating makes a held note shimmer rather than sit still. A whole semitone gives an interval rather than a shimmer, and that is what .bell uses it for.

waveform and source are the same setting written two ways. Reading waveform on a string voice gives .sine, and setting it makes the voice an oscillator again.

The geometric waves are corrected as they are produced. A sawtooth therefore still sounds like a sawtooth at the top of the keyboard, instead of ringing against itself.

Physical models

A wave is a shape repeated over and over. A physical model works out the thing itself as it goes, so what you hear comes out of the model rather than being dialed in.

Each model carries its own presets, and two of them share a name for the same reason: BowedString.sustained is a light bow a long way along the string, BlownTube.reedy a tube bitten tight, and Envelope.sustained the envelope that stays out of the way of either. There are four models, and they split into two kinds. A plucked string and a struck body are set going once and then left to fade, so the whole note is decided at its start. A bowed string and a blown tube are kept going instead. That note lasts as long as you keep driving it, and it can change while it sounds. Synth.pressure is that driving, and it is the difference this section is really about.

A plucked string

let synth = Synth(.steel)
synth.play("E3", for: 3)

A string is a disturbance running up and down a length of something under tension. It loses a little at each end, and it loses its top faster than its bottom. In code that is a delay line one period long with a filter in the loop. Everything a player recognizes comes out of that loop. It gives the attack, the way a held note darkens, and the difference between plucking near the bridge and over the hole.

var string = PluckedString.steel
string.position = 0.5          // halfway along
synth.voice = Voice(plucked: string)

position is the one that sounds least like a setting. A string held at a point cannot move there. Every harmonic with a node at that point is therefore missing. Plucking at 0.5 loses every even harmonic and comes out hollow. Plucking near the end keeps them all and comes out thin and nasal. A quarter of the way along is roughly where a guitar is played. Examples/Audio/Strings lets you click a string wherever you want to pluck it.

A string decides for itself how a note fades. The envelope only has to stay out of the way. Envelope.plucked is that envelope, and the string presets use it. Ask for a note long enough to let the string finish (for: string.decay), or the envelope's release cuts it off mid-ring.

Tuning is exact. The loop has to come out exactly one period long, and a whole number of samples cannot do that. An allpass filter supplies the fraction left over. The loop filter's own delay is counted in as well, so changing damping cannot move the pitch. Without that, notes go progressively sharper or flatter towards the top of the keyboard. At the top of the range, rounding the loop to whole samples is out by most of a semitone.

Everything else about the voice is unchanged. detune gives a second string slightly apart, and the filter still applies after it. A string is an ordinary Voice, so you can assign it between notes like any other.

A struck body

The second model is something hit rather than plucked. A struck object does not make a wave. It makes a handful of pure tones at once, each fading at its own rate, and its shape decides which tones those are.

let synth = Synth(.chime)
synth.play("C4", for: 4)

ModalBody is the value underneath, and .drum, .plate, .bar, .bell, .wood, and .glass are the ones with names. Its settings:

damping is most of what separates one struck thing from another. At 0 every tone fades together, which is what a bell does. At 1 a tone twice as high fades twice as fast, which is what most things do. Past 2 you hear a knock rather than a note.

A shape you drew, struck

The frequencies can come from geometry instead of a list. A flat shape held at its edge rings at frequencies decided entirely by its outline, and StruckShape works them out:

lazy var outline = textToShapes("O", at: Vector2(540, 540)).first!
lazy var bell = StruckShape(outline)             // built once, on first use

override func mousePressed() {
    synth.voice = Voice(struck: bell!.body(struckAt: Vector2(mouseX, mouseY)))
    synth.play("C4", for: 3)
}

Nothing chooses that sound, because the ratios come from the outline itself:

• A round one rings like a real drumhead, at 1, 1.59, 2.14, 2.30, and so on. Those are the zeros of the Bessel functions.
• A square one gives a square membrane's list, which is 1, 1.58, 2, 2.24.
• An outline nobody has a name for gives whatever its own geometry allows.

Only the ratios come from the shape. You decide the note when you play it, so one outline is an instrument rather than a single sound.

Three things are worth knowing:

• Measuring is the expensive part, and striking is free. Measuring a circle takes a few hundredths of a second in a release build. A sketch run straight from source is unoptimized, and there it takes a second or two. So measure once, in setup(), and keep the StruckShape. ModalBody(shape:) is the one-liner that measures every time, for when the shape never changes.
• Where you strike it decides which tones answer. A tone that holds still under your finger gets nothing. That is the same reason a string plucked in the middle sounds hollow. Strike a circle exactly in the middle, and every tone with a line of stillness through the center goes quiet. That is most of them.
• A symmetric shape rings at some tones twice. A pattern that fits at one rotation fits at another, and both are really there. That is why the circle's list above has each entry twice, apart from the ones with no rotation at all. A real drum does the same. Which of a pair takes a given strike is arbitrary, because they ring at the same frequency. So read their gains together rather than one at a time.

A body carries up to sixteen tones, which is what lets it reach the audio thread without allocating. A tone that would land above half the sample rate is dropped. Folding it back down the spectrum would add something that was never struck.

A bowed string

let synth = Synth(.cello)
synth.noteOn("G2")
synth.pressure = 0.7          // and keep moving it while the note sounds

The string is the same string as before. The difference is that a pluck happens once and a bow keeps happening.

One nonlinearity is what makes it sound bowed. Rosin grips harder when the bow and the string are traveling together than when they are sliding past each other. So the bow catches the string, drags it sideways, tears loose, lets it snap back, and catches it again, hundreds of times a second. That cycle is the tone, and it is why a bowed note comes out close to a sawtooth. The string spends most of each cycle stuck to the bow.

Two things fall out of the model rather than being settings, and both are worth knowing:

• Bow too fast for the force and the tone breaks. The string tears loose twice a cycle instead of once. The note jumps to the octave, which is exactly what over-bowing sounds like on a real instrument. Raise force or lower pressure and it settles back.
• Loudness comes from force as much as from speed. From where a note first speaks to full pressure the level moves by 5 to 10 dB, depending on the instrument. force moves it further. That is also true of a real bow, and it means force is the loudness parameter while pressure is the expressive one.
• At pressure 0 the bow is off the string. Over the bottom tenth of pressure, where no preset sounds yet, the bow lifts away, and at 0 the string rings down over its decay like a plucked one. A pressure that glides down to 0 slows the bow on the string first, which takes some of the ring, the way stopping a bow does.

A blown tube

let synth = Synth(.clarinet)
synth.noteOn("D4")
synth.pressure = 0.8

A column of air in a tube resonates, and a reed at one end keeps feeding it. The same pressure that drives the reed also pushes it shut, and that feedback is what makes the whole thing sound.

The tube is stopped at the reed and open at the far end, and that one fact is most of the sound. A tube closed at one end fits only a quarter of a wave, so it supports the odd harmonics and not the even ones. That is why it is hollow and woody rather than bright. It also sounds an octave and a fifth below an open tube of the same length, instead of an octave below. Nothing in the code decides that the even harmonics should be missing. They are missing because the tube is half as long as an open one.

Driving them

synth.pressure = 0.6 + 0.4 * abs(sin(time * 2))

pressure is 0...1, read once every block of a few milliseconds, and shared by every note the instrument is playing. That is right for one bow and one breath. At zero there is nothing to hear, because nothing is driving the model. A tube also needs a minimum breath before it speaks. Every tube speaks from 0.3, at every pitch and with any reed, and below that it is silent: where a note starts depends on how much its tube loses, so each note's start is worked out from its own loss and placed at 0.3 on the dial. What a tube sends out swings about zero, so a held breath leaves no standing offset to eat into the headroom of a mix. One limit stays: near the top of the range a tube blown hard leaves its note for the one a twelfth above, the way a clarinet overblows, and from about MIDI 87 up the breath that makes a note speak and the breath that tips it over are close together. The sources that are set going once (a wave, a plucked string, a struck body) ignore it entirely. Adding it changed nothing that already worked. A note pressed on its own (press) is driven by that instead, from the first press on, which is what a surface with a finger on every note wants.

This is the control an envelope cannot give you. An envelope is decided when the note starts, while pressure is whatever you are doing right now.

Patch

A Patch is an instrument you build rather than pick.

A Voice is a fixed chain. Something makes a wave, an envelope shapes it, and a filter takes part of it away. That covers a great deal, but you cannot rearrange it. A Patch is the tier underneath, where the routing itself is the value.

let bell = Patch.tone(.sine)
    .modulated(by: .tone(.sine, ratio: 3.5), amount: 4)

synth.voice = Voice(patch: bell, envelope: .percussive)

The relationship is the one the drawing side already has. The bare calls sit on a Drawer that can do more, and the voice presets sit on a Patch the same way. Nothing about Synth(.pluck) changes because Patch exists, and a one operator patch renders the same samples the plain oscillator does.

What an operator is

An operator is one oscillator with a frequency, a level, and possibly something modulating it. Its frequency is a ratio of the note rather than a pitch, so a patch is an instrument rather than a chord. Ratio 1 is the note, and 2 is the octave above. Ratio 3.5 is not a note at all, and that is where metallic sounds come from.

An operator's level means one of two things, depending on where it sits. On an operator that reaches the output it is a mix level. On one that modulates another it is how far it pushes, which is the difference between a slight waver and a bell. That is one number doing the two jobs a level does, and it is the convention rather than a shortcut.

Three of those are wrapped as whole voices to play straight away: Voice.fmBell, Voice.fmBrass, and Voice.fmBuzz.

Why modulation rather than a filter

A filter can only take harmonics away, and a sine has none to take. Modulation puts them in, which is why index turns a plain tone into brass and no amount of filtering will. Small values give a waver. Past about 2 you hear a new instrument rather than the old one wobbling.

Whole ratios stay musical, because their tones land on the note's own harmonics. Anything else gives the inharmonic tones that bells and metal are made of, which is why .bell uses 3.5.

Why it is a fixed size

A patch travels to the audio thread inside a note, through a queue of preallocated slots. It therefore has to be something that can be copied a word at a time, with no arrays, no references, and nothing to allocate. So the operators live in fixed lanes, and there are eight of them. That is the same trade ModalBody makes with its sixteen tones.

Eight is more than most instruments need. A patch that would exceed it comes back unchanged and says so, rather than quietly dropping an operator. A patch with a piece missing is a different instrument, and finding that out by ear is worse than reading a note in the log.

Modulators always end up in an earlier lane than what they push, so one forward pass evaluates the whole patch. That comes from how patches are built rather than from a check afterwards.

Sampled instruments

Everything else a Synth plays is worked out as it goes. A sampled instrument works the other way. Someone recorded the thing, so a note means finding the nearest recording and moving it to the pitch you asked for.

synth.instrument = SampledInstrument.builtIn
synth.voice = Voice(sampled: Sampler(), envelope: .plucked)
synth.play("C4", for: 1.5)

Ollin bundles one small instrument, so you can hear this working without downloading anything. It is a struck bar recorded at five pitches. Scripts/make-sample-instrument.swift generates it rather than sourcing it, so it is Ollin's own and carries nobody's license. It is a demonstration, not a library.

Why the instrument is set separately from the voice

A Voice travels to the audio thread inside a note and has to be copyable a word at a time. That is why a ModalBody caps at sixteen tones (ModalBody.maxModes) and a Patch at eight operators (Patch.maxOperators). Recordings are megabytes on the heap and cannot travel with it. So the recordings live on the Synth and the Voice says only how to play them.

The read head moves through a recording at whatever rate the pitch asks for. That moves its pitch and its length together, exactly as a tape does. That is also the limitation, because moving a recording far enough makes the instrument audibly change size. A real library therefore ships many recordings rather than one, and the nearest is always chosen.

Loading one

let piano = try SampledInstrument(sfz: "Piano.sfz", in: .module)   // bundled with the sketch
let other = try SampledInstrument(contentsOf: url)                 // anywhere on disk

The format is SFZ, a plain text file listing regions. Each region names an audio file and the notes it answers to, with the audio beside it. That region arrives as a SampledInstrument.Recording: the samples, the rootKey it was recorded at, and the lowKey to highKey range it answers over, which defaults to the root note and nothing else. It is the format most freely licensed libraries ship in. The map is read as UTF-8, or as Latin-1 when its bytes are not UTF-8, which is how older libraries spell an accented sample name.

Ollin reads the opcodes that decide which file plays and at what pitch. They are sample, lokey / hikey / key, pitch_keycenter, lovel / hivel, tune, transpose, volume, and the loop points. SFZ has hundreds of others covering filters, envelopes, round robins and modulation.

An unknown opcode is skipped rather than refused, so a library that uses them loads and plays, without the parts this does not model. A region whose audio file is missing costs you that part of the range rather than the whole instrument. The numbers are held to the ranges the format gives them. Keys and velocities run 0 to 127. tune stays within eight octaves and transpose within 127 semitones. volume runs from -144 to 48 dB, and a loop point is never below zero. A value past its range reads as the nearest end, and one that is not a number reads as the default.

Examples/Audio/OwnSampler is an instrument made rather than downloaded. It has three recordings generated beside the sketch and a three-region map, read with SampledInstrument(sfz:in:). Each recording is drawn with its loop region shaded. Examples/Audio/Sampler plays the bundled one.

Stretching a recording

let bar = SampledInstrument.builtIn!
bar.recordingCount                             // how many recordings it holds
bar.recordingRoots                             // the note each was recorded at
bar.recordingIndex(for: 60)                    // which one a note plays
let slow = bar.recording(at: 0, over: 0...127).stretched(by: 4)
synth.instrument = SampledInstrument(recordings: [slow])

A sampler moves pitch and length together, like a tape: a note an octave down lasts twice as long. stretched(by:) is the other axis. The recording comes back factor times as long at the same pitch, so it can be slowed into a drone or hurried into a grace note and still play at the note it was recorded at. The factor runs 0.25...16, and 2 is twice as long. A room stretches the same way, and a stretched room is a bigger room: every echo lands later, every resonance rings longer, and nothing moves in pitch.

It changes the length, so it is a thing done to a recording rather than an effect in the chain. Do it in setup(): it costs about what the recording's own length costs to play. The two effects that do the same work live, without changing the length, are the pitch shift and the freeze.

Where to find instruments

The licenses matter here, so each group below says what its terms are.

Free to use and to redistribute (CC0, public domain). You can bundle these into anything, including something you sell.

Free per item. Freesound is enormous, but every upload carries its own license, which is CC0, CC-BY, or CC-BY-NC. Filter by license and check each clip, because the three are mixed together.

Free to make music with, but not to redistribute. The Philharmonia Orchestra samples are very good. Their terms say they must not be made available "as is", meaning as samples or as a sampler instrument. That is fine for a work you release. Do not ship them inside a sketch you hand to someone else.

The SFZ format site documents the format itself and lists more libraries.

Wavetables

A wavetable is a row of single cycles that a note reads by position. An oscillator traces one shape, and a patch pushes a few into each other. A wavetable holds any shapes side by side, and a note reads the blend of the two its position lands between. Move the position while the note sounds and the wave itself changes shape.

synth.wavetable = .basic                                     // sine, triangle, sawtooth, square
synth.voice = Voice(wavetable: WavetableScan(position: 0.3))
synth.play("C3", for: 2)

synth.voice = .morph            // struck to the far end, settling back
synth.wavetable = .vowels       // five mouth shapes a note sings through

You set the table on the Synth, and the voice says where in it to read. That is the same split a sampled instrument's recordings use, and for the same reason. A voice travels to the audio thread inside a note, so it has to be copyable a word at a time. A table is hundreds of kilobytes. A wavetable voice played with no table set reads .basic.

Three tables are built in, and a table is easy to make:

let bend = Wavetable(name: "bend", frameCount: 8) { phase, frame in
    sin(.tau * pow(phase, 1 + 2 * frame))           // a sine bent harder each frame
}
let odd = Wavetable(harmonics: [[1], [1, 0, 1 / 3, 0, 1 / 5]])

frame(_:) hands back one cycle, and cycle(at:) hands back the blend a position reads, so you can draw what is playing. Each frame is scaled so its loudest point is 1. A bright frame and a plain one therefore play at the same level.

Why a high note reads a softer copy

A cycle with a corner holds harmonics past any sampling limit. Read fast enough, those harmonics fold back down the spectrum as a gritty ring that tracks pitch the wrong way. So every frame is kept at eleven strengths, each with half the harmonics of the one before. A note reads the strongest one whose top harmonic still fits under half the sample rate. That is how a sawtooth stays a sawtooth at the top of the keyboard. Every strength is built from the same harmonics, so nothing shifts when a note moves from one to the next. frame(_:harmonicsUpTo:) shows what a given pitch actually reads.

Grains

A grain is a piece of sound too short to have a pitch of its own, a few thousandths of a second. An envelope shapes it so it does not click at either end. Pile enough of them up and what you hear is the statistics of the pile rather than any one of them. That is the whole technique, and it buys one thing nothing else here can do: how fast the sound is read and how high it sounds stop being the same number.

Everything else that plays a recording moves both together, the way a tape does. A grain cloud has two clocks. The grains are read at whatever speed the note's pitch asks for. The place they are cut from travels at speed, which is its own control. Set speed to 0 and the position stops while the note keeps going: one moment of a sound, held, for as long as you like.

synth.grainSource = try GrainSource(contentsOf: url)
synth.voice = Voice(granular: GrainCloud(size: 0.08, density: 40, speed: 0))
synth.play("C4", for: 8)                            // that moment, held

synth.grainScrub = mouseX / width                   // and dragged through by hand

You set the sound on the Synth, and the voice says how to cut it up. That is the same split a sampled instrument and a wavetable use, and for the same reason: a voice travels to the audio thread inside a note and has to be copyable a word at a time, where a few seconds of sound is hundreds of kilobytes. A cloud played with no sound set reads GrainSource.builtIn.

Three presets are worth starting from. Voice.cloud is a held moment spread wide, Voice.smear is the sound crawling past at a fraction of its speed in pieces, and Voice.rain is short sharp grains heard one at a time.

The shape a grain is cut with

At these lengths the envelope is most of the character. The same sound through a soft bell and through a sharp tick is two instruments.

shape.level(at:) hands back the curve, the same table the grains are read through, so a sketch can draw the cut it chose the way Envelope.level(at:heldFor:) draws the note.

Where the sound comes from

rootKey is the note the sound reads at its own speed. A note above it reads each grain that much faster, which moves the pitch without touching how fast the cloud travels.

Things worth knowing

• The sound loops. A cloud traveling past the end comes round to the start, so position 1 is position 0 and a note does not go quiet part way through.
• Loudness rises with the square root of the density. Grains land on each other at random, so what adds is power rather than amplitude: four times as many is twice as loud.
• A strict clock is a pitch. With timingJitter at 0 the grains arrive on a clock, and a frozen cloud repeats the same piece of sound at that rate. The output is then periodic at density hertz, whatever the sound was. That is a real instrument rather than a fault, and turning timingJitter up is how you stop hearing it.
• A cloud drops grains rather than waiting for room. A note may have 48 sounding at once. Past that a new one is dropped, the same bargain everything on the audio thread makes. A cloud dense enough to reach it is already a texture, and one missing grain in it cannot be heard.
• detune runs two streams rather than two copies. Every other grain takes the offset, so two interleaved streams a fraction apart beat against each other the way two oscillators do.
• A cloud is read when the note starts. Changing the settings takes the next note, the way every other voice works. What does move a sounding note is grainScrub, which is read every few milliseconds.
• panSpread needs two channels to spread into. A Synth placed in the 3D scene is one stream by definition, because turning one into two is the listener's whole job, so a placed instrument folds the spread back to the middle.

Examples/Audio/Grains draws the sound with the band the grains are being cut from lit over it, and lets you drag that band through by hand.

Placing a sound

A sound can come from somewhere in the 3D scene, with the camera as the listener.

override func draw() {
    cameraShowcase(.autoOrbit())
    guard let eye = activeCamera else { return }

    synth.place(at: Vector3(2, 0, -3), heardFrom: eye)
    synth.hearingDistance = 1...20
}

You give both facts together, because neither means anything alone. A position says nothing until something is listening, and the sketch hears from wherever it is looking from. Call place(at:heardFrom:) every frame, so a moving object and a moving camera both work.

On headphones the placing is done the way the ear works it out, which is more than loudness. It uses how much later the sound reaches one ear than the other, and what a head does to a sound arriving round it. So a sound behind you is behind you rather than merely quiet. On speakers it falls back to a plain left and right.

Placing survives an export. The moves are written down as the frames are drawn, then applied when the soundtrack is rendered.

The first call rebuilds the instrument's audio chain, so make it before the first note if you can. Placing a sound is a different shape of graph rather than a setting on one. One stream has to arrive at something that knows where the ears are, and leave it as two. Once placed, an instrument stays placed.

Examples/Audio/Spatial is three chimes standing still and one walking past them.

Sound in an export

A sketch that plays sound carries it out of the window and into the file:

swift run --package-path Examples Example-Audio-SoundInAnExport --export-video sound.mp4 --frames 480

The file has the music in it, with no recording step and nothing to switch on. The exporters drive a sketch on a fixed clock with nothing playing, so the notes are written down instead of going to the speakers. The soundtrack is then rendered at the end, through the same code that would have fed the speakers.

That works because the renderer takes events and gives back samples, and it has no clock of its own. An export is that same code with the waiting taken out. That is also why the sound reproduces. Export twice and the audio comes back sample for sample identical, so you can come back to the work later.

What to know:

• --export-video and --export-loop carry sound. GIF has no way to hold any, and the still and vector exports are one frame.
• An instrument has to be held by the sketch by the end of setup(), so it can be found: as a property, or inside a struct, an array, an optional, a dictionary, or an object of the sketch's own that a property holds. One made and thrown away inside draw() cannot be found. A list is read through its first element, so a list that holds plain values first and an instrument later is read as the first says.
• A note's length travels in beats of real time, not samples, so the file is written at its own rate. The hardware rate does not decide it.
• Delay and reverb are in the file, because the export runs the same effects the output had.
• A placed sound is placed in the export too. Where it was and where it was heard from are written down each frame. The notes travel the same way, and the soundtrack is rendered through a listener afterwards. So a sound that walks past you live walks past you in the file. Place from the first frame, because the chain is fixed when the soundtrack machine is built. An instrument that starts playing before it is placed says so, rather than quietly coming out centered.
• An exported placing is a plain left and right, not the head model headphones get live. A file cannot know what it will be played back on.
• A sketch that holds no instrument writes exactly the file it wrote before, with no audio track at all.

Envelope

An Envelope is the shape of a note over time.

Envelope(attack: 0.001, decay: 0.4, sustain: 0, release: 0.1)   // a pluck
Envelope(attack: 0.8, decay: 1.0, sustain: 0.7, release: 1.5)   // a pad

• attack is the time from silence to full level.
• decay is the time to fall from there to sustain.
• sustain is the level a held note rests at (0...1, not a time).
• release is the time to fall back to silence once the note is let go.

The falling segments approach their target rather than arrive at it. decay and release name the time to come within a thousandth of where they are headed. That is what the ear reads as the note being over.

A sustain of 0 means the note plays out entirely on its own, and holding the key adds nothing. That is what struck things do, like bells, plucks, and drums.

Presets: .percussive, .organ, .swell, .standard.

A note let go early falls from wherever it had reached, and a voice retriggered while it is still sounding bends into the new note. Both of those exist so that neither clicks.

Voice.Filter

A filter is what is taken out of the wave, and how that moves while the note sounds. It is most of what a synthesizer sounds like. A note that is bright when struck and darkens as it decays is a filter closing, not a wave changing.

Voice.Filter.lowpass(cutoff: 2000, resonance: 0.3)          // does not move
Voice.Filter.sweep(from: 400, by: 3.5, resonance: 0.35)     // opens, then closes

keyTracking defaults to 0, so high notes come out duller than low ones, which is what real instruments do. At 1 the cutoff moves with the note step for step, and that is what makes an unpitched wave playable. A noise voice has no pitch of its own, so the filter is the only thing a note can move. That is how .breath works.

The filter is solved by integrating the circuit rather than by folding in a fixed answer. The cutoff therefore lands where you asked for it across the whole range, and it stays stable while it is swept. A filter that only behaves standing still is no use to a note that opens as it is struck.

Effects

The chain is everything done to the sound after it is made, in order.

synth.effects = [
    .distortion(Distortion(.softClip, mix: 0.3)),
    .delay(Delay(time: 0.28, feedback: 0.5)),
    .reverb(Reverb(.hall, mix: 0.4)),
]

The voice side of this library is routing as a value. A Patch says what an instrument is made of and how the pieces are wired. The chain is the same idea on the way out, so the effects finish an instrument rather than decorate it.

Order is the point. An echo of a distorted sound and a distorted echo are different. So are a reverb of an echo and an echo of a reverb. The first repeats a room, and the second puts repeats in one. Writing the chain as a list is what lets you say which one you mean.

The two that were here first

synth.reverb = Reverb(.hall, mix: 0.3)      // still works
synth.delay = Delay(time: 0.25)

delay and reverb are views over the chain. Reading one gives the first effect of that kind, and setting it replaces that effect where it already is or appends it. So a sketch that only wants one echo and one room never has to think about a chain. A sketch that wants more can use the chain instead.

Delay

Delay(time: 0.25, feedback: 0.4, mix: 0.3, damping: 6000)

Reverb

Reverb(.hall, mix: 0.3)

The rooms are .room, .hall, .plate, and .cathedral, biggest last. mix is how much of the result is the room. Any other room is a room of your own.

A room of your own

let stairwell = try ImpulseResponse.resource("stairwell", withExtension: "wav", in: .module)
synth.reverb = Reverb(stairwell, mix: 0.4, preDelay: 0.02)

A room is what it does to a click. Clap once in a stairwell and what comes back is the stairwell: every surface and every distance, all at once. That recording is an impulse response. An instrument played through it is heard in that room, because every sample of the sound starts its own copy of the click's answer. ImpulseResponse is the answer written down, and Reverb(room) is the convolution reverb that plays through it, which is the chain's .convolution kind. It sits in the chain like any other reverb, the reverb view reaches it, and it reaches an export.

A response reads itself back as channels (one or two of them, with channelCount saying which), sampleRate, and duration. ImpulseResponse.maxSeconds is the longest room a reverb takes, twenty seconds, past which it is a drone rather than a room, and a longer response is cut there.

A drawn room is whatever the rule says:

ImpulseResponse(seconds: 3) { t, noise in exp(-2.3 * t) * noise }            // a hall
ImpulseResponse(seconds: 2) { t, _ in exp(-3 * t) * sin(.tau * 440 * t) }  // a room that hums at A

The noise differs between the two sides, which is what gives a room its width. It comes from a seed, so the same rule draws the same room every run. A response is two channels, and a mono recording is heard the same on both sides.

Two things sit beside mix. preDelay is seconds of silence before the room answers, 0...1, and a little of it keeps the sound itself clear of the room. And every room is brought to unit energy on the way in, so a quiet recording and a loud one sit at the same level, and mix means the same for both. Changing mix on a sounding room leaves its tail alone. Changing the room or the pre-delay starts a fresh one, since those are the room itself. A room longer than twenty seconds is cut there.

Recording one is a microphone and a clap, a balloon, or a starter pistol in the space you want. Free libraries of measured rooms exist, and the license travels with the file the way it does with a sample. A room's cost is its length, and the longest one allowed runs in a few percent of real time.

Examples/Audio/Rooms is five rooms drawn from rules, each room's answer to a click drawn as a waveform.

Equalizer

Equalizer(lowGain: -6, highGain: 3)         // thinner and brighter
Equalizer.lowCut(below: 300)                // when a sound is muddy

There are three controls. Two of them are the bottom and the top, lowGain below lowFrequency and highGain above highFrequency, both edges in Hz. The third goes wherever the problem is: midGain at midFrequency, with midWidth deciding how narrow that band is, around 1 for a broad tilt and past 5 for a notch aimed at one thing. Gains are in decibels, so zero leaves the sound untouched. A few decibels is a much bigger change than it looks on the page. The presets are .warm, .bright, and .scooped.

Distortion

Distortion(.softClip, drive: -6, mix: 0.3)

.softClip gives warmth rather than damage. .overdrive, .bitCrush, .ring and .squeeze get progressively harsher. drive is how hard the sound is pushed in, in decibels, and mix dials the whole thing back to nothing.

The four that move

synth.effects = [.chorus(Chorus(rate: 0.8, depth: 0.5))]
synth.effects = [.flanger(Flanger(rate: 0.25, depth: 0.7, feedback: 0.5))]
synth.effects = [.phaser(Phaser(rate: 0.4, stages: 4))]
synth.effects = [.tremolo(Tremolo(rate: 5, depth: 0.6, spread: 1))]

Each of these is one slow wave and the thing it moves. rate is how many times a second the wave turns, in Hz, and depth how far it moves the thing, 0...1. Where there is a mix, it is how much of the result is the moved sound rather than the plain one, and a mix of 0 is the plain sound exactly.

Turn any setting while the sound plays and the motion carries on from where it was: the wave keeps its phase, and a copy keeps sliding. A setting change never restarts the effect.

The three that hold a level

synth.effects = [.compressor(Compressor(threshold: -18, ratio: 4, makeupGain: 6))]
synth.effects = [.gate(Gate(threshold: -40, hold: 0.08)), .limiter(Limiter())]

Where the four above move the sound, these three watch how loud it is and act on that. Every threshold is in decibels below full scale, where 0 is as loud as a sample can be and a level you would mix at sits somewhere under -12.

attack is how long the holding down takes and release how long the letting go takes, both in seconds, and between them they are most of what a compressor sounds like. A fast attack flattens every transient, and a slow release breathes. The level is read from both sides at once, so a loud note on one side pulls the other down with it and the image stays where you put it.

Two footnotes. The compressor follows the level from crest to crest rather than reading each sample, so a tone lands where its ratio says it should rather than a decibel short of it. And a key from somewhere else, the trick where one sound ducks another, is not here: each Synth runs its own engine, so there is nothing for a detector on one to listen to on another.

The two that work in the spectrum

synth.effects = [.pitchShift(PitchShift(semitones: 7, mix: 0.5))]
synth.effects = [.freeze(Freeze(amount: mouseIsPressed ? 1 : 0))]

Everything above works on the sound as a wave. These two take it apart first. The sound is read in frames of about forty milliseconds, each frame as the level and the exact frequency of every partial in it, and put back together with those partials moved or held. That is a phase vocoder. It is what lets the pitch move without the length, which a sampler cannot do, since a recording played faster is shorter as well as higher.

Both arrive one frame later than the sound went in, about forty milliseconds, and the unmoved half of a blend is delayed to match, so a harmonizer's two voices land together. Turn a setting while the sound plays and the frames in flight carry on: the pitch slides rather than jumps, and a held instant stays held while its amount moves.

Two things to know. A pitch shift moves the whole spectrum, so a voice an octave up is a small voice rather than a high one, and keeping the shape of a voice in place while its pitch moves is not here. And put a Freeze in the chain at 0 and raise it when there is something to hold, since a freeze that starts at 1 holds the silence before the first note.

The same taking apart, done once to a recording rather than live, is stretching a recording. Examples/Audio/Spectral is both effects and the stretch behind a parameter, with the freeze on the mouse.

An effect of your own

synth.effects = [
    .custom("fold") { sound in
        for i in 0..<sound.frameCount {
            sound.left[i] = sin(sound.left[i] * 3)
            sound.right[i] = sin(sound.right[i] * 3)
        }
    },
    .reverb(Reverb(.hall, mix: 0.3)),
]

The four kinds above cover the classics. .custom is the seam for everything else, in the way Shader is that seam for drawing. Ollin hands the closure each block of samples on its way to the speakers, and the closure is free to rewrite them. It sits anywhere in the chain, and it reaches an export like every other link.

The closure gets an AudioBlock. left and right are the two channels. On a mono sound they are the same channel, so code written against both is right either way. frameCount is how many samples each channel holds this time. sampleRate is what a frequency is measured against. time is seconds since the effect started running, which is what anything moving needs. Samples run -1...1.

An effect that has to remember something between blocks takes its memory as state: and gets it back inout every time. A filter, or an echo of your own, works this way:

// A one-pole lowpass: each sample pulled toward the one before it.
.custom("soften", state: (l: Float(0), r: Float(0))) { sound, held in
    for i in 0..<sound.frameCount {
        held.l += (sound.left[i] - held.l) * 0.08
        held.r += (sound.right[i] - held.r) * 0.08
        sound.left[i] = held.l
        sound.right[i] = held.r
    }
}

Why not a captured variable? The audio thread calls this closure, so it has to be @Sendable, and Swift will not let a @Sendable closure write into anything it captured. state: is the memory that is allowed.

Two rules follow from where the closure runs, which is the audio thread with the speakers waiting. First, keep the closure to arithmetic over the samples, with nothing allocated, nothing locked, and nothing reached back into the sketch. Second, to change how it behaves from a parameter, build a new effect and set the chain again. The new closure swaps onto the same link without touching the wiring, so the sound never stops to change. A new effect does start with fresh state:.

The effect is a value, with two footnotes. Putting the same one in two places makes both places share one memory. An effect that went through Codable comes back as a passthrough that still knows its name, because a closure cannot be written down.

Examples/Audio/Shaping is three of these behind a parameter. They are a wavefolder, a sample-holding crush with state:, and a wobble run off time.

What it costs to change one

Changing a setting costs nothing, because the chain keeps the same wiring and only the numbers move. Changing which effects are in the chain rewires it. That happens on the running engine rather than around a stop. Measured on this wiring, reconnecting while it runs costs nothing audible, and stopping costs the same.

The chain reaches an export as well, built the same way from the same list. An export that ran a different set of effects from the one the sketch was heard through would be a different work.

What is not here yet

These limits are said plainly, so you can plan around them rather than go looking:

• One instrument, one sound at a time. A Synth plays one voice. Several sounds at once means several Synths, which is fine and cheap.
• A patch is oscillators, not a whole modular rack. Operators push each other and mix. There is no filter, envelope, or effect inside a patch. Those live in the Voice around it, one per voice rather than one per operator.
• The chain is on the instrument, not on a note. Every note a Synth plays goes through the same effects. Two different treatments means two Synths.
• No key from another sound. A compressor or a gate listens to what passes through it and nothing else. Ducking a pad under a voice, or gating a sound off a drum, needs a detector on one instrument reading another, and each Synth owns its own engine.
• No sequencer. You ask for notes from draw(), on whatever clock the sketch keeps. Composition decides which notes and when. To run on someone else's clock, use TempoClock.
• A sampler, but not a sample editor. Recordings are read and played. Nothing here trims them, loops them by ear, or lays out a map for you. The map is the .sfz.
• A pitch shift moves the whole spectrum. A voice an octave up is a small voice rather than a high one. Keeping the shape of a voice where it is while the pitch moves, which is what makes a harmonizer sound like a singer rather than a cartoon, is not here.
• One recording at a time per note. There is no crossfading between velocity layers, or between neighboring recordings. A change of layer is a step rather than a fade.
• A grain cloud's density does not follow the note. Grains arrive at density hertz whatever is played. Tying the grain rate to the note's own pitch, so the rate is the pitch and the grain's content is a formant, is a different instrument and is not here.
• One sound at a time per instrument. A Synth has one grainSource, the way it has one instrument. Clouds over two different sounds means two Synths, which is fine and cheap.
• A wavetable's position is read when the note starts. The sweep moves it over the note. The position itself is not a live control, the way pressure is. To move a held note by hand, use the sweep or play a new note.
• No jet-driven tube. The blown tube is reed-driven. A flute is a jet of air splitting across an edge, which is a different excitation and is not here.
• One position per instrument. A Synth is placed as a whole. Several sounds in several places means several Synths, which is fine and cheap.
• No sound in a GIF. The format has no way to hold any.

^{Audio covers the listening side. MIDI covers playing from a keyboard and running on an external clock.}