37. Listening

Sound can move a picture once it becomes a few numbers: how loud the low, middle, and high parts are, and when a beat lands. You read those numbers from a microphone, a song, or a video, and learn which sound should move which part of a picture. The Resonator above throbs on every beat under a crown of spectrum spokes. The chapter then turns to the note being sung, and to the words and sounds a listener can name.

Loudness from the microphone: amplitude

Sound reaches a sketch through OllinAudio, a small library you import beside the framework. The simplest start is the microphone and one number, amplitude. It says how loud things are right now, roughly 0...1, and it is smoothed so it doesn't flicker.

import Ollin
import OllinAudio

final class Pulse: Sketch {
    let mic = AudioInput()

    override func setup() { try? mic.start() }

    override func draw() {
        background(.black)
        fill(.white)
        drawCircle(width / 2, height / 2, (60 + Double(mic.amplitude) * 700) * scale)
    }
}

Save it as MySketches/Pulse.swift and run it with swift run OllinLive MySketches/Pulse.swift, like any sketch. macOS asks about the microphone the first time, so say yes, and hum. The circle grows and shrinks with your voice. If it never moves, the reason is usually the permission, and mic.unavailableReason then holds a sentence that says so. A Mac with no microphone, such as a Mac mini, can listen to a song file or a tone instead, as Four sources shows. Every sketch in this chapter has the same shape. You make a source, keep it in a property, start it in setup(), and read its values every frame. The steps that follow add richer values to read.

Swift note. amplitude is a Float, because audio hardware works in 32-bit floats. Drawing wants Double, so you'll see Double(mic.amplitude) around every audio read. The conversion changes the type and keeps the value.

Four sources: the microphone, a file, a tone, and a video

The microphone is one source of sound among four. Each source hands its sound to an analyzer, which turns it into the numbers you read, and the reads work the same on each. So choosing a source is one line. Here track.m4a is a file in the sketch's folder, and player is a VideoPlayer property from Chapter 34. lazy lets the soundtrack use it, since a lazy property is made the first time it is read:

let mic = AudioInput()                                         // the room
let song = try? AudioPlayer(resource: "track", withExtension: "m4a", in: .module)
let tone = Tone(frequency: 220, waveform: .sine)               // a note of your own
lazy var sound = Soundtrack(of: player)                        // a video's sound

AudioInput is the microphone, permission and all, and it hears whatever the Mac's sound input is set to. AudioPlayer plays a file and analyzes it as it sounds, once you call its play(). It reads .m4a, .mp3, .wav, .aiff, and the other formats macOS plays. try? gives nil when the file is missing, so song is optional and you read it as song?.amplitude, the optional chaining from Chapter 9. The Audio/FilePlayer example plays a violin recording that ships with it.

Tone plays a steady wave at one pitch and feeds it to the analyzer as it sounds. It starts with play() too, and it lets a sketch hear itself with no permission and no file. On a Tone, amplitude is the volume you set, so its measured loudness is tone.analyzer.amplitude. The Audio/Spectrum example keeps a gliding Tone ready and draws its spectrum until the microphone is allowed. Soundtrack reads the sound of a playing video, so footage can drive the picture with its own music. An audio file you ship with a sketch needs the same license care as any other asset, so credit what you use.

The sources behave differently under an export. AudioPlayer follows the export's clock through its file, so a sketch driven by a file writes the same frames on every run. A video's sound reads as silence there. The microphone hears the room live while the frames render, so it never gives the same video twice. Chapter 41 covers exporting in full.

A band that plays the same every run: StageMic

A room sounds different each time you run a sketch in it. A sketch tuned against the microphone never hears the same second twice. While you build, a source that plays the same music every run helps. StageMic is one. It is a short class at the bottom of Anatomy.swift, so copy it into your sketch's file.

StageMic works out a little band as numbers and feeds them into an AudioAnalyzer, the same analyzer every source above uses. The band is a kick drum every half second, with a hi-hat between the kicks. A kick is the low, thumping drum, and a hi-hat is a short tick of cymbal. Over them sit a held bass note, a slow four-note melody, and a little hiss. StageMic makes no sound you can hear. Each call to listen() feeds the analyzer the next sixtieth of a second, and you read the numbers from its analyzer property:

let mic = StageMic()                  // a stored property

// in draw():
mic.listen()
let level = mic.analyzer.amplitude    // the same reads as any source

Every value comes out of the same kind of analyzer a microphone feeds, so the reads mean the same things on both. The numbers differ a little, because the microphone's analyzer reads shorter stretches of sound. To switch, swap StageMic() for AudioInput(), start it in setup(), and delete the listen() line, because a live source feeds its analyzer by itself.

What the analyzer hears: waveform, spectrum, and bands

Every source hands its sound to an analyzer, and the analyzer turns it into the numbers a sketch reads. Sound arrives as samples, measurements of air pressure taken tens of thousands of times a second, usually 44,100 or 48,000. The analyzer reads the most recent stretch of them three ways.

The top panel is the waveform, the samples themselves, about 46 milliseconds of them. Two or three slow swells carry fast wiggles on them. The swells are the kick's low thump, and the wiggles are the melody. Draw the waveform when you want the look of an oscilloscope, the instrument that draws a signal as a moving line. Most mappings read the two panels below it.

The middle panel is the spectrum. Sound is vibration, and pitch is how fast the air vibrates. A low note shakes the air a few dozen times a second, and a high note thousands of times. The rate is measured in hertz (Hz), meaning cycles per second. The spectrum splits the instant into how much energy sits at each rate, the way a prism splits light into colors. Once it is split, you can read the mix. The kick's 55 Hz thump, the bass note at 110 Hz, and the melody near 659 Hz each get their own spike. The two smaller spikes to the right are the melody at twice its frequency and the click at the front of each kick. The tool that does the split is the Fourier transform, which Chapter 21 used to read a picture as waves. The analyzer runs it for you as the samples arrive. spectrum is the result, an array of magnitudes from low frequencies to high. Each entry is a bin, an equal slice of frequency, about 47 Hz wide on the microphone.

Raw spectra are awkward to draw. The magnitudes have no fixed range, and the equal slices don't match how you hear. Hearing works in ratios, so every doubling of frequency sounds like one equal step, and that step is an octave. The three octaves of bass from 40 to 320 Hz get about six bins, while the top octave alone gets hundreds. So the bottom panel is the read most sketches use, bands(_:). The fragments in this section read from mic, the AudioInput of the first step. On StageMic, the same reads go through mic.analyzer:

for (i, level) in mic.bands(24).enumerated() {
    let h = Double(level) * 300 * scale           // level is already 0...1
    drawRect(Double(i) * 34 * scale, height - h, 30 * scale, h)
}

bands(24) gives 24 bars spaced the way hearing is, from about 40 Hz up, so the bass gets several bars instead of one. A gain, a multiplier that adapts to the music, scales each bar to roughly 0...1. Each bar rises fast and falls gently, so the bars move without jitter. So a bar's height maps straight to pixels, with no gain for you to tune. Ask once per frame with the same count every time, because a new count starts the scaling over.

Between the raw spectrum and the shaped bands sit three named reads: bass, mid, and treble. They give the energy in the low, middle, and high ranges. magnitude(in: 40...120) reads a range you pick yourself. These three have no fixed range, like the spectrum, so scale them to taste.

Hearing the beat: onsets

Loudness and spectrum answer "how much". Music also has arrivals. A drum hit is a moment, and a picture that flashes on the drum reads as listening in a way a level meter does not.

The detector compares each new spectrum with the one before it and adds up every rise. A sudden brightening across many frequencies at once makes that sum jump, and the analyzer counts it as a beat. The sum is called spectral flux. A drum hit, a plucked string, and the start of a note all do that. The name for such a moment is an onset. Three reads report it:

mic.beat            // a 0...1 pulse: snaps to 1 on each beat, fades over ~0.25 s
mic.beatCount       // how many beats so far
mic.timeSinceBeat   // seconds of audio since the last one

beat is the ready-made value, so multiply a radius by it and the picture throbs. beatCount is for firing something once per beat. Compare it with a count you stored, and act when it has gone up. timeSinceBeat lets you draw a fade of your own length.

Look at the timeline. Every kick lands, and so does each quiet hat between the kicks. Onset detection hears sudden changes in the sound. It does not measure loudness, and it does not know "the beat" a drummer would tap. A soft hat is as sudden as a loud kick, so both count. To hear fewer beats, raise beatSensitivity above its default of 1.5, and a beat then needs a stronger arrival before it fires. A held chord, several notes at once, doesn't set it off. Neither does a drone, one note held under the music. The same samples always beat in the same places.

The detector says that a beat arrived, and it keeps no sense of tempo. Following the tempo of the room, so a sketch can play in time with a record, is BeatFollower. Chapter 40 teaches it once the sketch has notes of its own to play. A beat that several programs share over the network is the Link clock in Chapter 38.

From sound to picture: levels, moments, and decay

You can now read a level for each band and a moment for each beat. A music visual is mostly the choice of which read moves what. A few habits make the picture look like it is listening, and the Resonator is built on them.

Which band drives what. Low sounds are big and slow, and high sounds are small and quick. So let the bass bands drive the big shapes, such as a scale, a glow, or the weight of a line. Let the treble bands drive small, fast things, such as sparkle, grain, or a flicker at the edges. The middle bands carry most melody and voice, and they suit color and shape. Wired the other way round, the canvas swells on every hat. The Resonator puts the low bands at the top, where the music is loudest and the spokes grow longest. The high ones sit at the bottom.

Levels and moments. A level such as a band or amplitude changes all the time. So map it onto something that can change all the time, like a length, a brightness, or a position. A moment, such as beatCount going up, happens once, so let it start something: a burst of sparks, a flash, a new color. A level used as a trigger fires again and again while the sound stays loud. A level that moves too fast calms down with the source's smoothing. It runs from 0 for the raw reading to near 1 for a heavily damped one, and the microphone starts at 0.8.

Decay. A flash that lasts one frame is gone before the eye finds it. So a moment sets a value to 1, and the value falls back over a fraction of a second. The Resonator's core does this with one line every frame, pulse *= exp(-deltaTime * 5). exp(-deltaTime * 5) is a number a little under 1, the fraction of the pulse to keep this frame. This is the exponential decay of Appendix B, scaled by deltaTime so it falls at the same speed at any frame rate. The pulse drops to about a third in a fifth of a second, and a larger number than 5 makes the tail shorter. A tail of a fifth to half a second reads as a hit, and a longer one starts to read as a level.

A feed from the mixer. At a show, a microphone in the room hears the crowd and the air conditioning. It also hears the speakers a moment late. Ask the sound engineer for a line from the mixing desk, a cable carrying the sound itself. Plug it into an audio interface, a box that brings sound cables into the Mac. Choose the interface on the Input tab of System Settings > Sound. Then set it to 48 kHz, 48,000 samples a second, in the Audio MIDI Setup app, since that is the rate AudioInput expects. The sketch doesn't change, and the line carries the music alone at a steady level.

Rehearsing on a recording. StageMic gives you a band that plays the same every run. When you are tuning for a particular show, rehearse on its music instead. Play a recording of the set through AudioPlayer. The same second then gives nearly the same numbers on every pass, and exactly the same ones under an export. Once the mapping looks right on the recording, swap in the microphone or the mixer feed.

Putting it together: the Resonator

The finished sketch reads StageMic, draws bands as a crown of spokes, and uses beatCount as the moment that throbs the core and throws sparks. Two parameters are ready for the knobs and faders of Chapter 38. Make MySketches/Resonator.swift and paste in StageMic from Anatomy.swift. The figure file with everything together is Resonator.swift:

import Ollin
import OllinAudio

final class Resonator: Sketch {
    let mic = StageMic()

    @Param(24...96) var spokes = 56
    @Param(0.6...2.4) var brightness = 1.4

    struct Spark {
        var position: Vector2
        var velocity: Vector2
        var life: Double
    }

    var sparks: [Spark] = []
    var lastBeat = 0
    var pulse = 0.0

    /// Where the instrument sits: a touch below center, to leave the crown room.
    var mid: Vector2 { center + Vector2(0, 60 * scale) }

    override func setup() {
        seed(20)                              // the sparks re-fly the same way
    }

    override func draw() {
        mic.listen()
        let audio = mic.analyzer

        background(Color(hex: 0x06040E))
        toneMap(.aces)
        blendMode(.add)

        // The beat: snap the pulse up when a new one lands, let it fade.
        pulse *= exp(-deltaTime * 5)
        if audio.beatCount > lastBeat {
            lastBeat = audio.beatCount
            pulse = 1
            spawnSparks()
        }

        // The spectrum, worn as a ring of spokes: bass at the top, treble at
        // the bottom, mirrored left and right so the ring stays symmetric.
        let levels = audio.bands(spokes / 2 + 1)
        withState {
            translate(mid)
            rotate(time * 0.06)
            for i in 0 ..< spokes {
                let band = i <= spokes / 2 ? i : spokes - i
                let level = Double(levels[band])
                let angle = Double(i) / Double(spokes) * .tau - .pi / 2
                let dir = Vector2(cos(angle), sin(angle))
                let inner = 175 * scale
                let len = (14 + level * 290) * scale
                fill(Colormap.magma.color(at: 0.2 + level * 0.75)
                    .withAlpha(0.25 + level * 0.75 * brightness))
                drawOrientedBox(dir * inner, dir * (inner + len),
                                thickness: (3.5 + level * 9) * scale)
            }
        }

        // Sparks from past beats, flying and fading.
        for i in sparks.indices {
            sparks[i].position += sparks[i].velocity * deltaTime
            sparks[i].life -= deltaTime
        }
        sparks.removeAll { $0.life <= 0 }
        noStroke()
        for spark in sparks {
            let a = max(0, spark.life / 0.8)
            fill(Color(hue: 0.09, saturation: 0.5, brightness: 1).withAlpha(a * 0.9))
            drawCircle(spark.position.x, spark.position.y, (1.5 + a * 5) * scale)
        }

        // The core: a warm glow that swells on the beat, a hot center.
        let ember = Color(hex: 0xFFB65C)
        let glowRadius = (185 + pulse * 150) * scale
        fill(.radial(center: mid, radius: glowRadius,
                     Ramp([ember.withAlpha(0.3 + pulse * 0.5), ember.withAlpha(0)])))
        drawCircle(center: mid, radius: glowRadius)
        let coreRadius = (80 + pulse * 60) * scale
        fill(.radial(center: mid, radius: coreRadius,
                     Ramp([Color.white.withAlpha(0.95), Color.white.withAlpha(0)])))
        drawCircle(center: mid, radius: coreRadius)

        blendMode(.normal)
    }

    func spawnSparks() {
        for i in 0 ..< 14 {
            let angle = Double(i) / 14 * .tau + random(-0.15, 0.15)
            let speed = random(200, 380) * scale
            let dir = Vector2(cos(angle), sin(angle))
            sparks.append(Spark(position: mid + dir * 180 * scale,
                                velocity: dir * speed,
                                life: random(0.45, 0.8)))
        }
    }
}

Here is how it composes the steps:

• The source. mic.listen() feeds the next sixtieth of a second of StageMic's band to its analyzer, and every read goes through audio, that analyzer. mid here is a property that works out a point a little below the center, where the instrument sits. It is a different thing from the mid band read. On a 120 Hz display draw() runs twice as often, so the band plays at twice its speed. mic.listen(seconds: deltaTime) keeps it in real time.
• The levels. bands(spokes / 2 + 1) asks for one more band than half the spokes. Spoke i reads band i down one side and band spokes - i back up the other. So the crown is a mirror image. The ? : is the compact if from Chapter 6. Band 0, the bass, starts at the top, since the angle starts a quarter turn back, at -.pi / 2. rotate(time * 0.06) inside withState then turns the crown slowly, so the bass drifts around. spokes sets how many spokes there are, and brightness scales how opaque they are. A spoke's length, width, color, and alpha all grow with its level. Colormap.magma runs from magenta for the quiet bands to pale gold for the loud ones. Each spoke is a drawOrientedBox, which fills a thick bar between two points at any angle.
• The moment. When beatCount has gone past lastBeat, the sketch stores the new count, sets pulse to 1, and calls spawnSparks(). pulse then falls away with the decay from the mapping step, and it swells both the glow and the white core.
• The sparks. Each spark flies outward at its own speed and loses life by deltaTime. removeAll { $0.life <= 0 } drops the spent ones, where $0 is each spark in turn. seed(20) makes the sparks fly the same way on every run.
• The light. The glow and the white core are circles filled with a radial Ramp, one of the gradients of Chapter 2. It fades to nothing at the edge. The additive blend and the ACES tone map from Chapter 19 make the overlaps glow like light instead of paint. blendMode(.normal) at the end puts the blend back for the next frame.

Then make it yours:

• Give it your ears. Swap StageMic() for AudioInput(), start it in setup(), and delete the mic.listen() line. Then play music at your Mac.
• Give it your music. Use an AudioPlayer with a track you like, and tune beatSensitivity until the sparks land on the drums.
• Give it your hands. Chapter 38 binds a knob on a MIDI controller, or a fader on a phone, to brightness.

To keep it, export a video:

swift run OllinLive MySketches/Resonator.swift --export-video resonator.mp4 --seconds 12

StageMic makes no sound, so the video is silent, and it comes out the same on every run. With your own song in an AudioPlayer, the export still follows the file frame by frame, and the video still has the picture only. To keep the music with the picture, record a run instead, with --record in place of the export flags. Chapter 43 shows how a take records the sound the sketch is playing.

The note being sung: pitch and pitch classes

The Resonator reads how loud each band is and when something arrives. A sketch can also ask which note is sounding, so the picture follows the melody or the harmony instead of the drums. One read names a single note, and another lists every note in the air at once.

One note at a time: pitch

pitch names the note a voice, a bowed string, or a whistle is holding. Use it to let a singer steer the picture: a height that follows the melody, or a color for each note.

A held note is made of a fundamental, the rate you hear as its pitch, plus harmonics at two, three, and more times that rate. The detector is YIN, the 2002 method of Alain de Cheveigné and Hideki Kawahara. It does not look for the loudest frequency. It looks for the shortest delay after which the waveform repeats. A note with harmonics repeats at its fundamental's period, even when the fundamental is the quiet part or missing altogether. The ear decides the same way. So a bowed string reads at the string's note rather than at its brightest harmonic. YIN takes the first delay that repeats well enough, instead of the best one, which keeps it from reporting a note an octave low.

The top strip is four seconds of a violin recording. The detector reads about 50 milliseconds at a time, a stretch called a window, and each dot is one window. A semitone is the step from one piano key to the next, and twelve of them make an octave. The dots sit on the semitone lines because the player is in tune. Each dot is as solid as the detector was sure. The long run at the start is a double stop, two strings bowed at once. So are the few later dots an octave or more below the melody. Two notes together repeat only at the period both share, and the reading lands there. So pitch reads one note at a time, and a chord has no single pitch.

pitch is nil when nothing is being sung: silence, a room, a noise with no repeating shape in it. So you read it with if let. When there is a note, it carries five things:

• frequency is the rate in hertz.
• nearestPitch is the nearest note name: a letter for the note, # for the black key just above it, and a number for the octave. A4 is the A above middle C, the C near the middle of a piano keyboard.
• cents is how far above or below that name the note sits, where a hundred cents is one semitone.
• confidence is how sure the detector is. A pure tone reads 1, a sung or bowed note around 0.9, and noise stays under 0.5, where reporting stops.
• midi puts the note and the cents back together as one number. MIDI is the message format music hardware speaks, and Chapter 38 introduces it. Its numbering is 60 for middle C and one more for each semitone up.

This draws the note's name, and a circle whose height follows the melody and whose size follows the confidence:

if let heard = mic.pitch {
    drawText("\(heard.nearestPitch)", width / 2, 80 * scale)             // "A4"
    let y = map(heard.midi, 48, 84, height, 0)                   // pitch as a height
    drawCircle(width / 2, y, (20 + Double(heard.confidence) * 40) * scale)
}

Map midi onto a position rather than frequency. The ear hears equal steps of midi as equal steps, and an octave is always twelve of them. An octave in hertz is a doubling, which has no fixed length on a ruler. A new note is heard about one window after it starts. The reading is not smoothed, since a held note holds steady on its own. A sketch that wants a slow needle eases toward midi itself. For the name alone, mic.nearestPitch is an optional Pitch, nil when nothing is sung, and a Pitch prints as its name.

The bottom of the figure is a tuner, the note and the cents at one instant. The Audio/Tuner example is that face, live, with the trace and the twelve classes under it. It follows the violin by default. Run it with --mic and sing at it. The Audio/GuitarTuner example is the six-string kind. It compares midi with the note of each open string, the string played with no finger on it, and lights the nearest. Its cents are read against that string's own pitch rather than the nearest piano key. So a string a semitone flat still points at its own tuning peg.

Every note in the air: chroma

chroma lists which of the twelve note names are sounding, whatever octave each one is in. Use it for harmony, the notes sounding together, where pitch has no answer. A palette can change with the chord, or twelve lights can show the key, the set of notes the music uses. It follows the pitch class profile Takuya Fujishima described in 1999. The classes are read off the spectrum's peaks, the way Emilia Gómez's harmonic profile of 2006 reads them.

The middle strip of the figure above is chroma over the same four seconds, and the melody draws itself through the note names. It is twelve values, C first, one per note name, each 0...1 with the loudest at 1, and every octave folded onto the same twelve. So a C major chord, C, E, and G together, lights those three whatever octaves they sound in.

for (i, level) in mic.chroma.enumerated() {                   // C first
    let h = Double(level) * 200 * scale
    drawRect(Double(i) * 40 * scale, height - h, 36 * scale, h)
}

Words, and what that noise was: speech and sound events

Everything the Resonator reads describes sound as a shape: how loud, which frequencies, when something arrived. A sketch can also ask what it is hearing, a word or a clap or a dog. Two listeners answer that. Each attaches to a microphone, a file player, a tone, or a playing video, and here both listen to the microphone:

let mic = AudioInput()
var speech: SpeechListener!
var ears: SoundClassifier!

override func setup() {
    speech = SpeechListener(of: mic)
    ears = SoundClassifier(of: mic)
    try? mic.start()
}

Two listeners on one microphone is fine. The source is tapped once, and the audio goes to everything attached to it.

Words as they are spoken: SpeechListener

SpeechListener turns talking into text as it is spoken. Use it for captions a performer speaks into being, or for words that change the picture ("red", "stop", "again"). It runs Apple's on-device speech recognizer, the SpeechAnalyzer framework, so nothing is uploaded, and it asks for no permission beyond the microphone's. The first use of a language may download its model, which takes a moment and needs the network once. Until the model is in place, unavailableReason says so.

Recognition guesses early and corrects itself as it hears more. Each line on the left of the figure is the same recognizer handed a little more of one sentence. Four fifths of the way through, its best guess was that the fox jumped over the lace, which fit what it had heard so far. So a listener gives you two reads, one for each job. In this fragment, ink is a Color property of your sketch that the word "red" changes:

drawText(speech.caption, at: center)              // the guess, which may change
for phrase in speech.phrases() {                  // what it committed to
    if phrase.text.lowercased().contains("red") { ink = .red }
}

Draw the guess, act on the commitment. caption is the running best guess, including its last words, which may still change. It is trimmed to its last few words, so it does not run off the canvas. phrases() drains what the recognizer has finished with, and hands out each phrase once, so it is safe to trigger from. A trigger on caption acts on a word that may be taken back.

Sounds with names: SoundClassifier

SoundClassifier names the sounds around the microphone. It knows about three hundred everyday ones, from clapping, dog_bark, knock, glass_breaking, and police_siren, through every family of instrument, to silence. Use it to let the room's sounds place marks: a clap, a door, a laugh. It is Apple's Sound Analysis framework and the classifier Apple ships with it.

The right half of the figure above is the caution. Its three sounds are a sine wave, a tap every quarter second, and bursts of noise. The classifier called them a tuning fork, a click, and a hammer, which is fair. But it always answers, whatever it hears, so a small number means very little. Read the top label, keep a threshold, and treat the rest as opinion.

Like SpeechListener, it gives you a level and a trigger. In this fragment, marks is an array in your sketch, and Mark is a small type of your own that remembers where to draw:

let musical = ears.confidence(of: "music")                   // rises and falls
for event in ears.events() where event.label == "clapping" { // happens once
    marks.append(Mark(at: center))
}

An event fires when a label's confidence crosses the threshold, 0.6 by default, from below. A sound that goes on is one event, not one per moment it is still going. events() drains what arrived since the last call, the way phrases() does. For a mark that fades, read timeSinceHearing("clapping"), which you can ask every frame without using anything up. An iPhone can listen the same way from across the room, and Chapter 36 reads its sounds with the same reads.

Both listeners are live only. Under an export nothing is playing, so nothing is heard, and both say so in unavailableReason instead of going quiet. When an exported sketch needs words, work them out first, in setup(). Here caption is an optional String property of your sketch, and voice.m4a sits in its folder:

override func setup() throws {
    caption = try? waitFor {
        try await SpeechListener.transcribe(resource: "voice", withExtension: "m4a", in: .module)
    }
}

waitFor runs the work and waits for its answer, as in Chapter 34. SoundClassifier.classify(resource:withExtension:in:) names the sounds in a file the same way, and it answers straight away, with no waitFor. These forms are deterministic, which is the promise the seed made in Chapter 4. The same audio gives the same words every time, so a captioned export renders the same on every run. A system update can change the recognizer, and with it the words. The Audio/Listening example is the live one, with a caption you can talk into and marks you can clap at.

Where this comes from

The idea that a signal splits into pure waves is Joseph Fourier's, from his 1822 book on heat. The fast algorithm that made it real-time, the FFT, is Cooley and Tukey's (1965), and Ollin runs Apple's implementation.

Detecting arrivals by spectral flux is a standard technique from music information retrieval. Bello and colleagues survey it in their onset-detection tutorial (2005). The real-time recipe Ollin follows is Böck, Krebs, and Schedl's online method (2012). The pitch and pitch-class entries name their own sources.

The audio-reactive visual has a long lineage. It runs from Oskar Fischinger's hand-drawn sound films through the oscilloscope and music-visualizer traditions. It reaches today's VJs, who mix video live to music, and live coding. Full credits are in the project's attribution notes.

Go deeper

• Audio: every source and read, bands, beats, the note heard as pitch and nearestPitch and the twelve classes as chroma, and feeding the AudioAnalyzer yourself.
• Listening: the caption and transcript reads, phrases as triggers, languages and their models, the sound vocabulary and its threshold, bringing your own classifier, and the deterministic one-shot forms.
• Synthesis: Synth and its voices, which Chapter 39 is about, since a sketch that listens usually ends up playing too.
• Appendix B draws this chapter's math, one picture per idea: Sound as numbers.
• Worked examples: Examples/Audio/Listening, Examples/Audio/Spectrum, and Examples/Audio/Tuner.

Contents · Previous: Chapter 36, The iPhone as a sensor · Next: Chapter 38, Controls and signals