D. The complete toolbox
This appendix lists everything Ollin can do, one row for each capability, in the order the guide teaches it. The rows are grouped by part and chapter. Each row says what the capability is in plain words, links the section that teaches it, and links its page in the reference.
Use it as an index when you remember a technique from the guide and want to find it again. You can also read it as a list of what you haven't tried yet. A capability taught in more than one chapter sits with the chapter that teaches it most fully. Its row also links the other sections, each marked with its chapter. If something you want isn't here, it isn't in the framework yet, and the roadmap lists what's ahead.
Contents: The ideas underneath · Part I: Seeing something move · Part II: Systems that come alive · Part III: Shapes, lines, and marks · Part IV: Pixels and light · Part V: The third dimension · Part VI: The world coming in · Part VII: Out into the world
The ideas underneath
The capabilities all assume a few ideas. These seven pages in the reference explain them, one screen each, and sit between a chapter and a function signature.
| Idea | What it says | Taught in | Reference |
|---|---|---|---|
| The frame | A drawing call records rather than paints, call order is draw order, and nothing is kept for you | Ch 1: Once, then every frame, Shapes and ink | The frame |
| Where a point is | The canvas frame and its units, transforms as a change of frame, and the other frames a sketch meets | Ch 1: Where things go, The canvas is not the window; Ch 6: Moving the paper: translate, rotate, and scale | Where a point is |
| Layers | An off-screen picture you draw into and then read back: what one costs, and when you need one | Ch 19: A drawing you can hold: render targets | Layers |
| What survives a frame | What carries into the next frame, and what a batch, a reload, and a checkpoint hold | Ch 19: The canvas that keeps everything: noClear, Record it once: batches; Ch 45: Picking up where it left off: checkpoints and @Saved | What survives a frame |
| Why a run repeats | The seed and the fixed export clock, and the four things that break a repeat | Ch 4: Seeds: randomness you can keep, Finding a seed to keep; Ch 3: When a frame takes too long; Ch 41: The recipe in the file | Why a run repeats |
| Light and color | Why the middle of a frame is linear light, and what tone mapping and dithering do at the end of it | Ch 9: A picture made of pictures; Ch 19: Brighter than the screen: toneMap | Light and color |
| Values and bare calls | The typed values under the short calls, and when to keep the value rather than make the call | Ch 10: An arrow you can draw: Vector2; Ch 15: Shapes you can hold | Values and bare calls |
Part I: Seeing something move
Chapter 1: Hello, Ollin
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Checking the machine | ollin doctor checks the toolchain, the GPU, the command, and the permissions, and prints the line that fixes each problem; shell completions; --list-params for the parameters a sketch declares | What you need, A shorter way to run things; Ch 41: Passing the parameters back in | Doctor |
| The examples gallery | swift run OllinExamples shows the examples in a tree and runs the one you pick, with its parameters beside it | The gallery | Examples/ |
| Live reload | swift run OllinLive Sketch.swift watches the file, and each save swaps the change into the open window | Your first sketch | Sketch |
| Swift itself | As much of the language as a sketch needs | Your first sketch; Appendix A | Swift quick reference |
| The sketch lifecycle | setup() runs once, and draw() runs every frame at the display's rate | Once, then every frame | Sketch |
| Canvas and window | canvasSize and its presets, and windowMode (auto, fixed, resizable) | The canvas is not the window; Ch 44: Filling the screen or fitting in it: windowMode | Canvas |
| Looking closer | viewControl() lets you drag to pan and scroll to zoom the live view, and leaves the canvas as it is | Looking closer | Drawing |
| Normalized coordinates | uv(u, v) gives the canvas point at fractions from 0…1 across and down, and scale, the shorter edge over 1000, sizes marks to the canvas | Placing things without pixels | Canvas |
| The shape catalog | Circles, rectangles, lines, ellipses, triangles, stars, hearts, arcs, rings, and n-gons, drawPolygon and drawPolyline through points you give, and about twenty more | Shapes and ink; Ch 6: Grids that aren't square: hexGrid and triangleGrid; Ch 7: The diagonal tile: ten print | Drawing |
| Ink state | fill, stroke, and strokeWeight, then caps and joins; the reference adds hollow band mode and stroke alignment | Shapes and ink; Ch 5: Putting it together: a meadow in the wind; Ch 7: No coins at all: kolam and sona | Drawing |
| Rounded rectangles | drawRect with cornerRadius:, or cornerRadii: for a different radius at each corner | Shapes and ink | Drawing |
| Mouse | mouseX, mouseY, and mouseIsPressed, the press and release hooks, and mouse and previousMouse as points | The mouse joins in | Input |
@Param parameters | A property marked @Param becomes a control in the inspector, chosen by its type: a slider, a stepper, a color well, a pad, and more | Putting it together: a breathing ring; Ch 38: One parameter, three hands: binding and smoothing | Parameters |
| Saving tuned values | The inspector's Save parameters button writes the values you tuned back into your .swift file | Saving the values you tuned | Parameters |
| Dragging a shape | Hold Command in the live window and drag a shape, and the numbers that placed it change in your file | Moving something by hand; Ch 43: Moving a shape on stage: the Command-drag | DragToEdit |
| One frame on request | redraw() draws one more frame of a sketch that noLoop() stopped, so a still sketch answers a click and costs nothing between | One more frame when asked: redraw() | Sketch |
| The pointer | pointerShape(_:) picks one of the system's pointer shapes over the canvas, and hidePointer() and showPointer() take it away and bring it back | The pointer's shape, or none | Input |
| The open panel | chooseFiles(withExtensions:) shows the system's open panel without stopping the frames; the pick arrives in chosenFiles() and filesChosen() | A file the person picks: the open panel | Input |
| The frame on the clipboard | copyFrame() puts the frame the window shows on the clipboard as a PNG | The frame on the clipboard: copyFrame() | Recording |
| Single-file sketches | ollin new <file>.swift and ollin <file>.swift: one file is a sketch, and it runs from any folder | A shorter way to run things | SingleFile |
| Project generator | ollin new <Name> and ollin generate write a folder from a template, and you can watch the template run first | When one file isn't enough | ProjectGenerator |
| A package's sketch, live | ollin in a package's folder runs its sketch in the live window, every .swift file of the target compiled together and a save to any of them reloading it; --target picks one of several | When one file isn't enough | SingleFile |
| The reference offline | ollin docs, ollin api, and ollin examples read these pages, the calls, and the examples in the terminal, with a search across them; ollin site writes them out as a website | Looking something up without leaving the terminal | Reference |
| Ollin on one page | The lifecycle, the calls by area, units and conventions, the command line, and the mistakes that fail with no error | Go deeper | Quick reference |
Chapter 2: Color that works
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Color values | Named colors, hex, RGB, HSB, and alpha | Naming a color, Thinking in hue | Color |
| Perceptual color | The OKLab family, and Color.mix, which blends the way your eye expects | Mixing you can trust | Color |
| Lightness and related colors | OKLCH and OKHSL keep hues at one lightness, and lighter(by:), darker(by:), and complement make colors related to one you have | Hues that weigh the same: lightness | Color |
| Palettes and ramps | ColorBrewer sets, harmonies built from one color, smooth ramps, and cosine palettes | Kits you carry: Palette and Ramp | Color |
| Colormaps | Viridis, magma, and turbo: ramps for turning data into color | Kits you carry: Palette and Ramp | Color |
| Gradient paint | Linear, radial, conic, and along-path gradients for any shape's fill or stroke | Gradients as paint | Color |
| Palettes from a file | loadPalettes reads every palette in a file of hex codes, CSV or TSV, JSON, or Adobe .ase swatches, and loadPalette reads the first | Palettes from a file | Color |
| Accessibility | See your colors as the three kinds of color blindness see them, check that a palette's colors stay apart, and read the reduce-motion setting | Will everybody see it?; Ch 3: When somebody would rather it stopped: reduced motion | Accessibility |
| Spectral color | Spectrum: color as light by wavelength, .paint mixing where blue and yellow make green, blackbody light, and thin-film, diffraction, and paint-mix effects | Blue and yellow make green: mixing in .paint, Light by wavelength: Spectrum | Spectral color |
| Shades that drift | Palette.shades and Ramp.shades build a ramp of one color whose hue warms toward the light and cools into the shadow | Shades that drift: Palette.shades | Color |
| Color distance | distance(to:) measures how far apart two colors look, in OKLab or by CIEDE2000; CIELab brings a swatch's numbers in; nearest(to:) finds the palette entry most like a color | How far apart two colors look: distance(to:) | Color |
Chapter 3: Motion and time
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| The clock | time, deltaTime, frameCount, and frameRate; noLoop() for a still | The clock; Ch 1: It moves on its own; Ch 44: Wallpaper and a menu-bar strip | Sketch |
| Everyday math | map, lerp, clamp, fract, step, smoothstep, and .tau | map and lerp: moving between ranges, The circle behind sin, Shaping time; Ch 11: Putting it together: the wrecking ball | Math |
| Loop phases | loopProgress and pingPong turn the sketch clock into a cycle from 0 to 1 | map and lerp: moving between ranges | Math |
| Easing | The catalog of curves, and @Eased and @Sprung values that glide or spring to what you assign | A catalog of curves, Values that chase a target: @Eased and @Sprung | Animation |
| Perfect loops | Declare loopDuration, and --export-loop renders one lap that ends where it began | Putting it together: a loop that never ends; Ch 41: Motion: video and GIF | Export |
| Sway | sway moves a value from one end of a range to the other and back once per lap, along one of five paths; wave names the parts of a sine | The sway you write over and over: sway | Animation |
Timeline | Keyframes with easing on each segment, for sequences you choreograph | Choreography: Timeline | Animation |
| Timers | every, after, and everyFrames say whether a moment has arrived, for events like adding a dot every two seconds | Doing something now and then: every, after, and everyFrames | Animation |
| Knowing an export | isExporting is true from setup() to the last frame written in any export, so work spread over live frames is done at once | When every frame is written: isExporting | Sketch |
Chapter 4: Randomness
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Random | Seeded random, Gaussian values, choices plain and weighted, and shuffles | Rolling dice, Seeds: randomness you can keep, Letting chance decide, Two shapes of chance: uniform and Gaussian | Random |
| Variations | variation: the seed a run grew from, stepped and rolled from the inspector's Variation card | Finding a seed to keep | Variations |
| Keyboard | key, keyCode with its typed KeyCode values, isKeyDown(_:), and the press and release hooks | Finding a seed to keep; Ch 31: Someone to be in there: the character controller | Input |
| Contact sheets | --export-grid tiles one frame per seed into a labeled sheet, and --seed renders the one you keep | Finding a seed to keep; Ch 41: Picking a keeper: a contact sheet and a fixed seed | Variations |
| Random walks | randomWalk, a tangle that pools in one patch; levyFlight, tight clusters joined by long jumps; and selfAvoidingWalk, a path on a grid that never enters a cell twice and fills its region | Three walks, three rules: randomWalk, levyFlight, and selfAvoidingWalk, The random walk, by hand | Walks |
| Markov chains | MarkovChain from a drawing sketch: a pick whose odds follow the last pick, with the table declared by hand | A roll that remembers the last one: MarkovChain; Ch 40: What comes next: a MarkovChain | Random |
| Blue noise | Poisson-disk scatter: points at random, but kept a set distance apart | Darts that keep their distance: poissonDisk | BlueNoise |
| Low-discrepancy sampling | Halton and Sobol points: even at any count, and adding points never moves the ones already placed | Points that never move: haltonPoints and sobolPoints | LowDiscrepancy |
| Breeding by eye | Population and Genome: a handful of genomes, bred from the ones you pick | Sixteen things and no opinion about them: Population | Evolution |
Chapter 5: Noise
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Noise | noise, signedNoise, the loop: forms that come back to their start (a value, a line, a field, or a volume), and layered fbm; curlNoise and curlField; noiseFields for work off the main thread | Random that remembers: noise, Drift that swings: signedNoise, Coming home: the loop: argument, Layering shape and detail: fbm; Ch 14: Points the field carries: advection | Noise |
| The noise family | simplexNoise, cellular worley (the nearest point, the second nearest, or the border between), ridgedFbm, turbulence, and warpedFbm, all pinned by noiseSeed; gaborNoise, which takes a seed: of its own; and tilingFbm for a texture that wraps | A family of fields; Ch 27: A picture from three sides: triplanar | Noise |
Chapter 6: Grids and repetition
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
Grid | Rows, columns, padding, and gutters in one loop instead of two | One loop, not two: grid | Geometry |
| Transforms and the state stack | translate/rotate/scale, scoped with withState { } | Moving the paper: translate, rotate, and scale, Symmetry by hand: rotating around the center | Drawing |
| Kaleidoscope symmetry | symmetry(n, mirrored:) repeats your 2D drawing in n wedges around the origin, with a mirrored copy in each when you ask | The fold, done for you: symmetry | Drawing |
| Rectangles and circles | Rectangle and Circle as regions: a clip, a text box, insetting, and hit testing | Drawing inside a shape: withClip; Ch 8: Both edges flush: the box form and justification | Geometry |
| Clipping | withClip(shape) { }: drawing inside the block lands only within the region; nesting intersects | Drawing inside a shape: withClip | Drawing |
| A view box | withViewBox clips to a region and moves the coordinates too, so the block inside draws as though the region were the canvas | A cell that is a whole canvas: withViewBox | Drawing |
| Hex and triangle grids | The other regular grids, with hex distance, neighbors, and picking the cell under a point | Grids that aren't square: hexGrid and triangleGrid | Tiling |
| Recursive subdivision | subdivide: uneven panels from splitting the canvas again and again, in halves or in quarters | Uneven panels by recursion: subdivide | Tiling |
| Ulam spiral | The counting numbers written in a square spiral, so a test on each one becomes a picture, and the primes fall on diagonals | Walking a grid: numbers in a spiral | UlamSpiral |
Chapter 7: Tiles that cover the plane
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Truchet tiles | drawTruchet puts one tile in each cell at a random turn, and loops and mazes appear; truchet hands back the strands as values | Tiles that agree at their edges: Truchet, Strands as values: truchet | Truchet |
| Ten print | One of two diagonals in each cell by a coin flip, joined into the long paths they form | The diagonal tile: ten print | TenPrint |
| Hitomezashi stitching | One bit per grid line sets where its dashes start; stitches and a two-tone cloth appear | One coin per line: hitomezashi | Hitomezashi |
| Mazes | Perfect mazes, with one route between any two cells, carved three ways and drawn as line work | One route between any two: perfect mazes | Tiling |
| Kolam and sona | One line bounced around a field of dots until it closes, in gcd(rows, columns) loops, with walls to steer the count | No coins at all: kolam and sona | Kolam |
| Celtic knotwork | The same line given width and an over-under rule, its bands broken where each cord passes under | The same line, woven: Celtic knotwork | Knotwork |
| Polyominoes | Pieces of joined squares, the twelve pentominoes, and a search that fits a set of them into a region exactly or says it cannot | Pieces that have to fit: polyominoes | Polyominoes |
| Wave Function Collapse | Tiles with labeled edges solved by constraint propagation, or a texture learned from a small sample picture | Every neighbor must agree: Wave Function Collapse, Or hand it a picture instead: overlapping WFC | WaveFunctionCollapse |
| Aperiodic tilings | Penrose kites/darts and rhombs with matching-rule arcs, Wang edge-matching quilts, girih star patterns over any polygons, and the spectre (the einstein) | Tiles that never repeat: aperiodic tilings | Aperiodic tilings |
| De Bruijn sequences | A cyclic run holding every window of a given length exactly once, with the reader that turns a window back into a position | Every window once: de Bruijn sequences | De Bruijn sequences |
| Hyperbolic tiling | Regular polygons meeting more to a corner than flat paper allows, drawn in the Poincaré disk, with a checkerboard by parity, rings by depth, and a viewpoint you can pan | More room than the page has: hyperbolic tiling | Hyperbolic tiling |
| Parquet deformations | A tiling whose tile changes shape as you read across it, the geometry carried on the lattice's shared edges so nothing comes apart | A tile that changes as you read: parquet deformations | Parquet deformations |
Chapter 8: Words
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Drawing text | drawText with size, alignment, wrapping in a box, justification, and text along a path | Saying something: drawText, Both edges flush: the box form and justification, Letters that move: drawText with a closure | Text |
| Three font kinds | Outline (any installed font), bitmap, and plotter stroke fonts | Three kinds of letters: outline, bitmap, and stroke fonts | Text |
| Text as geometry | textToShapes: letters become Shapes you can warp, resample, and rebuild | Text as geometry: textToShapes | Text |
| Letters one at a time | drawText with a closure runs your code once per letter, so each one can move, turn, or change color | Letters that move: drawText with a closure | Text |
| Every script, and columns | Arabic, Devanagari, Thai, and Japanese shaped by the system; columns down the page that fill right to left, as Japanese does, or left to right, as Mongolian does; and hanging punctuation at the edge of a box | Shaping and direction: text in any script, Down the page: vertical text, Both edges flush: the box form and justification | Text |
| Missing letters | fontsUsed(for:) names the faces a line of text borrowed, and textMissingCharacters lists the characters the current font cannot draw | What the font could not do: fontsUsed and textMissingCharacters | Text |
Chapter 9: Pictures and data
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Images | Load PNG, JPEG, HEIC, and more; draw, tint, scale, and crop them, fit one into a box of another shape, and draw one cell of a sprite sheet | A picture on the canvas: loadImage and drawImage, A working copy: resized and cropped, The box is never the right shape: fit | Images |
| Sample photographs | Twenty bundled pictures and a short film, with their credits, to try a technique on | A working copy: resized and cropped; Ch 34: When there is no camera: stills and footage | Sample photographs |
| Pixels | Read and write a pixel with image[x, y], and build an image from nothing | An image you can ask: image[x, y] | Images |
| Atlas text | textMode(.atlas) draws a lot of text quickly | Putting it together: a picture painted with type | Text |
| Files dropped on the window | filesDropped() runs when files land on the window, and droppedFiles() hands over their paths | A picture you drop on the window | Input |
| Seam carving | A picture made narrower or wider by taking away or duplicating the paths that carry the least, so what carries texture keeps its shape | Making it narrower without squashing it: seam carving | SeamCarving |
| Palettes from a picture | Palette(extractedFrom:) pulls a palette out of a photograph | Palettes from a photograph | Color |
| Histograms | histogram(of:) counts a picture's pixels by lightness or by one channel, and percentile(_:) reads its working black and white | Palettes from a photograph | Color |
| Dithering | dithered rebuilds a picture in fewer colors, scattering the ones it has so the eye reads the tones between | Fewer colors than the picture needs: dithering | Color |
| A small picture drawn big | imageSampling(.nearest) keeps every pixel a hard square when a picture is drawn larger than it is, with no enlarged copy | Drawn big, cell by cell | Images |
| Glyph mosaic | A picture rebuilt as a grid of characters, each chosen by its measured ink in the active font | One mark per cell: drawGlyphMosaic and drawHalftone | GlyphMosaic |
| Halftone | A picture as a print's dot screen: each dot sized to its tone on a turned grid, drawn as circles a plotter can follow | One mark per cell: drawGlyphMosaic and drawHalftone | Halftone |
| Photo mosaic | A picture rebuilt out of many smaller pictures, each cell taking the nearest by average color | A picture made of pictures | Photo mosaic |
| Stippling | Dots packed to match a picture's tone (weighted Voronoi) | A picture as one line: stippling, singleLine, and spanningTree | Stippling |
| Single-line drawing | One unbroken tour through a stipple: the picture as a single closed line (TSP art) | A picture as one line: stippling, singleLine, and spanningTree | SingleLine |
| Spanning-tree drawing | The same stipple joined by the minimum spanning tree: the picture as branching veins | A picture as one line: stippling, singleLine, and spanningTree | SpanningTree |
| Hilbert order | The same stipple sorted along a space-filling curve: one line by a single sort, neighbors kept together | A picture as one line: stippling, singleLine, and spanningTree | Hilbert order |
| String art | One continuous thread over rim pins, each chord wound toward the darkness that remains | A picture wound from thread: StringArt | StringArt |
| Pixel sorting | Runs of a picture's own pixels, bounded by brightness, reordered into streaks | Sorting the pixels | PixelSorting |
| Autostereogram | A depth map hidden in a repeating pattern, so a surface nobody drew stands out of the page | A picture that hides a shape | Autostereogram |
| Tables and JSON | loadTable reads a CSV or TSV with typed reads by column name and numbers(_:) for a whole column; loadJSON reads a document by name and index | Reading a table: loadTable, From numbers to marks, Documents with a shape: loadJSON | Data |
Part II: Systems that come alive
Chapter 10: Vectors, gently
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Vectors | Vector2 and Vector3, with operators for adding and scaling, length, normalized, lerp, rotated, and with(x:) for a copy with one part changed | An arrow you can draw: Vector2, Arrow arithmetic: adding, subtracting, scaling, and dividing, A point part of the way: lerp(to:), Length and direction: length, normalized, and limited(to:), Position, velocity, acceleration; Ch 26: Where things are in the world: Vector3 and world units | Geometry |
| Arrows | drawArrow(from:to:) draws a vector as an arrow, with a head you can size | An arrow you can draw: Vector2 | Drawing |
Chapter 11: Forces and physics
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Soft physics | Verlet particles and springs, with disk collisions: ropes, cloth, and blobs | A world that pushes back: World and particles, Springs: connect, rest length, and pin | Physics |
| Rigid physics | Bodies, colliders, and joints in the same World: stacks, chains, and hinged arms | Rigid bodies: Body and colliders, Hinges and a handle: .revolute and grab | Physics |
| Grabbing | grab: pull a rigid body with the mouse, through a joint | Hinges and a handle: .revolute and grab | Physics |
| Fracture | Breaking a shape or a solid into pieces that fit back together, evenly or crowded around a point of impact, each piece ready to be a rigid body | Breaking things; Ch 30: What hit what: contacts | Fracture |
| Inverse kinematics | IKChain: a segmented limb that reaches for a target, or a rope dragged by its tip | A limb that reaches: IKChain | Motion |
| Gravity at scale | NBody: thousands of bodies pulling on each other, with seeded disk and cluster scenes | Gravity at scale: NBody | Motion |
| Force-directed layout | ForceLayout: a graph that untangles itself into an even web you can grow, pin, and drag, drawn with drawGraph | A graph that lays itself out: ForceLayout | ForceLayout |
| Soft bodies from outlines | addSoftBody(from:): any closed Shape as a squishy body that keeps its area, springs back toward its shape, and keeps other outlines out | A squishy thing from any outline: addSoftBody(from:) | Physics |
Chapter 12: Flocks and swarms
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Steering vehicles | Seek, flee, arrive, pursue, wander, follow, and contain, as forces you add together | A creature that steers: Vehicle, Seek and arrive: aiming at a target and stopping there, Roaming: wander, and trails from noClear; Ch 10: Steering: the chase | Steering |
| Flocking | Boids from separation, alignment, and cohesion, which can also follow a field | The flock, assembled: Boids, Three rules make a flock: separation, alignment, and cohesion | Boids |
| Spatial index | What is nearest, what is within reach, and what is inside a box, found without reading every point | The trick that keeps it cheap: SpatialIndex | Spatial index |
| Pursuit | Runners that each head at the one they chase: the ring of dogs, whose paths are logarithmic spirals, and a pursuer after a quarry that runs straight | Everyone chasing somebody: pursuit curves | Pursuit |
| Crowds | Crowd: walkers that avoid each other by each taking half of every swerve, walls no walker passes, even in a jam, and lanes that form on their own | A crowd that makes room: Crowd | Crowds |
| Coupled oscillators | Kuramoto: oscillators that fall into step past a critical coupling, with coherence to watch it happen; each one listens to the whole crowd, or to its neighbors on a ring, a square or hex lattice, or a graph of your own | Falling into step: Kuramoto; Ch 24: In step with the neighbors: oscillators on a lattice | Coupled oscillators |
Chapter 13: Growing things
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| L-systems | Grammar rewriting drawn by a turtle, for ferns and other plants, and parametric rules that carry numbers | Rules that rewrite: L-systems, When the rules need arithmetic: parametric L-systems | LSystem |
| Space colonization | Branching growth toward scattered attraction points, as in the veins of a leaf | Growth that claims space: space colonization | SpaceColonization |
| Diffusion-limited aggregation | Walkers that freeze on contact into dendrites and frost | Growth by chance: diffusion-limited aggregation | DiffusionLimitedAggregation |
| Shape grammars | Rules over labeled shapes in place: ice-ray lattices, building fronts, nested figures | Rules over shapes, not symbols: shape grammars | ShapeGrammar |
| Dielectric breakdown | Lightning grown where the solved field is strongest, with eta choosing between a bush and a bolt | Growth by voltage: dielectric breakdown | DielectricBreakdown |
| Differential growth | A line that folds like coral as it grows, because it pushes away from itself | Growth by crowding: differential growth | DifferentialGrowth |
| Crack growth | Perpendicular cracks subdividing the plane into city blocks, with a wash of translucent grains beside each crack | Growth by collision: crack growth | CrackGrowth |
| Meander | A river migrating by curvature, cutting off oxbow lakes, and keeping its past channels as scars | Growth by wandering: meander | Meander |
Chapter 14: Fields and flow
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Flow fields | Direction fields, streamlines (free and evenly spaced), advection | A direction at every point: FlowField, Following the flow: streamlines, Lines that keep their distance: evenly spaced streamlines, Points the field carries: advection | FlowField |
| Isolines | Level curves of any field or an image's tone (marching squares): metaball outlines, contour maps | Where the field equals something: contours | Isolines |
| Chladni figures | The standing-wave patterns of a driven plate in closed form, as a field, a generator, or nodal contours | Standing waves: Chladni figures; Ch 18: Closed forms at every pixel: the pattern fields | Chladni |
| Line integral convolution | .lineIntegralConvolution(length:field:) and .streaked(along:length:field:) brush a layer's grain, or a picture, along a direction field, so the whole field shows at once as fine streaks | The whole field at once: line integral convolution | Effects |
| Fitting | RadialBasis, a smooth field of numbers, vectors, or colors that passes through values you know at a few scattered places, and Fit.minimize, a downhill walk that finds a few numbers against a cost you write | A field you pin down yourself: radial basis functions; Ch 15: The circle they were scattered around: Fit.minimize | Fitting |
| Slopes | A fitted field's exact gradient(at:) and derivative(at:along:), and gradient(at:field:) and derivative(at:function:) for any function you write | A field you pin down yourself: radial basis functions; Appendix B: A number field's slope: the gradient | Fitting, Math |
Part III: Shapes, lines, and marks
Chapter 15: Shapes as material
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Paths, shapes, and contours | Curved outlines you build, hold, and edit, and respace with resampled | Contours, shapes, and holes; Ch 8: Text as geometry: textToShapes | Geometry |
| Shape booleans and offsets | Union, intersect, subtract, xor; grow and shrink regions | Shape arithmetic: the booleans, Growing, shrinking, and thickening: offset and stroked | Geometry |
| Stroke as shape | Turn a stroked line into a filled region for booleans and plotting | Growing, shrinking, and thickening: offset and stroked | Geometry |
| Voronoi and Delaunay | Territories and neighbor networks from points, with Lloyd relaxation | Territories and neighbors: Voronoi and Delaunay | Voronoi |
| Power diagrams | Voronoi cells for points that carry a weight, so the line between two circles falls where a point is equally far outside both, and a circle that touches no other sits inside its own cell | Territories and neighbors: Voronoi and Delaunay | Voronoi and Delaunay |
| Hulls | The convex hull, the rubber band around a point set, and the tighter wraps: the concave hull and the alpha shape, with its islands and holes | What shape are these points? Hulls and alpha shapes | Geometry, Hulls |
| Medial axis | A shape reduced to its skeleton, each point carrying the radius of the largest circle that fits there | The skeleton inside: the medial axis | MedialAxis |
| Hatching | Hatching(spacing:angle:crossHatches:) fills a shape with lines a pen can draw | Lines for a pen: hatching | Export |
| Outline questions | Tangent, normal, nearest point, the piece between two fractions, crossings, simplify, and rounded or cut corners | Asking an outline where it goes: contour questions | Geometry |
| Bending outlines | An outline bent along a curve, pulled toward a point with its ends held, smoothed with its corners kept, cut into pieces at its corners, and traced into a few cubic curves | Bending an outline: warped, pulled, split, and fitted | Geometry |
| A curved patch | A four-sided region whose sides are curves (or four corners, the bilinear map), and an outline bent into it from a rectangle, such as lettering on a waving banner | Bending an outline into a patch: CoonsPatch | Geometry |
| Enclosed regions | Every patch a drawing's lines wall off, each as its own Shape, holes included, with Circle.contour(segments:) and Rectangle.contour as the ways in | The regions a drawing encloses: regions(enclosedBy:) | Geometry |
| Points inside a shape | randomPoints(in:count:) and poissonDisk(in:radius:) scatter points inside an outline, and randomPoints(along:count:) along it | Chance inside an outline: points in a shape | Geometry, BlueNoise |
| Shapes that cross | LineSegment, Arc, and Ellipse as values beside Circle and Rectangle, each with its measures and its nearest point, and crossings(with:) between any two of them through PlaneCurve | Shapes that know where they cross: segments, arcs, and ellipses | Geometry |
| Circle packing | Circles that grow until they touch, seeded so a run repeats | Packing: circles that grow until they touch | Packing |
| Apollonian gasket | A circle filled with circles that touch, smaller and smaller down to a size you set | Packing: circles that grow until they touch | Tiling |
| Ford circles | A circle for each fraction, touching its neighbors and overlapping none, over the Farey sequence, with a Fraction type | A circle for every fraction: Ford circles | Ford circles |
| Shape packing | Packing measured against the shapes' outlines, so small shapes settle into a star's notches, all at once or filled in over time | Packing shapes, not circles | ShapePacking |
| Rectangle packing | Rectangles of given sizes fitted into one bin at their own sizes, the placed and the ones that did not fit, with rotation as an option, for a contact sheet, a collage, or an atlas | Packing rectangles: packRectangles | RectanglePacking |
| Straight skeleton | The ridges a shrinking outline leaves, with mitered insets cut from one build | The straight skeleton | StraightSkeleton |
| SVG import | Vector artwork read into shapes: draw it as authored, or use its outlines as geometry | Shapes from a file: SVG import | SVG |
| Crease patterns | The flat sheet a folded or cut thing comes from: the Miura fold, folding rigidly in three dimensions, rotating-squares kirigami, and the two local rules a flat fold must obey | Mountains and valleys: CreasePattern and the two laws, A sheet that opens all at once: the Miura fold, Origami allowed to cut: rotating squares | CreasePattern |
Chapter 16: Curves and figures
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Hobby's spline | drawCurve(_:closed:spline:) with .hobby draws a curve through your points that reads as drawn by hand, and HobbySpline(through:) hands it back as a path | A curve that reads as drawn: Hobby's spline | Geometry |
| Classic curves | Phyllotaxis, Lissajous figures, roses, superellipses, supershapes, spirograph gears, guilloche rosettes, the harmonograph, spirolaterals, Chaikin corner-cut smoothing, and a moving average that keeps every point, none of them random | Curves you can write down: the classic curves, A walk that comes home: spirolaterals | Classic curves |
| Clothoid | The curve whose bend grows at a steady rate (the Euler or Cornu spiral): easements into a turn, smooth paths through waypoints, corners a vehicle could take, and the curve that fits two points and two headings | A corner a car could take: clothoids | Clothoid |
| Envelopes and caustics | The curve a moving line leans on, the bright curve reflected or bent light crowds along, and where a wavefront gets to next | The curve a family of lines leans on: envelopes and caustics | Envelopes and caustics |
| Fourier epicycles | A closed outline rebuilt as a chain of spinning circles, with the number of terms setting the detail | Circles on circles: Fourier epicycles | Epicycles |
| Shape morphing | One shape turned into another; holes grow in and out, and each in-between is a shape you can use | One shape becoming another: shape morphing | Morphing |
| Anamorphosis | A plate that reads only in a mirrored cylinder standing on the page, from one viewing spot: points, contours, and shapes mapped to the page, and the arc of the cylinder an eye can see | A drawing only a mirror can read: anamorphosis | Anamorphosis |
| Turtle | A pen you drive from code: forward, back, left, right, pen up and down, and a remembered pose for branches, every run an ordinary contour | A pen you drive: the turtle | Turtle |
Chapter 17: Marks and media
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Variable-width strokes | strokeProfile(_:): width shaped along the path, so a line tapers, swells, or carries a profile you wrote | A width that changes along the path: strokeProfile | Drawing |
| Stroke dynamics | StrokeMark and drawMark: width and opacity set by how fast and how hard a mark is made; pressure reads a Force Touch trackpad or a tablet, and stylus a pen's tilt and twist | Painting as it happens: stroke dynamics | Marks, Input |
| Brushes | strokeBrush(_:): a stamp repeated along the path instead of one continuous ribbon, with its own tip, spacing, jitter, and angle | Stamps along the path: brushes | Marks |
| Dashed strokes | strokeDash(_:): a line with gaps, the pattern walked along the path so corners and curves keep their rhythm | A line with gaps: strokeDash | Drawing |
| Paper marbling | Ink floated on a bath and raked: drops, tines, combs, and vortices, each a transform of vector outlines | Ink on water: marbling | Marbling |
| Watercolor | Pigment from one polygon, deformed and stacked at low opacity: dense cores and uneven edges | Pigment from a polygon: drawWatercolor | Watercolor |
Part IV: Pixels and light
Chapter 18: Your first shader
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| User shaders | Write shade(uv, info) and run it as a generator, a filter, or a two-input combine; #include pulls in a helper file several shaders share | The first shader: shade, uv, and generate, Drawing with distance: smoothstep as an edge, Time, mouse, and parameters: info and params, One helper, two shaders: the include line | Shaders |
| The shader library | Hashing, noise, SDFs, palettes, and OKLab, spliced into each shader and kernel | The shader library: fbm, hash12, palette, and the distance functions | ShaderLibrary |
| A shader as paint | fill(shader) and stroke(shader) run shade at every pixel a shape covers, uv across the shape's own box, with the edge distance, the fraction along a stroke, and the canvas position in info; a mesh takes it as its surface | Paint, not a layer: fill and stroke with a shader | Color |
| Design generators | Ten built-in shaders that draw a finished graphic at their defaults, among them mesh gradients, god rays, spirals, orbiting dots, grain gradients, and pulsing borders | Backdrops that come composed: the design generators | Effects |
| Pattern fields | Closed-form fields that move only by the phase you pass: quasicrystal, moire, gyroid, phyllotaxis, hex pulse, and Chladni | Closed forms at every pixel: the pattern fields | Effects |
| Post-process anti-aliasing | .antialias smooths the stepped edges a shader leaves, as a filter on the layer | The edges a shader leaves: post-process anti-aliasing | Effects |
| Visual chains | Per-pixel composition written as a chain: oscillators, warps, blends, and modulation, in one pass | Patching without typing Metal: Visual chains | Visuals |
| Bringing a shader over | ollin new --from-shader: a GLSL fragment shader translated into Metal, with a project written around it | Somebody else's shader: GLSL import | ShaderImport |
| Checking a shader | ollin check: compile a .metal file on the GPU and see the errors at your own line, what the shader is, and the parameters it reads | Trying a shader on its own: ollin check | ShaderCheck |
Chapter 19: Layers and effects
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Render targets | Off-screen layers: draw into one with withTarget, then draw it back onto the canvas | A drawing you can hold: render targets | Effects |
| The filter catalog | The built-in GPU filters: blurs, glows, color, stylize, retro, distortion, and design; postProcess applies one to the whole finished frame, and drawSheet lays labeled tiles out to compare them | Filters; Ch 20: More from the shelf: ink lines, flat regions, chromatic aberration, a lens, a wall, and a mixer | Effects |
| Generator fills | generate(_:) fills a layer with a built-in pattern such as .noise; the plain fills .checkers, .gridLines, .bars, and .cellular are in the reference | Filters; Ch 14: The whole field at once: line integral convolution | Effects |
| Blend modes | Add, subtract, multiply, screen, lightest, darkest, as drawing state | How new paint meets old: blend modes | Drawing |
| Accumulation | noClear(): the canvas keeps what earlier frames drew, in single precision with .float32 for a long faint exposure; Accumulator: a layer that keeps the running mean, so faint samples settle into a picture | The canvas that keeps everything: noClear, Converging instead of brightening: the running mean; Ch 12: Roaming: wander, and trails from noClear | Accumulation |
| HDR and tone mapping | Light past full brightness, brought back by toneMap (clamp, Reinhard, ACES, or the hue-preserving neutral curve) | Brighter than the screen: toneMap | HDR |
| Feedback | A layer that sees its own last frame: trails, tunnels, and video feedback | The canvas that remembers itself: feedback | Effects |
| The compose block | compose { layer { } … }: a stack of layers, filters, and blends written as one block | The whole stack in one block: compose | Effects |
| Profiling | The inspector's cost row: CPU against GPU on one frame's scale, the draw, pass and batch counts, and a frame handed to Xcode | Which half is slow: the cost row; Ch 3: When a frame takes too long | Profiling |
| Retained batches | makeBatch { } records static drawing once into a Batch, and drawBatch draws it again each frame at little cost, placed by the transform in force | Record it once: batches | Retained batches |
Chapter 20: Pictures restyled
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Brushwork | .brushwork repaints a picture in strokes that follow its edges (an anisotropic Kuwahara filter) | Paint that follows the picture: brushwork | Effects |
| Hatching a picture | .hatching draws a picture's tones as strokes that follow its shapes | Strokes that follow the picture: hatching | Effects |
| A look from a file | ColorLUT reads a .cube color lookup table, the grade a color tool exports, and Filter.lut applies it; a look built in code writes out as a .cube | A look from a colorist's suite: ColorLUT | Looks |
| A film look | Filter.halation puts film's warm fringe around highlights, and Filter.filmGrain lays grain that follows the tone and keeps the mean | A film look: halation and grain | Effects |
| Design filters | Fluted glass, water, and paper that transform a picture; liquid metal, heatmap, and gem smoke that read a shape's silhouette | The design filters: paper, glass, water, and a material from a silhouette | Effects |
| Ink lines | .xdog draws a line wherever the picture has an edge (an extended difference of Gaussians) | A line where there is an edge: xdog | Effects |
| Flat regions | .shock flattens a picture into regions with crisp edges (a shock filter) | Regions with crisp edges: shock | Effects |
| Chromatic aberration | .chromaticAberration pulls the color channels apart, so a white edge grows a colored rim, with a mode for each kind of picture | One filter, five pictures: chromatic aberration | Effects |
| Lens distortion | A layer bent into a barrel or a pincushion by a lens's radial model, with the corners left empty or the frame kept full: the .lensDistortion filter | Through a lens, onto a wall, and through a mixer: lensDistortion, cornerPin, and channelMixer | Effects |
| Corner pin | A layer laid onto four points with its straight lines kept, the .cornerPin filter, the same map installation mode fits a projector with | Through a lens, onto a wall, and through a mixer: lensDistortion, cornerPin, and channelMixer | Effects |
| Channel mixer | .channelMixer mixes each output channel from all four input channels plus an offset, through a ColorMatrix: a gray made to your own recipe, or two channels swapped | Through a lens, onto a wall, and through a mixer: lensDistortion, cornerPin, and channelMixer | Effects |
| Relight, two tones, and warps | .relight reads a picture's brightness as height and lights it, .dither(dark:light:) screens it into two colors, and .swirl, .bulge, and .ripple bend it around a center you give | Height, tone, and a center: relight, the two-tone dither, and the warps | Effects |
| Droste | .droste makes a picture contain a smaller copy of itself, again and again | A picture inside itself: droste | Effects |
| Luminance melt | A picture liquified by a warped noise field and poured through a palette, the .melt filter | The picture poured: melt | Effects |
Chapter 21: Pictures you solve
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Diffusion curves | A few marks held as color sources and the color let out between them, so a curve can carry a different color on each side | A picture made of a few marks: diffusion | Effects |
| Measured distance fields | How far each pixel is from the nearest drawn edge and which way it lies, read back as contours, grown or shrunk shapes, outlines, and a Voronoi built from the drawn shapes rather than from points | A field you measure: the distance field | Measured distance fields |
| Layer styles | A layer given an edge of its own from its alpha: an outline inside, outside, or centered on the edge, a drop or inner shadow, an outer or inner glow, and a lit bevel, each one filter that stacks in the order it is chained | A field you measure: the distance field | Layer styles |
| Combines | combined(with:_:) runs an effect that reads a second layer, such as a mask, a displacement, a mix, depth of field, ambient occlusion, and screen-space reflections | A field you measure: the distance field; Ch 26: What the depth buffer is for: ambient occlusion and defocus; Ch 33: Mirrors that see off screen: ray-traced reflections | Effects |
| Light in a flat sketch | Draw a scene in one layer and the lamps in another, and get back the light that reaches each pixel: shadows that soften with distance, falloff, beams through a gap, and the color a lit wall gives back | Light that works itself out: .light | Light in a flat sketch |
| Seamless clone | .seamlessClone sets a piece of one picture into another so the colors meet with no seam (Poisson image editing) | Putting a piece of one picture into another: .seamlessClone | Effects |
| The frequency domain | A picture read as a sum of waves instead of a grid of pixels: filtering by scale is a shape drawn over the spectrum, and a field can be built from a description of its energy (Filter.fourier/.inverseFourier/.spectrum) | A picture read as waves: the Fourier transform | The frequency domain |
| Local averages | The average of the square around each pixel at a price that does not grow with its size: a box blur of any radius, and a threshold that compares each pixel with its own neighborhood, so uneven light stops mattering | Averages of a neighborhood, at a flat price: the summed-area table | Local averages |
Chapter 22: Iterated forms
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Fractals | IFS chaos games, fractal flames, the Buddhabrot, circle-inversion lace, Kleinian limit-set curves, and Schottky circles | The same fern, played as a game: the chaos game, Circles used as mirrors: inversion limit sets, A group drawn as one curve: Kleinian limit sets, The chaos game with a twist: fractal flames, Circles that pair off: Schottky groups, What the escapers leave behind: the Buddhabrot | Fractals |
| Strange attractors | Lorenz, Clifford, de Jong, and others, drawn as orbits, and as a field a million particles follow | A formula that folds the plane: chaotic maps; Ch 25: A million following the same field: attractor flow | Attractors |
| The bifurcation diagram | IteratedMap: the logistic map and its one-dimensional relatives on their way into chaos, watched through bifurcation diagrams, cobweb staircases, and Lyapunov exponents | One dial away from chaos: the bifurcation diagram | Bifurcation |
| Billiards | Billiard: a ball bouncing in a circle, an ellipse, a polygon, or a stadium, with posts standing in it, and its path as geometry you can stroke | A ball in a room: billiards | Billiards |
| Double pendulum | DoublePendulum: two weights on two hinged arms under gravity, stepped by advance(); a run repeats exactly, yet a start a ten-thousandth of a radian off soon swings somewhere else | The classic chaos machine: DoublePendulum | Motion |
| Escape-time fractals | The Mandelbrot set and Julia sets, colored by how fast each point escapes, zoomable; orbit traps color by the closest pass to a shape instead | Iteration without memory: escape-time fractals and orbit traps | Effects |
| Newton's basins | .newton colors each point by the root Newton's method carries it to | Where it lands: Newton's basins | Effects |
| Domain coloring | .domainColoring paints a complex function over its own plane: color for direction, rings for size unless you ask for .phase alone; placeable zeros and poles, and the named functions with their branch cuts | A picture of a function: domain coloring | Effects |
| Complex numbers | Complex, a point of the plane that multiplies: the polar form, exp, log, pow, sqrt, and the trig functions, and ComplexPlane to place it on the canvas; the shader library has the same in its complex section | Multiplying turns: the complex plane, Iteration without memory: escape-time fractals and orbit traps | Complex |
Chapter 23: Simulations on a grid
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Simulation fields | SimField and withField: the Game of Life, Gray-Scott reaction-diffusion with feed and kill that can vary across it, and fluid, as state on the GPU that you seed by drawing into it | State that lives on the GPU: SimField and withField, The Game of Life: a first field, Two chemicals: reaction-diffusion, Water you can stir: fluid | Effects |
| Reading a field as data | .gradientMap, .levels, and .relight read a field's numbers as color, contrast, and height | A field is data: gradientMap, levels, and relight | Effects |
| Predator-prey field | .predatorPrey: prey and predators on one land, with predators released where you draw green, an invasion front, and the spiral waves behind it | Two species: predator and prey | Effects |
| Multi-scale Turing patterns | .multiScaleTuring runs one activator-inhibitor rule at several sizes at once, with a TuringScale for each | The same rule at many sizes: multi-scale Turing | Effects |
| Ripple pool | .ripples: the wave equation as an interactive water surface; drawn marks add height, rings spread and reflect | A pool you can drop things into: ripples | Effects |
| Self-warp | .selfWarp measures which way each part of a picture moved and carries what it showed along that motion, so moving things smear and still ones stay sharp | The picture dragging its past: self-warp | Effects |
| Falling sand | .fallingSand with SandMaterial: sand that falls, rolls, and piles, water that spreads flat and lets sand sink through it, and walls that hold still, settled in 2x2 blocks each pass | Sand that falls: the falling-sand automaton | Effects |
| Watercolor wash | watercolor(pigments:): wet paint on rough paper, with darkened edges, backruns, dry brush, and glazing | Paint that behaves: watercolor | Watercolor |
| A simulation of your own | Sim.shader runs a kernel you write as a Shader, whose shade returns each cell's next state from its neighbors; inject: decides what a drawn mark does | Life rewritten as a kernel: Sim.shader | Effects |
Chapter 24: Automata
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Percolation | percolation: a grid of coin flips whose open cells join into one cluster across the grid once the probability passes about 0.5927 | When chance acts as a crowd: percolation, Putting it together: the wildfire | Percolation |
| Forest fire | .forestFire: bare ground, trees, and fire, a field that finds its own critical density, or with growth and lightning at zero burns only what you plant and light | A forest that keeps burning: the forest fire, Putting it together: the wildfire | Effects |
| Cellular automata | elementaryCA and totalisticCA rules, and Turmite ants: small rules run for a long time | A single row of cells: Wolfram's elementary rules, An ant on a grid: turmites | Cellular automata |
| Lenia | .lenia: the continuous Game of Life, smooth mass that grows colonies and creatures | Life made continuous: Lenia | Effects |
| SmoothLife | .smoothLife: Life's own rule on a continuous field, a disc for the cell and a ring for its neighbors, with the paper's glider as the default regime | Life with soft edges: SmoothLife | Effects |
| Grid automata | .cyclic, .excitable, .briansBrain, .hodgepodge, .wireworld, .schelling, and .ising: state automata as fields, from spirals and gliders to a circuit, a neighborhood sorting itself, and a magnet against the heat | The medium that has to rest: excitable media, A circuit made of cells: Wireworld, A neighborhood sorting itself: Schelling's board, Spins against the heat: the Ising model | Effects |
| Sandpile | .sandpile: grains dropped on a grid, each cell toppling onto its neighbors when it holds four | A pile of sand: the Abelian sandpile | Effects |
Chapter 25: Simulations made of particles
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Compute kernels | Your own GPU code, written in the chapter as the Particles step snippet and the Simulation step over a texture; .metal files that reload as you edit and the typed ComputeKernel are in the reference | A million grains: GPU particles | Compute |
| GPU particles | Particles: a million particles moved by a Metal snippet run once per particle and drawn on the GPU, with stepParticles and drawParticles | A million grains: GPU particles | Compute |
| Artificial life | ParticleLife, PPS (the Primordial Particle System), Physarum, ParticleLenia, and SwarmChemistry, most of them on the public SpatialHash neighbor search | Particles that see their neighbors: Particle Life, the Primordial Particle System, Physarum, and the flock at scale, Agents that talk through the floor: Physarum, Matter that decides what shape to be: Particle Lenia, A rule that spreads by winning arguments: swarm chemistry | Artificial life |
| Steering at scale | Swarm: separation, alignment, cohesion, seek, flee, arrive, wander, and flow as weights over tens of thousands of agents | The flock, a thousand times bigger: Swarm | Swarm |
| Particle fluids | ParticleFluid: SPH liquid with a free surface, splashes, and grabbing | Liquids and jellies: SPH and soft bodies | Fluids |
| Jellies | SoftBodies: shape-matched jellies that squash and pile | Liquids and jellies: SPH and soft bodies | Fluids |
| Ant colony | A colony laying pheromone trails, narrowing a web of possible routes down to a short tour | A search you can watch: ant colony optimization | AntColony |
| Evolution | Evolution: GPU populations bred toward a target past obstacles, by tournament selection | Letting the sketch find it: evolution | Evolution |
| Kernel simulations | Simulation runs a Metal step once per cell of a texture, the grains' kind of snippet over a grid, stepped with stepSimulation(_:custom:), with substeps for several steps a frame and a kernel of your own for the first state | One step per cell: Simulation | Compute |
Part V: The third dimension
Chapter 26: 3D, gently
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| The camera | camera(_:) with a fixed pose such as .orbiting(...), perspective(eye:target:) to aim the eye from one point at another, and ortho(eye:target:height:), where far things keep their size | A camera and a sphere, Placing things: transforms compose | 3D |
| Camera control and moves | Orbiting with the mouse, the self-driving cameraShowcase, and snap views such as cameraView(.front); the cinematic CameraMove values are in the reference | A camera and a sphere, Keeping your bearings: views, the axis, and the ground grid | Camera |
| Spherical coordinates | spherical(azimuth, elevation, radius, around:) places a point by two angles and a distance, in the camera's own orbit convention; a Vector3's azimuth and elevation read the angles back | Appendix B: Two angles and a radius make a point in space | Math |
| Axis and ground grid | cameraAxis() shows a small x-y-z compass and groundGrid() a faint floor, drawn in the live window and left out of exports | Keeping your bearings: views, the axis, and the ground grid | Camera |
| The depth test | Solids nearer the camera hide the ones behind, in whatever order you draw them, over a floor from drawPlane or drawGround() | Depth that hides things: the depth test | 3D |
| Solid primitives | Box, sphere, torus, knots, Platonic solids, lathes, extrusions, and more | A catalog of solids | 3D |
| Wireframe | wireframe() draws a mesh as its triangle edges, which take no light; each vertex also carries a normal, the way the surface faces there, in the mesh's normals | What a solid is made of: triangles and normals | 3D |
| Lights | Directional, point, spot, and ambient lights, glowing panels, disks, and tubes, and lighting presets | Light: presets and the kinds of light, A light with a body: area lights | 3D |
| Light shaping | A fixture's measured throw on a light (IES files), and an image projected through a spot (a cookie, the stage gobo) | Light: presets and the kinds of light, Air you can see: fog and volumetric light | 3D |
| Several rigs in one frame | withLights gives a block its own lamps, and withoutLights draws it flat | Light: presets and the kinds of light; Ch 27: One solid, two shadows: shadow art | 3D |
| Shadows | Cast shadows that harden near contact, with a softness setting, ray-traced for point lights on capable GPUs | Shadows, and what they tell you, Two lights, two shadows | 3D |
| Materials | Stylized finishes (toon, iridescent, velvet, glitter), and physically based metal and gloss from metallic and roughness | Materials; Ch 28: Two numbers for most real surfaces: PBR | 3D |
| What stacks with what | The compatibility map across the 3D features | Materials; Ch 32: The light this needs: materials and shadows on a field | Combining |
| Contact shadows | contactShadows(length:) darkens the seam under a thing resting on another | The seam under a resting thing: contact shadows | 3D |
| Atmosphere | Fog that fades surfaces with distance and pools low with a height falloff, aerial perspective that turns far ground blue and brightens the air toward the sky's sun, and volumetric light that turns spot cones, cookies, and cast shadows into visible beams and shafts | Air you can see: fog and volumetric light; Ch 28: Distant air: aerial perspective | Atmosphere |
| The cartoon look | outline(width:color:) draws a line around a mesh's silhouette, and headlight() is a light that follows the camera | The cartoon look: outlines, and a light that follows the camera | 3D, 3D |
| Hidden-line solids | featureEdges(creaseAngle:) draws a mesh's faces in the fill and over them only its creases, its boundary, and its silhouette in the stroke, near solids hiding the edges of far ones, every copy of an instanced draw in one more draw | A draughtsman's drawing: hidden-line solids | Hidden-line solids |
| Matcaps | Shading read from a picture of a lit sphere, such as chrome, clay, or car paint, with no lights in the scene | Shading from a picture: matcaps | 3D |
| Depth compositing | Flat 2D drawing placed inside the 3D depth buffer, so nearer solids hide it: depth(at:), project, and billboards | Flat drawing that knows where it is: depth compositing; Ch 35: Flat marks at a depth: depth compositing | DepthCompositing |
| Lines in 3D | drawLine and drawPolyline over Vector3: the 2D stroke laid on the camera's picture, a depth at every point so solids hide it and it hides what lies behind, a color per point, a weight on screen or in world units with strokeWeight(_:in:), and drawAxes and drawGrid built from it | Flat drawing that knows where it is: depth compositing | Lines in 3D |
| Type as a solid | drawText3D, over the Mesh.text and Mesh.textGlyphs builders: a word extruded into a lit solid that throws a shadow, or the same word a letter at a time, each letter knowing where it sits | A mesh from a word: type as a solid | 3D |
Chapter 27: Meshes and maps
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Meshes from file | OBJ, glTF, USDZ, STL, and PLY, with materials and textures; normalized(scale:) recenters and fits whatever arrives | A mesh from a file | 3D |
| Textures | A mesh wrapped in an image through its UVs, or, with no UVs, through a triplanar projection | A picture wrapped around it: textured, A picture from three sides: triplanar | 3D |
| Subdivision surfaces | subdivided(levels:): a rough cage of few faces refined into a smooth solid (Catmull-Clark, or Loop for triangle meshes), with welding and open edges handled | Smooth from a cage: subdivision surfaces | Subdivision surfaces |
| Height maps | A height map carves depth, as shading with parallaxMapped or as geometry with displaced(by:scale:) | Depth from a picture: height maps, Putting it together: the raked garden | 3D |
| Detail maps and decals | A detail map keeps texture sharp in a close look, and a decal stamps a picture onto the scene | Texture that survives a close look: detail maps, A picture stamped onto the scene: decals | 3D |
| Maps on a surface | normalMapped adds relief the light sees, and surfaceMapped sets metal, roughness, occlusion, and glow texel by texel (glowing(_:) for a mesh's own colors) | Relief from a picture: normal maps, What the surface is, per texel: surface maps | 3D |
| Scenes from file | loadScene reads a glTF or USD scene: named nodes drawn in place with drawScene, the file's camera and lights as values, a node reached by name, and the animations, skins, and morph targets it holds | A scene with its camera and lights: loadScene, Motion the file remembers: animations, skins, and morph targets | Scenes |
| Bringing a scene over | ollin new --from-scene: a glTF or USD scene written out as the camera, light and placement calls that draw it | A scene you can take apart: the scene written as source | SceneImport |
| Cutting and joining meshes | union, intersection, subtracting, and symmetricDifference combine two closed solids, and Mesh.joined(_:) makes one mesh from several placed ones | Cutting one solid with another: mesh booleans, One mesh from several: joined and placed | 3D, 3D |
| Shadow art | shadowArt(fromFront:fromSide:) carves the largest solid that throws the silhouettes you ask for, and shadow(from:) shows the shadow it throws | One solid, two shadows: shadow art | Shadow art |
| Mesh growth | MeshGrowth: a surface that grows more area than it has room for and folds, driven evenly, by curvature, by your own field, or by a reaction-diffusion pattern in the surface | A surface that outgrows itself: differential growth | Mesh growth |
| Sweeps and strips | Mesh.sweep carries a Shape along a Curve3D, whose frames never twist, sized and turned on the way; Mesh.loft carries several shapes placed along it and blends the surface from each to the next; Mesh.strip is the ribbon between two lines; Pace spaces points along a curve by length, or along a strip by area | A shape along a path: sweeps and strips | Sweeps and strips |
| The Hopf fibration | A sphere's worth of circles, no two of which meet and every two of which are linked once, handed back as 3D paths to sweep | A shape from four dimensions: the Hopf fibration | The Hopf fibration |
Chapter 28: Materials and surroundings
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Environments | HDRI image-based lighting (eight bundled, twelve to download) and a procedural sky that needs no files, with its own sun as a light and clouds; the surroundings become the light | Surroundings as the light: environments, Weather in the sky: clouds | 3D |
| Brushed metal, clearcoat, sheen, and skin | Anisotropy stretches a highlight into a streak, clearcoat and sheen give paint and cloth, and subsurface scattering lets light into skin, wax, and stone | A streak instead of a dot: anisotropy, Paint and cloth: clearcoat and sheen, Skin, wax, and stone: subsurface scattering | 3D |
| Glass | A material that bends and tints the light passing through it | Glass | 3D |
| The scene through glass | sceneThroughGlass() draws the frame once more with the glass taken out, so a bottle in your scene shows what stands behind it on any Mac, with no ray tracing | Glass | 3D |
| Volumes | Volume, a box of density samples, drawn by drawVolume as smoke, cloud, fire, or ink: the medium blocks light, scatters the scene's lights, and glows, and solids inside it are veiled by what lies in front | A cloud in the scene: volumes | Volumes |
| Thin film | thinFilm and thinFilmThickness color a surface the way a soap bubble is colored, with no pigment | Color with no pigment: the thin film | 3D |
Chapter 29: Landscapes and multitudes
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Terrain | Heightfield: landscapes grown from noise or diamond-square, weathered by simulated rain and gravity, read out as a mesh, an image, or samples | A landscape you grow: heightfields and erosion | Terrain |
| Instanced meshes | One mesh drawn thousands of times in one draw call: MeshInstance placements the GPU applies to each copy, or a buffer a compute kernel writes | Ten thousand of the same thing: instancing | Instancing |
| Points on a surface | Scatter a mesh's skin by area, not its vertex list, each spot carrying its normal | Where the copies go: points on a surface | SurfaceSampling |
| Mesh fields | A world of placed meshes kept between frames and drawn by one call, each copy culled on the GPU against the camera (MeshField, drawMeshField) | A world the camera trims: MeshField | Instancing |
| Strand fields | Grass grown inside the draw call: bending, swaying blades with no geometry buffers, camera-culled and distance-graded (StrandField/drawStrands) | Grass that was never built: strand fields | Strands |
| Drainage | Rivers worked out from a Heightfield rather than drawn: hollows filled so water always has a way out, flow counted per cell, the rivers above a minFlow as polylines you can stroke, Strahler ordering, and basins | Where the water goes: drainage | Drainage |
| The ocean | A sea built from its own wave spectrum: one inverse Fourier transform makes the surface, drawn as water with no stored geometry, its wave height a measurement in world units (Ocean/makeOceanField/drawOcean) | The sea, from what a sea is made of: an ocean spectrum | The ocean |
| How far a light carries | reach: bounds a lamp to its own neighborhood, with no light past it, which lets a frame carry up to 256 lamps: past eight, the lights are culled per screen tile | A courtyard of lamps: many lights | 3D |
Chapter 30: Worlds with weight
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| 3D physics | World3D: rigid bodies that stack, tumble, and swing inside the 3D scene, picked up and thrown with the mouse | Things with weight: World3D and bodies | Physics3D |
| Motors, gears, and racks | drive(at:strength:) turns a hinge or a slider at a speed and drive(to:) holds it at a pose; .gear(teeth:and:) makes two hinges turn together, and .rackAndPinion turns a hinge's turning into a slider's travel | Machines out of joints: motors, gears, and racks | Physics3D |
| Collision groups | group: and ignoreCollisions(between:and:) let chosen bodies pass through each other | Things that pass through each other: collision groups | Physics3D |
| Asking the world | world.contacts says what hit what, a body made with isSensor: counts what is inside it, and raycast(from:to:), sweep, and bodiesOverlapping ask what is in the way | What hit what: contacts, A region that counts what is inside: sensors, What is in the way: rays, sweeps, and overlaps | Physics3D |
| Holding a body back | freedom: takes a direction of motion away, gravityScale gives a body a pull of its own, and checksPath keeps a fast body from passing through a thin wall | Taking a direction away: degrees of freedom, A pull of its own: gravityScale, Too fast for a thin wall: checksPath | Physics3D |
| More joints | A cart on a track with .path(through:looping:alignment:), a pulley, and .allowing(_:at:travel:), a joint written as a list of freedoms | A cart on a track: the path joint, A rope over two hooks: the pulley, The joint that is a list of freedoms: .allowing | Physics3D |
| Tensegrity | Tensegrity and addTensegrity: struts held apart by cables in their balanced forms, standing in the world as capsule struts joined by .cable joints | Standing on cables: tensegrity | Tensegrity |
| Buoyancy | world.water with Water(level:): bodies float by the weight of the water they push aside, on waves if you ask | Water, and what it holds up: buoyancy | Physics3D |
| Ground and scenery from elsewhere | .heightfield turns a landscape into ground, and addStaticBodies(from:) and addBodies(from:) read colliders from a scene or a USD file | Ground from a landscape: the heightfield collider, Scenery and bodies from a file: scenes and USD physics | Physics3D |
| Physics snapshots | snapshot() and restore(_:) keep a world as it stands, save(to:) writes it to a file, and assetName saves a large collider by name | A world saved as it stands: snapshot and restore, A large collider saved by name: assetName; Ch 31: Snapshots of figures, vehicles, and cloth | Physics3D |
Chapter 31: Characters, vehicles, and cloth
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| A character | addCharacter(radius:height:at:): a capsule steered by walk(at:), often from moveAxis, that jumps, climbs steps, and stops at slopes too steep for it | Someone to be in there: the character controller | Physics3D |
| Vehicles | addVehicle: wheels with suspension, an engine, steering, and a hand brake; isTracked: makes a machine that turns by driving its two sides at different speeds | Something to drive: the vehicle, Turning without steering: tracks | Physics3D |
| Cloth and soft bodies | addSoftBody(from:at:): cloth that drapes and balloons that hold their air, pinned where you say, touching the rigid world and floating like any body | Cloth that finds its own shape: soft bodies, A cloth among the contacts, A raft made of cloth: soft bodies that float | Physics3D |
| Ragdolls | addRagdoll(from:at:): a figure that falls, and with drive(toward:strength:), one that tries to stand | Letting a figure fall: the ragdoll, A figure that tries to stand: powered ragdolls | Physics3D |
| Cloth a figure carries | skinnedTo: and carriedBy: hang cloth on a moving figure, with sway and a limit on how far it stretches | A cape on someone's back: cloth a skeleton carries | Physics3D |
| Ropes | addRope(through:at:radius:pinned:): a rope that knows how far it is twisted as well as where it bends | A line that knows how it is turned: ropes | Physics3D |
Chapter 32: Sculpting with fields
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| 2D SDF combinators | Shapes as distance fields that merge, melt, morph, and repeat | A shape as a question: the signed distance field, Melting: the smooth minimum, Carving, overlapping, and halfway: the other verbs, Folding space: domain repetition | Combinators |
| Raymarched 3D fields | SDF3D drawn by sphere tracing through the camera, sharing depth with meshes | Into space: SDF3D and drawSDF3D, How the picture gets made: sphere tracing | Combinators |
| Field distortions | twisted(_:), bent(_:), displaced(amplitude:frequency:), and roughened(amplitude:frequency:) bend a whole field at once | Bending the whole form: distortions | Combinators |
| Joins like joinery | chamferUnion, stairsUnion, and columnsUnion join shapes with machined edges, and smoothUnion(k:) { } merges ordinary draw calls | Joins like joinery: the machined family, Merging ordinary draw calls: the block form | Combinators |
The sculpt { } block | Add, carve, and blend shapes with stateful verbs, in 2D or 3D, made for live coding | Building up and carving away: the sculpt block | Combinators |
| Fractal fields | mandelbulb, mengerSponge, and mandelbox as leaves of a raymarched field, each a distance estimate | Infinite detail: the fractal leaves | Combinators |
| Isosurfaces and metaballs | The surface where a field over space crosses a level, built as a Mesh by marching cubes, or by dual contouring to keep sharp corners: soft spheres that fuse, and noise volumes | Any field as a mesh: isosurface, Soft spheres that add up: metaballs, Corners that stay sharp: dual contouring | Isosurfaces |
Chapter 33: Traced light
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Ray-traced reflections | rayTracedReflections(): surfaces that reflect the scene's geometry, off-screen parts and hidden faces included, on a ray-tracing GPU; glossyReflections() blurs them, and reflectionBounces(_:) lets mirrors face mirrors | Mirrors that see off screen: ray-traced reflections, The floor that shows the room: glossy reflections, Mirrors facing mirrors: reflectionBounces | 3D |
| Global illumination | globalIllumination(intensity:) lets light bounce off one surface and color the next, on a ray-tracing GPU | Light that bounces: global illumination | 3D |
| Caustics | The light glass and polished metal focus, photon-traced: a lens's bright spot inside its own shadow, a chrome ring's folded fan, prism rainbows via dispersion | The light the glass takes, given back: caustics | Caustics |
| Temporal anti-aliasing | temporalAntialiasing(): 3D edges refined past the fixed samples per pixel by shifting the projection a fraction of a pixel each frame and averaging the frames; withMotion { } keeps a moving mesh's edges smooth | Edges that settle: temporal anti-aliasing | 3D |
| Motion blur | motionBlur(shutter:): the streak a shutter leaves, worked out from how far each surface moved since the frame before | The streak a shutter leaves: motion blur | 3D |
| Lens flare | The light the camera adds: the ghosts a lens prescription throws from a bright source and the star on the source, both shaped by the iris blades, fading as something covers the source | Light in the camera: lens flare | Lens flare |
| Path-traced export | --path-traced renders the 3D scene by tracing light: soft shadows, bounce color, mirror in mirror, glass, glowing meshes as lights, textures in the bounces, and a lens; --pt-noise lets a settled pixel stop early and spends the count where the grain is; --pt-clamp bounds what one bounce may add to a sample, so a polished scene's fireflies go, at a bias the recipe states; --pt-reuse carries the earlier frames' samples into each frame of a sequence, so the grain settles and holds still; --pt-seconds gives each frame a time in place of a count, and the recipe records the count it reached; --denoise filters out the grain | Light followed path by path: the path-traced export, Grain filtered out: denoising | Path-traced export |
| Specular anti-aliasing | specularAntialiasing(): highlights that hold still instead of crawling and sparkling as a surface turns | Highlights that hold still: specular anti-aliasing | 3D |
| Bokeh | .defocus(blades:catsEye:) shapes the out-of-focus blur by the iris blades, with the cat's-eye shape toward the edges | The shape of a blur: bokeh | Effects |
| Depth of field on the canvas | depthOfField(): the 3D frame blurred by its own depth through the camera's lens (aperture, focusDistance, apertureBlades), a blurred foreground turned into a veil | The frame through the camera's lens: depthOfField() | 3D |
| The overlay | withOverlay { }: a block drawn over the finished frame after every pass, so a caption or a readout stays sharp under the lens, the edge average, and the motion blur, and never joins an accumulating canvas | The frame through the camera's lens: depthOfField() | Effects |
| Depth of field from light | LineSpray and Bokeh: lines drawn as millions of scattered points through a lens into a running mean, until the lens's blur appears; the .light particle style and the develop print underneath | A lens made of samples: depth of field from light; Ch 25: Grains as light: the light style and the running mean; Ch 19: Converging instead of brightening: the running mean | Depth of field from light |
| Motion for a changing mesh | drawMesh(_:previous:) hands motion blur and anti-aliasing the mesh's shape from the frame before | A mesh that changes shape: drawMesh(_:previous:) | 3D |
| Temporal upscaling | temporalUpscaling(_:): render fewer pixels and let MetalFX build the frame back at full size, in the live window | Rendering fewer pixels: temporal upscaling | 3D |
| Frames between the drawn ones | frameInterpolation(): draw() runs every other refresh and the platform builds the frame in between, so a heavy scene drawn at thirty frames a second moves at the display's sixty, one refresh later | Drawing fewer frames: frame interpolation | 3D |
Part VI: The world coming in
Chapter 34: Seeing
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Camera feeds | The webcam, or any frame source, drawn with drawFrame and read by trackers as it runs | The webcam is an image, Trackers: attach, then read | Vision |
| Video playback | Video files as live textures you play, seek, and loop, frame for frame the same under an export | When there is no camera: stills and footage | Video |
| A feed with no camera | Camera.orStill and StillFrames, so a sketch that reads the world runs with nothing plugged in | When there is no camera: stills and footage | Vision |
| Motion trackers | Object tracking, the paths of thrown things, and dense optical flow | The picture as a field: optical flow, A patch you point at: ObjectTracker, Things that fly: TrajectoryTracker | Vision |
| People trackers | Faces with landmarks, hands, 2D and 3D body poses, and person segmentation | The body as a controller: hands, faces, and bodies, The person as pixels: lifting the subject | Vision |
| Still analysis | Most trackers also run once on an Image | The body as a controller: hands, faces, and bodies | Vision |
| Scene trackers | Contours, rectangles, barcodes, text (OCR), saliency, classification, and the subject lifted from a picture, whole or at a point you give | Reading outlines and print: contours, rectangles, text, and codes, Words for the scene: ImageClassifier, Where an eye would go: SaliencyTracker, The person as pixels: lifting the subject, A click cuts it loose: PointSegmenter | Vision |
ModelTracker | A Core ML model of your own: classifiers, detectors, image-to-image maps, and segmenters, with sketchResource(_:) to find the model file | Bringing your own model: ModelTracker, Models of your own | Vision |
| Depth from a picture | DepthTracker estimates depth from one camera and holds it steady from frame to frame; DepthClip reads a whole recorded clip first, so an export gets the same depth each run | Depth that holds still: DepthTracker, A clip read whole for an export: DepthClip | Vision |
| Words as numbers | ConceptTracker scores phrases you type against the picture, as numbers a sketch can use | Words as numbers: ConceptTracker | Vision |
| The camera in a 3D scene | Environment.feed lights a 3D scene with the camera's picture, and textured(_:) wraps a mesh in the live frame | The room as the light: Environment.feed, The room as the surface: textured | 3D |
| Screen capture | ScreenCapture: a display, an app, or one window on the Mac as a live frame source, with nothing asked of the app it captures; a tracker reads it like a camera, and excludesOwnWindows = false puts the sketch's own window in the picture as a feedback tunnel | Drawing with the screen: ScreenCapture | ScreenCapture |
| Slit scan | A rolling frame history read back through a per-pixel time delay: time as a spatial dimension | Every pixel its own moment: SlitScan | SlitScan |
Chapter 35: Depth
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| RGBD frames | Color plus depth plus intrinsics; unproject turns a pixel into a point in meters | What a depth camera sees, Standing the picture up | RGBD |
| Point clouds | Points drawn as instanced splats that face the camera, in the hundreds of thousands | Standing the picture up | 3D |
| World fusion | WorldCloud fuses depth frames into one world cloud by their poses, and ScanGraph recognizes a place the scan comes back to, so the scan agrees with itself | One world from many frames, When the camera loses its place: correcting drift, Coming back to where you started: closing the loop | Phone |
| A depth frame as the stage | drawDepthScene(_:) puts a depth frame behind the 3D scene as both backdrop and depth, with camera(.intrinsic(_:)) matched to its lens | Solids behind the scene: drawDepthScene | DepthCompositing |
| Surface reconstruction | reconstructSurface and particleSurface: a scanned or generated point cloud turned back into a Mesh, with holes in the data left open, and particle sets skinned as one blended body | A solid from a sweep: reconstructSurface, Points as their own material: particleSurface | Surface reconstruction |
Chapter 36: The iPhone as a sensor
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Recorded captures | Open .r3d depth recordings as point clouds, with the intrinsics of the camera that made them | Depth footage: Record3D clips and the live stream | Record3D |
| Live USB depth | A tethered phone's RGBD stream, frame by frame | Depth footage: Record3D clips and the live stream | Record3D |
| The capture app | Ollin's own iPhone app runs ARKit on the phone and streams typed results through PhoneDevice, starting with LiDAR depth and the phone's pose as it moves | Ollin's own app: Ollin Capture | Phone |
| A body in solids | The phone's skeleton in meters, each joint with an orientation, dressed in capsules, and read from a film with BodyTracker3D when no phone streams | A pose you can dress in solids: the body, When no phone is streaming: the film, Putting it together: the dancer in solids | Phone, Vision |
| The room from the phone | The phone's labeled room mesh, the flat surfaces it found, and the light it measured in the room | The room from the phone: its mesh, its surfaces, and its light, A surface the phone already built: the room mesh, Somewhere to stand: flat surfaces, The light in the room: PhoneLight | Phone |
| More of a person from the phone | Hands measured in meters, and where a look lands | More of a person from the phone: hands and a gaze, A hand you can reach in with: hands, Where a look lands: gaze | Phone |
| The picture from the phone | Text read and placed in the world, where an eye would go, how the picture is moving, and pictures the phone was told to look for | The picture from the phone: words, attention, motion, and a known print, The words on the wall: text, What draws the eye: attention, How the picture is moving: flow, A picture it knows: markers | Phone |
| The phone in your hand | The phone as a pointer, its Ray3 hitting things in the scene, the sounds it hears named, touches on its glass, and the air pressure around it | The phone in your hand: a wand, its ears, the glass, and the air, Pointing at it with the phone: the wand, What the phone hears: sound events, Playing the glass: touches, The air it is standing in: the barometer | Phone, Geometry |
Chapter 37: Listening
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Audio analysis | Amplitude, spectrum, waveform, and log-spaced bands | Loudness from the microphone: amplitude, What the analyzer hears: waveform, spectrum, and bands | Audio |
| Audio sources | The microphone, audio files (the same numbers under an export), test tones, and a video's soundtrack | Four sources: the microphone, a file, a tone, and a video | Audio |
| Feeding the analyzer yourself | An AudioAnalyzer fed samples you make with analyze(samples:count:), one frame's worth at a time, so its numbers are the same on every run; the chapter's StageMic feeds it a made-up band this way | A band that plays the same every run: StageMic | Audio |
| Beat detection | Onsets as events: beat, beatCount, timeSinceBeat | Hearing the beat: onsets | Audio |
| Pitch tracking | The note being sung or played as pitch (its hertz, confidence, nearest note, cents, and MIDI number) and nearestPitch, and the twelve pitch classes as chroma, with the octaves folded together | One note at a time: pitch, Every note in the air: chroma | Audio |
| Listening | Speech as a caption that corrects itself, phrases you can trigger from, and about 300 everyday sounds named as they happen | Words as they are spoken: SpeechListener, Sounds with names: SoundClassifier | Listening |
Chapter 38: Controls and signals
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| MIDI | Knobs, notes, and messages from hardware controllers, sent as well as read, and each held note's own bend, pressure, and slide from an MPE surface | Parameters from anywhere: MIDI and OSC; Ch 39: A note under the finger: expression | MIDI |
| OSC | OSCReceiver reads network control messages from tablets, DAWs, and other machines, and OSCSender sends them out | Over the network: OSC | OSC |
| MIDI clock | TempoClock(from:) follows the beat a drum machine, a DJ mixer, or a DAW sends as MIDI clock, as beats, bars, and a phase to animate with | Musical time: MIDI clock and Link | MIDI |
| Link | LinkClock joins the shared tempo session on the network, so every app that joins keeps one tempo and lands the same downbeat, with the reads TempoClock has | One beat for the whole room: Link | Link |
| Parameter binding | One @Param driven by MIDI (a knob or the pitch wheel), OSC, serial, Bluetooth, and the inspector alike, with smoothing, and show-rules that hide a parameter while another leaves it nothing to do | One parameter, three hands: binding and smoothing | Parameters |
| Game controllers | Sticks, triggers, and buttons read in draw(), plus motion and a touchpad on hardware that has them, with several players at once and centered sticks and unpressed buttons when nothing is plugged in | Something to hold: game controllers | Controller |
| The weather outside | Weather reads the sky over a place or a name in plain units, and Place.sun(at:) puts the sun where the clock says | The weather outside: Weather | Weather |
| Haptics | A pattern of taps and hums played from draw(), put together like a phrase, and turned into a trackpad's three feelings and one strength | Touch as an output: haptics | Haptics |
| Pinch and turn | Two fingers on the trackpad, or on the phone showing the sketch, read in draw() as this frame's change of size and turn, with a hook for each, recorded into a take, and zooming viewControl() and cameraControl() | Two fingers on the trackpad: pinch and turn | Input |
| Typed text | The keyboard read as a text field reads it: a dead key's accent, the emoji viewer's pick, and an input method's confirmed word arrive through typedText(), the reading still being composed in composingText, with a hook as text arrives | Text the way a text field takes it: typed text | Input |
| TUIO | Touches, tagged pieces, and shapes from a marker table, a touch wall, or a phone app | A table you put things on: TUIO | TUIO |
| Serial | A USB microcontroller's sensor lines read in draw() or bound to a parameter, lines written back to drive the hardware, a port that waits through unplugs and reopens itself, and FirmataBoard for a board running Firmata | A wire to the physical world: serial | Serial |
| Bluetooth | A Bluetooth Low Energy sensor read in draw() or bound to a parameter, BluetoothScan listing the devices in range, the formats that turn bytes into values, and a connection that waits and reconnects by itself | The same loop, without the wire: Bluetooth | Bluetooth |
| OSCQuery | The sketch's parameters published as a tree a control app browses and builds its own controls from, the values coming back over OSC | The sketch that says what it takes: OSCQuery | OSCQuery |
| Curves and bars | A ResponseCurve parameter shaped by dragging its points and read like an easing, and a list of numbers painted as a row of bars, both saved into the sketch and held by a cue | Shapes you draw: a curve and a row of bars | Parameters |
| Smoothing | @Smoothed calms a jittery input, such as the mouse, a tracker, or a sensor, as it arrives | A value that is not a parameter: @Smoothed | Animation |
| Live data | DataFeed reads one address over and over in the background, counting only the answers that changed; PushFeed holds a connection open for messages that arrive on their own | Numbers that keep arriving: DataFeed, Messages that arrive on their own: PushFeed | LiveData |
Part VII: Out into the world
Chapter 39: Making sound
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Synthesis | A Synth the sketch plays: notes by name or number, voice presets, envelopes, and filters | A sketch that plays | Synthesis |
| Physical models | A plucked string worked out sample by sample, a shape you strike that rings in its own modes, and a bowed string and a blown tube you keep playing | A string worked out sample by sample: the plucked string, A shape you can hit: modal synthesis, A note you keep playing: bowed and blown | Synthesis |
| The effects chain | synth.effects: distortion, delay, reverb, chorus, and a limiter in a chain after the note, in the order you list them, and a room of your own from a recording or a rule (convolution reverb) | After the note: the effects chain, A room you can draw: convolution reverb | Synthesis |
| Patches and FM | Patch builds an instrument from oscillators, and modulated(by:amount:) lets one oscillator push another's frequency back and forth (FM) | Building an instrument instead of choosing one: patches and FM | Synthesis |
| Sampled instruments | SampledInstrument: an instrument played from recordings, the built-in one or an SFZ set you load | An instrument somebody recorded: sampled instruments | Synthesis |
| Wavetables and grains | A wavetable read by position, and a grain cloud cut from a sound, so a moment can be held and dragged through | A wave you can draw: wavetables, A sound in pieces: grains | Synthesis |
| More effects | The four effects that move a sound (chorus, flanger, phaser, tremolo), the three that hold its level (compressor, limiter, gate), pitch shift, freeze, and stretch, and an effect you write yourself | Something that moves: chorus, flanger, phaser, tremolo, Something that holds a level: compressor, limiter, gate, Something that takes the sound apart: pitch shift, freeze, stretch, An effect nobody wrote for you: custom effects | Synthesis |
| Per-note expression | noteOn returns a PlayingNote, and bend(_:semitones:), press(_:_:), and slide(_:_:) shape that one note while it sounds | A note under the finger: expression | Synthesis |
Chapter 40: Music by rule
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Tempo and steps | Tempo turns the clock's seconds into beats and a NoteLength into seconds, and StepCounter hands back each step number the beats have passed | Beats and note lengths: Tempo and NoteLength, One step at a time: StepCounter | Composition |
| Euclidean rhythms | Rhythm spreads hits as evenly as it can over a number of steps, rotated to start where you say | Hits spread evenly: Euclidean rhythms with Rhythm | Composition |
| Swing | Every second step played late by the same fraction of its pair, through the wait a synth's play(_:velocity:for:after:) takes | Late on purpose: swing | Composition |
| Scales | Scale turns a degree into a note that stays in key | Staying in key: Scale | Composition |
| Chords and arpeggios | Chord names a chord by its root and kind, chord(on:) builds one on a degree of a Scale, and Arpeggio plays its notes one step at a time | Chords from the key: Chord and Arpeggio | Composition |
| Markov chains | MarkovChain counts what followed what in a motif and picks each next element by those counts, the same line for the same seed | What comes next: a MarkovChain; Ch 4: A roll that remembers the last one: MarkovChain | Composition |
| Chord progressions | Progression plays a key's chords by Roman numeral, or chords named by symbol from any key, with named progressions and wandering(_:) for a longer run that makes only the moves the original made | Chords that come out of a key: Progression | Composition |
| MIDI files | A Standard MIDI File read into notes a sketch plays or draws, and written back out so a phrase opens in a sequencer; its tempo map read from beats to seconds and back; a take recorded from a Synth as it plays | Keeping what it played: writing a MIDI file, Music that travels between programs: MIDI files | MIDI files |
| A step sequencer | StepSequencer: a grid of steps with swing, velocity, chance, and ratchets, read at the beat you hand it | A grid with a feel: StepSequencer | Composition |
| An arpeggiator | Arpeggiator plays the notes you hold as a pattern, one step at a time | A chord played as it is held: Arpeggiator | Composition |
| Tunings | Tuning: equal temperament, just intonation, Pythagorean tuning, other divisions of the octave, and Bohlen-Pierce | Other divisions of the octave: Tuning | Composition |
| Playing along with the room | BeatFollower finds the beat in the sound it hears, so the music keeps time with the room | Playing along with the room: tempo sync | Composition |
| Sonification | Numbers read out as notes: a table column, a terrain profile, or a picture row spread over a range of pitch and snapped to a scale | Numbers you can hear: sonification | Sonification |
| Spatial audio and sound in an export | place(at:heardFrom:) sounds a synth from a point in the 3D scene, heard from the camera; an export renders what the synths play into its soundtrack, the same on every run, with the placing kept | Sound from a place in the scene: spatial audio, Putting it together: the music box | Synthesis, Sound in an export |
Chapter 41: Finishing a sketch
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Stills and sequences | --export writes a frame as PNG or HEIC, and --export-sequence a folder of them | Every export runs without a window | Export |
| Paper sizes | canvasSize as a sheet such as .a4, .a3, or .usLetter, turned .landscape, at a .dpi(_:) you choose | Sized for the output: fractions of the canvas, and paper sizes | Canvas |
| Video and GIF | --export-video (H.264, HEVC, ProRes) and --export-gif, the same frames on every run | Motion: video and GIF; Ch 3: Putting it together: a loop that never ends | Export |
| SVG and PDF for plotters | --export-svg and --export-pdf write the drawing as vector paths, and --hatch turns fills into line work | Lines for a pen: SVG and PDF; Ch 15: How a plot runs | Export |
| Describable output | describe(_:) and describe(_:as:in:): the sketch says what it shows and what its parts are, so a screen reader has something to read | Saying what it shows: describable output | Accessibility |
| Reproducibility metadata | PNG, SVG, PDF, and video files carry their recipe: the seed, the @Param values, the git commit, and the frame; --capture-source keeps the code too | The recipe in the file | Export |
| Setting a parameter for one run | --param name=value sets a parameter for an export, and --list-params shows what a sketch declares | Passing the parameters back in | Export |
| Headless capture | OllinApp.image(of:frame:) renders a sketch to a CGImage in code, with no window; the export flags wrap it, and the guide's figures are made with it | Rendering from code: OllinApp.image(of:frame:) | Export |
| Drawing finer than you save | --render-scale draws each frame larger and averages it back down, for finer edges on filled shapes and outline text | Drawing finer than you save: supersampling | Export |
| Linear frames for a compositor | --export-exr writes the frame before the tone map: light past white, the frame's coverage, and, for a 3D scene, the distance from the eye as a Z channel | The frame before the tone map: OpenEXR | Export |
| Wide gamut and HDR output | colorOutput: Display P3 on screen, highlights brighter than white, HDR10 video, and HEIC stills with a gain map; Color(displayP3:green:blue:) for colors outside sRGB | Brighter than white: HDR output | Wide gamut & HDR output |
| Transparent output | background(.clear) keeps the canvas see-through, and PNG and the two video codecs that carry alpha, proRes4444 and hevcWithAlpha, keep that transparency, so the sketch lands over another layer | Leaving the background behind: transparent video | Export |
| Frame rates | --fps by number or by name (ntsc, film), a FrameRate in code, the broadcast fractions kept exact | The grid a broadcast asks for: named frame rates | Export |
| Slow motion in an export | --slow-motion renders the extra frames a slowed shot needs, so the motion stays smooth rather than stepping, and --made-frames has the GPU build the ones between for a heavy 3D scene | Slower than it happened: slow motion | Export |
| Settled frames | --settle lets a running mean settle before each frame of a video is written | Settled, then written: a running mean in a video | Export |
| Motion blur from subframes | --subframes draws each frame at moments across the open shutter and averages them, so anything a sketch draws streaks, 2D included, at that many times the render cost | Blurred the slow way: subframes | Export |
| A web page | --export-web records what the sketch draws and writes a page that plays it back in a browser, with the framework's shape shader rewritten in GLSL, as one file or a fragment for your own page | A page that plays it: the web export | Web page |
Chapter 42: Making it physical
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| G-code for machines | --export-gcode writes the frame's line work as a machine program: pen plotter, laser, or mill, with the pen-up travel planned short | Driving the machine itself: G-code | G-code |
| A named sheet | GCode(.plotter(), paper: .a4) and DXF(paper: .a3) size a drawing for a sheet with a margin clear on every side, holding its height as well as its width; --gcode-paper, --dxf-paper | Driving the machine itself: G-code | G-code |
| Line drawing | lineDrawing(of:): a 3D scene written down as 2D line work, with what the surfaces hide taken out: the silhouettes, creases, and boundaries a draftsman would draw, so a plotter, a laser, or a vector file can have the scene | A 3D scene on the plotter: lineDrawing(of:), Putting it together: the printed knot | Line drawing |
| Print separations | --export-separations: one grayscale master per spot ink, screened into dots with --screen if you ask, with an overprint preview | Printing one ink at a time: separations, Putting it together: the printed knot | PrintSeparations |
| 3D printing | mesh.write(to:) as STL, OBJ, or 3MF at the size you set, with printCheck() reporting whether the surface is a solid | Something you can hold: a 3D print | Fabrication |
| Embroidery for a machine | --export-embroidery writes the frame as the stitches an embroidery machine sews: strokes as running stitch, fills as rows, each color its own thread | Thread instead of ink: embroidery | Embroidery |
| A drawing for the shop | --export-dxf writes the frame as a DXF drawing a CAD program or a laser's software opens: lines, polylines, and circles in millimeters, each color on its own layer | A drawing for the shop: DXF | DXF |
| Print color | SoftProof and Filter.softProof: the canvas as a press profile will print it, the colors it cannot reach flagged, and --export-plates for the process plates | Seeing the print before you print it: soft proofs | PrintColor |
| Spatial models | --export-usdz writes a 3D frame, and write(to:) any Scene, as USDZ that Quick Look and Messages open and that stands in a room in AR | Something you can walk around: USDZ | Spatial |
| Spatial video | --export-spatial writes the motion as a stereo pair per frame (MV-HEVC), with stereoGeometry naming how far apart the eyes stand and how far away they agree | Something you can look into: spatial video | Spatial |
Chapter 43: Performing
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Live coding | swift run OllinLiveCoding: the code shown over the visuals, evaluated while it moves, with names completed under the text cursor | Performing the code itself: live coding, Finishing a name as you type: completion | LiveCoding |
| Live recording | startRecording() or ⌘⇧R keeps a run as it happens, sound included, through evaluations | Keeping the take | Recording |
| Cues | Each parameter's value saved under a name and called back at once or over a fade: saveCue, cue(_:over:), nextCue, the inspector's Cues card, a .cues.json file named after the sketch, and --cue | A look you come back to: cues | Cues |
| The host on a controller | The live-coding host's own actions, such as evaluate and record, answer to fixed OSC addresses and to MIDI you teach it | The host on a controller: OSC and learned MIDI | LiveCoding |
| Record & replay | --record-take writes down a run's seed, clock, inputs, and parameters; --replay plays it back the same, steps through it frame by frame, and renders it through any export | Playing the night again: replay | Replay |
| Keyframed parameters | automate($parameter) { } writes a parameter's values down over time, with named or Bezier curves, looped or at any speed, and rendered the same by any export | A value at each moment: keyframes | Automation |
| Parameter timeline | OllinLive's panel: drag the playhead, set a parameter, and click its diamond to place a key; lanes, curves shaped by eye, a rule typed in the parameter's own row, and an automation file OllinLive reads and other exports take with --automation | Directing by hand: the timeline panel | Timeline |
| Timecode | TimecodeClock follows the MIDI Time Code a deck or an editing system sends, so a sketch keeps to the show's position rather than its own clock | Following another timeline: timecode | MIDI |
| Parameters driven by a formula | drive($parameter, "190 + sin(time) * 80") reads a parameter's rule from text rather than from Swift source, so it can arrive from a file or a typed field, and one parameter can be worked out from another | A rule instead of keys: formulas, A rule for each part: parameters that hold several numbers | Formula |
| Syphon | Publish frames to (and read frames from) other Mac apps, GPU to GPU | Into another app: Syphon | Syphon |
| Virtual camera | publishVirtualCamera() makes the sketch a camera any app can pick, a video call or a web page included, once the Ollin Camera extension is installed | The sketch as a webcam: the virtual camera | VirtualCamera |
Chapter 44: Handing it over
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| A sketch as the screen saver | ollin new --kind screen-saver wraps a sketch as a .saver the system loads when the machine goes idle, with one script to build, sign, and install it | Living in the system: a screen saver | Screen saver |
| A sketch as the wallpaper | ollin new --kind wallpaper runs a sketch as the desktop: one window per display at desktop level, behind the icons, quit from its menu-bar mark | Wallpaper and a menu-bar strip | Wallpaper |
| A sketch in the menu bar | ollin new --kind menu-bar runs a sketch as a small live strip among the status items, at 30 frames a second, since it stays on screen all day | Wallpaper and a menu-bar strip | Menu bar |
| A sketch as a widget | ollin new --kind widget shows a sketch on the desktop as a widget: widgetTimeline says how far apart the pictures sit, one is drawn for each moment on the day's own clock, and --export-widget shows the run without waiting for it | A widget, and what it does to a sketch, A widget's clock: the time of day, How far apart the pictures sit: widgetTimeline, Seeing the run without waiting: --export-widget | Widget |
| A sketch as an app | ollin new --kind mac-app wraps a finished sketch as a .app, signed for this Mac, or with --sign and --notarize for any Mac without the toolchain, with a frame of itself as its icon | An app to hand somebody: signing and notarizing | A sketch as an app |
| The sketch on the phone | ollin phone puts a sketch on a paired iPhone or iPad and installs it again on each save, its clock and parameters carried across, with the parameters live on the Mac | In your pocket: the sketch on the phone | OnThePhone |
| The phone as the sketch's screen | device.show(self) keeps the sketch running on the Mac and shows its frames full screen on the capture app, sent down the cable as video, with the first finger on the glass as the sketch's pointer and nothing built for the phone | Leaving it on the Mac: the phone as the sketch's screen | Phone |
| Extensions | extend(_:) registers a SketchExtension that hears about setup and each frame before and after draw(), and can ask for the rendered image, for recorders, overlays, and guides | Adding behavior from outside draw(): SketchExtension | Sketch |
| Writing an extension | ollin new --kind extension: a library other sketches import, the ollinx- naming convention, and the four seams to build on | Giving it to somebody else: an extension package | Extensions |
Chapter 45: Installations
| Capability | What it is | Taught in | Reference |
|---|---|---|---|
| Running unattended | Installation puts a sketch on a wall: full screen, no pointer, the display kept awake, and, with a loop declared, a shader clock that starts over at the end of a lap so it stays exact for weeks (every sketch already treats a gap in the frames as a pause) | Leaving it running: Installation, The clock over a long run: gaps and precision | Installation |
| Resuming after a stop | Installation(checkpoint:) writes the @Saved properties, the seed, the clock, and every @Param value to a file on a cadence you set, so a relaunch carries on near where the sketch was | Picking up where it left off: checkpoints and @Saved | Installation |
| Restarting | Installation(restarts: .onFailure) starts the sketch again after a crash or a stall | When it falls over: restarting | Installation |
| Opening hours | Installation(schedule:) with .open(from:to:), or parts of the day named with .from(_:_:) and read back as scheduledPeriod, with .dark(from:) for the hours with nothing on screen | Gallery hours: the schedule | Installation |
| DMX lighting | Lights driven from draw() over Art-Net or sACN: a 512-channel universe filled through the fixtures you patch with DMXFixture and sent each frame, and a lighting console driving the sketch back | Light instead of pixels: DMX and LED maps, A console that drives the sketch: DMXReceiver | DMX |
| LED maps | LEDMap lays strips and matrices of LEDs over the canvas and sends each frame's pixels under them out as DMX | Light instead of pixels: DMX and LED maps | DMX |
| Show lasers | LaserProjector sends line work to a show laser through an Ether Dream: points spaced evenly, held at corners, the beam dark between shapes, and no light until arm() | Drawing with light: a show laser | Laser |
| Fitting a projector | Corner pinning with Command-K fits the picture to a wall, and Installation.Projection blends the edges where two projectors overlap | Fitting a projector: corner pinning, Two projectors, one picture: edge blending | Installation |
| Several displays | Installation(displays: .spanning) spreads one canvas over several displays | Several displays, one machine: .spanning | Installation |
| Several windows, one world | canvasOnScreen tells each window where its canvas sits in the space all the screens share, so several windows show one world | Several windows, one world: canvasOnScreen | Installation |
| Room | Several machines drawing one sketch: they find each other by a name with no server, share values and @Param parameters, agree on one clock so motion stays in step, and take a seat each across a wall | Several machines, one sketch: Room | Room |
| Remote surface | RemoteInspector serves the sketch's @Param parameters to a phone on the same Wi-Fi as touch controls, live both ways, for tuning an installation from in front of it | Tuning it from the floor: RemoteInspector | Remote |
| MQTT | The message bus a building speaks: a house's sensors subscribed to and read in draw() or bound to a parameter, lamps and machines published back to, retained values, a last will for when the sketch is cut off, and a connection that resubscribes and resends on its own | What the building already says: MQTT | MQTT |
Contents · Previous: Appendix C, Coming from p5.js and Processing · Next: Appendix E, Coming from openFrameworks and OPENRNDR