Screen capture

A sketch can take the Mac's own screen as material. Any display, any app, or any single window arrives as a live image, so the sketch can draw it, filter it, and analyze it. That means anything running on the machine can become material, whether that is a browser, a map, a video call, a terminal, or another sketch.

Syphon is the counterpart, and the difference is who has to agree. Syphon carries frames between apps that have both chosen to speak it. That is why it is fast and clean, and why it only reaches apps that publish. Screen capture asks nothing of the other app, so it reaches everything on the screen. In exchange it needs the one thing Syphon never does, which is the user's permission to record the screen.

Screen capture lives in a separate library. The drawing core therefore stays free of ScreenCaptureKit, and the permission belongs to the sketches that ask for it. Add import OllinScreen alongside import Ollin.

import Ollin
import OllinScreen

final class Mirror: Sketch {
    let screen = ScreenCapture(.mainDisplay)

    override func setup() { screen.start() }

    override func draw() {
        background(.black)
        drawFrame(screen)
    }
}

Frames arrive as GPU textures, the same way the frames of a video file do. Drawing one costs no trip through the CPU, and every filter applies to it. A capture is also a FrameSource. Every tracker in the vision catalog attaches to it the way it attaches to a camera, and reads whatever is on screen instead.

Two panels of a stand-in desktop. Left, a clean capture of a wallpaper with two windows. Right, the same capture with an Ollin sketch window on it showing the same picture, nested four levels deep

Permission - what the prompt actually covers for a sketch, which is not what you would expect
Choosing what to capture - naming a display, an app, or a window
Listing what is there - finding the names to write into the sketch
Capturing the screen you are drawn on - self-exclusion and the feedback tunnel
Settings - size, frame rate, cursor
Reading the screen with a tracker - vision over a captured feed
What is not here

Permission

Recording the screen needs the user's consent, and macOS grants that consent to an application, not to a piece of code. That distinction matters a lot for a creative-coding framework, because a sketch is usually not an application.

A sketch run from the terminal inherits the terminal's permission. Ollin builds plain executables with no bundle identifier of their own. That covers swift run, the ollin command, OllinLive, and the examples gallery. So macOS attributes their request to the process that launched them, which is Terminal, iTerm, Ghostty, or whichever you use. The prompt you see names your terminal, and so does the entry in System Settings. Allow it there once, and every sketch you run from that terminal can capture, with no further prompt. That single approval covers all of them.

That is convenient, and it is worth being clear about what you are agreeing to. If you allow a terminal to record the screen, everything you run from that terminal can record the screen. It is the same trade as giving a terminal Full Disk Access. That is why a sketch has to opt into the capability, by importing a library and calling start(). The drawing core never carries it.

Granting the permission does not reach a process that is already running. Allow it, then start the sketch again.

ScreenCapture.isAvailable          // is the permission in place?
ScreenCapture.unavailableReason    // a sentence to draw when it is not
ScreenCapture.requestAccess()      // ask (the system prompts once, ever)

start() asks for you, so most sketches never call requestAccess(). A capture with no permission is not an error, and it does not throw. It just has no frames. waitingMessage says why there are none, and drawFrame draws that message on the canvas for you.

override func draw() {
    background(.black)
    guard let rect = drawFrame(screen) else { return }   // draws the reason if there is one
    // …
}

To style the notice yourself, read the reason directly:

if let reason = ScreenCapture.unavailableReason {
    drawStatus(reason, style: .warning)
}

A sketch that ships as a real .app bundle gets its own entry in System Settings and its own prompt. That is the better experience for anything you hand to someone else.

Choosing what to capture

A ScreenSource is a plain value you write in the sketch. The same line always picks the same thing, so the sketch itself stays the record of what was captured:

ScreenCapture(.mainDisplay)                          // the display the menu bar is on
ScreenCapture(.display(displayID))                   // a display by its system id
ScreenCapture(.app("Safari"))                        // every window one app has open
ScreenCapture(.app("com.apple.Safari"))              // the same, by bundle identifier
ScreenCapture(.window(matching: "Shopping list"))       // one window, by its title
ScreenCapture(.window(matching: "Untitled", app: "Notes"))
ScreenCapture(.windowID(windowID))                   // one window by its id

Names are matched loosely, because that is what makes them worth writing down. An app matches on its name or its bundle identifier, ignoring case, and the whole name has to match. A window matches any title containing the text, ignoring case, so "Shopping list" keeps working when the title bar reads "Notes: Shopping list". When several windows match, the lowest window id wins, so a repeated run picks the same window instead of whichever one happens to be frontmost.

Capturing a window gives you that window alone, at its own size, with nothing in front of it, even when something covers it on screen. Capturing an app gives you its windows against their display, and keeps their positions and their overlap.

The source is a live property, so a sketch can switch targets from a parameter or a key without stopping:

screen.source = .app("Music")

A source that names something not on screen yet is not an error. The capture keeps looking. It starts by itself the moment the window appears, so you can name a window and then go and open it. isRunning stays true through the wait, where it means "wanted", and isReceiving tells you whether frames are actually arriving.

Listing what is there

To find out what a machine can capture right now, ask it. All three calls need the permission, and they return an empty array without it. All three are ordered, so a listing reads the same way twice:

await ScreenCapture.availableDisplays()   // [ScreenDisplay]  id, size, isMain, label
await ScreenCapture.availableWindows()    // [ScreenWindow]   id, title, appName, bundleIdentifier, frame, label
await ScreenCapture.availableApps()   // [ScreenApp]      id (pid), name, bundleIdentifier, label

The calls are asynchronous, so a sketch reads them in a Task and keeps drawing while it waits:

override func setup() {
    Task { @MainActor in
        for window in await ScreenCapture.availableWindows() {
            print(".window(matching: \"\(window.title ?? "")\")   // \(window.label)")
        }
    }
}

The intended workflow is to list once, find what you want, and then write the name into the sketch. The sketch then does not depend on anyone having clicked a menu. The ScreenCapture example prints each entry in the listing as the line of code that names that source, ready to paste.

Capturing the screen you are drawn on

By default the sketch's own windows are cut out of a display capture. Without that, a full-screen capture contains the window the sketch is drawn in, so the picture feeds back into itself:

screen.excludesOwnWindows = true    // the default

Turn the exclusion off and that is exactly what happens, which is the point:

screen.excludesOwnWindows = false

The sketch draws a screen containing a window, which draws a screen containing a window, receding until the innermost copy is a few pixels across. This is video feedback, the effect people have been getting by pointing cameras at monitors since the 1960s, and here it costs one boolean. How far it recedes depends on how fast the sketch draws compared with the capture. Move the window and the recursion drifts and smears, which is the part worth watching.

Self-exclusion works on the process, not on the window title. So it holds for a sketch with several windows, and for a sketch with no bundle identifier. It has no effect when you capture a single window or another app, because the sketch was never in that picture to begin with.

Settings

screen.scale = 0.5          // multiplier on the captured pixel size (default 1)
screen.frameRate = 30       // most frames per second (default 60)
screen.showsCursor = false  // draw the pointer into the frames (default true)

scale is the setting that matters on a large display. At 1 a capture arrives at the display's true backing resolution. On a Retina screen that is twice its size in points, so a 5K display gives you a 5120-pixel-wide texture every frame. Halving the scale quarters the number of pixels, which is the cheap way to feed a heavy effect chain.

Frames are delivered only when the captured content actually changes, so a still screen costs nothing whatever frameRate says. The cap is there to keep a busy screen from outrunning the sketch.

Reading the screen with a tracker

Because a capture is a FrameSource, every vision tracker attaches to it the way it attaches to a camera:

import OllinScreen
import OllinVision

let screen = ScreenCapture(.app("Safari"))
lazy var text = TextRecognizer(screen)
lazy var faces = FaceTracker(screen)

A sketch can then read the words on a page as they scroll, or find faces in a video call. It can follow motion across a map, or trace contours out of anything on screen. Map the results into the rectangle drawFrame returns, exactly as you would with a camera.

A tracker needs a CPU copy of each frame. That copy is made only while a tap is installed, so a capture you only draw never pays for it.

What is not here

System audio. ScreenCaptureKit can record what the machine is playing, and this capture does not. It captures pictures only. Sound belongs with the audio analyzer surface rather than with a video frame source, and it has not been built yet.

The system picker. macOS offers a Control Center panel for choosing what to share. It is built around an application singleton with an observer protocol. That suits an app with a bundle identity and a settings window, rather than a sketch. A choice made through a panel would also make the picture depend on a click nobody recorded. Naming the source in code is the reproducible path, and availableWindows() gives you the same discovery with no interface in the way.

Capturing while the screen is locked or asleep. The system does not allow it, and a capture reports itself as stopped.