Depth of field from light

A blurred photograph is not a sharp picture with a blur on top. It is many rays of light that each landed in a slightly different place, because they passed through the lens out of focus. Ollin can render a picture the same way. Draw a scene as millions of faint points. Move every point to a random spot in a ball, and let the ball grow with the point's distance from the plane of focus. Then add all the points up as light. Lines on the focal plane stay sharp, and the rest softens into bokeh. There is no blur filter. The blur comes from where the light fell.

LineSpray is that technique as one call. It is built from four smaller features that you can also use on their own. They are the .light particle style, the Accumulator, the Bokeh lens value, and the develop filter that prints the result.

Contents

• LineSpray - lines of light through a lens, converging over frames
• Bokeh - the lens
• Accumulator - the running mean under it
• The light particle style - particles that deposit light for an additive sum
• develop - printing accumulated light
• Building your own - the same pipeline from the compute primitives
• Notes

LineSpray

Give a LineSpray an array of SprayLines once, in setup(). Then draw it every frame while a camera is set:

var spray: LineSpray!

override func setup() {
    var lines: [SprayLine] = []
    for i in 0 ..< 400 {
        let a = Vector3(random(-5, 5), random(-5, 5), random(-5, 5))
        let b = a + Vector3(random(-1, 1), random(-1, 1), random(-1, 1))
        lines.append(SprayLine(from: a, to: b, color: .white, intensity: 0.5))
    }
    spray = makeLineSpray(lines, sampling: .byLength(pointsPerPass: 50_000), passesPerFrame: 5)
    spray.bokeh = Bokeh(focalDistance: 20, strength: 0.08, minSize: 0.02)
}

override func draw() {
    background(.black)
    camera(.orbiting(target: .zero, radius: 20, azimuth: 0.4, elevation: 0.3, fieldOfView: .pi / 8))
    drawLineSpray(spray)
    drawImage(spray.developed(exposure: 400, ground: Color(hex: 0x151010)).image, 0, 0)
}

Each frame, drawLineSpray scatters passesPerFrame passes of points along the lines on the GPU, and adds them into a running mean. Every point is a new spot on its line. Ollin then moves that spot to another spot inside the bokeh ball around it, and projects it through the sketch's Camera3D. The mean converges as passes add up, so the longer the sketch runs, the smoother the bokeh gets. When you change the camera, the lens, the lines, or the point size, the average restarts on its own. It restarts because the samples drawn before no longer describe the picture. spray.passes tells you how far it has converged.

A SprayLine is a segment with light at each end and a weight. Name the light as it is, or as a tone and a brightness:

SprayLine(from: a, to: b, light: radiance, weight: 1)                                 // linear RGB, per pass
SprayLine(from: a, to: b, color: .orange, endColor: .white, intensity: 2, weight: 1)  // a tone times a brightness

• light and endLight are linear RGB radiance emitted per pass, SIMD3<Double>, with no ceiling, so a line can be brighter than white. A scene that works out its light in linear units, a diffuse term times a tint, hands the numbers over unchanged.
• color and endColor are sRGB tones and intensity and endIntensity linear multipliers. The initializer converts them on the way in, color.linearRGB × intensity, so the two forms meet exactly: a white line at intensity 0.4 is the line whose light is 0.4 in every channel.
• A line's light per pass is its light, however many points draw it. Every point carries its share of that light. This keeps the exposure the same under either sampling mode.
• light is the whole segment's, not a brightness per unit of it. Length never enters the exposure: a meter-long line and a millimeter-long one at the same light lay down the same light per pass. A scene that works in light per unit length multiplies by its own extent before handing the number over. A line whose ends are the same point puts every sample in one place, which is how a dot is drawn.
• sampling picks how a pass divides its points among the lines. .byLength(pointsPerPass:) gives every line points in proportion to its length times its weight, and never fewer than one. So a long line is drawn as densely as a short one. .perLine(n) gives every line the same count.
• Under .perLine(n) every primitive gets n points, a quad as much as a line. Under .byLength(pointsPerPass:) the quads draw from a pool of their own, shared by area times weight, because a length and an area are not the same units and one budget split between them would mean nothing: the number is the same one, spent twice.
• A scene of dots wants .perLine(n). Under .byLength a share is the length times the weight, so a dot's share is zero however heavy it is. It falls to the floor of one point a pass. Its light is still right, since that one point carries all of it, but it fills in far more slowly than a line does. Where every line is a dot there is no length to share out, and each gets one point whatever pointsPerPass says.
• pointSize is the diameter that each sample's light spreads over, in canvas points. The total light is the same at any size, so a larger size softens the grain without changing the exposure. The default of 1 is a one-pixel point, which is the cheapest to draw.
• spray.setQuads(_:) puts surfaces in the same spray. A SprayQuad is the parallelogram two edges span from a corner, sampled over its area rather than along a length, so a scene can be made of faces instead of a wireframe of one. Lines and quads scatter into the same accumulator in the same pass and print together: one picture, not two.
• A quad's light is the whole quad's, exactly as a line's is. Scaling a quad spreads the same light over more surface rather than making it brighter, so a scene that works in light per unit area multiplies by its own extent on the way in.
• bokeh.aperture is the shape light scatters through, which is what a point too far from focus turns into. .round is the default and the ball every picture was made with before there was a choice. .blades(6) is an iris of six straight blades, the hexagonal sparkle a real camera draws; .blades(count:rotation:) turns it. Fewer than three blades reads round, since there is no shape to scatter in.
• .picture(_:) takes any Image as the aperture instead, read across the hole: where the picture is dark, no light passes. That covers an iris a sketch drew for itself and one it loaded, since a render target's image and a decoded file are the same thing here. Ollin bundles no masks.

spray.bokeh.aperture = .blades(6)                       // hexagonal sparkles
spray.bokeh.aperture = .blades(count: 5, rotation: .pi / 10)

let iris = makeRenderTarget(width: 128, height: 128)    // or drawn, and animated
withTarget(iris) {
    background(.black)
    fill(.white)
    drawPolygon((0 ..< 7).map { i in
        let a = Double(i) / 7 * .tau
        return Vector2(64 + cos(a) * blades, 64 + sin(a) * blades)
    })
}
spray.bokeh.aperture = .picture(iris.image)

• An aperture moves where a sample lands, never how much light it carries. A hexagon, a triangle and the round ball print the same total light; what changes is the shape it is spread into. A mask is the exception by its nature, since what it darkens does not pass.
• A flat aperture spreads a little wider than the round one at the same strength. The ball is a volume, so its projection crowds the middle, while a hole is flat and even across itself. Blades also scatter in the aperture's own plane rather than through a ball, so they carry no depth jitter.
• spray.picture is an Image the quads read. Each quad's pictureBounds names the patch of it that belongs to that quad, in 0...1, and the quad's light is multiplied by what it finds there, so a photograph or a sheet of glyphs becomes an object made of light for the lens to throw out of focus. The texels are read as linear light, the units a quad's own light is in. A quad with no pictureBounds reads nothing and stands on its light alone.
• spray.lines and spray.quads are the scene as it was last given, for reading back what a sketch put in.
• spray.image is the running mean as an Image in linear light, for when you would rather print it yourself.
• spray.setLines(_:) swaps in a new scene and restarts the average. It is built to be called every frame. The same lines again cost nothing and leave the average alone, which is what lets a sketch that rebuilds its scene each draw converge under --settle. Lines that moved restart it, and while every line keeps its point count (always under .perLine) nothing is allocated: the point table is kept and the records are rewritten into a ring of buffers the frames in flight are not reading. A scene whose point counts change, a different number of lines or .byLength shares that rounded differently, rebuilds the table. Under .byLength that is every frame for a scene whose lines stretch or whose surfaces change size, since the shares move with the lengths, and a rebuild writes one entry for every point of a pass. It is a small cost beside scattering the points, and a scene that only turns never pays it, since its lengths hold. The points themselves are kept from frame to frame with room to drift, so a total that moves by a few points allocates nothing.

spray.setQuads([
    SprayQuad(corner: Vector3(-2, -1, 0), edge1: Vector3(4, 0, 0), edge2: Vector3(0, 0, 4),
              color: .white, intensity: 0.4),                      // a lit floor
    SprayQuad(corner: Vector3(-1, 0, 1), edge1: Vector3(2, 0, 0), edge2: Vector3(0, 2, 0),
              light: SIMD3(repeating: 1),
              pictureBounds: Rectangle(x: 0, y: 0, width: 1, height: 1)),   // a picture, defocused
])
spray.picture = try? loadImage("poster.jpg", in: .module)

The Rendering/LineSpray example draws a whole scene through it, with the camera, the lens, and the print settings as parameters.

Bokeh

Bokeh describes the lens as a value:

Bokeh(focalDistance: 49, strength: 0.095, minSize: 0.015, power: 1, attenuation: 0)

• focalDistance is the distance from the camera to the plane of focus, along the view axis, in world units.
• strength is the ball's radius per unit of defocus, so it acts as the aperture. Every sample scatters inside a ball of radius strength × defocus^power, where defocus is the sample's distance from the focal plane. A strength of 0 is a pinhole.
• minSize is the smallest ball. It gives a line that is exactly in focus some width, rather than a hairline.
• power is the exponent applied to the defocus before it is scaled. A power of 1 grows the blur in proportion to distance. A power of 1.5 keeps the region near focus sharper and lets far things blur faster.
• attenuation fades light with distance from the focal plane. Each sample is multiplied by exp(-attenuation × defocus).

Accumulator

An Accumulator is a layer that keeps the running mean of everything drawn into it, frame after frame. The mean is what makes a picture built from faint samples converge instead of growing brighter without end. It is documented with the rest of accumulation. LineSpray owns one as spray.accumulator. The DepthOfField example drives one directly.

The light particle style

drawParticles(_:style: .light) draws a particle buffer as light rather than ink:

withAccumulator(light, passes: 4) {
    blendMode(.add)
    drawParticles(samples, style: .light)
}

• Coverage is the plain area ramp, so a particle deposits color × alpha × area wherever it lands. The deposit is exactly linear in the particle's size. It is the same whether the particle falls on a pixel's center or on its corner. The default .marks style instead remaps coverage to perceptual alpha. That is right for ink over a light ground, but wrong for a sum of light. In a sum of light it lifts a dot on a corner to more than twice the light of a dot on a center.
• color is read as linear radiance, not as an sRGB tone. If you want to write a tone in the kernel, convert it with srgbToLinear.
• A particle at or under one pixel takes a cheap path. Its quad is a single texel, so a million one-pixel points cost a million fragments rather than twenty-five million. The deposit is continuous across the one-pixel boundary, so a size that is animated through that boundary never steps in brightness.
• To deposit a point's whole light into its pixel, put the radiance in color and set alpha = 4 / (π × size²). The disc's area then cancels out, and the pixel receives the radiance itself.

develop

Filter.develop(exposure:ground:) prints a layer of accumulated light as a picture. It scales the layer's linear values by exposure, then rolls them off through the Reinhard curve x / (1 + x), so nothing clips. After the curve, it adds ground as a display color that the curve never touches. ground is the paper the light is printed on. The result is written as the display value itself rather than re-encoded. That is what gives a picture accumulated from faint light, such as a sandpainting, its deep midtones. Accumulator.developed(exposure:ground:) and LineSpray.developed(exposure:ground:) apply this filter to the mean.

drawImage(light.developed(exposure: 30, ground: Color(hex: 0x151010)).image, 0, 0)

Building your own

LineSpray samples lines. The same pipeline runs from the compute primitives over any geometry a kernel can evaluate, such as a curve, a surface, or a point cloud. Each frame, do three things:

1. Pack the camera with cameraParams() and append the lens after it. Dispatch a kernel with compute(_:buffers:count:params:), binding your scene buffer and a particle buffer. In the kernel, move a point into camera space with the packed view. Then scatter it with ballSample(seed) × radius, and project it with ollin_project_eye(p.camera, eye, u.resolution).
2. Draw the particles with drawParticles(buffer, style: .light) inside withAccumulator(acc, passes:) under blendMode(.add).
3. Print with acc.developed(exposure:ground:).

The Rendering/DepthOfField example is exactly that pipeline, over nine closed curves. It draws each sample as three particles scattered to slightly different radii, so the bokeh gets chromatic fringes. The compute reference covers the kernel side.

Notes

• A camera is required. drawLineSpray projects through the active Camera3D. Without one, it does nothing. The focal distance is measured along the view axis. So when the camera's target is what should be sharp, set focalDistance to the camera's radius.
• Tell the accumulator how many passes. The mean divides by the number of passes recorded. When a frame draws several passes' worth of samples, say how many, using passesPerFrame or withAccumulator(_:passes:). Otherwise the picture reads too bright by that factor.
• Exposure is a print setting. It scales the mean after the mean has converged, so changing it does not restart the average. Changing the lens or the camera does.
• A moving camera in an export. Every frame of a turning scene restarts the average, so a plain video export is grainy. --settle N holds the clock and draws each written frame N times before writing it (see Settled frames). For the ring sphere, 40 draws is 200 passes.
• Not on the web page. An accumulator keeps state that the recorded page cannot hold, so --export-web names it and stops. Export the sketch as video instead.
• Attribution. The technique is Anders Hoff's (inconvergent). Blurry by Domenico Bruzzese is the reference implementation. It was read for approach, and Ollin's version is written independently. See ATTRIBUTION.md.

^{Accumulation · HDR & tone mapping · Compute & GPU particles · 3D}