Playback

An arrangement plays when each of its tracks reads its clips on the audio clock: its recordings, at their places, through their gains and fades, round the loop. @audio-ui/core/playback is that reading, with no React and no audio node in it, so it runs the same in an AudioWorklet, in an offline render and in a test.

bunnpmpnpmyarn
Copy
pnpm add @audio-ui/core

Frames, on both sides

Every time is counted in whole frames: frames of the audio clock (a context's currentFrame) on one side, frames of the arrangement on the other. A PlaybackState joins them: from the clock's frame at, the arrangement's frame position plays, round its loop while it is on. Whole frames are what keep a loop of any length from drifting, and a recording played at speed 1 from being changed by a single bit.

Copy
import { playbackRuns, toFrames } from "@audio-ui/core/playback";

const rate = context.sampleRate;
const state = {
  at: Math.ceil((context.currentTime + 0.02) * rate), // a little ahead, so every track hears of it in time
  loop: [toFrames(8, rate), toFrames(16, rate)] as const,
  playing: true,
  position: toFrames(cursor, rate),
};

// The arrangement's frames a block of 128 plays, cut where the loop comes round.
playbackRuns(state, currentFrame, 128);

Every part of an engine reads the same state: each track's player, the scheduler of the notes (startsInWindow gives what starts in its look-ahead window, and the clock's frame it starts at), and the playhead (heardFrame turns what the speakers play now into a frame). They agree to the frame.

A player per track

TrackRenderer is a track's player without the audio thread. It reads clips as playedClip turns them into frames, the recordings they name, and the states of the transport as they come, each from its own frame.

Copy
import { playedClips, TrackRenderer } from "@audio-ui/core/playback";

const player = new TrackRenderer(rate, recordings); // Map of id to { channels, sampleRate }
player.setClips(playedClips(clipsOfTheTrack, (clip) => clip.recording, rate));
player.schedule(state);
player.render(output, currentFrame, 128); // adds into output, a Float32Array per channel

• A clip at 0 dB with no fade, not warped, whose recording is at the engine's rate, is copied as it is: what comes out is the recording, bit for bit. A split clip plays the same frames as the clip it was cut from.
• A gain multiplies, a fade-in rises along its shape from a silent first frame, and a fade-out is the same curve backwards.
• A looped clip plays to its loop's end, then the loop again, each pass on the frame after the last.
• A warped clip plays each stretch between two markers at the speed that joins them, its pitch with its speed, and reads its recording through a cubic interpolator; so does a recording at another sample rate.
• A stop or a jump fades what played out over 2 ms (DECLICK_SECONDS), and a jump fades the new place in. A start from a stop does not fade, and nor does the loop's own wrap.
• A state that arrives late takes over at once and plays the frames its clock says, so a track that heard of a play a block late joins it in step.

In a browser the player runs in an AudioWorklet, and the clip-player library does the wiring: it loads the worklet once per context, posts each recording once for all the players of a context (into shared memory when the page is cross-origin isolated, so an offline render reads the same samples), and says when a player has heard everything, which an offline render waits for before it starts.

bunnpmpnpmyarn
Copy
pnpm dlx shadcn@latest add @audio-ui/clip-player

Copy
import { playedClips } from "@audio-ui/core/playback";
import { createClipPlayer, loadClipPlayer } from "@/lib/clip-player";

if (await loadClipPlayer(context)) {
  const player = createClipPlayer(context);
  player.output.connect(trackInput); // stereo, into the track's chain
  player.setClips(playedClips(clips, (clip) => clip.recording, rate), recordings); // id to AudioBuffer
  player.schedule(state);
  await player.synced(); // before an offline context's startRendering()
}

Without the library, CLIP_PLAYER_WORKLET is the processor as the source of a module, whatever your bundler:

Copy
import {
  CLIP_PLAYER_PROCESSOR,
  CLIP_PLAYER_WORKLET,
  type ClipPlayerMessage,
} from "@audio-ui/core/playback";

const url = URL.createObjectURL(
  new Blob([CLIP_PLAYER_WORKLET], { type: "text/javascript" })
);
await context.audioWorklet.addModule(url);
URL.revokeObjectURL(url);

const node = new AudioWorkletNode(context, CLIP_PLAYER_PROCESSOR, {
  numberOfInputs: 0,
  outputChannelCount: [2],
});
const send = (message: ClipPlayerMessage) => node.port.postMessage(message);
send({ channels, id: "take-1", sampleRate: rate, type: "recording" });
send({ clips, type: "clips" });
send({ state, type: "state" });

A recording posted through one player is kept for every player of the context.

The mixer's arithmetic

Playing an arrangement

arrangement-engine plays the document useArrangement holds: each audio track through a clip player, each MIDI track through the instrument of its chain with a voice for each note it holds (so a monophonic instrument plays chords), every device chain, every send and return, and the main track, on one transport counted in frames of the audio clock. The document comes in as a value; the engine follows it while it plays, and the interface never touches the audio.

bunnpmpnpmyarn
Copy
pnpm dlx shadcn@latest add @audio-ui/arrangement-engine

Copy
import { useArrangement } from "@/hooks/use-arrangement";
import { useArrangementEngine } from "@/hooks/use-arrangement-engine";

const arrangement = useArrangement({ duration, initialTracks, tempo });
const engine = useArrangementEngine({
  doc: arrangement.doc,
  metronome: { on: clicking, rhythm }, // heard in playback, never in an export
  recordings, // a memoized Map of recording id to AudioBuffer
});

<Button onClick={() => (engine.playing ? engine.stop() : engine.play(cursor))}>Play</Button>;
<TimelinePlayhead source={engine.position} />; // seconds, as the speakers play them
<Meter source={useAudioLevels(engine.meter(track.id) ?? NONE)} />;

• engine.noteOn(track, pitch, velocity) and engine.noteOff(track, pitch) play a note on a MIDI track's instrument now, through its chain, as a key of a clip's piano roll does.
• play(from) starts 12 ms from now (START_SECONDS), so every track has heard of it before its first frame, round the document's loop when it is on; a note already sounding where it starts plays from there. stop() stops the clips and the notes and lets the effects' tails ring.
• position is a live source of the seconds the speakers play: the audio clock, less the output's latency and the engine's delay compensation, so the playhead is where the sound is. The engine stops itself at the end of the arrangement.
• status is the context's sample rate, its latency in seconds (the output's and the delay compensation's), and its state: what an engine line shows.
• A device switched off passes the sound on unchanged, as late as it would make it, so switching it moves nothing in time.

renderArrangement(doc, recordings, { from, to, sampleRate }) renders the main output to a buffer through the same engine on an offline context, faster than real time, scheduling the notes a second at a time with onProgress as it goes: the same frames come out as play, and a neutral clip comes out bit for bit. encodeWav from wav-file turns it into a file.

The mixer

mixer-engine is the mixing desk the engine plays through, usable on its own with sources of your own: a channel per track and per return, each with a device chain (setChain), a volume, a pan, a switch and a solo, sends taken before or after the volume, pan and switch as each return asks, delay compensation from the latency each chain declares, a meter per channel, and the main track to the output. Nothing limits the main track, so a mix over full scale shows on the meters as it would in the file.

Copy
import { createMixer } from "@/lib/mixer-engine";

const mixer = createMixer(context);
mixer.update({
  main: { gain: 1 },
  returns: [{ gain: 1, id: "hall", latency: 0, on: true, pan: 0, pre: false, soloed: false }],
  tracks: [{ gain: 0.8, id: "vox", latency: 288, on: true, pan: -0.2, sends: { hall: 0.3 }, soloed: false }],
});
source.connect(mixer.strip("vox")!.input);

The device library

device-library is every device of the registry with its engine, by shelf: libraryDevice("audioui.compressor") gives the description and create(context, state) its engine, which is how a host turns the ids a project stores into sound, and DEVICE_SHELVES is what a device browser lists.