# clip-player

> A track's player of audio clips on an AudioWorklet: its recordings to the frame on the audio clock, with gain, fades, loops, reverse and warp, in a running or an offline context

```bash
npx shadcn@latest add @audio-ui/clip-player
```

```ts
import { streamedRecording, wantPages, loadClipPlayer, createClipPlayer, forgetRecording } from "@/lib/clip-player";
```

- Files: `lib/clip-player.ts`
- Depends on: nothing; npm: `@audio-ui/engine`
- Used by: `arrangement-engine`, `block-audio-editor`, `block-device-chain`, `block-mixer`, `block-timeline`

| Export | Kind | What it is |
| --- | --- | --- |
| `StreamedSource` | interface | A recording kept in a file of the origin private file system and streamed from it, never decoded whole into memory: `path` from the root, the file holding its channels one after the other as 32-bit floats at the engine's rate, `length` frames each. |
| `RecordingSource` | type | A recording as a player takes it: decoded, its channels and rate, or streamed from a file. |
| `streamedRecording` | function | What the planner needs of a recording streamed on this page: its rate and length; nothing for one decoded in memory. |
| `wantPages` | function | Loads the pages of streamed recordings asked for, soonest first; resolves once they are in. |
| `ClipPlayer` | interface |  |
| `loadClipPlayer` | function | Loads the clip player into a context once: an AudioWorklet registered as `CLIP_PLAYER_PROCESSOR`. |
| `createClipPlayer` | function | A track's player on a context where `loadClipPlayer` resolved to `true`: an AudioWorkletNode with no input and a stereo output, which plays the track's audio clips from the recordings it is given, on the audio clock. |
| `forgetRecording` | function | Lets a recording go from a context's players, once no clip plays it. |

## Usage

`TrackRenderer` is a track's player without the audio thread. It reads clips as `playedClip` turns them into frames, the recordings they name, and the states of the transport as they come, each from its own frame.

```ts
import { playedClips, TrackRenderer } from "@audio-ui/engine/playback";

const player = new TrackRenderer(rate, recordings); // Map of id to { channels, sampleRate }
player.setClips(playedClips(clipsOfTheTrack, (clip) => clip.recording, rate));
player.schedule(state);
player.render(output, currentFrame, 128); // adds into output, a Float32Array per channel
```

- A clip at 0 dB with no fade, not warped, whose recording is at the engine's rate, is copied as it is: what comes out is the recording, bit for bit. A split clip plays the same frames as the clip it was cut from.
- A gain multiplies, a fade-in rises along its shape from a silent first frame, and a fade-out is the same curve backwards.
- A looped clip plays to its loop's end, then the loop again, each pass on the frame after the last.
- A warped clip plays each stretch between two markers at the speed that joins them, its pitch with its speed, and reads its recording through a cubic interpolator; so does a recording at another sample rate.
- A stop or a jump fades what played out over 2 ms (`DECLICK_SECONDS`), and a jump fades the new place in. A start from a stop does not fade, and nor does the loop's own wrap.
- A state that arrives late takes over at once and plays the frames its clock says, so a track that heard of a play a block late joins it in step.

In a browser the player runs in an AudioWorklet, and the `clip-player` library does the wiring: it loads the worklet once per context, posts each recording once for all the players of a context (into shared memory when the page is cross-origin isolated, so an offline render reads the same samples), and says when a player has heard everything, which an offline render waits for before it starts.

```ts
import { playedClips } from "@audio-ui/engine/playback";
import { createClipPlayer, loadClipPlayer } from "@/lib/clip-player";

if (await loadClipPlayer(context)) {
  const player = createClipPlayer(context);
  player.output.connect(trackInput); // stereo, into the track's chain
  player.setClips(playedClips(clips, (clip) => clip.recording, rate), recordings); // id to AudioBuffer
  player.schedule(state);
  await player.synced(); // before an offline context's startRendering()
}
```

The player plays every clip that sounds at a frame, so clips can overlap. `withCrossfades(played)`, from `@audio-ui/arrangement`, uses that: two clips that meet with fades of one length go on past the join, as far as both sources hold, and fade through each other, so what they share stays level. Give it what `playedClips` returns. `withEdgeFades(value)` gives a clip you lay down a fade of 4 ms at each edge.

Without the library, `CLIP_PLAYER_WORKLET` is the processor as the source of a module, whatever your bundler:

```ts
import {
  CLIP_PLAYER_PROCESSOR,
  CLIP_PLAYER_WORKLET,
  type ClipPlayerMessage,
} from "@audio-ui/engine/playback";

const url = URL.createObjectURL(
  new Blob([CLIP_PLAYER_WORKLET], { type: "text/javascript" })
);
await context.audioWorklet.addModule(url);
URL.revokeObjectURL(url);

const node = new AudioWorkletNode(context, CLIP_PLAYER_PROCESSOR, {
  numberOfInputs: 0,
  outputChannelCount: [2],
});
const send = (message: ClipPlayerMessage) => node.port.postMessage(message);
send({ channels, id: "take-1", sampleRate: rate, type: "recording" });
send({ clips, type: "clips" });
send({ state, type: "state" });
```

A recording posted through one player is kept for every player of the context.

**Playback**

| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `toFrames(seconds, sampleRate)` | `number` |  | A time as the nearest whole frame. |
| `arrangementFrame(state, frame)` | `number` |  | The frame of the arrangement that plays at a frame of the clock. |
| `playbackRuns(state, frame, count)` | `PlaybackRun[]` |  | The arrangement's frames a block plays, cut where the loop comes round. |
| `startsInWindow(state, frame, count, items, startOf)` | `{ at, item }[]` |  | The items that start in a block, each with the clock's frame it starts at, once per pass of the loop. |
| `rescaled(state, ratio, at)` | `PlaybackState` |  | The state that carries on after every time of the arrangement became `ratio` times what it was, as on a change of tempo: the playhead stays on its bar and beat. |
| `continuesFrom(before, after)` | `boolean` |  | Whether a new state carries on without a break, such as a new loop around the playhead. |
| `heardFrame(contextTime, sampleRate, delay?)` | `number` |  | The clock's frame the speakers play at a context time, less the frames the engine delays. |
| `playedClip(clip, recording, sampleRate)` | `PlayedClip` |  | A clip of the arrangement, in seconds, as a player reads it, in frames. |
| `playedClips(clips, recordingOf, sampleRate)` | `PlayedClip[]` |  | The clips of a track that play: not deactivated, with a recording. |
| `TrackRenderer` | `class` |  | A track's player: `setClips`, `schedule`, `render`. |
| `CLIP_PLAYER_WORKLET` | `string` |  | The player's processor, to load with `audioWorklet.addModule`, registered as `CLIP_PLAYER_PROCESSOR`. |

## Streaming long recordings

A recording decoded into memory takes 32-bit floats for every frame of every channel: an hour of stereo at 48 kHz is 1.4 GB, more than a tab can hold for a few of them. A recording can be streamed instead, read from pages that a Worker loads ahead of playback. The pages sit in shared memory, so the page needs to be cross-origin isolated (`Cross-Origin-Opener-Policy: same-origin`, `Cross-Origin-Embedder-Policy: require-corp`).

- **The file.** A streamed recording is a file of the origin private file system that holds its channels one after the other, each its frames as 32-bit floats at the engine's rate. Write it once, when the recording comes in.
- **Giving it to the engine.** With `clip-player`, give it as a `StreamedSource`, `{ path, channels, length, sampleRate }`, where you would give an `AudioBuffer`. The first one played starts the loader and a pool of 512 pages of 32,768 frames (128 MB) that every streamed recording of the page shares.
- **Pages ahead of time.** `arrangement-engine` asks for the pages of the next 4 seconds as it plays. `cue(from)` loads, while stopped, those a play from the cursor would read first: `useArrangementEngine` calls it as the position moves. `renderArrangement` waits for each second's pages before it renders that second.
- **A page not there.** It plays silence: a play that runs ahead of the disk loses sound, never time.

```ts
import type { StreamedSource } from "@/lib/clip-player";

const recordings = new Map<string, StreamedSource>([
  ["take-1", { channels: 2, length: 33_984_000, path: ["cache", "take-1.f32"], sampleRate: 48_000 }],
]);
const engine = useArrangementEngine({ doc, recordings });
```

Without the library, `@audio-ui/engine/playback` has the pieces:

- `createPagePool` and `createRecordingStream` make the pool and each recording's directory of pages.
- `RECORDING_LOADER_WORKER` is the loader, as the source of a module, to start as a Worker and post `RecordingLoaderMessage`s.
- `pagesAhead` plans the pages a track's clips read over the next frames, loops, reverse and warp included.
- The clip player's `stream` message gives it a recording to read from its stream.

**Streaming**

| Prop | Type | Default | Description |
| --- | --- | --- | --- |
| `createPagePool(slots, { channels, pageFrames, shared })` | `PagePool` |  | The pages every streamed recording of a page shares, in shared memory unless `shared` is false. |
| `createRecordingStream(pool, channels, length)` | `RecordingStream` |  | A recording read from the pool, with the directory of the slot each of its pages is in. |
| `pagesAhead(state, clips, frame, count, recordings, sampleRate, pageFrames)` | `PageRequest[]` |  | The pages a track's clips read over a stretch of the clock, the soonest first; stopped, those a play from the position reads. |
| `PageCache` | `class` |  | Loads the pages asked for into the pool and gives up those asked for longest ago: what the loader runs. |
| `RECORDING_LOADER_WORKER` | `string` |  | The loader, to start as a module Worker, which reads pages from files of the origin private file system. |