clip-player

bunnpmpnpmyarn
Copy
pnpm dlx shadcn@latest add @audio-ui/clip-player

Copy
import { streamedRecording, wantPages, loadClipPlayer, createClipPlayer, forgetRecording } from "@/lib/clip-player";

Usage

TrackRenderer is a track's player without the audio thread. It reads clips as playedClip turns them into frames, the recordings they name, and the states of the transport as they come, each from its own frame.

Copy
import { playedClips, TrackRenderer } from "@audio-ui/engine/playback";

const player = new TrackRenderer(rate, recordings); // Map of id to { channels, sampleRate }
player.setClips(playedClips(clipsOfTheTrack, (clip) => clip.recording, rate));
player.schedule(state);
player.render(output, currentFrame, 128); // adds into output, a Float32Array per channel

• A clip at 0 dB with no fade, not warped, whose recording is at the engine's rate, is copied as it is: what comes out is the recording, bit for bit. A split clip plays the same frames as the clip it was cut from.
• A gain multiplies, a fade-in rises along its shape from a silent first frame, and a fade-out is the same curve backwards.
• A looped clip plays to its loop's end, then the loop again, each pass on the frame after the last.
• A warped clip plays each stretch between two markers at the speed that joins them, its pitch with its speed, and reads its recording through a cubic interpolator; so does a recording at another sample rate.
• A stop or a jump fades what played out over 2 ms (DECLICK_SECONDS), and a jump fades the new place in. A start from a stop does not fade, and nor does the loop's own wrap.
• A state that arrives late takes over at once and plays the frames its clock says, so a track that heard of a play a block late joins it in step.

In a browser the player runs in an AudioWorklet, and the clip-player library does the wiring: it loads the worklet once per context, posts each recording once for all the players of a context (into shared memory when the page is cross-origin isolated, so an offline render reads the same samples), and says when a player has heard everything, which an offline render waits for before it starts.

Copy
import { playedClips } from "@audio-ui/engine/playback";
import { createClipPlayer, loadClipPlayer } from "@/lib/clip-player";

if (await loadClipPlayer(context)) {
  const player = createClipPlayer(context);
  player.output.connect(trackInput); // stereo, into the track's chain
  player.setClips(playedClips(clips, (clip) => clip.recording, rate), recordings); // id to AudioBuffer
  player.schedule(state);
  await player.synced(); // before an offline context's startRendering()
}

The player plays every clip that sounds at a frame, so clips can overlap. withCrossfades(played), from @audio-ui/arrangement, uses that: two clips that meet with fades of one length go on past the join, as far as both sources hold, and fade through each other, so what they share stays level. Give it what playedClips returns. withEdgeFades(value) gives a clip you lay down a fade of 4 ms at each edge.

Without the library, CLIP_PLAYER_WORKLET is the processor as the source of a module, whatever your bundler:

Copy
import {
  CLIP_PLAYER_PROCESSOR,
  CLIP_PLAYER_WORKLET,
  type ClipPlayerMessage,
} from "@audio-ui/engine/playback";

const url = URL.createObjectURL(
  new Blob([CLIP_PLAYER_WORKLET], { type: "text/javascript" })
);
await context.audioWorklet.addModule(url);
URL.revokeObjectURL(url);

const node = new AudioWorkletNode(context, CLIP_PLAYER_PROCESSOR, {
  numberOfInputs: 0,
  outputChannelCount: [2],
});
const send = (message: ClipPlayerMessage) => node.port.postMessage(message);
send({ channels, id: "take-1", sampleRate: rate, type: "recording" });
send({ clips, type: "clips" });
send({ state, type: "state" });

A recording posted through one player is kept for every player of the context.

Streaming long recordings

A recording decoded into memory takes 32-bit floats for every frame of every channel: an hour of stereo at 48 kHz is 1.4 GB, more than a tab can hold for a few of them. A recording can be streamed instead, read from pages that a Worker loads ahead of playback. The pages sit in shared memory, so the page needs to be cross-origin isolated (Cross-Origin-Opener-Policy: same-origin, Cross-Origin-Embedder-Policy: require-corp).

• The file. A streamed recording is a file of the origin private file system that holds its channels one after the other, each its frames as 32-bit floats at the engine's rate. Write it once, when the recording comes in.
• Giving it to the engine. With clip-player, give it as a StreamedSource, { path, channels, length, sampleRate }, where you would give an AudioBuffer. The first one played starts the loader and a pool of 512 pages of 32,768 frames (128 MB) that every streamed recording of the page shares.
• Pages ahead of time. arrangement-engine asks for the pages of the next 4 seconds as it plays. cue(from) loads, while stopped, those a play from the cursor would read first: useArrangementEngine calls it as the position moves. renderArrangement waits for each second's pages before it renders that second.
• A page not there. It plays silence: a play that runs ahead of the disk loses sound, never time.

Copy
import type { StreamedSource } from "@/lib/clip-player";

const recordings = new Map<string, StreamedSource>([
  ["take-1", { channels: 2, length: 33_984_000, path: ["cache", "take-1.f32"], sampleRate: 48_000 }],
]);
const engine = useArrangementEngine({ doc, recordings });

Without the library, @audio-ui/engine/playback has the pieces:

• createPagePool and createRecordingStream make the pool and each recording's directory of pages.
• RECORDING_LOADER_WORKER is the loader, as the source of a module, to start as a Worker and post RecordingLoaderMessages.
• pagesAhead plans the pages a track's clips read over the next frames, loops, reverse and warp included.
• The clip player's stream message gives it a recording to read from its stream.