Mediabunny Media Processing

Mediabunny is the underlying library used within the Agent Board ecosystem for high-performance, low-level media operations. It enables precise metadata extraction, frame-accurate thumbnail generation, and browser compatibility checks.

Retrieving Media Metadata

Obtaining accurate metadata is critical for dynamically sizing compositions or calculating the timeline. Use these utilities to fetch video and audio properties.

Getting Video Metadata

To retrieve the duration and dimensions of a video file, use the following pattern. This is commonly used inside calculateMetadata to adjust composition settings.

import { Input, ALL_FORMATS, UrlSource } from "mediabunny";

export const getMediaMetadata = async (src: string) => {
  using input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });

  const [durationInSeconds, videoTrack] = await Promise.all([
    input.computeDuration(),
    input.getPrimaryVideoTrack(),
  ]);

  return {
    durationInSeconds,
    dimensions: videoTrack ? {
      width: videoTrack.displayWidth,
      height: videoTrack.displayHeight,
    } : null,
  };
};

Extracting Frames

Mediabunny allows for extracting specific frames from a video source without rendering the entire file. This is ideal for generating filmstrips, hover-thumbnails, or processing frames via Canvas.

The `extractFrames` Utility

The extractFrames function iterates through a list of timestamps and provides a VideoSample for each.

import { ALL_FORMATS, Input, UrlSource, VideoSampleSink } from "mediabunny";

export async function extractFrames({
  src,
  timestampsInSeconds,
  onVideoSample,
  signal,
}: {
  src: string;
  timestampsInSeconds: number[];
  onVideoSample: (sample: any) => void;
  signal?: AbortSignal;
}) {
  using input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });

  const videoTrack = await input.getPrimaryVideoTrack();
  if (!videoTrack) throw new Error("No video track found");

  const sink = new VideoSampleSink(videoTrack);

  // Iterate through samples at requested timestamps
  for await (using videoSample of sink.samplesAtTimestamps(timestampsInSeconds)) {
    if (signal?.aborted) break;
    if (videoSample) onVideoSample(videoSample);
  }
}

Drawing Samples to Canvas

Once a sample is extracted, it can be drawn directly to a 2D context:

onVideoSample: (sample) => {
  const canvas = document.createElement("canvas");
  canvas.width = sample.displayWidth;
  canvas.height = sample.displayHeight;
  const ctx = canvas.getContext("2d");
  
  // Draws the specific frame to the canvas context
  sample.draw(ctx!, 0, 0);
}

Codec Validation

Before attempting to render or play a video, use Mediabunny to verify if the client's browser has the necessary hardware or software decoders for the specific container and codec.

Checking Decode Capability

The canDecode function validates both the container format and the individual video/audio tracks.

import { Input, ALL_FORMATS, UrlSource } from "mediabunny";

export const canDecode = async (src: string): Promise<boolean> => {
  const input = new Input({
    formats: ALL_FORMATS,
    source: new UrlSource(src),
  });

  try {
    await input.getFormat();
    
    const videoTrack = await input.getPrimaryVideoTrack();
    if (videoTrack && !(await videoTrack.canDecode())) return false;

    const audioTrack = await input.getPrimaryAudioTrack();
    if (audioTrack && !(await audioTrack.canDecode())) return false;

    return true;
  } catch {
    return false;
  }
};

Input Sources

Mediabunny supports various source types depending on where the media is hosted.

Example: Using a Blob Source

For client-side file processing before upload:

import { Input, ALL_FORMATS, BlobSource } from "mediabunny";

const validateFile = async (file: File) => {
  const input = new Input({
    formats: ALL_FORMATS,
    source: new BlobSource(file),
  });
  // ... proceed with metadata or decode checks
};

Resource Management

Mediabunny utilizes low-level browser resources (like WebCodecs). It is essential to use the using keyword (Explicit Resource Management) or manually call .close() on Input and VideoSample objects to prevent memory leaks.

// The 'using' keyword ensures the input is closed 
// as soon as the block scope is exited.
using input = new Input({ /* ... */ });