Frame & Thumbnail Extraction

Agent Board utilizes the mediabunny library to perform high-performance frame extraction from video sources. This capability is essential for generating preview thumbnails, creating filmstrip UI components, or performing frame-by-frame analysis.

Core Extraction Utility

The primary interface for this feature is the extractFrames function. It allows you to specify a set of timestamps and provides a callback that executes as each frame is successfully decoded.

Basic Usage

To extract frames at specific timestamps, pass an array of seconds to the timestampsInSeconds property.

import { extractFrames } from "./utils/extract-frames";

await extractFrames({
  src: "https://example.com/video.mp4",
  timestampsInSeconds: [0, 5, 10, 15],
  onVideoSample: (sample) => {
    // Process each frame (e.g., draw to a canvas)
    const canvas = document.createElement("canvas");
    canvas.width = sample.displayWidth;
    canvas.height = sample.displayHeight;
    const ctx = canvas.getContext("2d");
    
    sample.draw(ctx!, 0, 0);
    
    const imageUrl = canvas.toDataURL("image/jpeg");
    console.log(`Extracted frame at ${sample.timestamp}s`);
  },
});

Generating Filmstrips

For UI components like timelines or filmstrips, you often need to extract a dynamic number of frames based on the video's duration and the available display width. You can pass a function to timestampsInSeconds to calculate these values dynamically.

await extractFrames({
  src: "https://example.com/video.mp4",
  timestampsInSeconds: async ({ track, durationInSeconds }) => {
    const frameCount = 10; // Number of thumbnails to generate
    const timestamps: number[] = [];
    const duration = durationInSeconds ?? 0;

    for (let i = 0; i < frameCount; i++) {
      // Calculate midpoints for each segment
      timestamps.push((duration / frameCount) * (i + 0.5));
    }

    return timestamps;
  },
  onVideoSample: (sample) => {
    // Store or display the filmstrip frame
  },
});

API Reference

`extractFrames` Props

The `VideoSample` Object

The onVideoSample callback provides a VideoSample object with the following public interface:

displayWidth / displayHeight: The intrinsic dimensions of the frame.
timestamp: The actual timestamp (in seconds) of the extracted frame.
draw(ctx, x, y): A method to render the frame directly onto a 2D Canvas context.

Cancellation and Timeouts

Frame extraction can be resource-intensive. It is highly recommended to use an AbortSignal to prevent memory leaks or redundant processing when a user navigates away or starts a new extraction task.

const controller = new AbortController();

// Cancel if extraction takes longer than 10 seconds
const timeoutId = setTimeout(() => controller.abort(), 10000);

try {
  await extractFrames({
    src: videoUrl,
    timestampsInSeconds: [1, 2, 3],
    onVideoSample: (sample) => {
        /* ... */
    },
    signal: controller.signal,
  });
} catch (error) {
  if (error.name === 'AbortError') {
    console.log('Extraction cancelled by user or timeout');
  }
} finally {
  clearTimeout(timeoutId);
}

Best Practices

Memory Management: When drawing samples to a canvas in a loop, ensure you are not creating excessive DOM elements. Reuse canvases where possible.
Concurrency: Avoid running multiple extractFrames calls on the same high-resolution source simultaneously, as this can saturate the browser's hardware decoder.
Error Handling: Always wrap extractFrames in a try/catch block, as it may fail if the video codec is unsupported or the network request fails. Use the canDecode utility to pre-verify sources.