Canvas and WebGL rendering

Optimizing canvas and WebGL compositions for high-performance video rendering

Overview

Helios supports two rendering paths:

• DOM path - Screenshots HTML/CSS/SVG via Playwright
• Canvas path - Direct frame encoding via WebCodecs (faster)

For canvas-heavy compositions (Three.js, Pixi.js, raw Canvas API), the canvas path provides significantly better performance through hardware-accelerated encoding.

Automatic path detection

Helios automatically selects the rendering path:

// DOM path (screenshots)
helios.subscribe((state) => {
  document.querySelector('.box').style.opacity = state.time;
});

// Canvas path (WebCodecs)
helios.subscribe((state) => {
  ctx.clearRect(0, 0, width, height);
  ctx.fillRect(state.time * 100, 100, 50, 50);
});

The renderer inspects the composition and selects the optimal strategy.

Canvas rendering path

The canvas path encodes frames directly from a canvas element using the WebCodecs API.

How it works

1. Frame capture - Renderer calls your draw function
2. Canvas export - Pixels extracted via canvas.toBlob() or ImageBitmap
3. Hardware encoding - VideoEncoder compresses frame (GPU-accelerated)
4. FFmpeg muxing - Frames piped to FFmpeg for MP4 container

Basic setup

import { Helios } from '@helios-project/core';

const canvas = document.getElementById('canvas') as HTMLCanvasElement;
const ctx = canvas.getContext('2d')!;

const helios = new Helios({ duration: 5, fps: 30 });

helios.subscribe((state) => {
  const { currentFrame, fps, duration } = state;
  const progress = currentFrame / (duration * fps);
  
  ctx.clearRect(0, 0, canvas.width, canvas.height);
  
  // Draw based on progress
  ctx.fillStyle = `hsl(${progress * 360}, 100%, 50%)`;
  ctx.fillRect(100, 100, 200, 200);
});

Three.js optimization

Basic Three.js composition

import * as THREE from 'three';
import { Helios } from '@helios-project/core';

const canvas = document.getElementById('canvas') as HTMLCanvasElement;
const renderer = new THREE.WebGLRenderer({ 
  canvas, 
  antialias: true,
  alpha: false // Disable if no transparency needed
});

const scene = new THREE.Scene();
const camera = new THREE.PerspectiveCamera(75, 16/9, 0.1, 1000);
camera.position.z = 5;

// Add objects
const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshStandardMaterial({ color: 0x00ff00 });
const cube = new THREE.Mesh(geometry, material);
scene.add(cube);

// Lighting
const light = new THREE.DirectionalLight(0xffffff, 1);
light.position.set(1, 1, 1);
scene.add(light);

const helios = new Helios({ duration: 10, fps: 30 });

helios.subscribe((state) => {
  const time = state.currentTime;
  
  // Animate based on time (not delta)
  cube.rotation.x = time * 0.5;
  cube.rotation.y = time * 0.8;
  
  renderer.render(scene, camera);
});

Performance tips for Three.js

Use static scene graph

// Good: Modify existing objects
cube.rotation.y = time;

// Bad: Create new objects every frame
scene.add(new THREE.Mesh(geometry, material)); // Memory leak!

Disable auto-clear for layers

renderer.autoClear = false;

// Manual render order
renderer.clear();
renderer.render(backgroundScene, camera);
renderer.clearDepth();
renderer.render(foregroundScene, camera);

Reuse geometries and materials

const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshStandardMaterial({ color: 0xff0000 });

for (let i = 0; i < 100; i++) {
  const mesh = new THREE.Mesh(geometry, material);
  scene.add(mesh);
}

Use instancing for repeated objects

import { InstancedMesh } from 'three';

const geometry = new THREE.BoxGeometry();
const material = new THREE.MeshStandardMaterial();
const count = 1000;

const instancedMesh = new THREE.InstancedMesh(geometry, material, count);

const matrix = new THREE.Matrix4();
for (let i = 0; i < count; i++) {
  matrix.setPosition(Math.random() * 10, Math.random() * 10, Math.random() * 10);
  instancedMesh.setMatrixAt(i, matrix);
}

scene.add(instancedMesh);

Optimize shadow maps

renderer.shadowMap.enabled = true;
renderer.shadowMap.type = THREE.PCFSoftShadowMap; // Better quality
renderer.shadowMap.autoUpdate = false; // Update manually if static

// Only cast shadows from key lights
directionalLight.castShadow = true;
directionalLight.shadow.mapSize.width = 2048;
directionalLight.shadow.mapSize.height = 2048;

Pixi.js rendering

Pixi.js is a 2D WebGL rendering library optimized for sprites and 2D graphics.

import * as PIXI from 'pixi.js';
import { Helios } from '@helios-project/core';

const app = new PIXI.Application({
  view: document.getElementById('canvas') as HTMLCanvasElement,
  width: 1920,
  height: 1080,
  backgroundColor: 0x1099bb,
});

const sprite = PIXI.Sprite.from('assets/bunny.png');
sprite.anchor.set(0.5);
sprite.x = app.screen.width / 2;
sprite.y = app.screen.height / 2;
app.stage.addChild(sprite);

const helios = new Helios({ duration: 5, fps: 30 });

helios.subscribe((state) => {
  const progress = state.currentFrame / (state.duration * state.fps);
  
  // Animate sprite
  sprite.rotation = progress * Math.PI * 2;
  sprite.scale.set(1 + Math.sin(progress * Math.PI * 2) * 0.2);
  
  // Manually render (disable auto-render)
  app.renderer.render(app.stage);
});

// Disable Pixi's auto-render ticker
app.ticker.stop();

GPU acceleration setup

GPU acceleration is critical for rendering performance.

Browser configuration

When using @helios-project/renderer, GPU acceleration is enabled by default:

import { Renderer } from '@helios-project/renderer';

const renderer = new Renderer({
  mode: 'canvas',
  browserConfig: {
    headless: true,
    args: [
      '--use-gl=egl',              // Force OpenGL backend
      '--ignore-gpu-blocklist',     // Override GPU blacklist
      '--enable-gpu-rasterization', // Hardware rasterization
      '--enable-zero-copy',         // Zero-copy texture upload
    ]
  }
});

Verifying GPU acceleration

Use Helios.diagnose() to check WebGL support:

const report = await Helios.diagnose();

console.log('WebGL:', report.webgl);     // true if supported
console.log('WebGL2:', report.webgl2);   // true if supported
console.log('WebCodecs:', report.webCodecs); // true if supported

For renderer diagnostics:

const diagnostics = await renderer.diagnose();

console.log('FFmpeg:', diagnostics.ffmpeg.version);
console.log('HW Accel:', diagnostics.ffmpeg.hwaccels);
console.log('Codecs:', diagnostics.browser.codecs);

Hardware encoder selection

WebCodecs automatically selects hardware encoders when available:

const config = {
  codec: 'avc1.4d002a', // H.264 High Profile
  width: 1920,
  height: 1080,
  bitrate: 5_000_000,
  framerate: 30
};

const { supported } = await VideoEncoder.isConfigSupported(config);
console.log('H.264 supported:', supported);

The renderer will fall back to software encoding if hardware acceleration is unavailable.

WebGL best practices

Canvas resolution

Match canvas resolution to export resolution:

const width = 1920;
const height = 1080;

canvas.width = width;
canvas.height = height;

renderer.setSize(width, height, false); // Three.js

Avoid using CSS to scale canvas. This wastes GPU resources rendering at high resolution, then downscaling.

Texture management

Dispose unused textures

texture.dispose();
geometry.dispose();
material.dispose();

Use compressed textures

const loader = new THREE.CompressedTextureLoader();
const texture = loader.load('texture.ktx2');

Limit texture size

const maxSize = renderer.capabilities.maxTextureSize; // Usually 4096 or 8192

Avoid per-frame allocations

// Bad: Creates garbage every frame
helios.subscribe(() => {
  const temp = new THREE.Vector3(); // Allocates memory
  cube.position.copy(temp);
});

// Good: Reuse objects
const temp = new THREE.Vector3();
helios.subscribe(() => {
  temp.set(x, y, z);
  cube.position.copy(temp);
});

Frame timing patterns

Helios provides frame-accurate timing. Use currentFrame or currentTime for animations.

Linear interpolation

helios.subscribe((state) => {
  const progress = state.currentFrame / (state.duration * state.fps);
  cube.rotation.y = progress * Math.PI * 2;
});

Keyframe-based animation

function getValueAtTime(keyframes: [number, number][], time: number) {
  for (let i = 0; i < keyframes.length - 1; i++) {
    const [t1, v1] = keyframes[i];
    const [t2, v2] = keyframes[i + 1];
    
    if (time >= t1 && time <= t2) {
      const progress = (time - t1) / (t2 - t1);
      return v1 + (v2 - v1) * progress;
    }
  }
  return keyframes[keyframes.length - 1][1];
}

const keyframes: [number, number][] = [
  [0, 0],
  [2, 5],
  [5, -3],
  [10, 0]
];

helios.subscribe((state) => {
  cube.position.x = getValueAtTime(keyframes, state.currentTime);
});

Debugging canvas renders

Visual debugging

Render in headed mode to see what’s happening:

helios render composition.html --no-headless

Export canvas for inspection

helios.subscribe((state) => {
  // Draw frame
  renderer.render(scene, camera);
  
  // Export for debugging
  if (state.currentFrame === 30) {
    canvas.toBlob((blob) => {
      const url = URL.createObjectURL(blob!);
      console.log('Frame 30:', url);
    });
  }
});

Monitor performance

const stats = {
  drawCalls: 0,
  triangles: 0
};

helios.subscribe((state) => {
  renderer.info.reset();
  renderer.render(scene, camera);
  
  stats.drawCalls = renderer.info.render.calls;
  stats.triangles = renderer.info.render.triangles;
  
  if (state.currentFrame % 30 === 0) {
    console.log('Stats:', stats);
  }
});

Next steps

• Learn about custom drivers for specialized renderers
• Optimize with performance tuning
• Debug issues with diagnostics