Vision Camera

Run WGSL effects on live camera frames by importing each Vision Camera Frame as a GPUExternalTexture, sampling it in a shader, and presenting from the frame-output worklet.

Vision Camera owns the camera pipeline and exposes each captured image as a Frame hybrid object: a short-lived handle to GPU-shareable pixel data. React Native WebGPU sits on the consumer side: you take the frame's NativeBuffer, wrap it into a NativeVideoFrame, and sample it with importExternalTexture.

Vision Camera concepts

Before writing WGSL, understand what the Frame object gives you (full API reference):

Concept	Why it matters for WebGPU
`hasNativeBuffer`	Zero-copy GPU path - skip CPU pixel downloads
`getNativeBuffer()`	Returns a `NativeBuffer` with a native `pointer` (IOSurface / AHardwareBuffer)
`orientation`	Sensor rotation relative to your output - map to `importExternalTexture({ rotation })`
`isMirrored`	Front-camera horizontal flip - map to `importExternalTexture({ mirrored })`
`pixelFormat: "native"`	Request NV12-style surfaces via `useFrameOutput({ pixelFormat: "native" })`
`dispose()`	Release the frame back to the camera pool - call after GPU work finishes

Vision Camera documents the WebGPU import path directly on Frame.getNativeBuffer() - Examples:

if (.) {
  const  = .();
  const  = .(.);
  const  = .({
    : ,
    : "camera-frame",
  });
  // After submitting commands that sample externalTexture:
  .();
  .();
  .();
}

Frame processors run on a worklet runtime, so the wrapper is called through the RNWebGPU global, which the worklet captures like device; the createVideoFrameFromNativeBuffer export works on the JS thread only.

React Native WebGPU extends importExternalTexture with rotation and mirrored options so the external texture uprights the sensor image before your shader runs - see Native Extensions.

1. Request a device

Camera frames need Dawn's native frame import, which requestDevice() enables by default whenever the adapter supports it. Nothing has to be listed:

const  = await ..();
const  = await !.();

A few Android devices cannot import camera frames. device.features.has("opaque-ycbcr-android-for-external-texture") tells before the first frame; otherwise the first importExternalTexture() throws an error naming it.

2. Canvas + frame output

Use useFrameOutput (Vision Camera v5) with pixelFormat: "native" so each Frame carries a GPU-native buffer:

import { Canvas, useCanvasRef } from "react-native-webgpu";
import { useFrameOutput } from "react-native-vision-camera";

const ref = useCanvasRef();
// Build pipeline, sampler, uniform buffers once on main thread…

const frameOutput = useFrameOutput({
  pixelFormat: "native",
  onFrame: (frame) => {
    "worklet";
    if (!frame.hasNativeBuffer) {
      frame.dispose();
      return;
    }
    renderCameraFrame(frame, device, context, pipeline);
  },
});

3. Per-frame import and render

Each frame is a new external texture.

function (
  : Frame,
  : GPUDevice,
  : ,
  : GPURenderPipeline,
) {
  "worklet";

  const  = .();
  try {
    const  = .(
      .,
    );

    let : 0 | 90 | 180 | 270 = 0;
    if (. === "right")  = 90;
    else if (. === "down")  = 180;
    else if (. === "left")  = 270;

    const  = .({
      : ,
      ,
      : .,
    });

    // encode render pass sampling texture_external …
    ..([.()]);
    .();
    .();
    .();
  } finally {
    .();
    .();
  }
}

Order matters: submit → present → destroy → release. Skipping release() or frame.dispose() stalls the camera buffer pool.

4. WGSL - sample the camera

Bind the external texture and use textureSampleBaseClampToEdge, as on the web:

@group(0) @binding(0) var srcTex: texture_external;
@group(0) @binding(1) var srcSampler: sampler;

@fragment
fn fs_main(@location(0) uv: vec2f) -> @location(0) vec4f {
  return textureSampleBaseClampToEdge(srcTex, srcSampler, uv);
}

Apply cover-fit scaling in uniforms so a landscape sensor fills a portrait canvas without stretching.

5. Color conversion

The sample is RGB on both platforms: the external texture converts the camera's YUV while sampling, with the color space of the frame. On iOS the pixel buffer's attachments declare it (BT.709 limited range when they are missing). On Android a camera buffer carries no color metadata, so the conversion assumes what the platform reports for the buffer, BT.601 limited range by default, which is what Android cameras produce; sourceColorSpace overrides it. Pass colorSpace: "display-p3" to importExternalTexture to sample in Display P3 instead of sRGB.