React Native WebGPU

Vision Camera

Real-time GPU effects on live camera frames.

Run WGSL effects on live camera frames by importing each Vision Camera Frame as a GPUExternalTexture, sampling it in a shader, and presenting from the frame-output worklet.

Vision Camera owns the camera pipeline and exposes each captured image as a Frame hybrid object: a short-lived handle to GPU-shareable pixel data. React Native WebGPU sits on the consumer side: you take the frame's NativeBuffer, wrap its pointer into React Native WebGPU's own NativeBuffer (which holds a reference of its own), and sample it with importExternalTexture.

Vision Camera concepts

Before writing WGSL, understand what the Frame object gives you (full API reference):

ConceptWhy it matters for WebGPU
hasNativeBufferThe zero-copy GPU path, with no CPU pixel download
getNativeBuffer()Returns a NativeBuffer with a native pointer (IOSurface / AHardwareBuffer)
orientationSensor rotation relative to your output: map it to importExternalTexture({ rotation })
isMirroredFront-camera horizontal flip: map it to importExternalTexture({ mirrored })
pixelFormat: "native"Request NV12-style surfaces via useFrameOutput({ pixelFormat: "native" })
dispose()Releases the frame back to the camera pool: call it once the GPU work is submitted

Vision Camera documents the WebGPU import path directly on Frame.getNativeBuffer() examples:

if (.) {
  const  = .();
  const  = RNWebGPU.createNativeBufferFromPointer(.);
  const  = .({
    : ,
    : "camera-frame",
  });
  // After submitting commands that sample externalTexture:
  .destroy();
  .release();
  .();
}

Frame processors run on a worklet runtime, so the wrapper is called through the RNWebGPU global, which the worklet captures like device; the createNativeBufferFromPointer export works on the JS thread only.

React Native WebGPU extends importExternalTexture with rotation and mirrored options so the external texture uprights the sensor image before your shader runs; see Native Extensions.

1. Request a device

Camera frames need Dawn's native frame import, which requestDevice() enables by default whenever the adapter supports it. Nothing has to be listed:

const  = await ..();
const  = await !.();

A few Android devices cannot import camera frames. device.features.has("opaque-ycbcr-android-for-external-texture") tells before the first frame; otherwise the first importExternalTexture() throws an error naming it.

2. Canvas + frame output

Use useFrameOutput (Vision Camera v5) with pixelFormat: "native" so each Frame carries a GPU-native buffer:

import { Canvas, useCanvasRef } from "react-native-webgpu";
import { useFrameOutput } from "react-native-vision-camera";

const ref = useCanvasRef();
// Build pipeline, sampler, uniform buffers once on main thread…

const frameOutput = useFrameOutput({
  pixelFormat: "native",
  onFrame: (frame) => {
    "worklet";
    if (!frame.hasNativeBuffer) {
      frame.dispose();
      return;
    }
    renderCameraFrame(frame, device, context, pipeline);
  },
});

3. Per-frame import and render

Each frame is a new external texture.

function (
  : Frame,
  : GPUDevice,
  : ,
  : GPURenderPipeline,
) {
  "worklet";

  const  = .();
  try {
    const  = .(
      .,
    );

    let : 0 | 90 | 180 | 270 = 0;
    if (. === "right")  = 90;
    else if (. === "down")  = 180;
    else if (. === "left")  = 270;

    const  = .({
      : ,
      ,
      : .,
    });

    // encode render pass sampling texture_external …
    ..([.()]);
    .();
    .();
    .();
  } finally {
    .();
    .();
  }
}

Order matters: submit → present → destroy → release. Skipping release() or frame.dispose() stalls the camera buffer pool.

Importing each frame is the simplest path. The camera cycles through the same few buffers, though, so a render loop that runs for long can import each buffer once instead, keyed by its handle, and only open and close an access per frame; see Importing a buffer once.

4. WGSL: sample the camera

Bind the external texture and use textureSampleBaseClampToEdge, as on the web:

@group(0) @binding(0) var srcTex: texture_external;
@group(0) @binding(1) var srcSampler: sampler;

@fragment
fn fs_main(@location(0) uv: vec2f) -> @location(0) vec4f {
  return textureSampleBaseClampToEdge(srcTex, srcSampler, uv);
}

Apply cover-fit scaling in uniforms so a landscape sensor fills a portrait canvas without stretching.

5. Color conversion

The sample is RGB on both platforms: the external texture converts the camera's YUV while sampling, with the color space of the frame. On iOS the pixel buffer's attachments declare it (BT.709 limited range when they are missing). On Android a camera buffer carries no color metadata, so the conversion assumes what the platform reports for the buffer, BT.601 limited range by default, which is what Android cameras produce; sourceColorSpace overrides it. Pass colorSpace: "display-p3" to importExternalTexture to sample in Display P3 instead of sRGB.

Further reading