Vision Camera

Run WGSL effects on live camera frames by importing each Vision Camera Frame as a GPUExternalTexture, sampling it in a shader, and presenting from the frame-output worklet.

Vision Camera owns the camera pipeline and exposes each captured image as a Frame hybrid object: a short-lived handle to GPU-shareable pixel data. React Native WebGPU sits on the consumer side: you take the frame's NativeBuffer, wrap it for Dawn, and sample it with importExternalTexture.

Vision Camera concepts

Before writing WGSL, understand what the Frame object gives you (full API reference):

Concept	Why it matters for WebGPU
`hasNativeBuffer`	Zero-copy GPU path - skip CPU pixel downloads
`getNativeBuffer()`	Returns a `NativeBuffer` with a native `pointer` (IOSurface / AHardwareBuffer)
`orientation`	Sensor rotation relative to your output - map to `importExternalTexture({ rotation })`
`isMirrored`	Front-camera horizontal flip - map to `importExternalTexture({ mirrored })`
`pixelFormat: "native"`	Request NV12-style surfaces via `useFrameOutput({ pixelFormat: "native" })`
`dispose()`	Release the frame back to the camera pool - call after GPU work finishes

Vision Camera documents the WebGPU import path directly on Frame.getNativeBuffer() - Examples:

if (.) {
  const  = .();
  const  = .(.);
  const  = .({
    : ,
    : "camera-frame",
  });
  // After submitting commands that sample externalTexture:
  .();
  .();
  .();
}

React Native WebGPU extends importExternalTexture with rotation and mirrored options so Dawn uprights the sensor image before your shader runs - see Native Extensions.

1. Request the right device features

Camera frames need native surface import:

const  = await ..();
const : [] = [
  "rnwebgpu/native-texture" as ,
  "dawn-multi-planar-formats" as ,
];

// Android: opaque YCbCr path for camera AHardwareBuffer
if (. === "android") {
  .("opaque-ycbcr-android-for-external-texture" as );
}

const  = await !.({ :  });

2. Canvas + frame output

Use useFrameOutput (Vision Camera v4+) with pixelFormat: "native" so each Frame carries a GPU-native buffer:

import { Canvas, useCanvasRef } from "react-native-webgpu";
import { useFrameOutput } from "react-native-vision-camera";

const ref = useCanvasRef();
// Build pipeline, sampler, uniform buffers once on main thread…

const frameOutput = useFrameOutput({
  pixelFormat: "native",
  onFrame: (frame) => {
    "worklet";
    if (!frame.hasNativeBuffer) {
      frame.dispose();
      return;
    }
    renderCameraFrame(frame, device, context, pipeline);
  },
});

3. Per-frame import and render

Each frame is a new external texture.

function (
  : Frame,
  : GPUDevice,
  : ,
  : GPURenderPipeline,
) {
  "worklet";

  const  = .();
  try {
    const  = .(
      .,
    );

    let : 0 | 90 | 180 | 270 = 0;
    if (. === "right")  = 90;
    else if (. === "down")  = 180;
    else if (. === "left")  = 270;

    const  = .({
      : ,
      ,
      : .,
    });

    // encode render pass sampling texture_external …
    ..([.()]);
    .();
    .();
    .();
  } finally {
    .();
    .();
  }
}

Order matters: submit → present → destroy → release. Skipping release() or frame.dispose() stalls the camera buffer pool.

4. WGSL - sample the camera

Bind the external texture and use textureSampleBaseClampToEdge:

@group(0) @binding(0) var srcTex: texture_external;
@group(0) @binding(1) var srcSampler: sampler;

@fragment
fn fs_main(@location(0) uv: vec2f) -> @location(0) vec4f {
  return cameraDecode(
    textureSampleBaseClampToEdge(srcTex, srcSampler, cameraCoord(uv)),
  );
}

Apply cover-fit scaling in uniforms so a landscape sensor fills a portrait canvas without stretching.

5. Android YUV decode

On iOS, NV12 → RGB happens in hardware. On Android, Dawn returns raw [Y, Cb, Cr] - decode in WGSL:

fn cameraDecode(c: vec4f) -> vec4f {
  let y  = c.r - 0.0627451;
  let cb = c.g - 0.5;
  let cr = c.b - 0.5;
  let r = 1.164384 * y + 1.792741 * cr;
  let g = 1.164384 * y - 0.213249 * cb - 0.532909 * cr;
  let b = 1.164384 * y + 2.112402 * cb;
  return vec4f(clamp(vec3f(r, g, b), vec3f(0.0), vec3f(1.0)), 1.0);
}

Also flip UVs on Android if the buffer origin differs from iOS (cameraCoord helper).