/hz-unity-face-tracking

name: hz-unity-face-tracking
license: Apache-2.0
description: Drive ARKit-blendshape-rigged head/face models in Unity with the wearer's facial expressions on Meta VR via Meta Movement SDK (face tracking + A2E). Use when a user has an FBX with the 52 ARKit blendshapes (any prefix, _L/_R suffixes) and wants it to animate from face tracking on Quest Pro / Quest 3 / Quest 3S.

Unity Face Tracking for ARKit-Rigged Models (Meta Movement SDK)

End-to-end recipe to make a head/face model rigged with the standard 52 ARKit blendshapes animate from the wearer's face on Meta VR. Uses the **public** `OVRCustomFace` extension hook — no `OVR_INTERNAL_CODE`, ships to 3P.

The model's blendshape names must follow the ARKit naming convention (camelCase, `_L`/`_R` suffixes, e.g. `eyeBlink_L`, `jawOpen`, `mouthSmile_R`). An optional prefix like `blendShape2.eyeBlink_L` is automatically stripped.

When to use

• User has an FBX/GLB/mesh with ARKit-named blendshapes and wants it driven by Meta VR face tracking.
• User asks: "animate this head with my face", "drive these blendshapes from face tracking", "use Movement SDK A2E with my model", "wire ARKit shapes to Meta VR".
• Target device: Quest Pro, Quest 3, Quest 3S (Quest 2 is no-op — no face cameras).

Prerequisites checklist

1. **Quest face tracking-capable headset** (Pro / 3 / 3S). 2. **Packages** in `Packages/manifest.json`:

• `com.meta.xr.sdk.core` (Meta XR Core — provides `OVRFaceExpressions`, `OVRCustomFace`)
• `com.meta.xr.sdk.movement` (Meta Movement SDK — A2E + retargeting helpers)

3. **`Assets/Oculus/OculusProjectConfig.asset`** (verify via Project Settings → Meta XR):

• `faceTrackingSupport: 1` (Supported) or `2` (Required)
• `eyeTrackingSupport: 1` if the rig has gaze

4. **Android manifest** (`Assets/Plugins/Android/AndroidManifest.xml`) permissions:

• `<uses-feature android:name="oculus.software.face_tracking" android:required="false" />`
• `<uses-permission android:name="com.oculus.permission.FACE_TRACKING" />`
• `<uses-permission android:name="android.permission.RECORD_AUDIO" />` (required for A2E)
• For eye gaze: `oculus.software.eye_tracking` + `com.oculus.permission.EYE_TRACKING`

5. **OVRCameraRig** in the scene with `OVRManager.FaceTrackingDataSources` including `Audio` (A2E) — if you skip Audio, mouth motion is visual-only.

After any change to OculusProjectConfig, regenerate the manifest (see [Regenerating the manifest](#regenerating-the-manifest) below).

Approach (high level)

1. Drop the **`ARKitOVRCustomFace`** script (in `references/ARKitOVRCustomFace.cs`) into the project. 2. Add an `OVRFaceExpressions` component to the OVRCameraRig (or anywhere in the scene). 3. On the GameObject that has the model's `SkinnedMeshRenderer`, add `ARKitOVRCustomFace`. Adding the component triggers `Reset()`, which auto-populates the blendshape→FaceExpression mapping by scanning the mesh's blendshape names. 4. Wire the component's `FaceExpressions` field to the `OVRFaceExpressions` instance. 5. For eye gaze: drop **`ARKitEyeGazeBlendshapeDriver`** in (see [Eye tracking](#eye-tracking-do-this--face-expressions-alone-dont-work) below). 6. Press Play with Meta XR Link, or build APK and side-load. On first launch, accept the Face Tracking / Eye Tracking / Microphone permission prompts.

That's the whole flow. Details below.

Eye tracking (DO THIS — face expressions alone don't work)

**Don't rely on `OVRFaceExpressions.EyesLook*` for eye gaze.** Those fields are derived from face-camera visuals, not the dedicated eye tracker. On Quest Pro they're often zero or noisy even when face tracking is otherwise working. The right API is `OVREyeGaze`, which taps the eye tracker directly.

For ARKit rigs with `eyeLook*` blendshapes (no eye bones), this skill ships **`ARKitEyeGazeBlendshapeDriver`**:

• Reads gaze rotation from two `OVREyeGaze` components (one per eye, `TrackingMode = HeadSpace`).
• Computes rotation relative to a head reference (e.g. `CenterEyeAnchor`).
• Decomposes pitch → `eyeLookUp/Down_{L,R}`, yaw → `eyeLookIn/Out_{L,R}` (with the ARKit "_In = toward nose" convention).
• Writes weights in `LateUpdate`, so it overrides whatever `ARKitOVRCustomFace` wrote in `Update`.

For rigs with **eye bones** (no `eyeLook*` blendshapes), skip the blendshape decomposition and just parent `OVREyeGaze` to each eye bone with `ApplyRotation = true` — the component will rotate the bone directly.

Eye gaze setup (blendshape rigs)

1. Create two empty GameObjects under `CenterEyeAnchor` (or your head transform): `LeftEyeGaze`, `RightEyeGaze`. 2. Add `OVREyeGaze` to each. Set `Eye = Left` / `Right`, `TrackingMode = HeadSpace`, `ApplyPosition = false`, `ApplyRotation = true`, `ConfidenceThreshold = 0.5`. 3. Add `ARKitEyeGazeBlendshapeDriver` to the head's SkinnedMeshRenderer GameObject (alongside `ARKitOVRCustomFace`). Wire `leftEye`, `rightEye`, and `referenceFrame` (= `CenterEyeAnchor`). 4. Tweak `maxAngleDeg` (default 30°) and `smoothing` (default 0.4) to taste.

Step-by-step

1. Install the script

Copy `references/ARKitOVRCustomFace.cs` into `Assets/Scripts/ARKitOVRCustomFace.cs`. It defines the public, 3P-shippable ARKit ↔ OVR FaceExpression table and a `MapBlendshapes()` method that scans `SkinnedMeshRenderer.sharedMesh.GetBlendShapeName(i)`, strips any prefix before the last `.`, lowercases, and matches against the table. Unmatched mesh blendshapes are set to `OVRFaceExpressions.FaceExpression.Max` (sentinel — skipped at runtime).

2. Enable Movement SDK + permissions

Project settings:

Project Settings → Meta XR → Face Tracking Support = Supported
Project Settings → Meta XR → Eye Tracking Support = Supported (if needed)

Then regenerate the manifest.

Regenerating the manifest

Against a live Editor, call the SDK's generator through `run_script` — the type resolves directly, so no reflection is needed:

// AgentScripts/UpdateManifest.cs
public static class UpdateManifest
{