/3dgs-mcp-renderer

name: 3dgs-mcp-renderer
description: 'MCP protocol integration with 3DGS rendering pipeline: Agent-controlled Three.js/WebGPU rendering,
  voice-driven scene reconstruction, real-time parameter manipulation, light tracing backend. Use when: MCP rendering,
  agent-controlled 3DGS, voice-driven reconstruction, real-time 3DGS editing, Three.js 3DGS, WebGPU Gaussian splatting,
  interactive rendering control, speech-to-3D, light tracing, HiGS accelerated rendering.'
license: Apache-2.0
metadata:
  version: 1.1.0
  author: jaccen
  tags:
  - mcp
  - 3dgs
  - gaussian-splatting
  - rendering
  - three.js
  - webgpu
  - voice
  - agent
  - interactive
  - spec-first
  - sculpting
  - code-first
  - slat
  - latent-editing
  disable-model-invocation: true
  user-invocable: true

> Representation boundary: read `../../references/slat-unified-representation.md` before claiming learned encoding or conversion capabilities. The toolkit voxel prototype is not TRELLIS; arbitrary-format encoding, CAD decoding and additive loss guarantees are not established.

3DGS MCP Renderer — Agent-3DGS Interaction via MCP Protocol

Prototype specification for integrating MCP (Model Context Protocol) with 3DGS rendering pipelines, enabling AI Agents to directly manipulate Three.js/3DGS rendering parameters and achieve voice-driven 3D scene reconstruction.

Architecture

┌─────────────┐     ┌─────────────┐     ┌──────────────────┐     ┌──────────────────┐
│ Voice/Text  │────▶│   Agent     │────▶│  MCP Server      │────▶│  3DGS Renderer   │
│ (Whisper/   │     │ (Claude/    │     │  (Node.js/       │     │  (Three.js/      │
│  Prompt)    │     │  TeleClaw)  │     │   Python)        │     │   WebGPU/HiGS/   │
│             │◀────│             │◀────│                  │◀────│   DDF-GS)        │
└─────────────┘     └─────────────┘     └──────────────────┘     └──────────────────┘
                        │                      │                       │
                        │  Tool calls          │  WebSocket/HTTP       │  WebGL/WebGPU/
                        │  (MCP protocol)       │  transport            │  HiGS/DDF-GS

Spec-First Sculpting Pipeline (v0.9.0)

> **Design inspiration**: img2threejs (GitHub: img2threejs/img2threejs) — open-source AI Skill that converts a single image into an interactive Three.js 3D model via a stage-gated sculpting pipeline. We borrow two core principles: (1) **spec-first** — define quality criteria and component hierarchy before any rendering; (2) **stage-gated sculpting** — progressive refinement with acceptance checks at each stage.

Why Spec-First for MCP Rendering?

The original MCP pipeline was **reactive**: user issues a voice command → agent maps to a tool → render → verify. This works for single-step edits but fails for complex scene construction because:

• No upfront quality criteria → agent cannot self-assess before rendering
• No stage gates → errors compound across steps (bad camera → bad selection → bad edit)
• No component hierarchy → edits are flat, no part-level control

**The fix**: Introduce a `define_scene_spec` tool that runs *before* any sculpting/editing tools. This produces a machine-readable Object Spec that subsequent tools reference as acceptance criteria.

The 6-Stage Sculpting Pipeline

┌─────────────────────────────────────────────────────────────┐
│                    SPEC-FIRST SCULPTING                      │
│                                                             │
│  ┌──────────────┐                                           │
│  │ define_scene │  ← Object Spec: component hierarchy,      │
│  │ _spec        │    material system, quality criteria      │
│  └──────┬───────┘                                           │
│         │                                                   │
│         ▼                                                   │
│  Stage 1: blockout    → Bounding boxes, camera framing     │
│         │  gate: bbox coverage ≥ spec.target_coverage?      │
│         ▼                                                   │
│  Stage 2: structural  → Part decomposition, hierarchy      │
│         │  gate: part count & nesting matches spec?         │
│         ▼                                                   │
│  Stage 3: form        → Gaussian density/scale/rotation     │
│         │  gate: PSNR estimate ≥ spec.min_psnr?             │
│         ▼                                                   │
│  Stage 4: material    → PBR/SH assignment per part         │
│         │  gate: material count per part matches spec?      │
│         ▼                                                   │
│  Stage 5: surface     → Normal consistency, thin structures │
│         │  gate: normal consistency score ≥ spec.threshold? │
│         ▼                                                   │
│  Stage 6: lighting    → Environment, shadows, AO           │
│            gate: render quality score ≥ spec.target_score?  │
└─────────────────────────────────────────────────────────────┘

Each stage is an MCP tool call. The agent renders a frame after each stage, evaluates against the gate, and either advances or retries. This mirrors img2threejs's `blockout → structural → form → material → surface → lighting` flow.

Gate Evaluation Protocol

For each stage gate, the agent follows this protocol:

1. Execute stage tool (e.g., sculpt_form with parameters)
2. Call render_frame() to get current visual state
3. Call query_scene(query_type="stats") to get quantitative metrics
4. Compare metrics against spec gate criteria
5. If pass → advance to next stage
6. If fail → adjust parameters and retry (max 3 attempts)
7. If 3 failures → report to user with diagnostic info

Voice-Driven Sculpting Example

> **Loaded on demand** — See [mcp-tools-spec.md](references/mcp-tools-spec.md) for the full voice-driven sculpting example (desk scene with 8-step agent pipeline).

Code-First Rendering Philosophy (v0.9.0)

> **Design i