Design doc and reference implementation sketch for the reference layer

Design for a poseable mannequin plus glTF/GLB import overlay in Umamo:
world axes, orbit camera, draw path, UMA persistence, UI, risks.

docs/seams.md lists the RenderDevice additions the feature needs:
PrimitiveTopology, DepthState, depthAttachment, WorldToClip,
ReferenceMeshSpec.

sketch/ mirrors the algorithms in Kotlin. verify/verify.py proves them:
30 checks, all passing. Two findings: accessor byteOffset must be a
multiple of the component size, and the screen-right basis collapses to a
zero vector at pitch +/-90 degrees, so pitch is clamped to an open
interval.
This commit is contained in:
litruv
2026-11-03 18:14:07 +11:00
parent e95d7795ae
commit 4eb583b5f3
10 changed files with 1913 additions and 2 deletions
+89
View File
@@ -0,0 +1,89 @@
# PR stack
Six PRs, each reviewable and mergeable on its own, following the stack convention in
`umamo/README.md` § Contributors. Each one has a reason to exist before the next one uses it.
## PR 1 — `:format`: glTF/GLB reader
Read-only. Container rules, buffer views, accessors, scene graph, skins. No rendering, no UI.
Why first: the container rules are the part most likely to be wrong, and they are testable
without a GPU or an asset. Alignment rejection, chunk padding, truncation, and the
more-than-one-chunk case are all pure.
Depends on nothing. `kotlinx-serialization-json` is already a `:format` dependency.
Reuse `UmaManifest.kt`'s JSON depth guard (`UMA_MAXIMUM_JSON_DEPTH`) for the glTF JSON chunk.
An accessor that reads past its buffer view's `byteLength` is an error, not a zero fill.
Tests: `GlbContainerTest`, `GltfAccessorTest`. Corpus gate under `test/corpus/gltf/`, skipped
when empty, like every other corpus test.
## PR 2 — `:render`: seam additions
`PrimitiveTopology`, `DepthState`, `depthAttachment`, `WorldToClip`, `ReferenceMeshSpec`, the two
draw calls, and the GL implementation. Nothing uses them yet.
Why it must be its own PR: it touches every backend, including the GLES and Metal stubs, which
have to answer for the new `PipelinePurpose` values explicitly. A seam change that arrives
attached to a feature cannot be bisected when a backend breaks.
Defaults on `RenderPipelineSpec` keep every existing call site compiling unchanged, so this PR
has no behavioural diff for the puppet path. That is the test that it is honest.
Tests: `ReferenceDepthGateTest`, GL-gated, shaped like `WorldAxisLinesTest.kt` — a flat-shaded
triangle at a known camera must rasterize the expected pixels, and the depth test must reject a
triangle behind it. Skips via a JUnit assumption in a display-less environment.
## PR 3 — `:runtime` + `:edit`: the reference layer model and the pose
`ReferenceLayer`, the joint set with per-joint clamps, forward kinematics, CPU skinning. Pure
data and pure functions, no GL, no UI.
Why before the renderer: the pose is the part a test can pin. Bone length invariance, joint
origin invariance, orthonormality, and a weighted blend between two poses are all facts about
the model, not about the GPU.
Tests: `MannequinFkTest`, `ReferenceLayerTest`, `SkinningTest`.
## PR 4 — `:render`: the reference renderer
Consumes PR 1's scene and PR 3's pose. Builds the camera matrix, uploads the mesh, draws the
flat fill and the wireframe, resolves the pose on pose-change frames only.
Draw order `Above` and `Below` only. Depth-sorted occlusion against the puppet is not in this
PR: it needs the puppet to write depth from drawable Z, and the puppet path has no depth today.
Tests: `ReferenceDrawPlanTest` (pure), `ReferenceRenderGateTest` (GL-gated).
## PR 5 — `:ui`: the overlay
Overlays toggle in the viewport header, the Properties section, the commands through
`CommandRegistry`, keymap entries, `strings.xml` entries.
Why last of the functional ones: it is the only PR that changes what a user sees, and it depends
on everything above existing.
Commands: `reference.import`, `reference.reset-pose`, `reference.fit`,
`reference.toggle-visibility`, `reference.link-camera`, `reference.clear`.
Tests: `CommandTableOrderTest` and `CommandRoutingTest` cases for the new commands, plus the
area-scoped routing — the reference commands belong to the viewport area, not the UV editor.
## PR 6 — `:format` + `:ui`: UMA persistence
The `reference` entry, the bridge type, the writer and loader wiring.
Why last: persistence is only worth designing once the shape of the object is settled by PR 3,
and the entry must be additive and ignorable — a reader that does not know `reference` skips it
and the document still opens.
Tests: `UmaEntrySchemaTest` case, `UmaReferenceRoundTripTest`, and the export-contamination
gate: a document with a reference layer present must still produce a byte-identical CMO3 for
the puppet. That gate is the one that catches the layer leaking into the format.
## Out of order on purpose
PR 2 could be folded into PR 4 and PR 6 into PR 3. They are separate because a seam change and a
persistence change are the two things most likely to break something that does not use them, and
a stack that can be bisected is the point of the convention.
+143
View File
@@ -0,0 +1,143 @@
# Seam changes the reference layer forces
These are the parts of `module/render/src/commonMain/.../device/` that cannot support a 3D
reference layer as they stand. Each is shown as it is today, then as proposed.
## 1. `PrimitiveTopology`
Today the seam assumes triangles everywhere. `PipelinePurpose.WorldAxisLine` is already a line
draw ("attribute-less; two endpoints from the vertex index"), so lines exist in spirit but are
hardcoded into one purpose. A wireframe overlay needs lines as a general topology, which means
the pipeline spec has to say it.
```kotlin
public enum class PrimitiveTopology {
/** Triangles. Every existing pipeline. */
Triangles,
/** Line pairs from an index buffer. The wireframe overlay and the world axis lines. */
Lines,
}
```
Wireframe by barycentric attribute was considered and rejected: it needs a per-vertex attribute
and a shader branch, and it cannot draw a silhouette edge shared by one triangle. GL_LINES from
an edge index buffer built at upload is simpler and matches the axis-line path that already
exists.
## 2. `DepthState`
No pipeline can depth-test today, and no target can hold a depth attachment. Both are needed.
```kotlin
public enum class DepthFunc {
/** Standard: nearer wins. */
Less,
/** Always pass. Used when the layer draws over everything with no depth buffer. */
Always,
}
public data class DepthState(
val enabled: Boolean,
val func: DepthFunc = DepthFunc.Less,
)
```
`RenderPipelineSpec` becomes:
```kotlin
public data class RenderPipelineSpec(
val purpose: PipelinePurpose,
val blend: PipelineBlend,
val cullBackFaces: Boolean = false,
val topology: PrimitiveTopology = PrimitiveTopology.Triangles,
val depth: DepthState = DepthState.Disabled,
)
```
Both defaults keep every existing call site compiling unchanged. That is deliberate: the depth
work must not require touching the puppet path.
`RenderTargetSpec` gains `depthAttachment: Boolean = false`. Only the reference pass needs one.
Two rules that have to be stated, because they are not obvious from the API:
- `resolve` and `resolveUsed` copy color only. A depth attachment is never copied, so a pass
that reads a resolved target has no depth to read.
- The composite path samples screen-space textures and has no depth input. A depth-tested draw
must not be composited through a blend-mode layer; the blend would run against pixels the
depth test should have rejected. The reference layer draws directly.
## 3. `PipelinePurpose` additions
```kotlin
public enum class PipelinePurpose {
// ... existing six ...
/** Flat-shaded or vertex-normal-shaded reference triangles. */
ReferenceMesh,
/** Reference wireframe, line pairs. */
ReferenceWireframe,
}
```
The enum is closed on purpose — adding a value forces every backend to answer for it rather
than silently lacking one. That cost is real here: `GlesRenderDevice` and `MetalRenderDevice`
are stubs, so this feature is desktop-only until the GLES device exists. The stubs must answer
explicitly, not throw.
## 4. A 3D camera on the pass encoder
`WorldToNdc` is a 2D affine and cannot carry depth. The reference pass needs a full matrix.
```kotlin
public data class WorldToClip(val matrix: FloatArray) // 16 floats, row-major
public data class DepthRange(val near: Float, val far: Float)
```
```kotlin
fun setClipCamera(worldToClip: WorldToClip, depthRange: DepthRange, screenTexWidth: Int, screenTexHeight: Int)
```
`setCamera(worldToNdc, ...)` stays. `WorldToNdc` is its Z = 0 slice, and the two must agree
exactly at yaw = pitch = 0 — that agreement is verified in `verify/verify.py` and is what keeps
the reference layer aligned with the drawing plane when nothing is rotated.
## 5. A 3D mesh upload
`MeshSpec.restPositions` is interleaved x,y. A reference mesh needs three components and has no
UVs, no glue attributes, and no deform inputs.
```kotlin
public class ReferenceMeshSpec(
val positions: FloatArray, // interleaved x, y, z
val indices: IntArray,
val normals: FloatArray?, // interleaved x, y, z, or null for flat shading
val wireframeIndices: IntArray?, // triangle edges as line pairs, built at upload
)
```
Building `wireframeIndices` at upload rather than per frame is the point: the edge set is
static, and a per-frame rebuild would be the most expensive thing in the frame.
## 6. Draw calls
```kotlin
fun drawReferenceMesh(mesh: GpuMesh, fragment: FragmentUniforms)
fun drawReferenceWireframe(mesh: GpuMesh, fragment: FragmentUniforms)
```
`FragmentUniforms` already carries `colorRed/Green/Blue/Alpha` and `opacity`, which is all the
reference layer needs. It does not need `useTexture`, `useMask`, or the multiply/screen colors.
Reusing the struct is fine; the marshaller reads what it reads.
## What does not change
- The deform eval, pose resolution, glue, and render order are untouched.
- The read-back contract is untouched: RGBA8888, top row first. A depth-tested frame reads back
the same way, which is what makes the GL-gated test possible at all.
- `FrameEncoder.barrier` is untouched. The reference pass has no cross-pass dependency.
+204
View File
@@ -0,0 +1,204 @@
# 3D model reference layer
## What it is for
A drawing aid that sits in the viewport over the puppet. Five uses, in the order people
actually reach for them:
1. **Proportion check.** Is this head too big for this body. Umamo has no body, so today the
answer comes from memory or a photo pinned next to the window.
2. **Anatomy reference.** Where the elbow actually is in a pose, where the shoulder joint sits,
how a ribcage turns.
3. **Foreshortening.** A hand coming toward the camera. A 2D reference cannot show this; a
3D one can, and this is the one use that justifies the feature existing at all.
4. **Scale reference.** "This puppet is 160 cm tall." Everything else follows from that.
5. **Line-art underlay.** Pose the mannequin, lower its opacity, draw over it.
It is not a 3D modeler, and the distinction matters for every decision below.
## Non-goals
- No 3D puppet. The puppet stays a 2D deforming mesh set.
- No glTF export. `TODO.md` already lists glTF export as a far-future item; it is a different
problem and shares nothing with this except the word glTF.
- No lighting, no material system, no texture sampling on the reference layer. Flat fill,
wireframe, or vertex-normal shading. Three options, no more.
- No animation playback on the reference layer. Pose is a static state.
- No physics, no cloth, no bendy bones. `TODO.md` § Future Feature Wishes already wants a real
skeleton for the puppet; that is a separate project and this one must not become its
accidental first installment.
- Nothing from the reference layer reaches CMO3 or MOC3 export. It is editor-only. See § Risks.
## World-space convention
Umamo's world space is X right, Y depth, Z up. `PuppetModel.worldOriginX` and `worldOriginZ`
already exist, and the viewport already draws the X and Z axis lines through the world origin
(`module/render/src/jvmTest/.../gl/WorldAxisLinesTest.kt`).
The reference layer uses that same frame. No new axis convention, no conversion. A mannequin
standing at the world origin is already aligned with the drawing, and at yaw = pitch = 0 the
reference camera must agree exactly with `ViewportCamera.worldToNdc`. That agreement is the
anchor of the whole design: the reference layer at rest sits exactly in the drawing plane, and
rotating it is the only thing that separates it from the puppet.
Verified in `verify/verify.py`: at yaw 0 pitch 0 the camera basis is forward +Y, right +X,
up +Z, and a point one world unit right projects to NDC x = 2·zoom/viewportWidth, which is
`ViewportCamera.worldToNdc`'s value.
Units are model pixels: one world unit is one model pixel. A mannequin therefore needs a
scale stated in human units, not pixels. `referenceHeightCm` (default 160) plus the document's
puppet height in world units gives the scale, and that is what makes the proportion check
possible at all — a head is about 1/7.5 of standing height, and the mannequin encodes that as
data rather than as a guess.
## Data model
```
ReferenceLayer
source BuiltInMannequin(proportions) | ImportedGltf(assetKey, sceneIndex)
transform position (x, y, z), scale, rotation (yaw, pitch, roll)
pose Map<JointId, Euler(x, y, z)> degrees
style fill, wireframe, shading (Flat | VertexNormal | None), opacity
visibility visible, occlusion (Above | Below)
camera linked to the viewport camera, or an independent orbit
```
`ImportedGltf` stores an asset key and a hash, never the mesh. A missing asset degrades to
"asset missing" and the layer keeps its pose and transform. That is the same shape the source-
art binding already uses in `:reimport`: the mapping survives the pixels going away.
The built-in mannequin is procedural: a joint hierarchy plus primitive volumes. No bundled
asset, so no licensing question, and it is deterministic — the same proportions always produce
the same triangles, which a test can pin.
Joint set, minimum viable:
```
pelvis, spine, chest, neck, head
shoulder_l, elbow_l, wrist_l
hip_l, knee_l, ankle_l
shoulder_r, elbow_r, wrist_r
hip_r, knee_r, ankle_r
```
17 joints. Each has an anatomical clamp, so a pose cannot be built that a body cannot hold, and
the clamp is data the UI can show as a slider range rather than a silent rejection.
## Camera
`ReferenceCamera(target, yaw, pitch, zoom, projection)`, pure, returning a new instance,
touching no GL — the same discipline as `ViewportCamera`, so the maths unit-tests without a
context.
Orthographic by default. Perspective is an option with a focal length, because foreshortening —
the use that justifies the feature — is a perspective phenomenon. Orthographic is the default
because it agrees with the 2D plane exactly, and a reference that disagrees with the drawing's
projection is wrong in a way the user cannot see.
Two clamps, both found by the harness rather than assumed:
- `pitch` must be clamped to the open interval (−89.9°, +89.9°). At ±90° the camera looks along
the world up axis and the screen-right basis vector becomes a zero vector. The projection
still returns a number, so the failure is silent: the model collapses to a line and nothing
reports it. The clamp has to be in the camera, not in the UI drag handler.
- `zoom` reuses `ViewportCamera.MIN_ZOOM` / `MAX_ZOOM`.
Camera linking: when linked, the reference camera's target and zoom follow the viewport camera,
and only yaw and pitch are its own. When unlinked it is fully independent. Linked is the
default, because the useful case is "rotate the reference without losing the drawing".
## Draw path
The seam has no depth concept today. `RenderPipelineSpec` is `(purpose, blend, cullBackFaces)`;
`RenderTargetSpec` has no depth attachment; `PipelinePurpose` is closed at six values, none of
which is a 3D mesh. `MeshSpec.restPositions` is interleaved x,y.
Four additions, in `docs/seams.md`. They are not optional and they are not local: adding a
`PipelinePurpose` value forces every backend to answer for it, which is the stated intent of
that enum being closed. So this feature cannot land in `:render` alone.
Draw order:
- `Above` — draw after the puppet, no depth test. Always correct, no depth buffer needed.
Default.
- `Below` — draw before the puppet, no depth test.
- `Depth-sorted` — the mannequin occludes the puppet properly. Deferred: it needs the puppet to
write depth from drawable Z, and the puppet path has no depth today.
The reference layer must stay outside the layer-composite path. The composite samples
screen-space textures and blends in-shader with blending disabled; it has no depth input, so a
depth-tested draw composited through it would blend against pixels the depth test should have
rejected. Keep it a direct draw.
## Pose
Pose is a map of joint id to Euler triple, applied as `T(offset) · R(euler)` per joint, world =
parent world · local. Verified invariants in the harness:
- bone length is pose-invariant,
- a joint's rotation does not move its own origin,
- the rotation block stays orthonormal at scale 1, which is what skinning depends on.
Pose presets reuse the storage shape already planned for `TODO.md` § Pose Palette/Library —
Cubism 5.4's "Model state set" is a pose library, and the same container holds a mannequin pose
with no format work.
Skinning for imported glTF is CPU-side: joint matrices times weights, summed. Verified: a
70/30 blend lands between the two poses. CPU first matches the codebase's existing shape
(CPU deform eval as the oracle, GPU as the fast path), and it gives a testable oracle before
any GPU work.
## Persistence
A new UMA entry, `reference`. It stores the source, transform, pose, style, visibility, and
camera link. It does not store geometry — geometry comes from the asset or from the procedural
generator, both reproducible.
The entry must be additive and ignorable: a reader that does not know `reference` skips it and
the document still opens. Mirror `UmaEntrySchemaTest.kt` for the gate.
## UI
- Viewport header → Overlays toggle, which is already planned (`TODO.md` § Overlays Toggle) and
currently lists General Information, Wireframe, Grid, 2D Cursor. Add Reference Layer there.
- Properties panel → a Reference section.
- Commands through `CommandRegistry`, not hardcoded in a widget: `reference.import`,
`reference.reset-pose`, `reference.fit`, `reference.toggle-visibility`,
`reference.link-camera`, `reference.clear`.
- Settings keys, dotted namespace: `view.referenceLayer.style`, `.opacity`, `.wireframe`,
`.defaultHeightCm`, `.maxTriangles`, `.linkCamera`.
## Tests
Pure, `commonTest`:
- GLB container rules: header, chunk types, padding, truncation, accessor alignment.
- FK invariants above.
- Camera basis orthonormality, agreement with `ViewportCamera` at yaw 0 pitch 0, pitch clamp.
GL-gated, `jvmTest`, shaped like `WorldAxisLinesTest.kt`: a flat-shaded triangle at a known
camera must rasterize the expected pixels, and the depth test must reject a triangle behind it.
Skips in a display-less environment via a JUnit assumption, so the run reports SKIPPED rather
than a green pass that asserted nothing.
Tolerance, not bit-exactness, for anything with trig. The codebase's determinism rule is real,
but FK and the camera cannot be bit-identical across targets; the tests must say so explicitly
rather than assert exact floats and fail on a different machine.
## Risks
1. **Export contamination.** The reference layer must never reach the CMO3 writer. The
byte-identical round-trip gate is the thing that catches this, and it only catches it if the
gate is run with a reference layer present. Add that case.
2. **Closed enum cost.** `PipelinePurpose` is deliberately closed so a new pipeline forces every
backend to answer. The GLES and Metal devices are stubs, so this feature is desktop-only
until the GLES device exists (`TODO.md` § Deferred). That is fine, but the stubs must answer
explicitly rather than throw.
3. **Silent degeneracy.** The pitch collapse is the clearest example: a projection that returns
a number instead of failing. Any clamp added here belongs in the pure model, not in the
input handler.
4. **Memory.** An imported glTF can be 100k triangles. On a tablet that is not a reference
layer, it is a stall. Cap at import, report the cap as a notice, do not silently drop
primitives.
5. **Scope creep.** The feature's natural next step is a real 3D viewport, which is a different
project. The non-goals section exists to keep this one from becoming its first installment.