Design doc and reference implementation sketch for the reference layer
Design for a poseable mannequin plus glTF/GLB import overlay in Umamo: world axes, orbit camera, draw path, UMA persistence, UI, risks. docs/seams.md lists the RenderDevice additions the feature needs: PrimitiveTopology, DepthState, depthAttachment, WorldToClip, ReferenceMeshSpec. sketch/ mirrors the algorithms in Kotlin. verify/verify.py proves them: 30 checks, all passing. Two findings: accessor byteOffset must be a multiple of the component size, and the screen-right basis collapses to a zero vector at pitch +/-90 degrees, so pitch is clamped to an open interval.
This commit is contained in:
@@ -0,0 +1,89 @@
|
||||
# PR stack
|
||||
|
||||
Six PRs, each reviewable and mergeable on its own, following the stack convention in
|
||||
`umamo/README.md` § Contributors. Each one has a reason to exist before the next one uses it.
|
||||
|
||||
## PR 1 — `:format`: glTF/GLB reader
|
||||
|
||||
Read-only. Container rules, buffer views, accessors, scene graph, skins. No rendering, no UI.
|
||||
|
||||
Why first: the container rules are the part most likely to be wrong, and they are testable
|
||||
without a GPU or an asset. Alignment rejection, chunk padding, truncation, and the
|
||||
more-than-one-chunk case are all pure.
|
||||
|
||||
Depends on nothing. `kotlinx-serialization-json` is already a `:format` dependency.
|
||||
|
||||
Reuse `UmaManifest.kt`'s JSON depth guard (`UMA_MAXIMUM_JSON_DEPTH`) for the glTF JSON chunk.
|
||||
An accessor that reads past its buffer view's `byteLength` is an error, not a zero fill.
|
||||
|
||||
Tests: `GlbContainerTest`, `GltfAccessorTest`. Corpus gate under `test/corpus/gltf/`, skipped
|
||||
when empty, like every other corpus test.
|
||||
|
||||
## PR 2 — `:render`: seam additions
|
||||
|
||||
`PrimitiveTopology`, `DepthState`, `depthAttachment`, `WorldToClip`, `ReferenceMeshSpec`, the two
|
||||
draw calls, and the GL implementation. Nothing uses them yet.
|
||||
|
||||
Why it must be its own PR: it touches every backend, including the GLES and Metal stubs, which
|
||||
have to answer for the new `PipelinePurpose` values explicitly. A seam change that arrives
|
||||
attached to a feature cannot be bisected when a backend breaks.
|
||||
|
||||
Defaults on `RenderPipelineSpec` keep every existing call site compiling unchanged, so this PR
|
||||
has no behavioural diff for the puppet path. That is the test that it is honest.
|
||||
|
||||
Tests: `ReferenceDepthGateTest`, GL-gated, shaped like `WorldAxisLinesTest.kt` — a flat-shaded
|
||||
triangle at a known camera must rasterize the expected pixels, and the depth test must reject a
|
||||
triangle behind it. Skips via a JUnit assumption in a display-less environment.
|
||||
|
||||
## PR 3 — `:runtime` + `:edit`: the reference layer model and the pose
|
||||
|
||||
`ReferenceLayer`, the joint set with per-joint clamps, forward kinematics, CPU skinning. Pure
|
||||
data and pure functions, no GL, no UI.
|
||||
|
||||
Why before the renderer: the pose is the part a test can pin. Bone length invariance, joint
|
||||
origin invariance, orthonormality, and a weighted blend between two poses are all facts about
|
||||
the model, not about the GPU.
|
||||
|
||||
Tests: `MannequinFkTest`, `ReferenceLayerTest`, `SkinningTest`.
|
||||
|
||||
## PR 4 — `:render`: the reference renderer
|
||||
|
||||
Consumes PR 1's scene and PR 3's pose. Builds the camera matrix, uploads the mesh, draws the
|
||||
flat fill and the wireframe, resolves the pose on pose-change frames only.
|
||||
|
||||
Draw order `Above` and `Below` only. Depth-sorted occlusion against the puppet is not in this
|
||||
PR: it needs the puppet to write depth from drawable Z, and the puppet path has no depth today.
|
||||
|
||||
Tests: `ReferenceDrawPlanTest` (pure), `ReferenceRenderGateTest` (GL-gated).
|
||||
|
||||
## PR 5 — `:ui`: the overlay
|
||||
|
||||
Overlays toggle in the viewport header, the Properties section, the commands through
|
||||
`CommandRegistry`, keymap entries, `strings.xml` entries.
|
||||
|
||||
Why last of the functional ones: it is the only PR that changes what a user sees, and it depends
|
||||
on everything above existing.
|
||||
|
||||
Commands: `reference.import`, `reference.reset-pose`, `reference.fit`,
|
||||
`reference.toggle-visibility`, `reference.link-camera`, `reference.clear`.
|
||||
|
||||
Tests: `CommandTableOrderTest` and `CommandRoutingTest` cases for the new commands, plus the
|
||||
area-scoped routing — the reference commands belong to the viewport area, not the UV editor.
|
||||
|
||||
## PR 6 — `:format` + `:ui`: UMA persistence
|
||||
|
||||
The `reference` entry, the bridge type, the writer and loader wiring.
|
||||
|
||||
Why last: persistence is only worth designing once the shape of the object is settled by PR 3,
|
||||
and the entry must be additive and ignorable — a reader that does not know `reference` skips it
|
||||
and the document still opens.
|
||||
|
||||
Tests: `UmaEntrySchemaTest` case, `UmaReferenceRoundTripTest`, and the export-contamination
|
||||
gate: a document with a reference layer present must still produce a byte-identical CMO3 for
|
||||
the puppet. That gate is the one that catches the layer leaking into the format.
|
||||
|
||||
## Out of order on purpose
|
||||
|
||||
PR 2 could be folded into PR 4 and PR 6 into PR 3. They are separate because a seam change and a
|
||||
persistence change are the two things most likely to break something that does not use them, and
|
||||
a stack that can be bisected is the point of the convention.
|
||||
+143
@@ -0,0 +1,143 @@
|
||||
# Seam changes the reference layer forces
|
||||
|
||||
These are the parts of `module/render/src/commonMain/.../device/` that cannot support a 3D
|
||||
reference layer as they stand. Each is shown as it is today, then as proposed.
|
||||
|
||||
## 1. `PrimitiveTopology`
|
||||
|
||||
Today the seam assumes triangles everywhere. `PipelinePurpose.WorldAxisLine` is already a line
|
||||
draw ("attribute-less; two endpoints from the vertex index"), so lines exist in spirit but are
|
||||
hardcoded into one purpose. A wireframe overlay needs lines as a general topology, which means
|
||||
the pipeline spec has to say it.
|
||||
|
||||
```kotlin
|
||||
public enum class PrimitiveTopology {
|
||||
/** Triangles. Every existing pipeline. */
|
||||
Triangles,
|
||||
|
||||
/** Line pairs from an index buffer. The wireframe overlay and the world axis lines. */
|
||||
Lines,
|
||||
}
|
||||
```
|
||||
|
||||
Wireframe by barycentric attribute was considered and rejected: it needs a per-vertex attribute
|
||||
and a shader branch, and it cannot draw a silhouette edge shared by one triangle. GL_LINES from
|
||||
an edge index buffer built at upload is simpler and matches the axis-line path that already
|
||||
exists.
|
||||
|
||||
## 2. `DepthState`
|
||||
|
||||
No pipeline can depth-test today, and no target can hold a depth attachment. Both are needed.
|
||||
|
||||
```kotlin
|
||||
public enum class DepthFunc {
|
||||
/** Standard: nearer wins. */
|
||||
Less,
|
||||
|
||||
/** Always pass. Used when the layer draws over everything with no depth buffer. */
|
||||
Always,
|
||||
}
|
||||
|
||||
public data class DepthState(
|
||||
val enabled: Boolean,
|
||||
val func: DepthFunc = DepthFunc.Less,
|
||||
)
|
||||
```
|
||||
|
||||
`RenderPipelineSpec` becomes:
|
||||
|
||||
```kotlin
|
||||
public data class RenderPipelineSpec(
|
||||
val purpose: PipelinePurpose,
|
||||
val blend: PipelineBlend,
|
||||
val cullBackFaces: Boolean = false,
|
||||
val topology: PrimitiveTopology = PrimitiveTopology.Triangles,
|
||||
val depth: DepthState = DepthState.Disabled,
|
||||
)
|
||||
```
|
||||
|
||||
Both defaults keep every existing call site compiling unchanged. That is deliberate: the depth
|
||||
work must not require touching the puppet path.
|
||||
|
||||
`RenderTargetSpec` gains `depthAttachment: Boolean = false`. Only the reference pass needs one.
|
||||
|
||||
Two rules that have to be stated, because they are not obvious from the API:
|
||||
|
||||
- `resolve` and `resolveUsed` copy color only. A depth attachment is never copied, so a pass
|
||||
that reads a resolved target has no depth to read.
|
||||
- The composite path samples screen-space textures and has no depth input. A depth-tested draw
|
||||
must not be composited through a blend-mode layer; the blend would run against pixels the
|
||||
depth test should have rejected. The reference layer draws directly.
|
||||
|
||||
## 3. `PipelinePurpose` additions
|
||||
|
||||
```kotlin
|
||||
public enum class PipelinePurpose {
|
||||
// ... existing six ...
|
||||
|
||||
/** Flat-shaded or vertex-normal-shaded reference triangles. */
|
||||
ReferenceMesh,
|
||||
|
||||
/** Reference wireframe, line pairs. */
|
||||
ReferenceWireframe,
|
||||
}
|
||||
```
|
||||
|
||||
The enum is closed on purpose — adding a value forces every backend to answer for it rather
|
||||
than silently lacking one. That cost is real here: `GlesRenderDevice` and `MetalRenderDevice`
|
||||
are stubs, so this feature is desktop-only until the GLES device exists. The stubs must answer
|
||||
explicitly, not throw.
|
||||
|
||||
## 4. A 3D camera on the pass encoder
|
||||
|
||||
`WorldToNdc` is a 2D affine and cannot carry depth. The reference pass needs a full matrix.
|
||||
|
||||
```kotlin
|
||||
public data class WorldToClip(val matrix: FloatArray) // 16 floats, row-major
|
||||
|
||||
public data class DepthRange(val near: Float, val far: Float)
|
||||
```
|
||||
|
||||
```kotlin
|
||||
fun setClipCamera(worldToClip: WorldToClip, depthRange: DepthRange, screenTexWidth: Int, screenTexHeight: Int)
|
||||
```
|
||||
|
||||
`setCamera(worldToNdc, ...)` stays. `WorldToNdc` is its Z = 0 slice, and the two must agree
|
||||
exactly at yaw = pitch = 0 — that agreement is verified in `verify/verify.py` and is what keeps
|
||||
the reference layer aligned with the drawing plane when nothing is rotated.
|
||||
|
||||
## 5. A 3D mesh upload
|
||||
|
||||
`MeshSpec.restPositions` is interleaved x,y. A reference mesh needs three components and has no
|
||||
UVs, no glue attributes, and no deform inputs.
|
||||
|
||||
```kotlin
|
||||
public class ReferenceMeshSpec(
|
||||
val positions: FloatArray, // interleaved x, y, z
|
||||
val indices: IntArray,
|
||||
val normals: FloatArray?, // interleaved x, y, z, or null for flat shading
|
||||
val wireframeIndices: IntArray?, // triangle edges as line pairs, built at upload
|
||||
)
|
||||
```
|
||||
|
||||
Building `wireframeIndices` at upload rather than per frame is the point: the edge set is
|
||||
static, and a per-frame rebuild would be the most expensive thing in the frame.
|
||||
|
||||
## 6. Draw calls
|
||||
|
||||
```kotlin
|
||||
fun drawReferenceMesh(mesh: GpuMesh, fragment: FragmentUniforms)
|
||||
|
||||
fun drawReferenceWireframe(mesh: GpuMesh, fragment: FragmentUniforms)
|
||||
```
|
||||
|
||||
`FragmentUniforms` already carries `colorRed/Green/Blue/Alpha` and `opacity`, which is all the
|
||||
reference layer needs. It does not need `useTexture`, `useMask`, or the multiply/screen colors.
|
||||
Reusing the struct is fine; the marshaller reads what it reads.
|
||||
|
||||
## What does not change
|
||||
|
||||
- The deform eval, pose resolution, glue, and render order are untouched.
|
||||
- The read-back contract is untouched: RGBA8888, top row first. A depth-tested frame reads back
|
||||
the same way, which is what makes the GL-gated test possible at all.
|
||||
- `FrameEncoder.barrier` is untouched. The reference pass has no cross-pass dependency.
|
||||
+204
@@ -0,0 +1,204 @@
|
||||
# 3D model reference layer
|
||||
|
||||
## What it is for
|
||||
|
||||
A drawing aid that sits in the viewport over the puppet. Five uses, in the order people
|
||||
actually reach for them:
|
||||
|
||||
1. **Proportion check.** Is this head too big for this body. Umamo has no body, so today the
|
||||
answer comes from memory or a photo pinned next to the window.
|
||||
2. **Anatomy reference.** Where the elbow actually is in a pose, where the shoulder joint sits,
|
||||
how a ribcage turns.
|
||||
3. **Foreshortening.** A hand coming toward the camera. A 2D reference cannot show this; a
|
||||
3D one can, and this is the one use that justifies the feature existing at all.
|
||||
4. **Scale reference.** "This puppet is 160 cm tall." Everything else follows from that.
|
||||
5. **Line-art underlay.** Pose the mannequin, lower its opacity, draw over it.
|
||||
|
||||
It is not a 3D modeler, and the distinction matters for every decision below.
|
||||
|
||||
## Non-goals
|
||||
|
||||
- No 3D puppet. The puppet stays a 2D deforming mesh set.
|
||||
- No glTF export. `TODO.md` already lists glTF export as a far-future item; it is a different
|
||||
problem and shares nothing with this except the word glTF.
|
||||
- No lighting, no material system, no texture sampling on the reference layer. Flat fill,
|
||||
wireframe, or vertex-normal shading. Three options, no more.
|
||||
- No animation playback on the reference layer. Pose is a static state.
|
||||
- No physics, no cloth, no bendy bones. `TODO.md` § Future Feature Wishes already wants a real
|
||||
skeleton for the puppet; that is a separate project and this one must not become its
|
||||
accidental first installment.
|
||||
- Nothing from the reference layer reaches CMO3 or MOC3 export. It is editor-only. See § Risks.
|
||||
|
||||
## World-space convention
|
||||
|
||||
Umamo's world space is X right, Y depth, Z up. `PuppetModel.worldOriginX` and `worldOriginZ`
|
||||
already exist, and the viewport already draws the X and Z axis lines through the world origin
|
||||
(`module/render/src/jvmTest/.../gl/WorldAxisLinesTest.kt`).
|
||||
|
||||
The reference layer uses that same frame. No new axis convention, no conversion. A mannequin
|
||||
standing at the world origin is already aligned with the drawing, and at yaw = pitch = 0 the
|
||||
reference camera must agree exactly with `ViewportCamera.worldToNdc`. That agreement is the
|
||||
anchor of the whole design: the reference layer at rest sits exactly in the drawing plane, and
|
||||
rotating it is the only thing that separates it from the puppet.
|
||||
|
||||
Verified in `verify/verify.py`: at yaw 0 pitch 0 the camera basis is forward +Y, right +X,
|
||||
up +Z, and a point one world unit right projects to NDC x = 2·zoom/viewportWidth, which is
|
||||
`ViewportCamera.worldToNdc`'s value.
|
||||
|
||||
Units are model pixels: one world unit is one model pixel. A mannequin therefore needs a
|
||||
scale stated in human units, not pixels. `referenceHeightCm` (default 160) plus the document's
|
||||
puppet height in world units gives the scale, and that is what makes the proportion check
|
||||
possible at all — a head is about 1/7.5 of standing height, and the mannequin encodes that as
|
||||
data rather than as a guess.
|
||||
|
||||
## Data model
|
||||
|
||||
```
|
||||
ReferenceLayer
|
||||
source BuiltInMannequin(proportions) | ImportedGltf(assetKey, sceneIndex)
|
||||
transform position (x, y, z), scale, rotation (yaw, pitch, roll)
|
||||
pose Map<JointId, Euler(x, y, z)> degrees
|
||||
style fill, wireframe, shading (Flat | VertexNormal | None), opacity
|
||||
visibility visible, occlusion (Above | Below)
|
||||
camera linked to the viewport camera, or an independent orbit
|
||||
```
|
||||
|
||||
`ImportedGltf` stores an asset key and a hash, never the mesh. A missing asset degrades to
|
||||
"asset missing" and the layer keeps its pose and transform. That is the same shape the source-
|
||||
art binding already uses in `:reimport`: the mapping survives the pixels going away.
|
||||
|
||||
The built-in mannequin is procedural: a joint hierarchy plus primitive volumes. No bundled
|
||||
asset, so no licensing question, and it is deterministic — the same proportions always produce
|
||||
the same triangles, which a test can pin.
|
||||
|
||||
Joint set, minimum viable:
|
||||
|
||||
```
|
||||
pelvis, spine, chest, neck, head
|
||||
shoulder_l, elbow_l, wrist_l
|
||||
hip_l, knee_l, ankle_l
|
||||
shoulder_r, elbow_r, wrist_r
|
||||
hip_r, knee_r, ankle_r
|
||||
```
|
||||
|
||||
17 joints. Each has an anatomical clamp, so a pose cannot be built that a body cannot hold, and
|
||||
the clamp is data the UI can show as a slider range rather than a silent rejection.
|
||||
|
||||
## Camera
|
||||
|
||||
`ReferenceCamera(target, yaw, pitch, zoom, projection)`, pure, returning a new instance,
|
||||
touching no GL — the same discipline as `ViewportCamera`, so the maths unit-tests without a
|
||||
context.
|
||||
|
||||
Orthographic by default. Perspective is an option with a focal length, because foreshortening —
|
||||
the use that justifies the feature — is a perspective phenomenon. Orthographic is the default
|
||||
because it agrees with the 2D plane exactly, and a reference that disagrees with the drawing's
|
||||
projection is wrong in a way the user cannot see.
|
||||
|
||||
Two clamps, both found by the harness rather than assumed:
|
||||
|
||||
- `pitch` must be clamped to the open interval (−89.9°, +89.9°). At ±90° the camera looks along
|
||||
the world up axis and the screen-right basis vector becomes a zero vector. The projection
|
||||
still returns a number, so the failure is silent: the model collapses to a line and nothing
|
||||
reports it. The clamp has to be in the camera, not in the UI drag handler.
|
||||
- `zoom` reuses `ViewportCamera.MIN_ZOOM` / `MAX_ZOOM`.
|
||||
|
||||
Camera linking: when linked, the reference camera's target and zoom follow the viewport camera,
|
||||
and only yaw and pitch are its own. When unlinked it is fully independent. Linked is the
|
||||
default, because the useful case is "rotate the reference without losing the drawing".
|
||||
|
||||
## Draw path
|
||||
|
||||
The seam has no depth concept today. `RenderPipelineSpec` is `(purpose, blend, cullBackFaces)`;
|
||||
`RenderTargetSpec` has no depth attachment; `PipelinePurpose` is closed at six values, none of
|
||||
which is a 3D mesh. `MeshSpec.restPositions` is interleaved x,y.
|
||||
|
||||
Four additions, in `docs/seams.md`. They are not optional and they are not local: adding a
|
||||
`PipelinePurpose` value forces every backend to answer for it, which is the stated intent of
|
||||
that enum being closed. So this feature cannot land in `:render` alone.
|
||||
|
||||
Draw order:
|
||||
|
||||
- `Above` — draw after the puppet, no depth test. Always correct, no depth buffer needed.
|
||||
Default.
|
||||
- `Below` — draw before the puppet, no depth test.
|
||||
- `Depth-sorted` — the mannequin occludes the puppet properly. Deferred: it needs the puppet to
|
||||
write depth from drawable Z, and the puppet path has no depth today.
|
||||
|
||||
The reference layer must stay outside the layer-composite path. The composite samples
|
||||
screen-space textures and blends in-shader with blending disabled; it has no depth input, so a
|
||||
depth-tested draw composited through it would blend against pixels the depth test should have
|
||||
rejected. Keep it a direct draw.
|
||||
|
||||
## Pose
|
||||
|
||||
Pose is a map of joint id to Euler triple, applied as `T(offset) · R(euler)` per joint, world =
|
||||
parent world · local. Verified invariants in the harness:
|
||||
|
||||
- bone length is pose-invariant,
|
||||
- a joint's rotation does not move its own origin,
|
||||
- the rotation block stays orthonormal at scale 1, which is what skinning depends on.
|
||||
|
||||
Pose presets reuse the storage shape already planned for `TODO.md` § Pose Palette/Library —
|
||||
Cubism 5.4's "Model state set" is a pose library, and the same container holds a mannequin pose
|
||||
with no format work.
|
||||
|
||||
Skinning for imported glTF is CPU-side: joint matrices times weights, summed. Verified: a
|
||||
70/30 blend lands between the two poses. CPU first matches the codebase's existing shape
|
||||
(CPU deform eval as the oracle, GPU as the fast path), and it gives a testable oracle before
|
||||
any GPU work.
|
||||
|
||||
## Persistence
|
||||
|
||||
A new UMA entry, `reference`. It stores the source, transform, pose, style, visibility, and
|
||||
camera link. It does not store geometry — geometry comes from the asset or from the procedural
|
||||
generator, both reproducible.
|
||||
|
||||
The entry must be additive and ignorable: a reader that does not know `reference` skips it and
|
||||
the document still opens. Mirror `UmaEntrySchemaTest.kt` for the gate.
|
||||
|
||||
## UI
|
||||
|
||||
- Viewport header → Overlays toggle, which is already planned (`TODO.md` § Overlays Toggle) and
|
||||
currently lists General Information, Wireframe, Grid, 2D Cursor. Add Reference Layer there.
|
||||
- Properties panel → a Reference section.
|
||||
- Commands through `CommandRegistry`, not hardcoded in a widget: `reference.import`,
|
||||
`reference.reset-pose`, `reference.fit`, `reference.toggle-visibility`,
|
||||
`reference.link-camera`, `reference.clear`.
|
||||
- Settings keys, dotted namespace: `view.referenceLayer.style`, `.opacity`, `.wireframe`,
|
||||
`.defaultHeightCm`, `.maxTriangles`, `.linkCamera`.
|
||||
|
||||
## Tests
|
||||
|
||||
Pure, `commonTest`:
|
||||
|
||||
- GLB container rules: header, chunk types, padding, truncation, accessor alignment.
|
||||
- FK invariants above.
|
||||
- Camera basis orthonormality, agreement with `ViewportCamera` at yaw 0 pitch 0, pitch clamp.
|
||||
|
||||
GL-gated, `jvmTest`, shaped like `WorldAxisLinesTest.kt`: a flat-shaded triangle at a known
|
||||
camera must rasterize the expected pixels, and the depth test must reject a triangle behind it.
|
||||
Skips in a display-less environment via a JUnit assumption, so the run reports SKIPPED rather
|
||||
than a green pass that asserted nothing.
|
||||
|
||||
Tolerance, not bit-exactness, for anything with trig. The codebase's determinism rule is real,
|
||||
but FK and the camera cannot be bit-identical across targets; the tests must say so explicitly
|
||||
rather than assert exact floats and fail on a different machine.
|
||||
|
||||
## Risks
|
||||
|
||||
1. **Export contamination.** The reference layer must never reach the CMO3 writer. The
|
||||
byte-identical round-trip gate is the thing that catches this, and it only catches it if the
|
||||
gate is run with a reference layer present. Add that case.
|
||||
2. **Closed enum cost.** `PipelinePurpose` is deliberately closed so a new pipeline forces every
|
||||
backend to answer. The GLES and Metal devices are stubs, so this feature is desktop-only
|
||||
until the GLES device exists (`TODO.md` § Deferred). That is fine, but the stubs must answer
|
||||
explicitly rather than throw.
|
||||
3. **Silent degeneracy.** The pitch collapse is the clearest example: a projection that returns
|
||||
a number instead of failing. Any clamp added here belongs in the pure model, not in the
|
||||
input handler.
|
||||
4. **Memory.** An imported glTF can be 100k triangles. On a tablet that is not a reference
|
||||
layer, it is a stall. Cap at import, report the cap as a notice, do not silently drop
|
||||
primitives.
|
||||
5. **Scope creep.** The feature's natural next step is a real 3D viewport, which is a different
|
||||
project. The non-goals section exists to keep this one from becoming its first installment.
|
||||
Reference in New Issue
Block a user