suby pushed to oivo at suby/qmd
703371b723 fix(launcher): resolve dependencies beside symlinked dist
Verification:
- bash test/launcher-detection.test.sh
- bash test/launcher-symlinked-dist.test.sh
- ./bin/qmd --version
Session-Id: 01a047e1
- 03c43a2926 fix(embed): honor native Gemini key (omit-index + factory fallback)
QMD rejected Gemini OpenAI-compat responses that omit data[].index
(session 7fcfc297 on zdrowie) and ignored GEMINI_API_KEY unless
QMD_EMBED_ENDPOINT was set, so the crew rewrote config to ai.mm.mk and 401'd.
- Treat missing embedding index as input order
- If GEMINI_API_KEY/GOOGLE_API_KEY is set and no endpoint, use Gemini
OpenAI-compat (gemini-embedding-001)
- Status reports native-key / config.json, not only QMD_EMBED_ENDPOINT
Session-Id: 01a01263
- ab6440dc0e fix(embed): re-select documents whose later chunks failed to embed (i-xeekgx6h)
"Does this document need embedding?" was asked as "is there a vector at seq 0?",
while chunk inserts are per-chunk best-effort ("so a single bad chunk doesn't
drag down the rest"). Compose the two and a document whose chunk 0 landed and
whose chunk 3 did not is complete FOREVER: the 30-minute embed cron never selects
it again and the missing chunks are unreachable. Nothing errors — semantic search
just answers from a document with holes in it.
The trigger is any partial-failure window. Split out of i-7yayruey, where the
embeddings upstream vanished in five bursts; four were 100% failures, which is
the SAFE shape (nothing inserts, the document stays pending, the next run heals
it). The dangerous one is the mixed burst — 3505 requests at 26% failure.
Fix: record how many chunks a document was split into, at chunk time, and treat
"fewer vectors than chunks" as pending alongside "no chunk 0". Recording BEFORE
embedding rather than after is deliberate — a run that dies mid-document still
leaves the expectation behind for the next run to compare against.
The predicate now lives in ONE place (PENDING_EMBEDDING_JOINS/PREDICATE) used by
both getPendingEmbeddingDocs and getHashesNeedingEmbedding. Those two each
carrying their own copy of the rule is the shape that let this hole exist, so a
test asserts the filtered and unfiltered counts agree.
Documents embedded before the table existed have no recorded count, so they are
NOT re-selected: a whole-corpus re-embed on upgrade would cost far more than the
holes it heals. That limitation is deliberate, tested, and repaired by --force or
the next re-chunk.
Tests (5, all passing): chunk 0 embedded + later chunks dropped leaves the
document pending; a later healthy pass fills it and the count returns to 0. Two
CONTROLS carry the weight — a fully embedded document is NOT re-selected
(docsProcessed 0, chunksEmbedded 0, i.e. no corpus re-embed every tick), and a
legacy document with no recorded count is left alone. Mutation-checked: reverting
the predicate to the old "no chunk 0" turns 3 of the 5 red.
Scoped runs: embed-partial-chunks 5/5, embed-collection-filter 9/9 (same
predicate), store + embedding-store-integration 193 passed / 13 skipped.
dist/ rebuilt and committed with src/ per CLAUDE.md — the fleet bundler ships
this repo's dist/ as-is.
Refs: i-xeekgx6h
Refs: i-7yayruey
Session-Id: 7a7ae5c5
- View comparison for these 3 commits »
15 timmar sedan
suby pushed to oivo at suby/qmd
- 4f1e3deb9b merge(qmd): land the stranded 429-abort fix onto the commercial-policy head (i-yghj098h)
89f3222 ("stop one 429 from aborting a whole embedding run") was committed on
branch `oivo` and then walked away from: HEAD was checked out detached at the
divergent 34c9570 ("enforce commercial-api-only model operations"), so every
shipping `qmd embed` still ran the original defect — dist/store.js:1314/1322
`errors += remaining` booking never-attempted chunks as failures, with the
cumulative >80% ratio re-tripping on each following document.
This merges 34c9570 into `oivo` (textually clean, zero conflicts in
src/embedding/openai.ts and src/store.ts) and rebuilds dist/ from the merged
source, so the branch now carries BOTH the 429 handling and the
commercial-api-only policy, and the bundle that actually runs carries the fix.
Verified on the merged tree:
- grep -c BulkLaneGate dist/embedding/openai.js -> 3 (was 0)
- grep -c 'errors += remaining' dist/store.js -> 0 (was 2, lines 1314/1322)
- positive control: grep -c embedding dist/embedding/openai.js -> 31
- vitest test/embedding-openai.test.ts test/embedding-store-integration.test.ts
test/commercial-model-policy.test.ts -> 96/96 passed
- negative control: the recurrence test at
test/embedding-store-integration.test.ts:420 run against pre-merge 34c9570
fails with errors=6 (expected 1), so the test can see the defect.
Part 1 of the issue (the qmd-embed.mm.mk vhost / gateway per-caller budget)
remains open and undecided — this lands the client-side half only.
Refs: i-yghj098h
Session-Id: ab2d1bd2
- 34c95700f7 feat(qmd): enforce commercial-api-only model operations
- 89f322244e fix(qmd): stop one 429 from aborting a whole embedding run (i-yghj098h)
Bulk reindex traffic shares the interactive ai.mm.mk token bucket, and a
single rate-limit response could poison an entire pass:
- src/store.ts: the >80% error-rate guard compared CUMULATIVE errors and
fed its own un-attempted remainders back into `errors`, while `break`
exited only the inner batch loop — so every following document re-tripped
the guard and added its remainder too. 1 real 429 was measured booking
740,364 "failures" across 52 aborts. Now: the rate is judged over a
sliding window of recently ATTEMPTED batches, un-attempted chunks are
reported as `skipped` (new, additive field) instead of `errors`, and one
abort ends the run exactly once.
- src/embedding/openai.ts: 429s get their own retry budget and honour the
server cooldown (Retry-After header, or the "Retry after 29s" prose that
ai.mm.mk puts in the body) — the old 1s/4s/16s schedule could not outlast
a 30s bucket refill. 429s no longer count toward the circuit breaker:
backpressure must not become a 5-minute hard OPEN.
- Bulk lane (BulkLaneGate): a 429 on any worker now pauses the WHOLE pool
for one coalesced cooldown and halves in-flight concurrency (AIMD, floor
1, recovering one unit per 8 successes), so reindex traffic yields the
bucket to interactive callers instead of re-draining it. Multi-input
batches are attributed `via=embeddings-bulk` in X-AI-Caller so the gateway
can budget the two lanes separately.
Gateway-side per-caller budget and the dedicated qmd-embed.mm.mk vhost
(issue part 1a/1b) are unchanged and still open — this is the client-side
half. dist/ regenerated by the pre-commit build; no reindex run performed.
Refs: i-yghj098h
Session-Id: d83f4a3e
Resolves: i-yghj098h
Push-Allowed: yes
- View comparison for these 3 commits »
2 veckor sedan
suby pushed to repair/commercial-api-only-20260809 at suby/qmd
- 34c95700f7 feat(qmd): enforce commercial-api-only model operations
- 332ba839b2 Enforce commercial-only QMD model policy
- d84af60a89 fix(qmd): use shared Oivo cache for direct CLI
Direct qmd shell calls now default to /srv/.cache when present and ignore EPIPE from closed pipes.
Session-Id: 019ed817
Push-Allowed: yes
- bff0e428a6 fix(qmd): prune orphaned vectors without vacuum
Session-Id: 019ed817
Push-Allowed: yes
- 649a30b420 fix(llm): never compile llama.cpp at query time — prebuilt-only (i-tgac7ig3)
getLlama() used build:"autoAttempt", which compiles llama.cpp from source
per-GPU the moment a prebuilt fails to load. On the code LXC (libvulkan.so.1
present, no GPU device, no glslc) the GPU auto-probe picks Vulkan, the prebuilt
Vulkan binary is host-incompatible, and autoAttempt then runs a doomed cmake
build (no glslc) — a 30-60s+ blocking stall that hangs interactive 'qmd query'
and leaves a half-built localBuilds/ dir.
Switch to build:"never": node-llama-cpp loads prebuilt binaries only and falls
through the candidate list (Vulkan -> CUDA -> CPU) to the prebuilt CPU binary.
Verified on code: getLlama({build:"never",gpu:"auto"}) loads CPU in ~2.7s; a
full 'qmd query' runs in 6.3s and creates 0 localBuilds/ files. Completes
i-c28wngnd (which only covered the QMD_EMBED_ENDPOINT=cpu path) for the
interactive / gpu:"auto" path. Also fixes the same footgun in bench-rerank.ts.
Generated with [Claude Code](https://claude.ai/code)
via [Oivo](https://oivo.com)
Co-Authored-By: Claude <noreply@anthropic.com>
Session-Id: b6802844
2 veckor sedan
suby created new branch repair/commercial-api-only-20260809 at suby/qmd
2 veckor sedan
suby pushed to master at suby/genrecon
- 5233b929aa feat: reconstruction survives service restarts + phone-friendly glb + network-first SW
Owner's video 'stopped' on his phone but actually FINISHED server-side — his client
just lost the view and, on an old cached build, couldn't reattach. Three fixes:
- Detached restart-surviving runner: run_job.py runs each job in its own session,
serialized on the GPU by an flock; unit gets KillMode=process; app startup
re-attaches to a live runner or relaunches a dead one. Verified: a
'systemctl restart genrecon-app' mid-COLMAP left the same runner PID alive and the
job progressing. Nothing kills a job now.
- Phone-first glb: default simplify 0.12->0.04 (~900k->~300k tris) and textures
2048->1024, so the walkthrough doesn't exhaust mobile WebGL memory (blank scene).
- SW: app shell (HTML/JS/CSS) is now network-first so code updates land immediately
online (a stale cache-first app.js is why ?job= showed nothing); heavy vendor/glb
stay cache-first.
1 månad sedan
suby pushed to master at suby/genrecon
- d731854369 fix(client): reconstruction is a durable server-side job; the client is just a view
Root cause of 'it stopped': the phone lost its polling connection mid-COLMAP, but
the server-side worker kept going and the job actually finished (web-optimized glb
and all). The client just had no way to find its job again on reconnect.
- Persist the active job to the URL (?job=) AND localStorage, so a reload / lost
signal / reopening the app hours later resumes the SAME job (progress or scene).
- Auto-resume on load from URL or localStorage.
- Polling shrugs off transient fetch failures (never freezes); 404 → back to upload.
- Re-sync on visibilitychange / online / focus (mobile freezes backgrounded timers).
- Bump SW cache v2→v3 so clients pick up the new shell.
1 månad sedan
suby pushed to master at suby/genrecon
- 8a5f9b7797 fix(viewer): clearance-based spawn facing the scene (no more camera buried in geometry)
resetPlayer spawned blindly at the horizontal centre → for object-centric captures
the camera started inside the mesh (dark close-up). chooseSpawn() now samples floor
points across the scene, keeps the standing spot with the most horizontal clearance,
and yaws toward the scene centre with a slight downward tilt. Bump SW cache v1→v2 so
clients pick up the new viewer.js/app.js shell.
1 månad sedan
suby pushed to master at suby/genrecon
- 48e119fb5e chore(deploy): setup_gltftool.sh for the web-optimize step + README note
1 månad sedan
suby pushed to master at suby/genrecon
- 8584e69c71 feat(backend+viewer): real reconstructions end-to-end + web-ready walkthrough
Validated the full path on a real room scene (Mip-NeRF360 bonsai): video →
COLMAP → GenRecon SLAT (DINOv3 + TRELLIS) → 8M-face mesh → 58MB walkthrough glb,
renders 939k tris in-browser with floor collision + WASD movement.
- rgb/ input dir: GenRecon Iphone mode reads frames from <job>/rgb/, not images/.
- scale-adaptive point-cloud cleaning (radius_m=auto): plain-video COLMAP is only
up-to-scale, so a fixed metric radius wipes the cloud; derive it from median NN.
- web-optimize glb (weld → simplify → webp) so the 100s-of-MB raw mesh loads in a
browser; raw kept as scene_raw.glb; best-effort, never fails a job.
- friendly reconstruct errors (no depth / no overlap / OOM) instead of tracebacks.
- viewer: /?job=<id> deep-link for shareable/reloadable scenes.
1 månad sedan
suby pushed to master at suby/genrecon
- c6d70dc961 fix(backend): make GenRecon reconstruct actually run on the genrecon LXC
- COLMAP headless: QT_QPA_PLATFORM=offscreen + CPU SIFT/matching (apt COLMAP
aborts on Qt/EGL in a headless container).
- Checkpoint run-dir layout: the released .pt files are raw LoRA state_dicts
with no embedded config; point --ss/shape/tex_ckpt at
ckpts/{ss,shape,tex}/checkpoints/*.pt so load_train_config finds the matching
configs/gen training config.json.
- Pass the README-validated Iphone cleaning knobs (chunk_size_factor,
stat_std_ratio, radius_nb_points, radius_m, proj_batch_voxels), config-driven
so plain-video (non-ARKit, arbitrary-scale) captures can loosen radius_m.
1 månad sedan
suby pushed to master at suby/genrecon
- 2dbaf7647b GenRecon PWA + reconstruction backend (prebuilt for i-opavuyaq)
Phone video -> ffmpeg -> COLMAP -> GenRecon reconstruct_scene -> chunked_to_glb -> scene.glb,
then walkable in-browser (three.js first-person, Octree collision). PWA installable + offline shell.
Backend has a MOCK mode; locally smoke-tested: render + walk + full job flow, clean console.
Refs: i-opavuyaq
Session-Id: c652fcc3
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 månad sedan
suby created new branch master at suby/genrecon
1 månad sedan
suby skapade utvecklingskatalog suby/genrecon
1 månad sedan
suby pushed to master at suby/json-render
- 7c39f39038 docs(license): keep LICENSE as pristine Apache-2.0 for GitHub auto-detection
Fork attribution lives in NOTICE + README "License" section instead; editing
the LICENSE body broke licensee detection (NOASSERTION). Reverts the header
block only — NOTICE/README unchanged.
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan
suby pushed to main at suby/json-render
- 7c39f39038 docs(license): keep LICENSE as pristine Apache-2.0 for GitHub auto-detection
Fork attribution lives in NOTICE + README "License" section instead; editing
the LICENSE body broke licensee detection (NOASSERTION). Reverts the header
block only — NOTICE/README unchanged.
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan
suby pushed to master at suby/json-render
- 8fc30559f2 docs(license): note Apache-2.0 upstream fork — LICENSE header + NOTICE + README license section
Keeps upstream Apache-2.0 + Vercel copyright intact; adds fork attribution
(vercel-labs/json-render) and an Apache §4(b) modifications notice.
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan
suby pushed to main at suby/json-render
- 8fc30559f2 docs(license): note Apache-2.0 upstream fork — LICENSE header + NOTICE + README license section
Keeps upstream Apache-2.0 + Vercel copyright intact; adds fork attribution
(vercel-labs/json-render) and an Apache §4(b) modifications notice.
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan
suby pushed to master at suby/json-render
- 6e038cc6b6 docs(readme): use markdown images so Gogs rewrites relative paths (HTML <img> wasn't resolving)
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan
suby pushed to main at suby/json-render
- 6e038cc6b6 docs(readme): use markdown images so Gogs rewrites relative paths (HTML <img> wasn't resolving)
Session-Id: 481d7b9c
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2 månader sedan