Engineering handoff · Perfit_UR_Server + PerFit-Monorepo
Try-On Pipeline: Test Report
Four load runs, 50 paid generations, a 30-case input-validation sweep, and a full visual audit of all 20 outfits. What broke, why it broke, what was fixed, and what is still open.
Summary of all four runs
Every run used the REST API a customer would use — POST /tryon then poll — never GraphQL. All 50 jobs succeeded; nothing failed, nothing was aborted.
| Run | Jobs | What | Mean | p50 | Slowest | Re-rolls | Billed |
|---|---|---|---|---|---|---|---|
| 1 | 5 | image + video, 5 outfits, UR test photo | 119.3s | 110.6s | 235.4s | 2 | $4.00 |
| 2 | 5 | images, verify new cost counters live | 73.9s | 88.9s | 96.4s | 0 | $0.50 |
| 3 | 20 | images, all 20 outfits, blind re-roll | 45.1s | 35.8s | 135.3s | 5 | $2.00 |
| 4 | 20 | images, all 20 outfits, corrected re-roll | 51.7s | 41.9s | 123.6s | 5 | $2.00 |
The three things that matter most
- Concurrency is not a problem; the judge panel is the cost. 20 jobs at once ran faster than 5 at once. The image model answers in ~10s — everything else is our own VLM quality checks, 4 to 8 per image, which nobody prices.
- Per-job cost was understated by ~20%. A re-rolled image generated twice and billed once. Fixed and deployed; margin reads 43.8%, not the ~55% previously reported.
- Whole garments render reliably. Attachments do not. 18 of 20 outfits rendered every item. What fails is belts on shorts, one of two bags, and category labels — plus socks invented in 7 of 20.
Run 1 — 5 jobs, image + video
Outfits 3, 7, 11, 15, 19. One person photo (UR's test model), five different outfits, all submitted at the same second. Image then video for each. System idle at start: 0 jobs in flight, 25 warm Modal containers.
| Outfit | Body path | Image | Video | Total | Result |
|---|---|---|---|---|---|
| 3 | image_gen | 60.9s | 72.1s | 133.0s | done |
| 7 | image_gen | 110.6s | 76.3s | 186.9s | done |
| 11 | smpl_fit | 235.4s | 89.5s | 324.9s | done, but stalled 183s |
| 15 | image_gen | 127.3s | 106.4s | 233.7s | done |
| 19 | smpl_fit | 62.0s | 76.7s | 138.7s | done |
What went wrong
A job sat undispatched for 183 seconds
What happened. Outfit 11 showed status: preflight from t=10.7s to t=195.4s. It looked alive. It was not — Modal's run.started event for that job did not fire until t=183.4s. The job had not entered the pipeline at all. Once it did start it finished its image in 40.5s, the fastest of the five.
Why. All five jobs were enqueued within 448 milliseconds of each other. The dispatcher pulls the queue in batches of 5 and retries an empty pull only while nothing has been spawned yet:
msgs = _pull_messages() # batch_size=5
if not msgs and spawned == 0: # <-- retries ONLY on the first pull
for _ in range(15): ...
if not msgs: break # <-- 5th message not visible yet -> give up
The first pull returned four messages. The fifth had not become visible in the Cloudflare queue yet. Because spawned was no longer zero the retry was skipped, the loop broke, and the job fell through to the 30-second backstop cron — where it took roughly six cycles to be picked up.
Where. Perfit_UR_Server/modal_app.py:1400-1412.
Status. NOT FIXED. It is pipeline code and was left alone deliberately. Two consequences: a customer polling sees a job that looks busy and isn't, and the dashboard cannot tell an undispatched job from a working one because both read preflight.
The AI provider was never the bottleneck — our own quality checks were
What we measured. Every image call went to one provider (fal) with zero failures and returned in 8.9 to 13.3 seconds. But end-to-end image time was 60.9s at best and 235.4s at worst. Between 82% and 96% of the customer's wait was not generation.
Why. After the image is generated, verify_and_retry runs four VLM judges in parallel — garment, footwear, identity, accessory presence. If any complains, it runs up to four more confirm judges (a two-vote rule). If two votes agree, it regenerates the whole image and judges again. The telemetry emitted one event after all of that, carrying only the duration and price of the first generation.
So the pipeline was doing 4–10 model calls per image and reporting 1.
Proof — the same job shape, before and after. Both generated two images:
BEFORE (run 1) price_usd_estimate = 0.045 <-- one generation's price
no judge count, no generation count, no re-roll count
AFTER (run 3) outfit 1: gens=2 judges=7 rerolls=1 image_cost_usd=$0.090
outfit 3: gens=2 judges=8 rerolls=1 image_cost_usd=$0.090
outfit 2: gens=1 judges=3 rerolls=0 image_cost_usd=$0.045
---------------------------------------------------------
20 jobs -> 25 generations, 96 judge calls, real cost $1.125
old telemetry would have reported $0.900 (understated 20%)
Status. FIXED — counters added and deployed. See Code shipped.
Video: detail disappears when the camera moves
All five videos were 6.0s, 24fps, 498×1050, and every garment persisted across all 144 frames. But on outfit 19's 360° rotation the cream shearling collar is fully visible from the front and completely absent from the back view. A real shearling collar wraps the neck.
Status. NOT FIXED. Trim and hardware are the first things to go when the camera moves. Also note the video resolution is 498×1050; a July session recorded 720×1280, so this dropped at some point.
Run 2 — 5 jobs, counter verification
Outfits 2, 5, 9, 14, 18. Images only, using the customer-supplied photo. Purpose was not load — it was to confirm the new cost counters actually reach the database after deploy.
| Outfit | Time | Generations | Judge calls | Re-rolls | Recorded cost |
|---|---|---|---|---|---|
| 14 | 28.3s | 1 | 3 | 0 | $0.045 |
| 2 | 62.3s | 1 | 3 | 0 | $0.045 |
| 18 | 88.9s | 1 | 3 | 0 | $0.045 |
| 5 | 93.8s | 1 | 3 | 0 | $0.045 |
| 9 | 96.4s | 1 | 4 | 0 | $0.045 |
Counters land, and customer bills did not move
16 judge calls for 5 images — 4 where the outfit has a non-shoe accessory, 3 where it does not, which is exactly the arithmetic. Billing rose by precisely $0.50 (5 × $0.10), because customer billing counts fal.request rows and we still emit one row per image. Only our internal cost number changed.
Zero re-rolls in this run, so the doubled-cost path was not exercised live here — it was only unit-tested at this point.
Identity is preserved well
The customer photo is a different person from UR's test model. Across these renders the face structure, hair, and even a braided headband carried through consistently. The identity judge raised complaints on three outfits across later runs and all three were correctly overturned. This part of the pipeline works.
Run 3 — 20 jobs, one per outfit
All 20 outfits, each used exactly once, same customer photo, images only, all fired at the same moment. System idle at start. This was the first run with the new cost counters live.
20 at once ran faster than 5 at once
p50 was 35.8s here versus 110.6s in run 1. Modal scaled from 25 to 54 containers with zero backlog; the image provider served all 20 with no failures at 12.1s average. There was no dispatch stall this time.
Strip out the 5 jobs that re-rolled and the whole range collapses to 25.0–49.2s. The latency tail is the re-roll tail, not a concurrency limit.
What went wrong
Three of five re-rolls failed — we paid twice and shipped the flaw anyway
What happened. Outfits 1, 11 and 20 came back mismatch_after_reroll: the judge rejected the first render, we generated a second image, the judge rejected that too, and the pipeline shipped the original flagged. Only outfits 3 and 6 recovered.
Why. The re-roll was a byte-identical second call:
regenerate=lambda out: generate_image(
prompt=prompt, ref_paths=ref_paths, ...) # same prompt, same refs
The judge's complaint was computed, written to telemetry, and then thrown away. The model was never told what it got wrong, so a re-roll was another roll of the same dice rather than a correction.
Status. FIXED and deployed between run 3 and run 4 — see run 4 for the measured effect.
A confirmed defect records no reason; a false alarm does
What happened. Outfit 6 shipped with footwear_fidelity: "mismatch" and identity_fidelity: "mismatch" and no explanation for either. Outfits 3 and 20 shipped accessory_fidelity: "mismatch", also with no reason.
Why. In src/quality/garment_fidelity.py the issue text is written into meta only on the split->match path — that is, when the second vote overturns the first. When both votes agree it is a real defect, the reason stays in a local variable and is discarded.
So the system keeps the explanation for the cases that turned out to be nothing, and throws it away for the cases that were real. Backwards. It makes any shipped defect impossible to diagnose without opening the image.
Status. NOT FIXED. Small change, high value for whoever debugs the next customer complaint.
Footwear complains constantly and is almost always wrong
| Judge | Outfits checked | Complaints | Confirmed real | Overturned |
|---|---|---|---|---|
| Footwear | 20 | 8 | 1 | 7 |
| Garment | 20 | 5 | 3 | 2 |
| Identity | 20 | 5 | 1 | 4 |
| Accessory | 9 | 2 | 2 | 0 |
Footwear complains on 40% of outfits and is right once in eight. Accessory complains twice and is right both times. The two-vote rule is what stops the noisy judge from triggering pointless re-rolls — no re-roll in either 20-job run was caused by a false positive — but each complaint still costs up to four extra VLM calls.
The accessory judge only runs on 9 of 20 outfits. That is correct, not a bug: the other 11 have no accessory other than shoes, which has its own dedicated judge.
Run 4 — the same 20, with the re-roll fix live
Identical setup to run 3: same photo, same 20 outfits, images only. The only change was the deployed re-roll correction. This is the before/after.
| Measure | Run 3 (blind retry) | Run 4 (corrected retry) |
|---|---|---|
| Jobs succeeded | 20/20 | 20/20 |
| Wall clock | 143s | 129.5s |
| Mean / p50 | 45.1s / 35.8s | 51.7s / 41.9s |
| Re-rolls | 5 | 5 |
| Failed re-rolls | 3 | 2 |
| Judge calls | 96 | 82 |
| Generations / cost | 25 / $1.125 | 25 / $1.125 |
40% → 60% is not a result
The re-roll population changed almost completely between runs, because the judges are non-deterministic:
run 3 re-rolled: 1, 3, 6, 11, 20 run 4 re-rolled: 1, 8, 12, 18, 20 in both: 1, 20
Outfit 11 came back clean in run 4 with no re-roll at all — it was not "fixed by the correction", it simply did not fail that time. So the improvement is 3 successes versus 2, on a sample of 5. That is noise, and the mean-latency regression (45.1s → 51.7s) is the same noise pointing the other way.
If the fix rate matters commercially it needs several hundred jobs, not forty.
Outfit 20 went from no bag to one bag
Outfit 20 re-rolled and failed in both runs on the same fault, which makes it the only controlled comparison in the data. It is not a null result:
Outfit 20 declares two bags. The blind retry produced neither; the corrected retry produced one — which is why it still counts as a failure in the table and why the pass/fail number hides the improvement. The judge's own wording tracks it: run 3 said "Accessories from the left are not present on the person"; run 4 said "One of the reference bags is missing from the render." The fault shrank from all to one of two.
Whole-object faults get fixed; details and crowding do not
All three new re-rolls in run 4 succeeded, and their complaints share a shape — a whole item in the wrong place:
"The reference garment is a dress, but it has been rendered as a top." — outfit 8, fixed
"bag is missing" — outfit 12, fixed
"The dark grey textured cardigan is not present on the model" — outfit 18, fixed
The two that stayed broken were not like that. Outfit 1's was a texture detail — "the cardigan has a different pattern on one side". Outfit 20's is an attention-budget problem, described under Open problems.
Input validation — 30 cases, $0
Every case built from one real customer photo, differing in exactly one way. All sent to POST /tryon/validate, which runs the same preflight checks but creates no job and calls no provider — so the whole sweep was free.
25 of 30 behaved correctly: low resolution, blur, multiple people, no person, selfie crop, rotated, upside-down, truncated bytes, non-image bytes, empty body and malformed base64 were all caught with the right message. 16-bit PNG, TIFF, PNG-with-alpha, grayscale, a 12-megapixel upload, heavy JPEG artifacts and a letterboxed phone screenshot all passed correctly.
HEIC — the iPhone's default format — was rejected outright
What happened. A genuine HEIC file came back valid:false — "could not decode image".
Proof it was the container, not the photo. The same picture re-saved as JPEG came back valid:true, full_body. Only the wrapper differed.
Why. cv2.imread cannot read HEIC. HEIC has been what iPhones save by default since iOS 11, so any customer uploading straight from their camera roll — unless their app converts first — got a hard rejection on a perfectly good photo. It fails as an ordinary rejection, so nobody would report it as a bug.
Fix. New src/image_io.py::imread_any, a drop-in for cv2.imread. OpenCV runs first; PIL + pillow-heif is tried only after OpenCV declines, so the normal JPEG/PNG path is unchanged and costs nothing. Applied at both places that read the customer photo (quality/preflight.py, body/smpl_fit.py) so a format preflight now accepts cannot fall over one stage later.
Subtlety worth knowing. The fallback also applies ImageOps.exif_transpose. OpenCV honours the EXIF rotation tag; PIL does not unless asked, and phone photos are stored rotated with a tag. Without it the fix would have swapped "could not decode image" for "no person detected" — worse, because that reads as a real verdict.
Proof — the same three files through POST /tryon/validate, before and after the deploy:
BEFORE
24-heic.heic 200 valid:false rejected "could not decode image"
X3-heic-as-jpeg.jpg 200 valid:true full_body <-- same photo, JPEG container
AFTER
24-heic.heic 200 valid:true full_body <-- fixed
01-clean.jpg 200 valid:true full_body <-- unchanged
27-not-an-image.jpg 200 valid:false rejected "could not decode image"
<-- gate did NOT get looser
Status. Merged (PR #54) and deployed. Verified live afterwards: HEIC now returns valid:true, full_body, and genuine garbage still returns valid:false. Customer docs updated in EN and 中文.
Two error messages name the wrong cause
A dark photo is told it is blurry. A heavily underexposed image returns "image is too blurry — please upload a sharper photo". The brightness gate never fires, because a dark image has low Laplacian variance and the blur check catches it first. The customer retakes the photo sharper, and gets rejected again.
A small person in a bright frame is told the image is blown out. Returns "almost entirely blown out or black" when the real problem is that the subject is too small. "Person standing far away outdoors" is a normal customer photo.
Status. NOT FIXED. Both are message-ordering problems, not detection problems — the gates work, they just report in the wrong order.
A data: prefix actually works
The integration docs say person_image_b64 must have no data: prefix. Sending one was accepted and validated correctly; base64 with line breaks also works. The endpoint is more forgiving than documented.
The docs were updated to add HEIC but deliberately not to promise prefix tolerance — writing that down would turn an accident of lenient decoding into a contract we then have to keep.
One case untested: animated GIF. The test file was generated at 0 bytes, so that input has never been exercised.
Per-outfit audit — all 20
Every render checked by eye against the exact item list in its request template — not the marketing card set, which disagrees for 7 outfits. Based on run 4.
Genuinely missing items — only 2 of 20
| # | What is missing |
|---|---|
| 1 | The brown leather tie/belt threaded through the shorts. Cardigan, shorts, red sneakers and bag are all present — the belt exists only as a detail inside the shorts' product photo, not as its own reference. |
| 20 | One of two bags — the black leather hobo (UAYB60132BA1). The studded bag rendered; the hobo did not. |
UAYB60132BA1 is missing.18 of 20 outfits rendered every declared item.
Present, but wrong
| # | Problem |
|---|---|
| 3 | Bag proportions differ — the reference is wider and longer; the render is more compact. The silver clasps, zip detail and hanging leather tag all match. Flagged by the judge in run 3 and passed in run 4; on the evidence below, run 4 is probably right and run 3 was the false positive. |
| 19 | Shorts rendered as a skirt. The reference has a clear centre split and double button rows. Reproduced identically in both runs. |
| 15 | The studded belt is rendered twice — correctly at the waist, and again as a studded collar on the jacket. Tee graphic text also smeared and illegible. |
| 7 | Boots render mid-calf; reference is knee-high. Hidden by the skirt in practice. |
| 15, 18 | Hems run long — cargo shorts below-knee instead of above; pleated skirt below-knee instead of above. |
Socks are invented in 7 of 20 renders
Socks appear on outfits 2, 14, 15, 17, 18, 19 and 20. No product in the Final 20 includes socks.
This is more widespread than any missing item, and it is a retailer problem: the render shows a shopper wearing something the shop does not sell. The footwear judge flags it occasionally — "the person is wearing socks that are not present in the product photo" — and is overturned every time by the second vote.
Clean — nothing missing, nothing wrong
Outfits 4, 5, 8, 9, 10, 11, 12, 13, 16. Two worth noting: outfit 11 improved markedly in run 4 (ankle booties at the correct height, and the flared trousers visible and flared, where run 3 rendered knee-high boots that swallowed them); outfit 10's cow-print sneakers render correctly.
Could not verify
Outfit 6 declares 4 pieces in its top block and only 2 are distinguishable in the render. Not necessarily missing — layers may simply be hidden — but it cannot be confirmed visually.
Whole garments are reliable. Attachments are not.
Tops and bottoms rendered correctly in every one of the 20 outfits, across both runs. What fails clusters into four kinds:
- Details attached to a garment — a belt threaded through shorts (outfit 1) is dropped, because it is not its own reference image.
- Duplicates within one outfit — when an outfit declares two bags, one goes missing (outfit 20).
- Category labels — shorts described as "pants" render as a skirt (outfit 19).
- Invented extras — socks, in 7 of 20.
Code shipped this session
All merged to main and deployed to Modal ur-perfit-dummy, except the last item.
| Change | Repo / PR | Status |
|---|---|---|
Cost counters. verify_and_retry now returns judge_calls and image_generations; cost_usd carries unit price × generations, so a re-rolled image bills for both draws. | UR #52 | merged, deployed |
Failed re-rolls uploaded no audit image. The path pointed at _fidelity_rejected2.png, which the single-attempt loop never writes. The case most worth reviewing — paid twice and shipped flagged — left no evidence. | UR #52 | merged, deployed |
Re-roll count went missing on failure. garment_fidelity_rerolls was set inside the success branch only. | UR #52 | merged, deployed |
HEIC decode via src/image_io.py::imread_any, with EXIF transpose. | UR #54 | merged, deployed |
| Integration docs — HEIC added to the accepted-format list, EN and 中文 in parity. | Monorepo #26 | merged |
Re-roll correction. regenerate(out, complaints) — the retry prompt now carries every confirmed judge complaint. | UR, no PR yet | deployed, UNCOMMITTED |
A stacked PR merged into a branch that was already gone
PR #53 (HEIC) was stacked on PR #52. They were merged nine seconds apart, #52 first. GitHub normally retargets a stacked child once its base merges — here #53 was merged into the old base before that happened, so its merge commit landed on a dead branch and main never received the code. It showed as MERGED while src/image_io.py did not exist on main.
Caught by verifying the file on main after pulling, not by trusting the PR status. Re-opened as #54 targeting main directly. If you stack PRs here, verify the file landed after merging.
Open problems, prioritised
Nothing below is fixed. Ordered by what I would do first.
| # | Problem | Where | Why it matters |
|---|---|---|---|
| 1 | Bottom garments are hardcoded as "pants" — for all 20 outfits, regardless of what the template says. 11 of 20 are actually a skirt or shorts. | modal_app.py:662block(req["bottom"], "bottom", "pants") | Outfit 19's shorts render as a skirt. The label is one of the few textual signals the model gets about silhouette. |
| 2 | The prompt tells the model to read a size chart that is never attached. The Final 20 ship size_chart: [{"size_label":"M"}] with no measurements, so the prompt takes its no-measurement branch, which says "Read that chart's row for size M and translate it into a realistic on-body fit". No chart image is sent — sizeChart is null for all 20. | src/fit/outfit_prompt.py | This is the mechanism behind every hem-length and silhouette error. |
| 3 | A confirmed defect records no reason. The issue text is stored only when the second vote overturns the first. | src/quality/garment_fidelity.py | Every shipped defect is undiagnosable without opening the image. |
| 4 | Outfit 20 declares the same shoe as three separate accessory blocks (UAYS60044SA1, _2, _3) plus two bags — 10 refs across 5 blocks. Each block emits its own "worn on BOTH feet" instruction. | assets/outfit_requests/20.json | Three redundant shoe instructions crowd out two bags. This is the fault the re-roll correction cannot reach. It is a data edit, not code. |
| 5 | Socks are invented in 7 of 20 renders. | prompt / model behaviour | Shows shoppers a product the shop does not sell. Most widespread defect found. |
| 6 | The dispatch batch race — a job can sit undispatched for minutes while reporting preflight. | modal_app.py:1400-1412 | Customers see a job that looks alive and isn't; the dashboard cannot tell stalled from working. |
| 7 | Two error messages name the wrong cause — dark photos told they are blurry, small subjects told the frame is blown out. | src/quality/preflight.py | Customers retake the photo fixing the wrong thing and get rejected again. |
| 8 | Marketing cards disagree with render input for 7 of 20 outfits. Outfit 10's card UAWA60229 and outfit 20's UYV660128K0A are shown to shoppers but never rendered on the body. Two files (650.jpg, 249.png) are named by something other than their SKU, so they can never be matched to a card. | assets/final20/items.json vs assets/outfit_requests/*.json | The front end promises items the render does not deliver. |
Things that were suspected and proven fine
- Missing R2 assets. All 95 referenced garment keys exist; nothing is orphaned. Perfect 1:1.
- Missing style references. All 20 outfits have a
style_ref/look.png. - Cold starts. Preflight completed in 0.4–0.6s on every job — no cold container can do that.
- Provider capacity. Zero provider failures across 50 jobs; image calls average ~10–12s.
- Identity preservation. Consistent across all 20 outfits; all three identity complaints were correctly overturned.
- Customer billing. Unchanged and exact —
billedUsdcounts rows and we still emit one row per image.
How to reproduce
Everything here used the public customer API with one token. No internal access required.
Submit a job
POST https://logger-service.elbeg9amar.workers.dev/tryon
X-Api-Token: <customer token>
{ "person_image_b64": "<raw JPEG/PNG/HEIC bytes, base64>", "outfit_id": 1 }
-> 202 { "jobId": "...", "status": "queued" }
Poll it
GET /tryon/:jobId status, currentStep, progress, terminal GET /tryon/:jobId/image the render (202 while still working)
Check a photo for free — no job, no cost
POST /tryon/validate
{ "person_image_b64": "..." }
-> 200 { valid, category, reason }
Read the counters back
The per-job counters (image_generations, judge_calls, rerolls, unit_price_usd) ride on the fal.request row and inside run.completed → data.summary.outfits[0]:
GET /v1/logs?type=run.completed&limit=60
Note: /v1 deliberately strips costUsd and spendUsd for customers — those are our provider costs. Our own cost is readable from the summary blob, or via GraphQL behind the superadmin guard.
50 paid jobs · $8.50 billed · $2.25 provider cost across the two 20-job runs · 0 failures · 30 free validation cases
Modal
ur-perfit-dummy · logger-service 5a12702d · image lane 100% fal, video lane 100% veo, 0 provider failures