Matthew Kim
← All projects

Historic Fashion Archive — Research Assistant & Full-Stack Build

Related-item quality 0.70 → 0.93 nDCG@5, blind-judged" (recommended lead)

A long black gallery lit by rows of ceiling spots, with scanned garments standing along a raised runway into the distance
The 3D museum: a walkable gallery, one wing per era, with garments rendered from real photogrammetry scans standing along the runway. Drag to look, click the floor or use WASD to walk, click a garment to inspect it.

drag to turn · scroll to zoom

A garment scan from the capture pipeline, simplified to flat tones for the portfolio’s workshop. The Archive’s own viewer renders the full-detail scans.
Role
Research Assistant, UVA Historic Fashion Archive (2024–present)
Stack
Next.js 16 (App Router), React 19, TypeScript 5, Tailwind CSS v4; Three.js / @react-three/fiber / @react-three/drei v10 for the 3D viewer
Backend
CollectiveAccess (Providence), custom profile on the Costume Core metadata standard; cookie + Basic Auth integration, 5-minute in-memory cache with static-JSON fallback
Scope
both the backend metadata schema and configuration and the public-facing museum site, end to end
Recommendations (learned)
a re-ranker over FashionCLIP image embeddings and extracted attributes replaced the original fixed-weight score (era 30%, type 25%, color 20%, material 15%, decade 10%). nDCG@5 rose from 0.70 to 0.93 (paired 95% CI +0.15 to +0.31) on 50 query garments with 731 blind relevance judgments, under nested cross-validation. Gains were largest for accessories (0.50 to 0.91), where one catch-all "accessory" type had made the old score close to random
Search (deployed)
keyword (BM25) and FashionCLIP text-to-image retrieval fused by reciprocal rank fusion, served by a CPU-only FastAPI service on a VPS behind a Cloudflare Tunnel (about 100 ms per query). nDCG@5 0.80 vs 0.68 for keyword search on 40 blind-judged queries; a local-LLM version (Qwen3-8B parsing queries into filters, with grounded, citation-checked answers) reached 0.88 but needs a GPU
Attributes
material, color and technique extracted from 1,400+ catalog descriptions (F1 0.95-0.99 against 200 hand-labeled records), and predicted from photos for undescribed garments by a linear probe trained on those weak labels (color F1 0.58 vs 0.39 zero-shot)
Evaluation practice
dev/test splits frozen before tuning, blind pooled judging, an intra-rater check (weighted kappa 0.82-0.87), a blind re-judge when one round of grades turned out non-blind, and GitHub Actions regression gates on a committed fixture
ML stack
Python, PyTorch, OpenCLIP (Marqo FashionCLIP), scikit-learn, FastAPI, Ollama, MLflow
3D viewer
garments rendered from real photogrammetry scans (see garment-capture-pipeline.md); per-scan front-facing calibration; the home page is a walkable 3D museum with a wing per era (pre-1920 onward), first-person drag/WASD/click-to-walk controls and click-to-inspect garments
Data pipeline
filtering, search, analytics tracking, JSON/CSV/PDF export for the collection
Security work
HMAC-signed sessions, rate limiting, CSP headers, timing-safe auth checks, hardened from an initial shared-secret header into a real session system
Live site
uvahcc.org

The problem

Before this, the Archive's public-facing presence was 2D photography and text metadata — you could see a photo of a garment and read its catalog entry, but you couldn't actually examine it: turn it, compare its silhouette to another piece, or get a sense of how it moved or draped. That's a real limit for a fashion archive, where shape and construction are the point. I built this to close that gap: a 3D viewer built from real photogrammetry scans, a recommendation engine that surfaces related garments by actual visual and material attributes instead of shared tags, and a backend metadata schema to support all of it.

What I'm proudest of

What I'm proudest of is that this wasn't a frontend skin on someone else's backend — I designed the CollectiveAccess metadata schema, built the auth and caching layer, and built the public site on top of it, so I own the system end to end. It's also the first time I've worked on software inside a research context rather than a typical CRUD app, which changed how I thought about the problem: the "users" are curators and researchers who need the data model to reflect real museum cataloging practice (the Costume Core standard), not just whatever fields were convenient to build.

How I knew it worked

The honest answer is that I didn't trust any number until I'd tried to break it. There was no click data to learn from, so I built the ground truth myself: I hand-labeled 200 catalog descriptions, then judged 731 garment pairs blind, with candidates from every system pooled and shuffled so I couldn't tell which one had suggested what. Most of what I learned came from catching my own mistakes. My first text extractor scored almost perfectly, and it took a while to see why: I'd been cleaning the labels with a keyword checker, so the labels had quietly drifted toward exactly what keywords see. I froze a test set, stopped using the checker on it, and the real score turned out lower. When the learned re-ranker beat everything, I realized I'd graded its new picks knowing whose they were, so I re-graded them blind, mixed in with pairs I'd already judged. The result held, and the re-grade doubled as a consistency check on my own judging. The finding I didn't expect was the split: the old weighted score was already good for clothing and nearly random for accessories, because the catalog files fans, shoes and hats under one type. Photos fixed that, and the learned model combined both. The search service now runs the same code the benchmark measured, and a regression check fails the build if quality drops.

What's next

The most concrete next step is feeding the garment-capture pipeline's output back into CollectiveAccess — right now the 3D pipeline and the Archive site are connected by shared demo garments, but not yet by a real data pipeline between them. Beyond that, growing the number of garments in the 3D viewer past the initial pilot set is the other clear milestone.

The archive's home page: "Step into the archive." over the empty gallery, with buttons to enter the galleries or browse the collection
The front door: the home page opens on the gallery itself, and "Enter the galleries" hands you the controls.
A garment page: the textured 3D scan of a dress in an orbitable viewer beside its catalog record of era, type and collection
A garment's own page: the photogrammetry scan in the 3D viewer beside its catalog record. This is one of the pilot scans, still filed under a demo record.
Dot plot of nDCG@5 for five related-garment systems: the original fixed weights at 0.70, image similarity 0.77, fixed weights with attributes 0.81, a clothing/accessory switch 0.87, and the learned re-ranker highlighted at 0.93, each with a 95% interval
How often the related garments are right: the learned re-ranker against every simpler system, scored on 50 query garments I judged blind. Whiskers are 95% bootstrap intervals over queries.
Slope chart from clothing to accessory queries: the fixed weights fall from 0.90 to 0.50, image similarity rises from 0.70 to 0.84, and the learned re-ranker stays high, 0.95 to 0.91
The finding I didn't expect: the original score worked for clothing and was close to random for accessories, because the catalog files fans, shoes, hats and purses under one type. The learned model keeps the best of both.
Pipeline diagram: catalog rows and photos become attributes and FashionCLIP embeddings, which feed a learned re-ranker and a search service that serve uvahcc.org, with a box listing the blind evaluation behind every number
From catalog rows to the live site: what runs offline on a GPU, what serves visitors from a CPU-only server, and the blind evaluation behind each number.