1 · Hypergraph base
HIST learns per-class Gaussian prototypes and a hypergraph neural network over each batch — a genuinely relational loss, not just anchor–positive–negative triplets.
sfora · Polish, a hound pack
A pack of small models
that out-hunts the state of the art.
SFORA is a deep-metric-learning library. Its method, HERD — Hypergraph EMA-teacher Relational Distillation — plus a multi-model ensemble beat the reported same-architecture SOTA on CUB-200 by more than one point.
75.34SFORA · CUB-200 R@1 · best (9-model)
vs
73.4best reported (PFML)
+1.9
…and it's not just CUB: our EMA-teacher distillation improves any base, soit beats Proxy Anchor on every dataset →
The result
Beating the best reported number — honestly.
CUB-200, ResNet-50 / 512-dim, zero-shot split, cosine Recall@1, best-over-training (the same protocol the papers use). The three baseline rows are the papers' reported figures — including ones we could not reproduce. We compare against the highestreported bar (the harder target), and clear it.
| Method | R@1 | In our hands |
|---|---|---|
| Proxy Anchor | 69.7 | ✓ reproduces (we get 0.695) |
| HIST | 71.4† | ✗ we get ~70.1 |
| PFML | 73.4† | ✗ collapses in our runs |
| HERD — single model | 71.6 | our run · best/mean 71.6/70.5 |
| SFORA — 5-model ensemble | 74.68 | our run |
| SFORA — 9-model ensemble | 75.34 | our run |
† Reported number we could not reproduce with the same ResNet-50/512 backbone: our faithful PFML runs collapse during training, and HIST tops out near 70.1 rather than the reported 71.4. Proxy Anchor, by contrast, reproduces cleanly (0.695). We deliberately still compare against the papers' publishedfigures — the harder bar — not our lower reproductions.
0.73352
0.73943
0.74264
0.74685
0.75297
0.75349
reported SOTA · PFML 0.734
models in the SFORA pack → · bars start at R@1 0.72
A pack, without the storage cost of a pack.The 9-model pack concatenates to 4608 dimensions — but you can fold it back to a single 512-dim vector and keep almost all of it. The best fold isn't PCA: generalized-Procrustes alignment (rotate every model into a shared frame, then average) keeps 0.749 — 99.4% of the pack, beating PCA even at double the dimensions, still +1.5 over reported PFML, at single-model size. That 99.4% is a transductive upper bound (the alignment uses the test geometry); a deployable fold fit only on the disjoint train classes still recovers 98.0%. The last ~0.4pt is genuine cross-model disagreement; we do not close it by fitting to the eval set.See every compression method →
Two datasets
Does it transfer? Yes — our distillation beats PA on both.
HERD's real contribution isn't the HIST loss — it's the EMA-teacher relational distillation, a training procedure that improvesany base loss. HIST is the stronger base on CUB, Proxy Anchor on Cars — and best-base + our distillation beats plain PA on both. R@1, ResNet-50 / 512-dim, zero-shot split.
| Method | CUB-200 | Cars196 |
|---|---|---|
| Proxy Anchor (our run) | 69.5 | 88.8 |
| HIST (our run) | 70.0 | 87.1 |
| best base + our distillation | 71.6 | 89.6 |
| SFORA — ensemble (ours) | 74.68 | 91.7 |
All numbers are our in-harness runs (same code, same protocol), so the comparison is fair. Adding our distillation lifts every base onevery dataset: on CUB, HIST 70.0→71.6 (that is HERD); on Cars, HIST 87.1→88.4 and PA 88.8→89.6 (it also improves PA on CUB in a matched-recipe run). The "best base + our distillation" row is the winner per dataset — HERD (HIST + distillation) on CUB, PA + distillation (0.8961, reseeded mean over 3 seeds) on Cars — and both beat plain Proxy Anchor. No single fixed loss wins both (a fused HIST+PA model is a compromise, worse than each base); the unifying method is the procedure. The SFORA ensemble then stacks on top of the winning base per dataset: 74.68 on CUB (HERD ensemble), 91.7 on Cars (PA+distill ensemble). The HIST-based HERD ensemble reaches 90.3 on Cars — below the PA-based one, since HIST is the weaker Cars base — and is the run visualized cluster-by-cluster in the Cars separation viz →.
Explore the methods
How it behaves — curves, not just a headline.
One representative run per method (the benchmark numbers are the best across several seeds, so they sit a touch higher). Real runs, CUB-200 test retrieval, ResNet-50 / 512-dim.
- HERD70.9best @ ep 55
- Proxy Anchor68.0best @ ep 30
- HIST70.2best @ ep 35
Mean R@1 over random N-model packs; error bars are ±1 std (±0.56 at 1 model → ±0.16 at 7). The curve bends — the first few seeds buy the most, and get steadier.
bars start at R@1 0.70 (not 0); whiskers are the test-set standard error ±√(p(1−p)/N). Read the exact value on the right.
Alignment is the trick: a naive average (0.727) barely beats one model, but generalized-Procrustes alignment (iteratively rotate every model to a shared consensus, then average) reaches 0.749 — 99.4% of the full 4608-dim pack(0.753), in a single model's footprint with no concatenation, and beats PCA even at 1024 dims (0.744). The last ~0.4pt is genuine cross-model disagreement no single 512-vector can hold — we do not close it by fitting a projection to the eval set. Reproduce: ensemble_eval.py --compare-methods 512.
Note: PCA axes and the alignment rotations use only the embeddings' geometry (no labels/retrieval), but are computed on the test embeddings, so the aligned/PCA rows are a transductive upper bound; concat, random projection, naive mean and single model use no fitting at all.
How it separates
The same birds, five different spaces.
Eight held-out CUB-200 test classes — species the model never saw in training — the same 360 photos in every panel, projected to 2D with t-SNE. Each dot is one bird. Hover or tap a dot to see the photo; the same bird lights up across all five spaces.
before training
Frozen ResNet-50
prior baseline
Proxy Anchor
HERD's base loss
HIST
ours — single
HERD
ours — 9-model pack
SFORA (9 models)
Frozen ImageNet features already know something about birds, but the species smear together. Every trained objective — Proxy Anchor, HIST, HERD — pulls them into tight clusters, and on eight easy classes they look much alike: the 8-class silhouette is noisy and doesn't even rank them (it can put Proxy Anchor above HERD). The honest signal is retrieval over all 100 classes, bottom-left of each panel: 0.568 → 0.680 → 0.697 → 0.709 → 0.753 as you move frozen → Proxy Anchor → HIST → HERD → the 9-model SFORA pack. We show the silhouette tie on purpose — no cherry-picked angle.
Same viz, another dataset
The same cars, three different spaces.
Cars196, eight held-out test classes — car models the network never saw in training. The same 360 photos in every panel. On Cars the HIST-based HERD trails Proxy Anchor — but our distillation on the PA base (PA + distillation) beats it.
prior baseline
Proxy Anchor
ours — single
HERD
ours — 3-model pack
SFORA (3 models)
An honest contrast with CUB: HIST is the weaker base on Cars, so the HIST-based HERD single model (mean 0.8835) sits just below our own Proxy Anchor run (0.8879). But the distillation procedure still wins here — put it on the stronger base and PA + distillation reaches 0.8961 (reseeded mean), above plain PA. And the 3-model SFORA pack reaches 0.903. So our method beats PA on Cars too — via the PA base, not the HIST one.
In plain English
The whole idea, explained like you're five.
No jargon. (The jargon has its own glossary further down.)
01
The task
Show the model a photo of a bird. It should turn that photo into a list of numbers (an embedding) such that two photos of thesame species get nearby numbers, and different species getfar-apart numbers. Then "find similar images" is just "find the nearest numbers." The catch: it's tested on species it never saw in training — so it must learn general bird-ness, not memorize.
02
What everyone did before
Prior methods (Proxy Anchor, HIST, PFML) are all clever ways of tugging the numbers around: pull same-species together, push others apart, using one learnable "anchor" per species, or a graph of relationships. In our hands, none of these reproduciblypush past ~71% on this benchmark — and the best reported numbers (like PFML's 73.4) turned out not to reproduce at all.
03
What HERD adds our idea
Keep a slow-motion copy of the model — the EMA teacher. It changes only a tiny bit each step, so it's calmer and more consistent than the fast-learning student. Each step, the teacher looks at a batch and says "for this photo, here's which other photos are its neighbours, and how strongly." The student is trained to agree with the teacher's neighbourhood map. Copying relationships (who's near whom) — not just labels — is what teaches the model something that transfers to unseen species. That change lifts a single model off the plateau (0.716 best), and ensembling several of them is what beats reported SOTA.
04
The pack
Train several of these models from different random starts — apack (that's what sfora means). Glue their number-lists together and you get a stronger combined judgement than any single model, because they make different mistakes. And you can squeeze that combined list back down to the size of one model while keeping 99.4% of the pack's score (0.749 of 0.753) — by rotating the models into a shared frame first, then averaging.
Net: previous methods rearranged one model's view of onecrop. HERD changes what information the model gets each step (a teacher's neighbourhood hints), and a pack of them — compressed — beats the best number anyone has reported.
The method
HERD Hypergraph · EMA-teacher · Relational · Distillation
One idea nudged the single-model plateau that ~16 loss-geometry tweaks could not: change the information per training step, not the loss surface. The ensemble then does the decisive SOTA-beating work.
2 · EMA-teacher distillation the new idea
A slow momentum copy of the model produces a soft distribution over each anchor's batch neighbors. The student matches it. Distilling relational structure — not labels — is what transfers to unseen classes.
3 · A pack, not a hero
Independently-seeded HERD models are ensembled by concatenating their L2-normalized embeddings. Diversity buys several points — feature-concatenation ensembling is an established DML paradigm (BIER and related boosted-embedding methods).
Reproduce
Train the pack, then ensemble it.
train the pack & ensemble it
# 1 · train the 9-model pack (each seed saves its best embeddings)
for S in 0 1 2 3 4 5 6 7 8; do
uv run --extra research sfora image-end-to-end \
--protocol proxy-anchor-resnet50-512 --dataset-name cub \
--objectives hist --embedding-layer-norm \
--ema-distill-weight 1.0 --ema-momentum 0.999 \
--samples-per-class 8 --hist-lr-ds 0.03 --train-epochs 60 \
--eval-test-interval-epochs 5 --seed $S \
--save-test-embeddings reports/emb/ema_seed$S.npz
done
# 2 · feature-concatenation ensemble → the headline Recall@1
uv run python scripts/ensemble_eval.py reports/emb/ema_seed*.npz
#=> ENSEMBLE of 9 models: R@1=0.7534 (just 5 already clears SOTA at 0.7468)Honest science
What reproduced, what didn't, and what failed.
The reported SOTA numbers on this benchmark are optimistic. We say so plainly.
Reported vs. what we actually got
| Method | Reported | Ours | Status |
|---|---|---|---|
| Proxy Anchor | 69.7 | 69.5 | ✓ reproduces |
| HIST | 71.4 | 70.1 | ✗ not reproduced |
| PFML | 73.4 | collapses | ✗ not reproduced |
Proxy Anchor reproduces faithfully, so the harness is sound. HIST's 71.4 only reaches ~70.1 (matching the ML-Reproducibility-Challenge 2023), and faithful PFML collapses. We still beat the best reported number (PFML 73.4) — while being clear it does not independently reproduce.
Approaches that did not work
- Sub-center proxies — fragmenting a class into modes hurts zero-shot transfer (0.675).
- Un-normalized physics potentials — electrostatic / long-range potentials collapse without a partition-function softmax.
- DINO multi-crop — incompatible with frozen-BatchNorm metric learning: small crops hit ImageNet-224 BN stats, go out-of-distribution, and collapse training.
- Bigger ImageNet-V2 weights, 100-epoch schedules, and HIST re-tuning all under-performed plain HERD.
Nomenclature
Every term on this page, defined.
- Embedding
- The list of numbers (a vector) a model turns an image into. Similar images should get nearby vectors.
- Recall@1 (R@1)
- The headline score: the % of test images whose single nearest neighbour is the same class. Higher is better.
- Zero-shot split
- Train on one set of classes, test on a completely different set. Measures generalization, not memorization.
- Same-arch
- Same backbone (ResNet-50) and embedding size (512-dim) for every method, so the comparison is fair.
- Best-over-training
- Report the peak test score seen during training — the protocol every paper here uses.
- EMA teacher
- Exponential Moving Average: a slow copy of the model updated as θt ← m·θt + (1−m)·θs. Calmer, more consistent targets.
- Distillation
- Training one model (the student) to match another's (the teacher's) outputs.
- Relational distillation
- Matching who-is-near-whom in a batch (a neighbourhood distribution) rather than hard class labels.
- Hypergraph loss (HIST)
- The base loss HERD builds on: per-class Gaussian prototypes plus a graph over the batch. CVPR 2022.
- Proxy Anchor / PFML
- Prior metric-learning losses (a per-class anchor; a physics-style potential field). The baselines we compare against.
- Ensemble
- Combining several independently-trained models — here by concatenating their (L2-normalized) embeddings.
- Hypersphere
- Because embeddings are L2-normalized to length 1, they live on the surface of a sphere — that's what the hero visual shows.
References
Standing on the pack's shoulders.
- 01Proxy Anchor Loss for Deep Metric Learning
- 02Hypergraph-Induced Semantic Tuplet Loss (HIST)
- 03Potential Field-based Metric Learning (PFML)
- 04Emerging Properties in Self-Supervised ViTs (DINO)
- 05Understanding Contrastive Learning through Alignment and Uniformity
- 06BIER — Boosting Independent Embeddings Robustly