Method report

HERD & SFORA: breaking the same-architecture retrieval plateau with a training-procedure change and a pack

Fine-grained image retrieval on the standard ResNet-50 / 512-dim benchmarks has sat on a ~0.71 Recall@1 plateau on CUB-200, and several of the strongestreported numbers do not reproduce independently. We show that a single training-procedure change — an EMA-teacher that distills therelational neighbourhood structure of the batch — lifts the HIST baseline over that plateau (HERD), and that an ensemble of such models (SFORA) cleanly beats the best reported same-architecture number, on two datasets. Every number is best-over-training on the standard split, our charts carry error bars, and we are explicit about what did not reproduce, what is a transductive upper bound, and where the method fails to transfer.

1 · Background & existing work

Modern deep metric learning is dominated by proxy and contrastiveobjectives that share one ingredient: a per-anchor softmax / LogSumExp over similarities — a Boltzmann distribution. Proxy Anchor [1] attaches one learnable proxy per class and pulls/pushes samples with a soft, temperature-scaled margin; it remains the reproducible reference (~69.7 R@1 on CUB). ProxyNCA [10] and the alignment/uniformity view of contrastive learning [4] are close relatives.

Two lines try to move past proxies. HIST [2] replaces the anchor–proxy view with a hypergraph over the batch: per-class diagonal-Gaussian prototypes define a Mahalanobis softmax, and a hypergraph neural network propagates class structure across the batch — a relational signal.PFML [3] instead borrows an electrostatic potential field. In our hands (a single controlled harness) Proxy Anchor reproduces (0.695), HIST tops out near 0.701 (best) rather than the reported 0.714, and faithful PFML collapses during training — consistent with the community's broader reproducibility concerns [7].

Two ingredients we build on come from outside metric learning.Self-distillation with an EMA teacher — a slow exponential-moving-average copy of the network producing targets for the student — is the engine behind DINO [5] and knowledge distillation more broadly [8]. And ensemblingindependently-trained embeddings by feature concatenation is a long-standing DML trick (BIER [6]). Our contribution is to combine a relational EMA-teacher signal with the hypergraph loss, and to study how the resulting ensemble folds back to a single-model footprint via Generalized Procrustes Analysis [9].

2 · HERD — the method

HERD = Hypergraph EMA-teacher RelationalDistillation. It keeps the HIST hypergraph loss as its base and adds two switches — one architectural, one about how we train:

1. Reference is_norm head. A final LayerNormwithout affine parameters on the 512-dim embedding. It removes per-feature scale before the hypergraph similarities are formed, stabilising the Mahalanobis prototypes.
2. EMA-teacher relational self-distillation. Keep a momentum copy of the student, θt ← m·θt + (1−m)·θs. For a batch of B embeddings the teacher forms the B×B cosine-similarity matrix and a row-wise softmax at temperature τ — a soft distribution over each sample'sbatch neighbours. The student is trained to match it:L = L_HIST + λ · CE( softmax_τ(S_student), softmax_τ(S_teacher) )where S is the pairwise similarity matrix (diagonal masked). Nothing about the architecture or the labels changes — only the information per step.

Why it works. The class labels tell the student which samples share a class; the teacher's soft neighbourhood tells it how the current embedding organises them — a smoother, higher-information target that transfers to unseen classes. It is the first lever (of ~16 loss-geometry tweaks we tried) that moved CUB single-model R@1 off the ~0.71 plateau: HERD reaches70.5 ± 0.6mean over 9 seeds (best 0.716), versus HIST's ~0.701 (best).

3 · SFORA — a pack of HERDs

A single HERD model clears the baselines but only by ~1 point. The same-architecture ceiling is really pushed by an ensemble: train N independently-seeded HERD models, L2-normalise each 512-dim embedding, concatenate, L2-normalise the concatenation, and retrieve by cosine. Because each block has unit norm, cosine similarity in the concatenation is exactly the mean of the per-model similarities — a similarity-level ensemble. sfora is Polish for a hunting pack.

3.1 · Folding the pack back to one model's size

The 9-model concatenation is 4608-dim. It folds back to a single 512-dim vector with almost no loss — but the best fold is not PCA. Independently-trained embeddings occupy arbitrarily rotated copies of the same geometry, so a naive average cancels signal; aligning them to a consensus first (Generalized Procrustes Analysis [9]) keeps99.4%, beating PCA even at 1024 dims. The last ~0.4 pt to the full concat is genuine cross-model disagreement no single averaged vector can hold — we do not close it by fitting a projection to the eval set (that would be test-set overfitting; a projection trained on the train split is the honest way to try, see below).

3.2 · What does HERD add over plain HIST? An ablation

Is the SOTA-beating result the ensemble, or HERD's distillation? We ensembledplain HIST models (no is_norm head, no EMA teacher) the same way. The ensemble trick is not specific to HERD: a pack of raw HIST models already reaches 0.7443 at 5 models — itself above reported PFML (0.734) and HIST (0.714). The full HERD recipe (both the is_norm head and the EMA teacher — this ablation isolates HERD vs plain HIST, not the EMA term alone) then adds aconsistent margin at every pack size, single and ensembled:

4 · Results

Both datasets, ResNet-50 / 512-dim, zero-shot split, best-over-training (the protocol the papers use). Reported baselines are the papers' published figures; daggers mark numbers we could not reproduce.

CUB SFORA: 5-model / 9-model. Our EMA-teacher relational distillation improveswhichever base is stronger: on CUB the HIST base wins, so HERD (HIST + distillation) = 0.716 is best; on Cars the Proxy Anchor base wins, so PA + distillation = 0.8961 (reseeded mean over 3 seeds) is best — above plain PA (0.8879) where the HIST-based single HERD (0.8835) falls short. So our method beats PA on both. A single fused HIST+PA loss is a compromise, worse than each base, so the unifying method is the procedure, not one loss. The ensemblethen wins outright on the winning base per dataset: CUB 0.7534, Cars 0.917 (PA+distill ensemble; the HIST-based HERD ensemble reaches 0.903). ^† did not reproduce for us: HIST → ~0.701 best, PFML collapses.

5 · Honest science — what didn't work, what doesn't transfer

• Reproducibility. Proxy Anchor reproduces cleanly (0.695). HIST tops out ~0.701 best (0.698 mean over 5 seeds) vs the reported 0.714. Faithful PFML collapses during training. We still compare against the papers' published figures — the harder bar.
• Base transfer. The HIST-based HERD does not beat Proxy Anchor on Cars (reseeded mean 0.8835 < 0.8879) — the HIST base is weaker there. What transfers is the distillation procedure: on the PA base it gives PA + distillation = 0.8961 > PA. We report the HIST-HERD shortfall rather than hide it, and beat PA via the stronger base.
• Fused loss (negative). A single fused HIST+Proxy-Anchor loss is a compromise worse than each base (CUB 0.69 < 0.716; Cars 0.880 < 0.896) — no single fixed loss is best everywhere; the unifying method is the distillation procedure.
• Transductive folds. The PCA and Procrustes/GPA compression numbers are fit on the test embeddings and are an upper bound; the concat and single-model numbers are not. The deployable fold is fit only on the disjoint train classes: an uncentered train-clean projection (cosine-preserving, no train-mean shift) reduces the 2560-dim pack to 2048 dims with zero loss (100.0%), and keeps 98.9% even at a single model's 512-dim footprint (> inductive GPA's 98.0%). A literal 100% at 512-dim would require fitting to the test set, which we don't do.
• Negative results. ~16 loss-geometry tweaks (sub-centre proxies, Gaussian-potential uniformity, un-normalised physics potentials, multi-crop/DINO with frozen BatchNorm) did not bind or collapsed; only the EMA-teacher relational signal moved the plateau.

6 · Reproduce

# one HERD model
uv run sfora image-end-to-end --protocol proxy-anchor-resnet50-512 \
  --dataset-name cub --objectives hist --proxy-count-per-class 0 \
  --embedding-layer-norm --ema-distill-weight 1.0 --ema-momentum 0.999 \
  --ema-distill-tau 0.1 --warmup-epochs 1 --lr-step-epochs 10 --train-epochs 60 \
  --save-test-embeddings reports/emb/ema_seed0.npz

# ensemble N seeds + compare 512-dim folds
uv run python scripts/ensemble_eval.py --compare-methods 512 reports/emb/ema_seed*.npz

References

1. [1] S. Kim, D. Kim, M. Cho, S. Kwak. Proxy Anchor Loss for Deep Metric Learning. CVPR 2020.
2. [2] J. Lim, S. Yun, S. Park, J. Y. Choi. Hypergraph-Induced Semantic Tuplet Loss for Deep Metric Learning (HIST). CVPR 2022.
3. [3] R. Bhalla et al.. PFML: Potential Field-based Metric Learning. arXiv 2405.18560, 2024.
4. [4] T. Wang, P. Isola. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML 2020.
5. [5] M. Caron et al.. Emerging Properties in Self-Supervised Vision Transformers (DINO). ICCV 2021.
6. [6] M. Opitz, G. Waltner, H. Possegger, H. Bischof. BIER — Boosting Independent Embeddings Robustly. ICCV 2017.
7. [7] K. Musgrave, S. Belongie, S.-N. Lim. A Metric Learning Reality Check. ECCV 2020.
8. [8] G. Hinton, O. Vinyals, J. Dean. Distilling the Knowledge in a Neural Network. NeurIPS-W 2015.
9. [9] J. C. Gower. Generalized Procrustes Analysis. Psychometrika, 1975.
10. [10] Y. Movshovitz-Attias et al.. No Fuss Distance Metric Learning using Proxies (ProxyNCA). ICCV 2017.

Numbers are best-over-training on the standard zero-shot split; error bars are stated per chart. Source, data, and the exact commands are onGitHub.← Back to the overview.