Functional Alignment Can Mislead: Examining Model Stitching | arXiv
Appreciation
--
Importance
--
Date Added
8.24.26
TLDR
Successful model stitching shows functional compatibility, not that two models encode the same information or share aligned representations. They show this by constructing cases where the upstream model could not plausibly contain the same semantic representation as the downstream model.
2 Cents
--
Tags
- Model stitching: take the first part of model A, connect it through a small learned adapter to the second part of model B, and ask whether the combined system still performs the task well
- How? The learned stitch can essentially recode whatever features A provides into inputs that B's later layers can exploit.
- Examples
- A model trained only on birdsong spectrograms scores 6.59% on a 10-class ImageNet task by itself, but reaches 88.40% once a linear stitch feeds it an ImageNet model's representations (higher than its own 77.98% on birdsong)
- Note this birdsong model has never seen a natural image and cannot have encoded visual object structure
- Likewise for controlled experiments on MNIST (e.g., model A only classifies colors, model B only classifies B/W digits).
- In response to stitched accuracy ≥ model B’s baseline being the benchmark for “compatibility”:
- Swap model A for 1000 clusters of pure random noise and a linear stitch still gets model B, a pretrained ImageNet ResNet-50, to 100% on classifying those clusters
- A model trained only on birdsong spectrograms scores 6.59% on a 10-class ImageNet task by itself, but reaches 88.40% once a linear stitch feeds it an ImageNet model's representations (higher than its own 77.98% on birdsong)