(vec2vec) Harnessing the Universal Geometry of Embeddings | arXiv
Appreciation
--
Importance
--
Date Added
6.26.26
TLDR
Given embedding vectors from an unknown text encoder, with no idea what text produced them, you can learn to translate them into the space of an encoder you have full access to, with zero paired examples, well enough to read sensitive content back out. They present the Strong Platonic Representation Hypothesis: not only does a universal embedding space exist, you can leverage it to translate vectors from one space to another. Concretely, map the foreign vectors into a shared latent (in the limit, the platonic representation), pass through a learned transform inside that space, then map back out into a space you have the model for.
2 Cents
Although very interesting, not sure how useful this is besides providing nice empirical result to strengthen PRH. See note below about how you still need access to both models during training stage. Also, cross-modal results are weaker but it’s not clear how much of this is because CLIP is worse at text embedding.
Tags
-
vec2vec pipeline: For a model pair (M1, M2), four small adapters around one shared backbone T. Two input adapters map each native space into the shared latent Z; T transforms within Z; two output adapters map back out. Translation is just compose-through-T: M1 to Z to M2.
-
There are four losses:
- Adversarial: translations look native in the target space, and the two branches' latents are indistinguishable in Z (forces the shared space to actually be shared).
- Reconstruction: a vector into Z and back to its own space recovers itself (keeps adapters invertible).
- Cycle-consistency: translate over and back returns the original (unsupervised replacement since there are no pairs of data points).
- Vector-space preservation: pairwise dot-products survive translation (under PRH, the geometries of the spaces should be retained)
-
Interestingly enough, if we remove any one of these 4 losses performance tanks.
-
NOTE: to train the pipeline you need access to both models.
- Specifically, you query both models to generating a large pile of embeddings (~1M per model) from each. The four adapters and shared transform T are trained jointly against four losses to force a shared latent that preserves geometry (without seeing any matched pairs). So while inference doesn’t need either model (unless you want to use the “known” model to recover text), training assumes you have both models freely accessible.
- Parameter sizes can be found in repo, ~21M params for full translator (~13M for one direction), and discriminators are ~16M more. Adapters are small MLPs (4 linear layers), the shared backbone T is 5 linear layers.
- Took them 1-7 days to train each vec2vec, but with an H100 and caching embeddings you could get each one to a few hours certainly