There are lots of non-vector embeddings. “Embedding” just means one mathematical structure stuck inside another in such a way as to preserve its algebraic properties. So for example the real number line is embedded into the Cartesian plane along both axes. That’s a non-vector embedding. I think that field extensions (like the rationals in the reals) also count as an embedding although I may be wrong about that.
In ML, embeddings are used as a way to work with text using the tools of linear algebra. So of course people use (and want to use) vector embeddings, because that’s what gives you what you need in order to “do linear algebra on text”.
Thank you for the details. I was thinking about ML embeddings in particular, but your answer does help me understand the core mathematical nature of an embedding at a deeper level. E.g. the word "embedding" makes more sense now in the context of "one mathematical structure stuck inside another."