Is embeddings enough to preserve privacy? If I run the encoder/decoder on device and only communicate with server in embeddings?
For a short time maybe, but that kind of activity might look like something interesting enough for someone to target you and remove all privacy from your entire existence.
No, the original text can largely be recovered from embeddings[0] if you know which embedding model was used.
[0] https://arxiv.org/abs/2310.06816