Ternlight – 7 MB embedding model that runs in browser (WASM)

Hobby project, I wanted to "ship a useful model in a web browser". so I distilled a small sentence encoder from MiniLM with ternary quantization-aware training. Also wrote the inference engine from scratch and shipped in Rust → WASM SIMD.

It's an embeddings model, not an LLM: text goes in, a 384-dim vector comes out, and cosine similarity between two vectors tells you how related the texts are — regardless of shared words ("reset my password" ↔ "I forgot my password" → 0.88). Used for semantic search, FAQ/intent matching, and clustering. Running it on-device means search-as-you-type semantic search is performant with no API dependencies.

Demo (2k React docs, fully on-device): https://ternlight-demo.vercel.app

Two tiers on npm: - @ternlight/base (7 MB, ~5 ms/embed, more capable embedings) - @ternlight/mini (5 MB wire, ~2.5 ms/embed).

Bundled for Node and browsers.

Repo - see technical details (MIT, training pipeline included): https://github.com/soycaporal/ternlight

Curious if this is something useful, what are the use cases for on-device embeddings.

FWIW -- Granite r2 small is a 30M model, still small enough to run on CPU, and a good baseline for fine tunes.

This is cool!

but also maybe you could put a button on the landing page to trigger the demo because it's a bit startling to hear my fans go crazy when opening a webpage.

Thank you for this! Local models will bring privacy at some point, and I already know an excellent use case for such a small embedding model (cheap and fast search in a product base). Relying on the CPU is also a plus in my case.

I added an offline search engine to app.wazzup.im/search (no login or payment required).

First search downloads the model from the internet and subsequent runs are from the cache.

The model is very small so it's not the best for everything but it's good for basic math and coding.

Give it a try.

Can the 30 second embedding time be done beforehand and sent to the browser?

Inference is nice and quick after that.

Great, now my websites are gonna push entire LLMs onto my browser in order to use my CPU to make inferences about my shopping habits or whatever.

Interesting project. Happy to see someone who shares an interest in tiny vector embeddings models. I've worked on tiny (1MB - 4MB, 250K - 950K parameters) embeddings models called BERT Hash https://huggingface.co/blog/NeuML/bert-hash-embeddings

Keep up the great work!

That's really impressive, congratulations. It's nice to see novel applications of browser models.

Why do these things download into the browser automatically? This could be used to distribute malware and also or hog excessive browser memory.

Very cool! I'd love to point it at my own corpus to index/embed. Would be cool if you could give it a link to a markdown file or even a website to crawl.

What we need is a W3C LLM API like the one Chrome already offers: https://developer.chrome.com/docs/ai/built-in

[flagged]

[dead]