Tree-sitter vs. Language Servers

(Hi, I’m on the rust-analyzer team, but I’ve been less active for reasons that are clear in my bio.)

> Language servers are powerful because they can hook into the language’s runtime and compiler toolchain to get semantically correct answers to user queries. For example, suppose you have two versions of a pop function, one imported from a stack library, and another from a heap library. If you use a tool like the dumb-jump package in Emacs and you use it to jump to the definition for a call to pop, it might get confused as to where to go because it’s not sure what module is in scope at the point. A language server, on the other hand, should have access to this information and would not get confused.

You are correct that a language server will generally provide correct navigation/autocomplete, but a language server doesn’t necessarily need to hook into an existing compiler: a language server might be a latency-sensitive re-implementation of an existing compiler toolchain (rust-analyzer is the one I’m most familiar with, but the recent crop of new language servers tend to take this direction if the language’s compiler isn’t query-oriented).

> It is possible to use the language server for syntax highlighting. I am not aware of any particularly strong reasons why one would want to (or not want to) do this.

Since I spend a lot of time writing Rust, I’ll use Rust as an example: you can highlight a binding if it’s mutable or style an enum/struct differently. It’s one of those small things that makes a big impact once you get used to it: editors without semantic syntax highlighting (as it is called in the LSP specification) feel like they’re naked to me.

Side note, but thanks for the note about not using AI to write your articles. I'm tired of looking for information online, finding an article that may answer it, and not being sure about the author's integrity (this is so rampant on Medium).

>It is possible to use the language server for syntax highlighting. I am not aware of any particularly strong reasons why one would want to (or not want to) do this.

Hmm, the strong reason could be latency and layout stability. Tree-sitter parses on the main thread (or a close worker) typically in sub-ms timeframes, ensuring that syntax coloring is synchronous with keystrokes. LSP semantic tokens are asynchronous by design. If you rely solely on LSP for highlighting, you introduce a flash of unstyled content or color-shifting artifacts every time you type, because the round-trip to the server (even a local one) and the subsequent re-tokenization takes longer than the frame budget.

The ideal hygiene could be something like -> tree-sitter provides the high-speed lexical coloring (keywords, punctuation, basic structure) instantly and LSP paints the semantic modifiers (interfaces vs classes, mutable vs const) asynchronously like 200ms later. Relying on LSP for the base layer makes the editor feel sluggish.

I love tree-sitter+eglot but a few of the languages/schemes I work in, simply don't have parsers:

    > pacman -Ssq tree-sitter
    tree-sitter
    tree-sitter-bash
    tree-sitter-c
    tree-sitter-cli
    tree-sitter-javascript
    tree-sitter-lua
    tree-sitter-markdown
    tree-sitter-python
    tree-sitter-query
    tree-sitter-rust
    tree-sitter-vim
    tree-sitter-vimdoc

This is like the difference between an orange and fruit juice. You can squeeze an orange to extract its juices, but that is not the only thing you can do with it, nor is it the only way to make fruit juice.

I use tree-sitter for developing a custom programming language, you still need an extra step to get from CST to AST, but the overall DevEx is much quicker that hand-rolling the parser.

I thought that the blog's domain looked familiar. The author maintains an awesome and well maintained Emacs starter kit https://codeberg.org/ashton314/emacs-bedrock

one topic not mentioned is creating refactoring tools. My sense is that LSPs generally have the advantage here because they have the full parsed tree, but I suspect it would be possible to build simple syntactic refactorings in TS with the potential to be both faster and less sensitive to broken syntax.

Tree-sitter is great. It powers Combobulate in Emacs. Structured editing and movement would not have been easily done without it.

- as a guy who is absolutely not familiar with the idea of how code editors work and has to make a browser based code editor, what are the things that you think I should know?

- i got a hint of language server and tree sitter thanks to this wonderfully written post but it is still missing a lot of details like how does the protocol actually look like, what does a standard language server or tree sitter implementation looks like

- what are the other building blocks?

I'm doing a project with tree sitter right now

Any tips for keeping the grammar sizes under control? I'm distributing a CLI tool that needs to support several languages, and I can see the grammars gradually bloating the binary size

I could build some clever thing where language packs are opt-in and distributed as WASM, maybe. But that could be complex

I think the big reason to put syntax highlighting in the language server is you have more info, ex you can highlight symbols imported from a different file in one color for integers and a different for functions

Tree-sitter does incremental parsing which will speed up the Language Server having to otherwise re-parse an entire file.