I faced this problem when designing my own notation [1].

Solved it by surrounding code with more ticks than maximum number of consecutive ticks inside its text. This allows arbitrary nesting.

Postgres solves it by using `$something$ whatever $something$` [2].

[1] https://github.com/PratikDeoghare/brashtag [2] https://www.postgresql.org/docs/current/sql-syntax-lexical.h...