as far as I can tell, fossil doesn't support that. but I could be wrong.

Not without having a degraded git experience like shallow clones, or using hacks like LFS or Xet, and then you're back at the initial problem of depending on "something else besides your repo".