You got it. Scala had a shot being an early mover in the JVM functional programming space, but they really shot themselves in the foot with their version transition problems and tooling issues you allude to. Java is probably "good enough" for most shops now, and if you are not bound to the JVM I really don't understand why you would go with Scala today.
for spark
Scala's decline started before Scala 3, which brought its share of breakage (sometimes for dubious reasons, like the new syntax) but also fixed many warts. Tooling has improved a lot lately, but it's too late.
> if you are not bound to the JVM I really don't understand why you would go with Scala today.
Scala's metaprogramming abilities coupled to a powerful type system are still unmatched. Among mainstream languages, only TypeScript gets somewhat close. For your typical service oriented architecture, libraries such as Tapir or ZIO HTTP are pretty nice. I haven't found anything as pleasant in other languages.
That said if an LLM can write 95% of your code today, this point is a bit moot, sadly.