It's built as a full 3D game, and has extensive use of nice shaders for transparency effects like the crystals, water effects, particle effects, and more. This is all pretty standard game development stuff, you'd be able to do this pretty quickly with an off the shelf engine like Godot, Unity, or Unreal. Godot is a good place to start if you want to learn some of this stuff!
However Order of the Sinking Star is also made in a custom programming language and custom engine. This is not at all normal for game development. There also doesn't appear to be a scripting system. Normally you have an engine built in C++ or something fast, and then a scripting system where you write a higher level language to script how the game runs. Engine development is more separate from scripting development. With Order of the Sinking Star, the game engine appears to be much more just like a library, and the same codebase runs the engine and game logic, with no scripting layer, as far as I can figure out from public hints.
I am a little curious if there any interplays between Global Illumination, rastering and shaders that they are exploiting on account of having complete engine control. Techniques that something special about their engine might be allowing.
I only wonder because I recall the reason they built an engine from scratch for The Witness was to achieve certain lighting goals, and that specific feel and style I have never encountered quite like that in any other game.
A lot of that is down to art and direction, but there's a raw quality of the lighting that just feels different to any game I have played. You can often feel a game's engine out if it's one of the big ones, through the feeling of its lighting. I suspect since all engines have certain default approaches and settings in their lighting setups and devs only move the knobs as far as they need to meet their artistic goals, leaving much alone.
This game is also on consoles, can't imagine how much time it took to port the engine to all the proprietary SDKs
Speaking of the custom programming language (Jai), thought it was supposed to be released (beta goes public) with the game.
For how much Blow complains about modern software performance and bloat I was disappointed to hear a lot of reports that the performance is apparently bad, even for a type of game popular in the 8-bit era
Yeah, Blow has a generally unfavorable opinion of scripting in game engines. Jai (the language) gets around that by being very fast to compile debug builds, about 150 kloc/s, which he says he intends to optimize up to 1 mloc/s. Order of the sinking star compiles in about 2 seconds for a clean build, so iteration can be basically as fast as what you'd expect from scripting.