It's just Wolfram explaining how he likes stuying things that can be describe by simple rules and how complexity can emerge in spite of (or because of?) the seeming simplicity of those rules. He came up with a word for it, and while I think "ruliology" sounds a bit silly, it does say what's on the tin.
To me it sounds like this stuff:
https://en.wikipedia.org/wiki/Chomsky_hierarchy
But maybe it is more like fractals and emerging complex systems?
The word he's looking for is "formal system".
For some reason he doesn't like doing mathematical demonstrations so he shuns the practice of doing them, and invented a new word to describe that way of using formal systems.
https://en.wikipedia.org/wiki/Formal_system