I never played Thief, so I can't compare two. I was really impressed w/ System Shock 2.
re: the graphics - I don't remember being particularly underwhelmed by the graphics, but I didn't play many contemporary titles to compare it to. I was drawn-in by the System Shock franchise, but otherwise didn't really game.
For me the horror element came from the voice acting on the audio journals, the voices of "the many", and those goddamned cyborg assassins.
The weird graphics is mostly because they used extreme low poly models with low res textures compared to contemporary titles. This is a bigger complaint with the relatively late System Shock 2, which was just a few months apart from e.g. Deus Ex.
Compare:
SS2 unmodded: https://imgur.com/psb30TT
Deus Ex unmodded: https://imgur.com/TWjoNSV
Obviously neither game is a marvel of graphical fidelity, but SS2 has these distorted proportions that make the characters look like inflatable sex dolls that were stored in the same supply closet as the Necronomicon and now they're all weird and whispering of incomprehensible horrors. Like the polygons don't really correspond to the anatomy of the character at all, which is something that contemporary games were beginning to figure out.
I really like SS2 (and Thief 1 & 2, and Deus Ex for that matter), and I don't really mind them not looking that great. But man they did not look that great by late 90s early 00s standards..