OpenSolaris was an official OSS version of Solaris which was essentially Solaris, but developed in the open by Sun.

Illumos started as "remove all close source bits and replace with OSS", after Oracle closed down OpenSolaris, Illumos became a full-on fork and Solaris-like rather than another version of Solaris.

From there, multiple distros were born (because Illumos didn't want to be distro), notably OpenIndiana and SmartOS. OpenIndiana being a general purpose distro of Illumos. While SmartOS went with something like "OS for HCI datacenters"

So it's Solaris > OpenSolaris > Illumos.

I believe SmartOS is a distro of Illumos (in the same way that debian is a distro of linux).