The "OS" (or rather "kernel") was actually the VM which was implemented in microcode and BCPL. The Smalltalk code within the image was completely abstracted away from the physical machine. In today's terms it was rather the "userland", not a full OS.