alt Hacker NewsEDSAC as early as 1949 had an "initial orders" module. A boot-up ROM. There was a bank of rotary selector switches, to set the octal digits in each ROM word.
David Wheeler (of subroutine fame) figured out the most sensible thing to put in the tiny ROM was a paper tape loader and mini-assembler that loaded the rest of itself from paper tape. So at switch-on the machine would start reading the tape input and run. Programs could be written on the teletype with mnemonics and relative addresses in octal.
It seems a little strange that more advanced later machines like the original PDP-11 from 1970, couldn't "autostart" from ROM like that. You had to toggle in a bootloader.
I think that was because core memory was non-volatile. (EDSAC didn't use core; it had a refreshing DRAM-like delay line memory.) With core, you only had to toggle the bootloader into once and, provided your software didn't accidentally overwrite it, it was still there after a power cycle. So re-toggling in the boot loader was not really that common, despite how much lore surrounds it.
DEC supplied optional boot rom cards for UNIBUS, starting with the BM792, which arrived as a 32 x 16 matrix of diodes. You had to remove some of the diodes by hand to set the boot code.
Later cards used PROMs, and were pretty fancy. The M9312 had 512 words and included some basic CPU diagnostics. Different variants handled boot from disk, paper tape, console, etc, and you had to specify the options you wanted when you bought the card.