They were upgradable but the Apple hard disk utility would only format Apple drives. There were several third party utilities and they were usually bundled when buying e.g. Quantum drives, but if you took a drive from elsewhere you'd have to have a copy of the utility.

If there was custom firmware, I think it was just the model ID or some other license key system.