You've also got tons of prefixes with opcode dependent rules on what's allowed there, the opcode field itself is variable length (I've seen up to four bytes), you've got instructions that treat that immediate field as additional opcode bytes, etc.
The opcode is 5 maps (8, actually, but only 5 are occupied) of 10-bit opcodes, with the presence or absence of 66/F2/F3 prefixes providing 2 of those bits. If you ignore how the manual describes prefixes and look at it like that (which is suggested by the VEX encoding process), the decoding process becomes a lot simpler. In fact, with one singular exception, this is sufficient information to index into a map to figure out how long the immediate field is and whether or not ModR/M is present.
The opcode is 5 maps (8, actually, but only 5 are occupied) of 10-bit opcodes, with the presence or absence of 66/F2/F3 prefixes providing 2 of those bits. If you ignore how the manual describes prefixes and look at it like that (which is suggested by the VEX encoding process), the decoding process becomes a lot simpler. In fact, with one singular exception, this is sufficient information to index into a map to figure out how long the immediate field is and whether or not ModR/M is present.