> MNIST digits in FPGA

Your example suffers from one of the same problems in the paper. They exclusively used text book level problems in their eval of the models HDL coding ability. These are effectively poisoned in the training data with many many copies of solutions done in different ways.