You need highly educated individuals, a massive amount of energy expenditure, a massive facility to house your centrifuges, and an active mine to dig up nuclear materials.
It isn't impossible to keep such a secret, but practically it would be incredibly difficult just through the energy requirements and mining scale which would be hard to hide without anybody asking what exactly are you mining and processing.
"mining scale"
Don't need much area, depends on the concentration of radioactives. I have a small mine that's just a pegmatite body about the size of a house which produces almost marble-sized chunks of a thorium-uranium mixed metamict mineral (I suspect samarskite but Raman and XRD can't give any ID,) you'd barely notice it from a private airplane's typical flying height, however you could dig the entirety of it up and you'd have enough unprocessed uranium for some real fun.