It's the opposite.

Frameworks like pytorch are really flexible. You can implement any architecture, and if it's not enough, you can learn CUDA.

Keras it's the opposite, it's probably like you describe things.