It is hard.

I can hear a tune and immediately sing it or whistle it. But I can't immediately play it on the guitar. It's much easier for children than adults.

It's also hard to force yourself to practice the relevant skills. You can play scales all day but that won't necessarily help your ear. What you need is to force your brain to make the connection.