You can only castle if neither the king nor the rook have been moved (and none of the three squares the king uses may be under attack, and all the squares between the rook and the king must be empty).

Since you could move either rook somewhere and then back to their starting squares, you have to track their eligibility separately. If the king moves, both rooks lose eligibility.