You could decide if the en passant location is plausible from the position and color of the pawn on ranks 3 & 6, since it's only available of a pawn has moved two squares, so must be on rank 3 or 6, and it hasn't been promoted(another way it could reach those ranks)