I agree in general but the way this has worked for me in practice is that I approach things hierarchically up and down. Any specific hallucinations tend to come out in the wash as the same question is asked from different layers of abstraction.