This is why I really appreciate tools like Atuin. It augments your history with extra data such as the working directory, exit code, time to run command.