I tried something similar with a roguelike I was prototyping last year. Ended up being more useful for finding edge cases than actual gameplay feedback - the agent would do things no human would ever try, like walking into walls repeatedly or hoarding useless items. Still caught a bunch of bugs I never would have found otherwise.