alt Hacker NewsSuch a gem, thanks to the author for sharing it's findings :)
The only problem I have with planing in latent space is that it can be really noisy and not representative of the positions in the game (the latent are trained for semantic, so the optimizer can focus a set of specific features and can skip positions, which means it cannot know "where" to go by optimizing on the latents directly).
Hey, author here! Thanks so much for reading :) I totally agree, the latent captures details useful for prediction, but not necessarily for control, making planning noisy. Still, I was surprised by how well it understood horizontal position after less than two hours of training on one A100!