> We know that the Inca didn't build Sacsayhuaman because they said that they didn't.
Where are you getting this from? The Spanish chroniclers report Inca tradition that the 15th century leader Pachacuti initiated the building. The wiki article has a few long excerpts from Pedro Cieza de Leon's Cronicas del Peru, including details of how many labourers were involved and some of the methods for quarrying, transporting stone, and construction:
You can’t reason with ancient astronaut conspiracy. They have a loophole answer for everything.