The civilization found at the Inca structures was most probably descended from barbarians who conquered the civilization that built them. It's like someone arriving in Europe, finding the Franks ruling over it, and assuming they built the Roman Empire.