it's not clear to me how they determined that these were in fact different species and not just variation/evolution within the same species
This is a broader fundamental question which you are better off asking AI to give you an overview. It involves various techniques like;
1) Paleo genomics i.e. DNA study of any surviving material.
2) Statistical study of Human Morphology i.e. variation w.r.t. current day human (and close primates?) biological structures. Eg. Current-day human cranium sizes vary within known limits. So if we find a fossil cranium which is outside these limits then there is a high chance it might be a different species.
3) Fossil dating shows different species (w.r.t. above data) occupying the same geography and within overlapping time periods.
When biologists, botanists, etc. use the term "species", they are not using it in a principled way. The distinctions are oftentimes quite arbitrary, I'm very sorry to say.
For what it's worth, there isn't even much agreement on what constitutes a genus or species among professional philosophers of science (i.e. academics who care about giving a principled definition for species/genus), even for those that share many of the same views otherwise (like ur-Platonists, Aristotelians, Thomists, Scotists, etc.).