Why do so many supposedly smart humans think that we can make an artificial mind that is capable of AGI (or even something close to it), but from a completely detached evolutionary history and biological needs, and somehow force it to "align" to our human/biological/societal priorities?

Have none of these people ever had or been a teenager? At least teens have some overlapping biological requirements with non-teens that will force some amount of alignment.