One other aspect to consider is whether people actually want to have more than two kids per couple, even in an ideal world. Raising children is a huge effort and biologically and mentally very taxing on the parents, especially the mother. But we need much more than two children per couple to be above replacement rate.