It's a house, not a wall. And it's not being built by a robot. Adding the brick is inconsequential.
Go to any building site and ask a bricklayer if adding a single custom brick to the construction will stop them building the house and he'll laugh in your face.
You're missing GP's point. The objection is not to the inclusion of swift bricks in new houses but the belief that it is sufficient to stabilize/restore the population, because relatively few new houses are being constructed.