Why does being so deep in the gravity well pose an issue? If you are assuming the Dyson swarm is intended to go back up the well then sure, but that isn't necessary.