2 main reasons

1) Populations are their most expensive at their oldest age and each subsequent generation is smaller and needs to pay for an old generation larger than their own

2) infrastructure and many of the things a government provides is not scalable down and up. A road is not (much) cheaper to maintain because less people drive on it

One word: ageing. Population declines but also age, so working population declines much more than total population.