alt Hacker NewsQuestion: Why does AWS need to roll back estimated bills to a "last known good" state? I get wanting to do that for ACTUAL billing mistakes, but for estimates, they're just that -- approximations. I guess it's fine for predictive purposes to store estimates so they can be compared to actual usage and optimized. But why would AWS bind the values of present estimates to the estimates made earlier in the month. The calculation should always be:
1. Current month's usage * applicable rates; + 2. Estimated future usage for the month * applicable rates.
And Item 1 obviously requires proper data persistence, but Item 2 is just a projection. If they don't have Item 1 correct, AWS's whole system is in question, but I don't think that's the issue. I'm going to guess now -- looking forward to reading the root cause analysis -- that the problem is that someone decided to get too fancy with the estimates, and built a latent requirement that all prior estimates for the month must be available to compute the current estimate. Without estimates working, no estimates are available, and some denominator in an averaging or smoothing or normalizing function goes to 0; then everyone's estimated bill explodes without bound (subject to floating-point arithmetic) resulting in trillion-dollar estimates.
Thats not really how estimates work. The actual metering data is ingested in near real time. The metering * pricing plan is processed within a few hours; thats what youre seeing for “estimated spend” IIRC. The actual billing accumulation is done later, at the end of the cycle, because pricing has cross service discounts, price tranches, credits tied to total spend, etc.
“Rolling back” estimated bills is reprocessing the historic metering data by an older or newer pricing plan version. As i mentioned in another comment someone will have messed up a metering type vale (eg GB/B). Thats why theyll need a few hours to redrive the metering data.