The mean means nothing: data visualization to debug a latency problem

Median also lies more than 99th/100th percentiles because in many workloads the same client is making multiple parallel requests and often waiting for the slowest.

See this piece from Jeff Dean [0]

Quote:

""" Component-Level Variability Amplified By Scale

A common technique for reducing latency in large-scale online services is to parallelize sub-operations across many different machines, where each sub-operation is co-located with its portion of a large dataset. Parallelization happens by fanning out a request from a root to a large number of leaf servers and merging responses via a request-distribution tree. These sub-operations must all complete within a strict deadline for the service to feel responsive.

Variability in the latency distribution of individual components is magnified at the service level; for example, consider a system where each server typically responds in 10ms but with a 99th-percentile latency of one second. If a user request is handled on just one such server, one user request in 100 will be slow (one second). The figure here outlines how service-level latency in this hypothetical scenario is affected by very modest fractions of latency outliers. If a user request must collect responses from 100 such servers in parallel, then 63% of user requests will take more than one second (marked “x” in the figure). Even for services with only one in 10,000 requests experiencing more than one-second latencies at the single-server level, a service with 2,000 such servers will see almost one in five user requests taking more than one second (marked “o” in the figure).

"""

[0] https://cacm.acm.org/research/the-tail-at-scale/#body-4

It's little surprising that the author was doing perf work and not already comparing distributions. As the rest of the article outlines, you learn a lot more with more data!

It's complicated, but really worth learning how prometheus and grafana heatmaps combine if you want dashboards for real time service data. Multimodal distributions are basically the expected outcome given all the caching done in distributed systems.

Instead of the CDF I like to use one minus the CDF, the fraction of requests not yet completed at any given time. Then you can make it a log log plot showing the entire tail with details invisible in the CDF because in the CDF the tail is essentially a horizontal line at one.

An interesting statistical example if you haven’t seen it before is the Anscombe Quartet :

https://en.wikipedia.org/wiki/Anscombe%27s_quartet

Four sets of X,Y datapoints that have exactly (or very close) common statistical parameters (mean, variance, correlation, linear regression, R^2), but with vastly different spatial distributions and “behavior” when looked at visually.

With only having read the headline (as it is customary), I dare say: Aggregation without distribution always means nothing.