Vault 📚

05_metrics_and_analysis

7 min read

Metrics Analysis

In modern distributed systems, the performance and reliability of communication channels, APIs, and network infrastructure are critical factors that determine user experience. Metrics and analysis offer insights into system behavior under varying loads, help identify bottlenecks, and guide capacity planning. Capturing the right metrics and interpreting them correctly leads to robust, scalable architectures. This document explores the core metrics for communication, API usage, and network layers, alongside common formulas, analysis techniques, and best practices in monitoring and diagnostics.

Why Metrics Matter

Data-driven decisions about scaling, optimization, and resource allocation rely on the proper collection and interpretation of metrics. Without measurable indicators, developers and operators are left guessing about system performance and reliability. Metrics also facilitate:

  1. Troubleshooting: Pinpointing service or network segments causing latency spikes or errors.
  2. Capacity Planning: Predicting future infrastructure needs based on historical load.
  3. Service-Level Objectives (SLOs): Establishing performance targets and tracking whether services meet those targets.
  4. Trend Analysis: Spotting gradual performance degradations before they become outages.

Common Terminology and Formulas

Latency and Response Times

Latency is the time it takes for a request to travel from a client to a server and for the server to respond, often including network travel time and server processing. Commonly measured as:

  • p50 (Median): Half of requests complete below this time.
  • p95, p99: High percentiles that reveal the “worst” performance user experiences.
  • p100 (Max): Maximum observed latency (sometimes too spiky to be useful).

A simplified latency equation for an HTTP request might be:

Total_Latency = RTT_network + Server_Processing_Time + (Possible_Queueing_Delay)
  • RTT_network: Round-trip time for packets to travel between client and server.
  • Server_Processing_Time: The time the server spends handling the request (compute, I/O, DB calls, etc.).
  • Possible_Queueing_Delay: Delays in load balancers, proxies, or OS-level buffers.

Throughput and Requests per Second (RPS/QPS)

Throughput indicates how many requests or messages a system can handle over a given time, often expressed as Requests per Second (RPS) or Queries per Second (QPS). If N_req is the total number of requests within a measurement window T, throughput can be approximated as:

Throughput = N_req / T

Concurrency

Concurrency measures how many requests or connections a system is serving simultaneously. A system that supports large concurrency can handle many in-flight requests at once, but each active request may consume memory and CPU resources.

Error Rates

Common ways to measure errors:

  • 4XX Error Rate: Fraction of client errors (e.g., 400, 404, 401).
  • 5XX Error Rate: Fraction of server errors (e.g., 500, 503).
  • Overall Error Rate: (Number_of_Error_Responses) / (Total_Requests).

Uptime and Availability

Often expressed as a percentage of time the service is fully operational:

Availability (%) = 100 * (Uptime / Total_Time)

For critical services, SLOs might require 99.9% or 99.99% availability, corresponding to allowable downtime of minutes or seconds per month.

Packet Loss and Network Metrics

In lower-level network contexts (TCP, UDP), metrics like packet loss, retransmissions, and bandwidth utilization are key:

Packet_Loss_Rate = (Packets_Lost / Packets_Sent) * 100

Excessive packet loss degrades application performance, especially for real-time or streaming protocols.

Metrics for APIs

HTTP/RESTful APIs

For RESTful services, typical metrics include:

  1. Request Latencies by endpoint and method (GET, POST, etc.).
  2. Request Throughput in requests per second or per minute.
  3. Response Codes: Monitoring distribution of 2xx, 4xx, 5xx for quick error detection.
  4. Payload Sizes: Large payloads can affect network usage and latency.
  5. Caching Metrics: Cache hit/miss rates if a caching layer is in use.

A representation of data flow with potential metric collection points:

Client (Browser/App) ----> [Load Balancer] ----> [API Server(s)] ----> [Database/Cache]
     |                        |                       |                      |
     | metrics??             | metrics??             | metrics??            | metrics??
     v                        v                       v                      v

Logging & Monitoring Infrastructure (e.g., Prometheus, Grafana, ELK stack)

At each stage, logs and performance counters gather metrics on request durations, error rates, and resource usage (CPU/memory).

GraphQL APIs

Similar to REST, but the queries can be more dynamic. Important metrics include:

  1. Resolver Latencies: Identifying slow resolvers.
  2. Field Usage: Finding which fields in the schema are frequently queried.
  3. Depth and Complexity: Monitoring how complex queries might degrade performance.

gRPC

Because gRPC uses HTTP/2 and Protobuf, typical metrics include:

  1. RPC Latencies (per method).
  2. Inbound/Outbound Traffic (in bytes).
  3. Deadline/Timeout Exceeded events.
  4. Streaming Metrics (message count, stream duration).

Network-Level Metrics

TCP Metrics

  • RTT (Round Trip Time): Base measure of latency.
  • Window Size: Congestion window for flow control.
  • Retransmissions: High retransmissions can indicate congestion or packet loss.

UDP Metrics

  • Packet Loss: Particularly relevant in real-time streaming contexts.
  • Jitter: Variation in packet arrival times, crucial for voice/video.
  • Datagram Throughput: How many datagrams per second can be handled?

Bandwidth and Utilization

Bandwidth is the theoretical max data rate, while utilization measures how much of that bandwidth is in use. Monitoring helps avoid saturation. If you see throughput near the link’s capacity, you risk increased latency and packet drops.

Analyzing and Visualizing Metrics

Dashboards and Tools

Systems like Prometheus, Graphite, InfluxDB, or DataDog store metrics. Tools like Grafana or Kibana help create real-time dashboards. A typical setup might ingest counters and histograms from applications, store them in a time-series database, and visualize them in charts.

Histograms vs Averages

Relying on averages can be misleading—some users might experience extreme delays while the average remains fine. Histograms reveal distribution across multiple buckets, giving better insight into tail latencies (p95, p99).

Error Budgets and SLIs/SLOs

  • SLI (Service Level Indicator): A measured metric (e.g., request success rate).
  • SLO (Service Level Objective): A target threshold for that SLI (e.g., 99.9% success rate).
  • Error Budget: The allowable number of errors/time of unavailability before the SLO is violated.

Example formula for an SLO around error rate:

Error_Rate_SLI = (Number_of_Error_Requests / Total_Requests)
Target: Error_Rate_SLI <= 0.1%

If the error rate goes beyond 0.1%, you exceed your error budget.

Load and Stress Testing

Performance analysis relies on simulating realistic traffic:

  1. Load Testing: Evaluate normal to peak loads, measuring throughput, latency, and resource usage.
  2. Stress Testing: Push the system beyond capacity to see how it fails and recovers.
  3. Soak/Endurance Testing: Sustain high load for an extended period, revealing memory leaks or degrade over time.

Tools like Apache JMeter, Locust, or k6 let you define test scripts that emulate real client behavior. Metrics from these tests guide capacity planning and highlight scaling bottlenecks.

Distributed Tracing

In microservice architectures, a single request can span multiple services. Distributed tracing with solutions like Jaeger or Zipkin tracks how requests hop between services. The system collects timestamps and metadata at each node:

   [Service A] -- calls --> [Service B] -- calls --> [Service C]
       |                         |                        |
       v                         v                        v
   (Trace A)                (Trace B)                 (Trace C)

Traces aggregated and visualized in a central UI

This reveals which segments of a request path consume the most time or fail often. Tracing complements standard metrics by delivering a request-centric timeline rather than aggregated counters.

Putting It All Together: Example Monitoring Architecture

Below is a conceptual diagram of how metrics might flow:

+---------------------+                 
|   Various Clients   | -- Make Requests -->  [Load Balancer] -> [API/Services]
+---------------------+                                           |
                                                                   | Generate logs
                                                                   | and metrics
                                                                   v
                                                        +-----------------------+
                                                        |   Metrics Exporter    |
                                                        | (Prometheus, StatsD)  |
                                                        +-----------+-----------+
                                                                    |
                                                                    v
                                                        +-----------------------+
                                                        |   Time-Series DB      |
                                                        | (Prometheus, InfluxDB)|
                                                        +-----------+-----------+
                                                                    |
                                                                    v
                                                        +-----------------------+
                                                        |   Visualization       |
                                                        |   (Grafana, Kibana)   |
                                                        +-----------------------+

Steps in this pipeline:

  1. Requests Enter: Clients call the API through a load balancer.
  2. Services Process: Each service logs data or updates counters/histograms.
  3. Metric Scraping: A metrics exporter (Prometheus) scrapes data from the services.
  4. Storage: The time-series database retains historical metrics.
  5. Dashboards: Operators view performance, latencies, error rates, etc., in real time.

Best Practices and Recommendations

  1. Capture the Right Granularity: Overly detailed metrics can overwhelm storage, but too coarse metrics hide issues. Start with endpoint-level latencies, throughput, and error rates.
  2. Measure Tail Latencies: Focus on p95 or p99 to ensure outliers are not ignored.
  3. Define Clear SLOs: Tying metrics to business outcomes helps teams prioritize fixes.
  4. Automate Alerts: Set up threshold-based or anomaly-based alerts for critical metrics (e.g., 5xx rate spikes, high latencies).
  5. Integrate Tracing: Combine logs, metrics, and traces for a complete observability picture.
  6. Plan for Scale: As systems grow, you’ll need a robust architecture for metric collection and storage.
  7. Stress and Chaos Testing: Explore system limits and resiliency under adverse conditions (random instance termination, network partition).

Graph View