APM Tips

KQL · latency

Request latency p50, p95 and p99 by operation

Median, p95 and p99 server response time for every operation over the last 24 hours, weighted for sampling, slowest p95 first.

Updated

requests latency 

let lookback = 24h;
requests
| where timestamp > ago(lookback)
| summarize
    requests = sum(itemCount),
    p50_ms = percentilew(duration, itemCount, 50),
    p95_ms = percentilew(duration, itemCount, 95),
    p99_ms = percentilew(duration, itemCount, 99)
    by operation_Name
| order by p95_ms desc

Returns: One row per operation name with its request count and its p50, p95 and p99 duration in milliseconds, slowest p95 first.

Pitfalls

  • duration is in milliseconds in the classic requests table. In the workspace AppRequests table the same column is DurationMs.
  • Count with sum(itemCount), never count(): with sampling on, each stored row stands for itemCount requests.
  • Plain percentile() treats every stored row equally. percentilew() with itemCount as the weight keeps the result honest when the sampling rate changed during the window.
  • Operations with a handful of requests produce noisy p99s. Add | where requests > 100 before the order step when you only care about busy endpoints.
  • timestamp is UTC. ago(24h) is the last 24 hours from now, not since midnight.

How it works

Averages hide the requests users complain about. This query gives each operation three numbers instead: the median (what a typical request feels like), p95 (one request in twenty is slower than this) and p99 (the tail that pages people at night).

It reads the requests table, which holds one row per incoming request your app recorded, and groups by operation_Name, the route-level name such as GET /orders/{id}. Grouping by name gives the same result for most web apps; operation_Name also lines up with dependencies and exceptions from the same operation, which helps when you follow a slow route into its calls.

Two details make the numbers trustworthy under sampling. The request count uses sum(itemCount), because a sampled row carries the number of requests it represents. And the percentiles use percentilew(), weighted by itemCount, so a period when more was sampled away still counts at its true size.

Reading the result

Sort order is p95 descending, which surfaces routes that are slow for a meaningful share of users rather than routes with one freak request. A large gap between p50 and p95 on the same row usually means the route has two paths, such as a cache hit and a cache miss, or a cold start. A p99 far above p95 with a low request count is often a single timeout; check failed requests by result code for the same window.

Variations

  • Over time: replace by operation_Name with by bin(timestamp, 1h), operation_Name, sort by timestamp and add | render timechart.
  • One service of several: add | where cloud_RoleName == "checkout-api" after the time filter.
  • Workspace tables: in a Log Analytics workspace query, use AppRequests, TimeGenerated, DurationMs, ItemCount and OperationName.

If every row is a single URL with an ID in it, the operation names need fixing before the percentiles mean anything; the request naming guide explains how operation_Name is set and how to change it.

Sources, checked

  1. percentile(), percentiles() (Kusto)learn.microsoft.com · checked
  2. percentilew(), percentilesw() (Kusto)learn.microsoft.com · checked
  3. Telemetry sampling in Application Insightslearn.microsoft.com · checked
  4. Migrate an Application Insights classic resource to a workspace-based resource (table schemas)learn.microsoft.com · checked

  • Request duration p50 and p95 as a timechart

    requests latency

    Returns: One point per 5-minute interval with the p50 and p95 request duration in milliseconds, drawn as two lines over the last 24 hours.

  • Request latency p50, p95 and p99 by cloud role

    requests latency

    Returns: One row per cloud role name with its request count and its p50, p95 and p99 duration in milliseconds, slowest p95 first.

  • The 20 slowest requests in the last hour

    requests latency

    Returns: Up to 20 individual requests from the last hour, slowest first, each with its operation name, duration in milliseconds, result code, cloud role and instance, and the operation_Id that opens its trace.

  • Failed requests by result code

    requests failures

    Returns: One row per result code among failed requests, with the sampling-adjusted failure count, the number of distinct operations affected and one example operation.