APM Tips

KQL · latency

Requests slower than a threshold, by operation

Count how many requests per operation took longer than a limit you set, and what share of the operation's traffic that is, sampling-adjusted.

Updated

requests latency

let lookback = 24h;
let threshold_ms = 1000.0;
requests
| where timestamp > ago(lookback)
| summarize
    requests = sum(itemCount),
    slow = sumif(itemCount, duration > threshold_ms)
    by operation_Name
| where slow > 0
| extend slow_pct = round(100.0 * slow / requests, 1)
| order by slow desc

Returns: One row per operation that had at least one request over the threshold, with its total requests, the number over the threshold and that number as a percentage.

Pitfalls

  • Sort order matters. By slow count, busy routes rise to the top; by slow_pct, rarely called routes with two slow requests out of three do. Look at both before deciding what to fix.
  • Counts use sum(itemCount), so the numbers stand for real traffic when sampling is on. A plain count() would report only the stored rows.
  • Operations are only as useful as their names. If operation_Name holds raw paths with IDs in them, every row is one URL and nothing groups; fix the naming first.
  • The threshold is a strict 'greater than'. A request that took exactly the threshold is not slow.

How it works

A service-level objective is usually phrased as a threshold: “99% of checkout requests finish in under a second”. This query measures exactly that, per operation. Where a percentile answers “how slow is the slow tail?”, a threshold count answers “how many requests broke the promise?”, which is the number a product owner asks for.

Line by line

  • let lookback = 24h; and let threshold_ms = 1000.0; are the two knobs. Writing the threshold as 1000.0 keeps it a real number, the same type as duration.
  • requests = sum(itemCount) is the operation’s total traffic, counted the sampling-aware way.
  • slow = sumif(itemCount, duration > threshold_ms) adds up itemCount only for rows over the limit, so slow requests are weighted the same way as the total.
  • by operation_Name groups by route-level name. The data model describes the request name as a low-cardinality value such as GET /values/{id}, which is what makes grouping work.
  • where slow > 0 drops operations that never broke the limit.
  • slow_pct is the share of the operation’s requests over the threshold. Multiplying by 100.0 before dividing keeps the division in real numbers; integer division would round most shares to zero.

When to use it

Use it to check an objective, to pick which routes to optimise first, and as the body of an alert. For an alert, keep the summarize, add | where slow_pct > 1 and requests > 200, and let the rule fire when any row comes back; the alerts guide explains how log search alerts count result rows.

If routes show up as one row per URL, the request naming guide explains how operation_Name is set and how to fix it.

Variations

  • One route over time: filter to one operation_Name, group by bin(timestamp, 1h) and chart slow_pct with | render timechart.
  • Graded thresholds: the Apdex-style score adds a tolerating band at four times the target.
  • Workspace tables: use AppRequests, TimeGenerated, DurationMs, ItemCount and OperationName.

Sources, checked

  1. sumif() (Kusto)learn.microsoft.com · checked
  2. Let statement (Kusto)learn.microsoft.com · checked
  3. Telemetry sampling in Application Insightslearn.microsoft.com · checked
  4. Application Insights telemetry data model (request name and duration)learn.microsoft.com · checked