APM Tips

KQL · failures

Server errors vs client errors by hour

Split each hour's HTTP errors into 5xx and 4xx responses, sampling-adjusted, with the operations that returned server errors in that hour.

Updated

requests failures

let lookback = 24h;
requests
| where timestamp > ago(lookback)
| extend status = toint(resultCode)
| where status >= 400
| summarize
    server_5xx = sumif(itemCount, status between (500 .. 599)),
    client_4xx = sumif(itemCount, status between (400 .. 499)),
    operations_with_5xx = make_set_if(operation_Name, status >= 500, 5)
    by bin(timestamp, 1h)
| order by timestamp asc

Returns: One row per hour that had HTTP errors, with the number of 5xx and 4xx responses and up to five operations that returned a 5xx in that hour.

Pitfalls

  • resultCode is a string. For HTTP requests it holds the status code; for other request types it can hold other values, which toint() turns into empty values that the status filter drops.
  • This splits by status code, not by the success flag. By default a 401 counts as a successful request even though it is a 4xx here.
  • operations_with_5xx is a sample of up to five names, not a ranking. Use the companion query below for the top operations.
  • Hours with no errors produce no row, so a quiet hour is a gap rather than a zero.

How it works

A 5xx is your fault; a 4xx is usually the caller’s. Lumping them together hides both stories: a wave of 404s from a crawler can bury the ten 500s that matter, and a deploy that breaks request validation shows up as 400s, not 500s. This query keeps the two apart, hour by hour.

Line by line

  • extend status = toint(resultCode) turns the result code into a number. The data model defines resultCode as the HTTP status code for HTTP requests, but stores it as text, and other request types can put other values in it.
  • where status >= 400 keeps error responses only, which also drops rows whose code isn’t a number.
  • sumif(itemCount, status between (500 .. 599)) counts server errors, sampling-adjusted; the 4xx column does the same for 400 .. 499. between includes both ends.
  • make_set_if(operation_Name, status >= 500, 5) collects up to five distinct operation names that returned a 5xx in the hour, so you know where to look without a second query.
  • by bin(timestamp, 1h) gives one row per hour (UTC).

Reading it

A 5xx count that jumps in one hour and stays there points at a deployment or a dependency outage that started then. A 4xx count that jumps is worth reading too: a new client version calling an old endpoint, an expired token, or a scan. Remember that success follows its own rule, a response under 400 or a 401 counts as successful by default, so this view and the failure rate can disagree on 401s.

Top failing operations

The hourly view says when; this companion says where, over the same window:

let lookback = 24h;
requests
| where timestamp > ago(lookback)
| extend status = toint(resultCode)
| where status >= 400
| summarize errors = sum(itemCount) by operation_Name, class = iff(status >= 500, "5xx", "4xx")
| top 10 by errors desc

Variations

  • Chart it: drop the make_set_if line and add | render columnchart.
  • One service: add | where cloud_RoleName == "orders-api" after the time filter.
  • Workspace tables: use AppRequests, TimeGenerated, ResultCode, ItemCount and OperationName.

For the full list of codes behind the failures, see failed requests by result code; the alerts guide shows how to alert on a 5xx count rather than watching for it.

Sources, checked

  1. sumif() (Kusto)learn.microsoft.com · checked
  2. make_set_if() (Kusto)learn.microsoft.com · checked
  3. The between operator (Kusto)learn.microsoft.com · checked
  4. Application Insights telemetry data model (resultCode and success)learn.microsoft.com · checked
  5. Migrate an Application Insights classic resource to a workspace-based resource (table schemas)learn.microsoft.com · checked