APM Tips

KQL · failures

Failed requests by result code

Count failed requests in the last 24 hours by HTTP result code, with how many operations each code touched, sampling-adjusted.

Updated

requests failures

let lookback = 24h;
requests
| where timestamp > ago(lookback)
| where tostring(success) =~ "false"
| summarize
    failed = sum(itemCount),
    operations = dcount(operation_Name),
    example_operation = take_any(operation_Name)
    by resultCode
| order by failed desc

Returns: One row per result code among failed requests, with the sampling-adjusted failure count, the number of distinct operations affected and one example operation.

Pitfalls

  • The classic requests schema documents success as a string, and Microsoft's own examples compare it to 'False'; the workspace AppRequests table stores Success as a boolean. tostring(success) =~ "false" works against either.
  • Count with sum(itemCount), not count(), so sampled data reports true volumes.
  • resultCode is a string. For HTTP it holds the status code, but other request types can store other values, so don't cast it to a number blindly.
  • success is set by the SDK and can be changed by your own code (for example a telemetry initializer that treats 404s on some routes as successful), so filter on success rather than inferring failure from resultCode.
  • dcount() is an estimate. It is exact enough for a handful of operations; don't use it for billing-grade counts.

How it works

When the failure rate alert fires, the first question is which failures. This query answers it in one table: every result code that failed in the last day, how many requests it hit, and how widespread it is.

operations is the quick triage signal. A code that fails across dozens of operations points at something shared (a database, an auth service, a deployment); a code concentrated in one operation points at that route’s own code. example_operation gives you one place to start reading traces.

Turning it into a rate

A count alone can’t tell a bad hour from a busy one. To see each code’s share of all traffic, count everything first and filter afterwards:

requests
| where timestamp > ago(24h)
| summarize
    total = sum(itemCount),
    failed = sumif(itemCount, tostring(success) =~ "false")
    by resultCode
| where failed > 0
| extend failed_pct = round(100.0 * failed / total, 2)
| order by failed desc

Here total is per result code, so failed_pct answers “how often does a request that ended in this code count as failed”, which is useful for codes like 404 where the answer depends on the route.

Variations

  • Trend: group by bin(timestamp, 15m), resultCode, sort by timestamp and add | render timechart to see whether a code started at a deployment.
  • Per service: add cloud_RoleName to the by clause when several services share one resource.
  • Workspace tables: use AppRequests, TimeGenerated, Success == false, ResultCode, ItemCount and OperationName.

To be told about failures rather than look for them, the alerts guide shows how a log search alert runs a query like this on a schedule.

Sources, checked

  1. Metrics in Application Insights (log-based metric queries)learn.microsoft.com · checked
  2. Migrate an Application Insights classic resource to a workspace-based resource (table schemas)learn.microsoft.com · checked
  3. sumif() (Kusto)learn.microsoft.com · checked