APM Tips

KQL · cost

Sampling rate in effect, per telemetry type

Read the effective sampling percentage for each telemetry type from itemCount, to confirm whether sampling is on and how much it keeps.

Updated

requestsdependenciespageViewsbrowserTimingsexceptionstracescustomEventsavailabilityResults cost

let lookback = 24h;
union requests, dependencies, pageViews, browserTimings, exceptions, traces, customEvents, availabilityResults
| where timestamp > ago(lookback)
| summarize
    stored_rows = count(),
    items = sum(itemCount),
    retained_pct = round(100.0 / avg(itemCount), 1)
    by itemType
| order by retained_pct asc

Returns: One row per telemetry type with the rows stored, the items they stand for, and the percentage of items kept by sampling over the last 24 hours.

Pitfalls

  • retained_pct is an average over the window. Adaptive sampling changes its rate with traffic, so a day's figure hides busy-hour rates far below it; chart it by hour to see them.
  • 100% means the stored rows of that type weren't sampled. Data dropped by filtering, a daily cap or a processor that discards items doesn't show here at all.
  • Performance counters, custom metrics and sessions are never sampled, so they aren't in the union.
  • Each SDK or service instance samples on its own; group by cloud_RoleName as well to compare services.

How it works

Sampling is on by default in several SDKs and Distros, at rates that move with traffic, and it is easy to lose track of what your app actually sends. You don’t need the configuration to find out: every stored row carries itemCount, the number of items it stands for. If rows of a type average an itemCount of 4, one in four items of that type was kept.

Line by line

  • union ... reads the eight tables whose rows carry itemCount. Microsoft’s own version of this check reads requests, dependencies, page views, browser timings, exceptions and traces; custom events and availability results are added here because they carry the column too.
  • stored_rows = count() is how many rows were kept; items = sum(itemCount) is how many items they represent.
  • retained_pct = round(100.0 / avg(itemCount), 1) is Microsoft’s formula for the retained percentage, 100/avg(itemCount). It equals 100 * stored_rows / items.
  • by itemType gives one row per telemetry type: request, dependency, trace and so on.
  • order by retained_pct asc puts the most heavily sampled type first.

Reading the result

Microsoft’s rule is simple: if the retained percentage for a type is below 100, that type is being sampled. Expect to see it on busy apps that use the classic ASP.NET and ASP.NET Core SDKs, which turn adaptive sampling on by default (the classic ASP.NET configuration samples custom events at a separate rate from everything else), and on apps using the Microsoft OpenTelemetry Distro, which defaults to rate-limited sampling. A type at 100% on a busy app may not be sampled at all, which is worth knowing before the bill arrives; telemetry volume by table shows what it costs.

The sampling guide explains the types and defaults, and the pricing guide shows how a retained percentage turns into gigabytes.

Over time

The average hides the busy hours, when adaptive sampling bites hardest:

let lookback = 24h;
union requests, dependencies, traces
| where timestamp > ago(lookback)
| summarize retained_pct = 100.0 / avg(itemCount) by bin(timestamp, 1h), itemType
| order by timestamp asc
| render timechart

Variations

  • Per service: add cloud_RoleName to the by clause.
  • Workspace tables: union the App* tables and group by Type (the table name, the workspace form of itemType), using ItemCount and TimeGenerated.

Sources, checked

  1. Telemetry sampling in Application Insights (knowing whether sampling is in operation)learn.microsoft.com · checked
  2. Sampling in Application Insights with OpenTelemetrylearn.microsoft.com · checked
  3. union operator (Kusto)learn.microsoft.com · checked
  4. Migrate an Application Insights classic resource to a workspace-based resource (table schemas)learn.microsoft.com · checked