Custom metrics in Application Insights: pre-aggregated vs log-based

Application Insights has three kinds of metrics. Standard metrics (request rate, response time, failures) are collected for you and pre-aggregated into one-minute time series that sampling doesn’t touch. Log-based metrics are charts built at query time from logs, so sampling and filtering reduce their accuracy. Custom metrics are the ones you define. Getting custom metrics right is mostly about how they’re aggregated and where each copy ends up.

Pre-aggregated vs one item per value

There are two ways to send a number. One item per value (TrackMetric() in the classic SDK) sends a telemetry item for every observation, each going through initializers and processors. Pre-aggregation keeps a running count, sum, minimum and maximum per metric series in memory and sends one summary per minute.

The difference is volume. A service that records 20 values per request at 500 requests per second tracks 10,000 values a second, which is Microsoft’s own example. Sent one by one, that’s a real ingestion bill and a risk of hitting the 32,000 events per second throttle, where dropped points can mean missed alerts. Pre-aggregated, it’s one item per series per minute, whatever the rate.

Classic SDK 2.x

GetMetric() returns a handle that aggregates locally and submits a summary at a fixed one-minute interval. Cache the handle on hot paths, and add dimensions by name.

A pre-aggregated metric with one dimension in the 2.x SDK:

private readonly Metric _ordersPlaced;

public OrderService(TelemetryClient telemetry)
{
    _ordersPlaced = telemetry.GetMetric("OrdersPlaced", "Channel");
}

public void Place(Order order)
{
    // ... place the order ...
    _ordersPlaced.TrackValue(1, order.Channel); // "web", "app", "partner"
}

What arrives once a minute is a MetricData item whose data point holds value (the sum), count, min, max and stdDev. Things to know about the 2.x API:

• Dimensions. Up to 10. More than three need a MetricIdentifier. By default each metric is capped at 1,000 series and 100 values per dimension; when a cap is hit, TrackValue returns false and the value is dropped, so check it when dimension values come from user input.
• No gauges or histograms. GetMetric aggregates sums and counts; it can’t track a last value or a distribution.
• Context isn’t attached. Metrics don’t pick up the TelemetryClient’s context, such as the operation name. Use the special dimension names in MetricDimensionNames.TelemetryContext when you need them.
• Not sampled. Neither GetMetric nor TrackMetric values are sampled, which makes them safe for alerts.

Today: SDK 3.x and the Microsoft OpenTelemetry Distro

SDK 3.x

SDK 3.x keeps GetMetric and TrackMetric but emits them through OpenTelemetry. Only the simplified overloads remain: the ones that took MetricConfiguration or MetricAggregationScope are gone, so the per-metric series caps above are no longer yours to set. Metric names and namespaces must now follow OpenTelemetry’s instrument name syntax: start with a letter, then letters, digits, _, ., - or /, up to 255 characters, no spaces. A 2.x metric called Orders Placed has to be renamed before the upgrade, and the chart and alert that used the old name with it. The overloads of TrackEvent, TrackException and TrackAvailability that took a metrics dictionary were removed; send those numbers as metrics.

The Microsoft OpenTelemetry Distro

With the Distro, custom metrics use the .NET System.Diagnostics.Metrics API. Create a Meter, register its name, and record with the instrument that matches the question:

A counter and a histogram with the Distro in ASP.NET Core:

using System.Diagnostics.Metrics;
using Microsoft.OpenTelemetry;
using OpenTelemetry.Metrics;

var builder = WebApplication.CreateBuilder(args);
builder.Services.ConfigureOpenTelemetryMeterProvider((sp, metrics) => metrics.AddMeter("Contoso.Orders"));
builder.Services.AddOpenTelemetry().UseMicrosoftOpenTelemetry(o => o.Exporters = ExportTarget.AzureMonitor);

var meter = new Meter("Contoso.Orders");
var ordersPlaced = meter.CreateCounter<long>("orders.placed");
var orderValue = meter.CreateHistogram<double>("orders.value", unit: "EUR");

// in the handler
ordersPlaced.Add(1, new KeyValuePair<string, object?>("channel", "web"));
orderValue.Record(129.90, new KeyValuePair<string, object?>("channel", "web"));

The meter name passed to AddMeter() must match the Meter exactly, or nothing is collected. Microsoft notes that Azure Monitor flattens histograms into those five aggregations; percentiles of a custom histogram aren’t available from the metrics store.

Python follows the same model with the OpenTelemetry API:

from microsoft.opentelemetry import use_microsoft_opentelemetry
from opentelemetry import metrics

use_microsoft_opentelemetry(enable_azure_monitor=True)
meter = metrics.get_meter_provider().get_meter("contoso.orders")
orders_placed = meter.create_counter("orders.placed")
orders_placed.add(1, {"channel": "web"})

In Python, setting APPLICATIONINSIGHTS_METRIC_NAMESPACE_OPT_IN to true groups metrics in Metrics Explorer under the meter’s name as a custom namespace.

Where custom metrics land

Every custom metric is stored twice:

1. The customMetrics table (AppMetrics in the workspace), one row per series per interval, with all dimensions in customDimensions. Query it with KQL for anything detailed.
2. The metrics store, under the azure.applicationinsights namespace in Metrics Explorer, which is what metric charts and metric alerts read. This copy keeps no dimensions unless you select With dimensions under Send custom metrics to Azure Metric Store on Usage and estimated costs. That option is billed separately (Microsoft calls it a preview), and only data sent after you enable it has dimensions.

Auto-instrumentation doesn’t change this, but it limits what you can add: Microsoft lists custom metrics as not supported with ASP.NET Core, Node.js and Python auto-instrumentation, while the Java agent sends metrics from popular libraries (Micrometer among them) as custom metrics. For your own numbers, instrument in code; the auto-instrumentation guide covers when to switch.

Billing follows the copies. For billing purposes Microsoft treats all OpenTelemetry metrics, including the ones instrumentation libraries collect automatically, as custom metrics. The customMetrics rows are ordinary ingested log data. Dimensionless custom metrics in the metrics store are free up to a quota; dimensional ones cost extra. The metrics store also enforces Azure Monitor’s custom-metric limits: 10 dimension keys per metric, 256 characters per name or value, and 50,000 active time series per subscription and region, where an active series is any combination of metric and dimension values seen in the past 12 hours. A CustomerId dimension breaks that limit quickly; put identifiers in logs, not metric dimensions.

Query customMetrics correctly

Each row is already an aggregate, so count() counts rows, not observations. Use the aggregate columns. In the workspace schema valueSum becomes Sum, valueCount becomes ItemCount, valueMin and valueMax become Min and Max, and the value column is gone.

Orders per channel per five minutes, with the true average order value:

customMetrics
| where timestamp > ago(1d) and name == "orders.value"
| extend channel = tostring(customDimensions.channel)
| summarize orders = sum(valueCount),
            revenue = sum(valueSum),
            maxOrder = max(valueMax)
  by bin(timestamp, 5m), channel
| extend avgOrder = round(revenue / orders, 2)

Average the sums, not the averages: avg(value) over rows weights a quiet minute the same as a busy one. To alert on a custom metric, a metric alert on the metrics-store copy is the fastest and cheapest choice; see Azure Monitor alerts explained.

Troubleshooting