Telemetry channels: in-memory vs server channel, and the OpenTelemetry exporter

When telemetry goes missing from a batch job, a console tool or an app that restarts often, the cause is rarely the network. It’s the last stretch of the pipeline: the component that holds telemetry in memory, batches it, and tries to get it to the ingestion endpoint before the process ends.

What a channel does

In the classic SDK, a channel is any class implementing ITelemetryChannel. Its Send(ITelemetry item) method is called after every initializer and processor has run, so an item a processor drops never reaches it. Send doesn’t transmit straight away: the channel buffers items and sends them in batches over HTTPS. Live Metrics has its own separate channel and isn’t affected by any of this.

Classic SDK 2.x

The .NET Classic API SDK 2.x is deprecated and retires on March 31, 2027. Two channels ship with it:

How the server channel behaves under pressure

ServerTelemetryChannel packs up to 500 serialized, compressed items into a transmission and sends up to 10 transmissions in parallel. When telemetry arrives faster than that, or the network is slow, transmissions queue in memory up to 5 MB (MaxTransmissionBufferCapacity), then on local disk up to 50 MB. Network failures also send transmissions to disk. Only what’s on disk survives a crash; it’s sent when the app starts again.

The disk location is StorageFolder. On Windows it defaults to %LOCALAPPDATA% or %TEMP%; elsewhere %TMPDIR%, /var/tmp/ or /tmp/. Point it somewhere persistent and writable on hosts where temp folders are cleared or read-only. In ASP.NET Core, register the channel before adding Application Insights:

using Microsoft.ApplicationInsights.Channel;
using Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel;

builder.Services.AddSingleton(typeof(ITelemetryChannel),
    new ServerTelemetryChannel { StorageFolder = @"D:\telemetry-buffer" });
builder.Services.AddApplicationInsightsTelemetry();

Classic ASP.NET sets the same thing under <TelemetryChannel> in ApplicationInsights.config; see the ApplicationInsights.config guide.

Flush in short-lived processes

Microsoft is clear that both channels lose buffered telemetry if it isn’t flushed before shutdown, and equally clear that you shouldn’t call Flush() routinely in a web app or service. The rule: flush on shutdown, in a last-chance exception handler, and at the end of jobs and command-line tools. For a console app, the documented pattern:

telemetryClient.Flush();
// ServerTelemetryChannel flushes asynchronously; give it time before the process exits.
await Task.Delay(TimeSpan.FromSeconds(5));

Or await telemetryClient.FlushAsync(), which Microsoft’s sample uses without the delay.

If you want “send each item now” for a test harness, InMemoryChannel with an explicit synchronous Flush() is the closest thing 2.x has to a synchronous channel. Developer mode, which sends items immediately while a debugger is attached, is for debugging only; disabling Application Insights in development explains why. And don’t turn on System.Diagnostics.Trace autoflush in web.config: Microsoft warns it can send each trace as a separate request and exhaust network and storage on the server.

Node.js 2.x

The retired Node.js 2.x SDK batches too: up to maxBatchSize items (default 250) or maxBatchIntervalMs (default 15,000 ms), so telemetry waits about 15 seconds before it’s sent. Short-lived processes call appInsights.defaultClient.flush() before exiting. When the SDK detects a crash it flushes with isAppCrashing, saving the buffer to disk to send on the next start.

Today: SDK 3.x and the Microsoft OpenTelemetry Distro

SDK 3.x removes ITelemetryChannel, both channels and TelemetryConfiguration.TelemetryChannel. It uses the Azure Monitor exporter internally, which Microsoft describes as the closest analogue to ServerTelemetryChannel: it batches and persists data itself. The disk settings moved to StorageDirectory and DisableOfflineStorage on TelemetryConfiguration, ApplicationInsightsServiceOptions or applicationinsights.config, and TelemetryClient.Flush() still exists. Unit tests that mocked ITelemetryChannel should switch to an OpenTelemetry in-memory exporter.

The Microsoft OpenTelemetry Distro uses the same exporter. When the app can’t reach Application Insights, the exporter caches telemetry to disk and retries for up to 48 hours; under heavy load it may drop telemetry that exceeds the time or file-size limit, favouring recent events. To move or disable the cache in .NET:

using Microsoft.OpenTelemetry;

builder.UseMicrosoftOpenTelemetry(options =>
{
    options.Exporters = ExportTarget.AzureMonitor;
    options.AzureMonitor.StorageDirectory = "/var/lib/myapp/telemetry";
    // options.AzureMonitor.DisableOfflineStorage = true;
});

The other languages have the same switch under different names: azureMonitorExporterOptions.storageDirectory and disableOfflineStorage in Node.js, azure_monitor_exporter_storage_directory and azure_monitor_exporter_disable_offline_storage in Python. The Java agent writes .trn files to a telemetry folder under java.io.tmpdir, keeps up to 50 MB by default and deletes files older than 48 hours.

Flushing in a console app means owning the SDK’s lifetime: create it with OpenTelemetrySdk.Create(...), keep the returned object for the whole run, and dispose it only at shutdown. Disposing flushes pending telemetry; disposing early stops collection. Desktop and console apps shows the whole pattern.

Batch timing follows OpenTelemetry. Microsoft documents OTEL_BSP_SCHEDULE_DELAY (spans) and OTEL_BLRP_SCHEDULE_DELAY (logs) as export intervals for Python on App Service; the OpenTelemetry specification’s defaults are 5,000 ms and 1,000 ms. Microsoft doesn’t state whether every distro honours them. Metrics are exported every 60,000 ms by default (OTEL_METRIC_EXPORT_INTERVAL), matching the 60-second interval at which Azure Monitor Metrics ingests custom metrics.

A second destination was a TelemetrySink in 2.x. Now it’s another exporter: ExportTarget.AzureMonitor | ExportTarget.Otlp in the .NET distro, with the OTLP endpoint in OTEL_EXPORTER_OTLP_ENDPOINT. Microsoft notes it doesn’t officially support the OTLP exporter or anything downstream of it.

Proxies and relays

An outbound proxy is configured where the HTTP client lives:

• Classic ASP.NET: Microsoft recommends proxy settings in the app’s web.config.
• Node.js 2.x: config.proxyHttpsUrl, which defaults to the https_proxy environment variable.
• SDK 3.x and the .NET distros: the Azure Monitor exporter package depends on Azure.Core, so the Azure SDK’s proxy rules apply. Set HTTPS_PROXY (or ALL_PROXY, with NO_PROXY for exceptions) for the process.

export HTTPS_PROXY="http://proxy.internal:8080"
export NO_PROXY="localhost,127.0.0.1"

A relay you run yourself, such as Azure API Management in front of ingestion, needs no SDK change: override IngestionEndpoint in the connection string. The instrumentation key still names the destination resource, and the SDK uses the endpoint exactly as given:

InstrumentationKey=00000000-0000-0000-0000-000000000000;IngestionEndpoint=https://telemetry-relay.example.com/

Microsoft documents this pattern for browser telemetry when local authentication is disabled: the relay authenticates to Application Insights with a managed identity and forwards to the regional endpoint. More on the string itself in setting the connection string.

Troubleshooting