APM tips blog

Blog about application monitoring.

Track Dependencies in Console Application

| Comments

Update (11/17/2015): package RuntimeTelemetry was renamed to Microsoft.ApplicationInsights.DependencyCollector and some class names and namespaces were changed correspondingly.

Dependencies tracking in Application Insights is powerful feature that allows to see what SQL and Http calls your application makes. I’ve mentioned that you need to install Status Monitor or Azure WebSites extension to enable it for your web application. I don’t like magic and tools that configures something that I don’t quite understand. I think most of developers and especially devops thinks the same way. Hopefully after this post you can better understand how this feature works and will trust it more.

The main purpose of Status Monitor and Azure WebSites extension is to simplify Application Insights enablement for web applications. When you host your ASP.NET application in IIS or as Azure WebSite it has very predictable structure. So most of enablement steps can be automated. In this post I’ll show how to enable dependencies tracking feature for console application manually so you know what Status Monitor and Azure WebSites extensions automation makes under the hood. You can apply similar steps for any other application type - be it Windows Service, Worker Role or anything else.

Let’s assume you just created a simple console application called “TestDependency.exe” in Visual Studio. As it tests dependencies tracking - I make a simple http call in a Main method of this application:

1
2
3
4
5
6
7
var request = HttpWebRequest.Create("http://bing.com");
var response = request.GetResponse();
using (var s = new StreamReader(response.GetResponseStream()))
{
    Console.WriteLine(s.ReadToEnd().Length);
}
Console.ReadLine();

Now you need to include and configure Application Insights for this application and then enable dependencies tracking feature.

Install Application Insights

Every big feature in Application Insights is implemented in a separate nuget so you can choose what kind of monitoring you want to enable for your application. Dependencies tracking feature implemented in RuntimeTelemetry nuget package. So let’s install it:

1
Install-Package Microsoft.ApplicationInsights.RuntimeTelemetry -Pre

Once installed new telemetry module will appear in ApplicationInsights.config. This telemetry module is responsible for dependencies tracking. We call it remote dependencies module:

1
<Add Type="Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry.RemoteDependencyModule, Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry"/>

Lastly, as it is a console application, you need to set instrumentation key manually:

1
TelemetryConfiguration.Active.InstrumentationKey = "ec126cb1-9adc-4681-9cd4-0fcad33511c9";

When you compile your application plenty of new files will appear in bin/Debug folder. You’ll need to carry these files alongside with your application’s binary when you’ll deploying it. Here are the list of files you’ll need for 0.12 version of Application Insights SDK.

Application Insights configuration:

1
ApplicationInsights.config

Application Insights Core and dependencies:

1
2
3
4
5
Microsoft.ApplicationInsights.dll
Microsoft.Diagnostics.Tracing.EventSource.dll
Microsoft.Threading.Tasks.dll
Microsoft.Threading.Tasks.Extensions.Desktop.dll
Microsoft.Threading.Tasks.Extensions.dll

Files responsible for dependencies tracking:

1
2
3
4
Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry.dll
Microsoft.ApplicationInsights.Extensions.Intercept_x64.dll
Microsoft.ApplicationInsights.Extensions.Intercept_x86.dll
Microsoft.Diagnostics.Instrumentation.Extensions.Intercept.dll

Small note here - you may notice that RuntimeTelemetry nuget has a dependency to another nuget that is hidden in nuget.org: “Microsoft.Diagnostics.Instrumentation.Extensions.Intercept”. The only reason why it is hidden is that there is no reason to install it by itself, it only has utility features used by other Application Insights nugets.

Enable dependencies tracking feature

Dependencies tracking feature is based on code instrumentation in runtime. Application Insights decorates every call to http or SQL with prefix and postfix callbacks. Timer starts in prefix callback and on postfix - all information regarding dependency call like url and duration being send to Application Insights service. The only way today to notify CLR (common language runtime) to allow code instrumentation is to set certain environment variables before you run your application.

To enable dependencies tracking feature you should set these environment variables:

1
2
3
SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}
SET MicrosoftInstrumentationEngine_Host={CA487940-57D2-10BF-11B2-A3AD5A13CBC0}

These variables tells CLR to load certain COM object as a “profiler”. We call this object Runtime Instrumentation Agent as it’s main purpose is to enable code instrumentation in runtime, without application re-compilation. Settings above will only work on machine where Status Monitor installed as GUIDs points to COM objects that should be registered on computer.

If you are not a big fun of registering COM object (like me) - you can copy content of folder “%ProgramFiles%\Microsoft Application Insights\Runtime Instrumentation Agent” to any other folder, for instance output folder of your application:

Surely you should tell CLR where to look for COM objects by specifying path to corresponding dlls:

1
2
3
4
5
SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}
SET COR_PROFILER_PATH="folder"\x86\MicrosoftInstrumentationEngine_x86.dll
SET MicrosoftInstrumentationEngine_Host={CA487940-57D2-10BF-11B2-A3AD5A13CBC0}
SET MicrosoftInstrumentationEngine_HostPATH="folder"\x86\Microsoft.ApplicationInsights.ExtensionsHost_x86.dll

That’s it. Just start console window, set environment variables and run your application. In my case I see this dependency event being sent to Application Insights:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
"ver":1,
"name":"Microsoft.ApplicationInsights.RemoteDependency",
"time":"2015-01-05T05:03:46.4427348+00:00",
"iKey":"ec126cb1-9adc-4681-9cd4-0fcad33511c9",
"device":{"id":"id is a required field for Microsoft.ApplicationInsights.Extensibility.Implementation.DeviceContext"},
"internal":{"sdkVersion":"0.12.0.17386","agentVersion":"0.12.0"},
"data": {
  "type":"Microsoft.ApplicationInsights.RemoteDependencyData",
  "item":{
      "ver":1,
      "name":"http://bing.com/",
      "kind":"Aggregation",
      "value":592,
      "count":1,
      "dependencyKind":"HTTP",
      "success":true,
      "async":true,
      "source":2,
      "properties":{"DeveloperMode":"true"}
  }
}

Looking at this JSON you see that we now send two versions - sdk version (0.12.0.17386) and agent version (0.12.0). Agent version here is a version of Runtime Instrumentation Agent.

Web applications and dependencies tracking

So what exactly Status Monitor and Azure WebSites extension are doing.

Status Monitor:

  1. Installs Runtime Instrumentation Agent binaries to %ProgramFiles%\Microsoft Application Insights\Runtime Instrumentation Agent
  2. Registers binaries as COM objects. COM registration ensures that CLR will pick dll of the correct bittness as IIS may run in either.
  3. Sets environment variables for IIS by setting registry keys HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W3SVC\Environment and HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\WAS\Environment
  4. Suggest to restart IIS to apply those environment variables

Azure WebSites extension:

  1. Unpack Runtime Instrumentation Agent binaries to extension folder
  2. Detects site bittness to set proper environment variables
  3. Set environment variables for IIS by modifying applicationhost.config

Note, in both cases environment variables are set globally for all applications. So Runtime Instrumentation Agent may work with different versions of Application Insight SDK even if they are loaded in a single process. Basically, the only purpose of it is to enable code injection by SDK. You can enable runtime instrumentation agent for any application and if this application is not using Application Insights - Runtime Instrumentation Agent will do nothing.

Application Insights Requests Tracking - More Than Just a Begin and End

| Comments

I’ve already mentioned that Application Insights using http modules to track requests data. Tracking requests is actually quite straightforward task and can be easily implemented in couple lines of code on begin of request:

1
2
3
4
5
6
private static void BeginRequest(HttpContext ctx)
{
    var sw = new Stopwatch();
    sw.Start();
    ctx.Items["request-tracking-watch"] = sw;
}

and simple code on end of request:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
private static void EndRequest(HttpContext ctx)
{
    var sw = (Stopwatch)ctx.Items["request-tracking-watch"];
    sw.Stop();
            
    RequestTelemetry rt = new RequestTelemetry(
        name: ctx.Request.Path,
        timestamp: DateTimeOffset.Now,
        duration: sw.Elapsed,
        responseCode: ctx.Response.StatusCode.ToString(),
        success: 200 == ctx.Response.StatusCode
    );
    rt.Url = ctx.Request.Url;
    rt.HttpMethod = ctx.Request.HttpMethod;

    TelemetryClient rtClient = new TelemetryClient();
    rtClient.TrackRequest(rt);
}

Note: we actually send request start time as a timestamp. In the code example above I simplified code a little bit and send end time as a timestamp.

You can track requests from console application, worker role or OWIN middleware. It is easy and straightforward. However Application Insights Web nuget has more logic. Here is the list of ApplicationInsights.config settings controlling additional data collection (on the moment of this writing - version 0.12 of Application Insights SDK):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
<TelemetryModules>
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.TelemetryModules.WebRequestTrackingTelemetryModule, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.TelemetryModules.WebExceptionTrackingTelemetryModule, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.TelemetryModules.WebSessionTrackingTelemetryModule, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.TelemetryModules.WebUserTrackingTelemetryModule, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.WebApplicationLifecycleModule, Microsoft.ApplicationInsights.Extensibility.Web" />
</TelemetryModules>
<TelemetryInitializers>
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.TelemetryInitializers.WebOperationNameTelemetryInitializer, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.TelemetryInitializers.WebOperationIdTelemetryInitializer, Microsoft.ApplicationInsights.Extensibility.Web" />
</TelemetryInitializers>

<ContextInitializers>
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.AzureRoleEnvironmentContextInitializer, Microsoft.ApplicationInsights.Extensibility.Web" />
</ContextInitializers>

Here is a short overview of what else http module is doing under the hood and what kind of data is collected by these telemetry modules, context and telemetry initializers.

  1. Smart request name calculation (WebRequestTrackingTelemetryModule). Request name is used to group and correlate requests in UI. For instance, you can see average duration grouped by request name. Or search traces and exceptions for “similar” requests where similar means the same name. That’s why request name for MVC application is reported as “VERB Controller/Action” (for example “GET Home/Index”). It should not contain any unique identifiers, otherwise there will be too many groups in UI and it will be less usable. update: see new post for more information
  2. Track Operation ID for traces inside request(WebOperationNameTelemetryInitializer & WebOperationIdTelemetryInitializer). I’ve already mentioned default telemetry initializers that Web nuget installs. These telemetry initializers ensures that all traces, events and dependencies will be associated with request they called from.
  3. Exceptions collection (WebExceptionTrackingTelemetryModule). Application Insights http module attempts to collect exceptions happening in your application. I’m saying attempts as there are many cases when exceptions object will be cleared out by the moment http module’s code executes. This article shed some light on how to enhance exception collection logic for different technologies.
  4. Track users of your application (WebSessionTrackingTelemetryModule, WebUserTrackingTelemetryModule). Whenever Application Insights SDK get a request that doesn’t have application insights user tracking cookie (set by Application Insights JS snippet) it will set this cookie and start a new session. Application Insights SDK sets cookies carefully so cacheability of ASP.NET pages wouldn’t be broken. With user and session information you’ll see usage charts even if your application is a REST service called via AJAX by another web application.
  5. Get azure role name for cloud services (AzureRoleEnvironmentContextInitializer). When application initializes it attempts to get azure role name in case it is running on azure.
  6. Something else (WebRequestTrackingTelemetryModule, WebApplicationLifecycleModule et al). There are some logic, specific to http modules and IIS implementation that helps us ensure the best data collection. For instance, when routing happening we making sure that you will get one request data item even if multiple handlers were executed and http module received multiple notifications. We also collect some diagnostics information in case something goes wrong and send it to the portal so you can take an action.

I plan to cover some of these data collection aspects in more details going forward. Please leave a comment if you have questions or want more information on a particular aspect of data collection.

BTW, Application Insights will definitely support asp.net vNext so OWIN-based implementation of web requests tracking is on the way.

Application Insights Extension for Azure Websites…

| Comments

…and configuring Application Insights when you don’t have access to sources.

There is no right answer on whether Service Oriented Architecture is good or bad. Some people likes it. Some are using Azure WebSites to build applications with this paradigm in mind as Azure WebSites are cheap, easy to manage, fast to deploy and scale. With service oriented architecture for every particular web site it’s important to understand not only how this web site behaves, but also how your “dependencies” - services you are calling into - are doing.

Application Insights allows to monitor dependencies for your application. Today to track dependencies Application Insights using Profiling API to inject code into every http and sql call. You will need to use Status Monitor to enable it. However Status Monitor can’t be used for azure web site. That’s why we just released Azure WebSite extension.

This very first release of Application Insights Azure WebSite extension works in assumption that your application has Application Insights of version 0.12+ already configured. Once installed for your WebSite (go to extensions tab and click “Add”), it will enable collection of dependencies information for your Azure WebSite.

In future releases Application Insights Azure WebSite extension will enable Application Insights for any application - even if it doesn’t have Application Insights configured. I want to show you a small hack that you can use to enable Application Insights for an WebSite that you cannot recompile. This post will not have many technical details - just step-by-step instruction with the pictures.

Ok, imagine you have an application. For example it may be Orchard CMS blog from gallery:

  • Create Azure WebSite from gallery. Use Orchard CMS as a template

"create orchard1 application"

  • Configure Orchard. I’ve used Azure SQL server as Application Insights wouldn’t monitor SQL CE

"configure orchard"

  • You can always connect to your web site using WebMatrix. Here magic starts. Download your web site locally

"open in webmatrix"

  • Once downloaded I created new web site on my local IIS server

"create web site in IIS"

  • This was a hack that I mentioned. Now as you have a web site on your local server you can use Status Monitor to add Application Insights to it

"enable application insights for web site"

  • Now when you’d attempt to upload changes - it will upload all changes Status Monitor did

"upload changes"

  • I haven’t installed extension for my Azure WebSite. So I see requests to Orchard CMS, but do not see dependencies yet

"no dependencies yet"

  • Now you go to extensions blade and enable Application Insights extension

"enable extension"

  • Very next request to your azure web site will run a bit longer - Application Insights instrumenting your application. After this you most probably wouldn’t notice the noise created by monitoring dependencies. You can see how long your Azure WebSite spent in every dependency. You can see that Orchard has two dependency out of the box - http and sql

"dependencies chart"

  • You can also see what exact external calls were made for every particular request:

"dependencies for request"

It is very easy to use Application Insights Azure WebSite extension. It is also easy to configure Application Insights from Visual Studio. You can also enable Application Insights to your web site even if you cannot recompile it - Status Monitor and a simple hack - registering it in local IIS - will help you.

Use Azure WebSites to Host Custom Maven Repository

| Comments

I was trying to set up a temporary maven repository to share java libraries. It turns out you can easily do it for free using azure WebSites. Here are step-by-step instructions:

  • Create Azure WebSite. Name “m2” was already taken so I took dotm2 - here it is: http://dotm2.azurewebsites.net/.
  • Obtain ftp location and deployment credentials. I’d recommend to use site-specific credentials, not user ones. The difference is explained here. In ibiza portal you need to open web site blade and find “Get publish profile” button in blade’s title. It is not visible by default so you’ll need to click on “…” in the blade’s title. I got file like this:
1
2
3
4
5
6
7
8
9
10
11
<publishData>
  <morePublishingOptions>
  <publishProfile
      profileName="dotm2 - FTP"
      publishMethod="FTP"
      publishUrl="ftp://waws-prod-dm1-003.ftp.azurewebsites.windows.net/site/wwwroot"
      ftpPassiveMode="True"
      userName="dotm2\$dotm2"
      userPWD="password goes here"
  </publishProfile>
</publishData>

Now you can access web site content by FTP.

  • Since I’ve already opened FTP window to try out deployment credentials - I’ll make one last piece of website configuration. Enable directory browsing. Just create a web.config under “site/wwwroot” with this content:
1
2
3
4
5
6
7
8
9
10
<configuration>
  <system.webServer>
      <directoryBrowse enabled="true" showFlags="Date,Time,Extension,Size" />
      <staticContent>
          <mimeMap fileExtension=".pom" mimeType="application/xml" />
          <mimeMap fileExtension=".md5" mimeType="text/plain" />
          <mimeMap fileExtension=".sha1" mimeType="text/plain" />
      </staticContent>
  </system.webServer>
</configuration>

update(1/3/2015): At the moment of publication I forgot to configure mime types so pom files wasn’t accessible and gradle wasn’t able to resolve dependencies

  • Now FTP deployer can be used to upload jars to my repository. If you are using gradle maven plugin can be configured like shown below (build.gradle file):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
apply plugin: 'maven'

configurations {
    deployerJars
}

repositories {
    mavenCentral()
}

dependencies {
    deployerJars "org.apache.maven.wagon:wagon-ftp:2.8"
}

version = "1.0.0-SNAPSHOT"
group = "com.microsoft.applicationinsights"

uploadArchives {
    repositories {
        mavenDeployer {
            configuration = configurations.deployerJars

            repository(url: "ftp://waws-prod-dm1-003.ftp.azurewebsites.windows.net/site/wwwroot/repository/") {
                authentication(userName: "dotm2\\\$dotm2", password:javamavenuserpassword)
            }
         }
    }
}

Note, I’ve used variable “javamavenuserpassword” that you’ll need to supply to gradle to upload artifacts.

  • Build and upload artifacts:
1
gradlew uploadArchives -Pjavamavenuserpassword=<password goes here>

That’s it. Now you can use artifacts from your repository like this:

1
2
3
4
5
6
7
repositories {
    mavenCentral()

    maven {
        url 'http://dotm2.azurewebsites.net/repository/'
    }
}

ApplicationInsights.config File Search Order

| Comments

I’ve already mentioned how to programmatically set instrumentation key and mentioned how to configure telemetry initializers. In fact you can completely remove ApplicationInsights.config file and make all configuration programmatically. However it is handy to have a single file containing all the settings so you don’t need to recompile an application to change them.

When you add application insights to your .NET application - ApplicationInsights.config file being created and marked as a content. So it will be copied to output directory of your application. In general it will follow the same behavior as app.config file - copied side by side with executable for windows application and next to web.config for web application.

So it will be logical to expect that Application Insights SDK will be searching this file next to app.config. However for certain scenarios this algorithm doesn’t work and there is a gotcha here. Application Insights SDK is using the following order of searching ApplicationInsights.config file:

  1. bin folder of application - from Application Insigths Core assembly it’s Assembly.GetExecutingAssembly().CodeBase
  2. base directory - side by side with web.config for ASP.NET applications (AppDomain.CurrentDomain.BaseDirectory)

This behavior caused some problems already. There were cases when some random ApplicationInsights.config file was deployed to the bin folder of web application as well as correct one into content folder. So modifications of configuration file you’d expect to be used will not take an effect:

Proxy Application Insights Events

| Comments

Sometimes you want to check that telemetry data is collected by Application Insights SDK and being sent to the endpoint. It is easy to do in Visual Studio - you will see JSON of telemetry items in debug window or use Fiddler to see data being sent by application running in IISEpress. However if your application is running in IIS under application pool identity or local system account Fiddler will not pick up events. It is also challenging to do when application is running in production environment and you don’t want to install fiddler there.

Here is how you can do it.

Install and configure reverse proxy. I’m using Fiddler as a reverse proxy. It is very simple to configure it. If you are troubleshooting production environment you can have fiddler on machine accessible from one running your application.

In my example I have fiddler on machine where my application is running. So I used this code snippet in OnBeforeRequest handler:

1
if (oSession.host.toLowerCase() == "localhost:8888") oSession.host = "dc.services.visualstudio.com";

You may need to replace “localhost” to the name of machine running fiddler.

Redirect application Insights SDK traffic to this proxy. Here is a piece of XML you need to insert into ApplicationInsights.config file of your application. Note, this will work for application Insights SDK 0.12. It had different format for previous versions and may change in future:

1
2
3
4
5
<TelemetryChannel>
  <InProcess>
      <EndpointAddress>http://localhost:8888/v2/track</EndpointAddress>
  </InProcess>
</TelemetryChannel>

Update: Please note, format of configuration file changed in 0.13 SDK. Now you should not specify “InPorcess” tag:

1
2
3
<TelemetryChannel>
  <EndpointAddress>http://localhost:8888/v2/track</EndpointAddress>
</TelemetryChannel>

Again, if you are using different machine you will need to replace localhost to the name of machine running your proxy server (or fiddler in my case).

Redirect JavaScript-generated traffic. It is rare, but sometimes you may want to redirect JavaScript events. It is rare since JavaScript events may be easily viewed using browser tools and fiddler will always pick them up. However if you want to proxy JavaScript-generated telemetry data from all your customers this may be handy. All you need to do is to add endpointUrl property in Application Insights JavaScript code snippet:

1
2
3
4
{
    instrumentationKey: "e62632f1-81af-41d2-b8e8-9df22f10d9c3",
    endpointUrl: "//localhost:8888/v2/track"
}

Now you can see all Application Insights telemetry in fiddler (or other reverse proxy tool).

On Vacation

| Comments

On vacation for couple weeks… Leave a comment if you like this blog and want me to keep posting after vacation.

Tracking Static Content With Application Insights HttpModule

| Comments

Application Insights Web Nuget registers itself as http module to hook up in IIS request processing pipeline and start collecting requests information. You may notice that it registers both - classic mode http module and integrated mode since we don’t know at design time what mode your application will be running in. One caveat here is that this nuget also sets validateIntegratedModeConfiguration flag to false so IIS wouldn’t complain on classic mode registration when running in integrated mode.

1
2
3
<system.webServer>
  <validation validateIntegratedModeConfiguration="false" />
</system.webServer>

Since most application these days running in integrated mode you can safely remove this setting and one of http module registration to keep your web.config clean.

Since application insights is just a regular http module you can configure it even further. For instance, you may have some static html pages, served by IIS, not by ASP.NET, that you want to monitor - see who and when have accessed them. When running in integrated mode of IIS you can do it by setting runAllManagedModulesForAllRequests to true and remove precondition=”managedHandler” for Application Insights http module. This will start tracking ALL requests to static files as requests. There is no filtering capability so you’ll see all static content and will need to use search explorer on ibiza portal to see specific pages. Here is how your modules section should look after modifications:

1
2
3
4
5
6
<system.webServer>
  <modules runAllManagedModulesForAllRequests="true">
    <!--<add name="ApplicationInsightsWebTracking" type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebRequestTrackingModule, Microsoft.ApplicationInsights.Extensibility.Web" preCondition="managedHandler" />-->
    <add name="ApplicationInsightsWebTracking" type="Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebRequestTrackingModule, Microsoft.ApplicationInsights.Extensibility.Web" />
  </modules>
</system.webServer>

Once configured you’ll start seeing static content as requests collected on portal:

Since this blog is statically generated using octopress and hosted as azure web site I plan to use this technique to monitor requests made to it. I’ll post an update when will try it.

Telemetry Initializers

| Comments

Application Insights .NET SDK has number of extensibility points. One of them is called telemetry initializer. Telemetry initializer is a class implementing ITelemetryInitializer interface. The only method of this interface “Initialize” is called whenever a TraceFoo method is called for one of telemetry data items (Event, Metric, Request, Exception, etc.).

The Application Insights web SDK comes with two default telemetry initializers - web operation name and Id initializers:

1
2
3
4
<TelemetryInitializers>
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.TelemetryInitializers.WebOperationNameTelemetryInitializer, Microsoft.ApplicationInsights.Extensibility.Web" />
  <Add Type="Microsoft.ApplicationInsights.Extensibility.Web.TelemetryInitializers.WebOperationIdTelemetryInitializer, Microsoft.ApplicationInsights.Extensibility.Web" />
</TelemetryInitializers>

These initializers are used to mark every collected telemetry item with the current web request identity so that traces and exception can be correlated to corresponding requests:

The trace telemetry in this example, has the following context populated by telemetry initializers mentioned above:

1
"operation":{"id":"1940098063557174680","name":"GET Home/Index"}

It is easy to implement your own telemetry initializer. Say, you want to mark every telemetry data item with ETW ActivityID and System.Diagnostics ActivityID. To do this, first, you create a class that implements the ITelemetryInitializer interface. In the interface’s Initialize method you can fill out telemetry data item properties:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
namespace ApmTips.Tools
{
    using Microsoft.ApplicationInsights.Extensibility;
    using Microsoft.Diagnostics.Tracing;
    using System.Diagnostics;

    public class ExtendedIDTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(Microsoft.ApplicationInsights.Channel.ITelemetry telemetry)
        {
            telemetry.Context.Properties["ETW.ActivityID"] = EventSource.CurrentThreadActivityId.ToString();
            telemetry.Context.Properties["E2ETrace.ActivityID"] = Trace.CorrelationManager.ActivityId.ToString();
        }
    }
}

You then need to register your telemetry initializer using one of the following two options: Adding it to ApplicationInsights.config file:

1
2
3
<TelemetryInitializers>
  <Add Type="ApmTips.Tools.ExtendedIDTelemetryInitializer, ApmTips.Tools" />
</TelemetryInitializers>

or programmatically:

1
Microsoft.ApplicationInsights.Extensibility.TelemetryConfiguration.Active.TelemetryInitializers.Add(new ExtendedIDTelemetryInitializer());

This is what every data item will be marked with after you start your application with the new telemetry initializer configured:

1
"properties":{"ETW.ActivityID":"00000000-0000-0000-0000-000000000000","E2ETrace.ActivityID":"00000000-0000-0000-0700-0080000000f9"}

And here is how it looks like in UI:

Telemetry initializers are a powerful, but dangerous tool. They are called synchronously and block program execution flow; if written poorly they can harm application performance.

The following example demonstrates synchronous execution of telemetry initializers. It traces every telemetry initializer into the file with the stack trace where telemetry data item was created from:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
namespace ApmTips.Tools
{
    using Microsoft.ApplicationInsights.Extensibility;
    using System;
    using System.Diagnostics;
    using System.IO;

    public class DiagnosticsTraceTelemetryInitializer : ITelemetryInitializer
    {
        public void Initialize(Microsoft.ApplicationInsights.Channel.ITelemetry telemetry)
        {
            var stack = new StackTrace();

            using (StreamWriter sw = new StreamWriter(Environment.ExpandEnvironmentVariables("%tmp%\\ai-log.txt"), true))
            {
                sw.WriteLine(telemetry.GetType().Name + " was traced");
                sw.WriteLine(" from " + stack.ToString());
            }
        }
    }
}

Here is the output this telemetry initializer generates for a single request with Trace statement in controller. You can see that the Trace.Write method was called from the home controller (WebApplication3.Controllers.HomeController.Index). This in turn called the Application Insights trace listener which finally called Track method and our telemetry initializer:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
TraceTelemetry was traced from
   at ApmTips.Tools.DiagnosticsTraceTelemetryInitializer.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Track(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TraceListener.ApplicationInsightsTraceListener.Write(String message)
   at System.Diagnostics.TraceInternal.Write(String message)
   at System.Diagnostics.Trace.Write(String message)
   at WebApplication3.Controllers.HomeController.Index()
   at lambda_method(Closure , ControllerBase , Object[] )
   at System.Web.Mvc.ActionMethodDispatcher.Execute(ControllerBase controller, Object[] parameters)
   at System.Web.Mvc.ReflectedActionDescriptor.Execute(ControllerContext controllerContext, IDictionary`2 parameters)
   at System.Web.Mvc.ControllerActionInvoker.InvokeActionMethod(ControllerContext controllerContext, ActionDescriptor actionDescriptor, IDictionary`2 parameters)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.ActionInvocation.InvokeSynchronousActionMethod()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<BeginInvokeSynchronousActionMethod>b__39(IAsyncResult asyncResult, ActionInvocation innerInvokeState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`2.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethod(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3d()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.AsyncInvocationWithFilters.<>c__DisplayClass46.<InvokeActionMethodFilterAsynchronouslyRecursive>b__3f()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass33.<BeginInvokeActionMethodWithFilters>b__32(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeActionMethodWithFilters(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<>c__DisplayClass2b.<BeginInvokeAction>b__1c()
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.<>c__DisplayClass21.<BeginInvokeAction>b__1e(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResult`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncControllerActionInvoker.EndInvokeAction(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<BeginExecuteCore>b__1d(IAsyncResult asyncResult, ExecuteCoreState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncResultWrapper.End(IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Controller.EndExecuteCore(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.<BeginExecute>b__15(IAsyncResult asyncResult, Controller controller)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncResultWrapper.End(IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Controller.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.Controller.System.Web.Mvc.Async.IAsyncController.EndExecute(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.<BeginProcessRequest>b__5(IAsyncResult asyncResult, ProcessRequestState innerState)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncVoid`1.CallEndDelegate(IAsyncResult asyncResult)
   at System.Web.Mvc.Async.AsyncResultWrapper.WrappedAsyncResultBase`1.End()
   at System.Web.Mvc.Async.AsyncResultWrapper.End[TResult](IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.Async.AsyncResultWrapper.End(IAsyncResult asyncResult, Object tag)
   at System.Web.Mvc.MvcHandler.EndProcessRequest(IAsyncResult asyncResult)
   at System.Web.Mvc.MvcHandler.System.Web.IHttpAsyncHandler.EndProcessRequest(IAsyncResult result)
   at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

RequestTelemetry was traced from
   at ApmTips.Tools.DiagnosticsTraceTelemetryInitializer.Initialize(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.TelemetryClient.Track(ITelemetry telemetry)
   at Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.TelemetryModules.WebRequestTrackingTelemetryModule.OnEndRequest(RequestTelemetryContext state, HttpContext platformContext)
   at Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebPlatformModuleAdapter.ExecuteStepExceptionSafe[TX,TY](String stageName, Action`2 stage, TX state, TY platfromContext)
   at Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebPlatformModuleAdapter.OnEndRequest(HttpContext platformContext)
   at Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebRequestTrackingModule.OnCallbackExceptionSafe(String callbackName, HttpContext platformContext, Action`1 action)
   at Microsoft.ApplicationInsights.Extensibility.Web.RequestTracking.WebRequestTrackingModule.OnEndRequest(Object sender, EventArgs eventArgs)
   at System.Web.HttpApplication.SyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
   at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
   at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error)
   at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb)
   at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)
   at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags)

Use telemetry initializers wisely for properties that changes from one data item to another. There are other mechanisms to set global properties that will have the same value for all data items in the process.