APM tips blog

Blog about application monitoring.

Where Are the Telemetry Context Properties?

| Comments

Recently the change was made. Upgrading from Application Insights SDK version 0.13 to the version 0.14 you may notice that some public interfaces were changed. Specifically, public property Properties was removed from TelemetryContext class. Yes, one that mentioned in the documentation and blog post. One that is very important to enable many scenarios.

It is not a change we plan to keep for a long time. Public interface will be reverted back soon.

I thought a lot on how to explain this change and what led to this. Now I know what was the motivation of this change, I can tell that semantic versioning is designed to experiment with API surface, I know why it wasn’t immediately reverted. However I better not to go into details. I want to assure you that we understand that such a big API change should not happen again without the notice.

Version 0.14 of SDK brings great new features like ability to monitor custom performance counters. If you want to use this features and need custom properties I’d recommend to wait for the new version of SDK. If you don’t want to wait - here is a workaround (it will only work in 0.14 SDK).

If you are using context initializer - convert it to telemetry initializer. Now in telemetry initializer use ISupportProperties interface to set properties for telemetry item:

1
2
3
4
5
6
7
8
9
10
11
public class WorkaroundTelemetryInitializer : ITelemetryInitializer
{
    public void Initialize(ITelemetry telemetry)
    {
        var propsTelemetry = telemetry as ISupportProperties;
        if (propsTelemetry != null)
        {
            propsTelemetry.Properties["environment"] = "development";
        }
    }
}

Again, this workaround will only be needed for the version 0.14 and will not work in the next version of SDK.

Javascript Snippet Explained

| Comments

Big thanks to Scott Southwood who helped to prepare this post.

For end user monitoring Application Insights requires to add this JavaScript snippet to the page:

1
2
3
4
5
6
7
8
var appInsights=window.appInsights||function(config){
    function s(config){t[config]=function(){var i=arguments;t.queue.push(function(){t[config].apply(t,i)})}}var t={config:config},r=document,f=window,e="script",o=r.createElement(e),i,u;for(o.src=config.url||"//az416426.vo.msecnd.net/scripts/a/ai.0.js",r.getElementsByTagName(e)[0].parentNode.appendChild(o),t.cookie=r.cookie,t.queue=[],i=["Event","Exception","Metric","PageView","Trace"];i.length;)s("track"+i.pop());return config.disableExceptionTracking||(i="onerror",s("_"+i),u=f[i],f[i]=function(config,r,f,e,o){var s=u&&u(config,r,f,e,o);return s!==!0&&t["_"+i](config,r,f,e,o),s}),t
}({
    instrumentationKey:"d2cb4759-8e2c-453a-996c-e65c9d0e946a"
});

window.appInsights=appInsights;
appInsights.trackPageView();

This snippet will make initial set up for end user tracking and then download the reset of the monitoring logic from CDN.

There are number of reasons why this script needs to be injected into the page html code. First, placing script into the page will not require additional download at the early stage of the page loading phase. So page loading time will not be affected. Second, it provides you API to track metrics and events so you don’t need to check whether the full Application Insights script is already loaded or not. Third, this script is working in application domain so it can subscribe on onerror callback and get a full error stack. Due to security restrictions browser will not give you the full error stack if you subscribe on onerror callback from the script downloaded from the different domain. It also takes cookies from application domain so they can be used for user/session tracking. Here is more detailed explanation of what it is doing.

First, we check that Application Insights object haven’t been created yet. If so - we will create it:

1
var appInsights = window.appInsights || function (config) {

Next goes a helper function that will define a new callback with the name passed as an argument. Methods like appInsights.trackEvent will be created using this helper. Initial implementation of this method is to put an object into the queue for further processing. “Real” implementation will come with Application Insights javascript file downloaded from CDN later:

1
2
3
4
5
6
7
8
   function s(config) {
         t[config] = function () {
                var i = arguments;
                t.queue.push(function () {
                       t[config].apply(t, i)
                })
         }
   }

Now real appInsights object is defined under the pseudo name t. Initially it only contains config field. More fields and methods will be created later:

1
   var t = { config: config },

Bunch of constants and variables:

1
2
3
4
5
6
                r = document,
                f = window,
                e = "script",
                o = r.createElement(e),
                i,
                u;

In the beginning of this for loop snippet will create script element in DOM model so Application Insights javascript file will be download later from CDN:

1
2
   for (o.src = config.url || "//az416426.vo.msecnd.net/scripts/a/ai.0.js",
                r.getElementsByTagName(e)[0].parentNode.appendChild(o),

Store domain cookies in appInsights object:

1
                t.cookie = r.cookie,

Create an events queue:

1
                    t.queue = [],

And now there is an actual loop. Iterating thru the collection methods like appInsights.trackEvent will be created (remember helper s in the very beginning of the snippet):

1
2
3
4
                i = ["Event", "Exception", "Metric", "PageView", "Trace"];
                i.length;
                )
                s("track" + i.pop());

Now, subscribe to onerror callback to catch javascript errors in page initialization. You can disable thie logic by setting disableExceptionTracking property to false:

1
   return config.disableExceptionTracking ||

Using the same helper from the beggining of script define appInsights._onerror method:

1
2
                (i = "onerror",
                s("_" + i),

Now save existing window.onerror (f is window, see above in constants section) callback from the page and replace it with the new implementation. New implementation will chain the call to appInsights._onerror and call to initial window.onerror implementation:

1
2
3
4
5
                u = f[i],
                f[i] = function (config, r, f, e, o) {
                       var s = u && u(config, r, f, e, o);
                       return s !== !0 && t["_" + i](config, r, f, e, o), s
                }),

Return freshly created appInsights object from the method:

1
2
                t
}

Constructor function is created. Now call it, providing initial configuration. The only required piece of configuration is instrumentationKey:

1
2
3
({
       instrumentationKey: "@RequestTelelemtry.Context.InstrumentationKey"
});

Now we store appInsights as a global variable and save into the queue pageView event:

1
2
window.appInsights = appInsights;
appInsights.trackPageView();

Read more on out-of-the box usage analytics. Inforamtion on tracking custom events and metrics here.

Status Monitor for Cloud Services

| Comments

Cool article on how to install Status Monitor on your web role. Don’t forget to install Microsoft.ApplicationInsights.Web NuGet package for your web project.

Now in order to track dependencies on worker roles you need to do the same and one additional step - set environment variables to tell worker role where those new components are:

1
2
3
4
5
6
7
8
9
10
11
<ServiceDefinition name="MyService" xmlns="http://schemas.microsoft.com/ServiceHosting/2008/10/ServiceDefinition">
   <WorkerRole name="<name>">
      <Runtime>
         <Environment>
            <Variable name="COR_ENABLE_PROFILING" value="1" />
            <Variable name="COR_PROFILER" value="{324F817A-7420-4E6D-B3C1-143FBED6D855}" />
            <Variable name="MicrosoftInstrumentationEngine_Host" value="{CA487940-57D2-10BF-11B2-A3AD5A13CBC0}" />
         </Environment>
      </Runtime>
   </WebRole>
</ServiceDefinition>

More on how to set environment variables for your worker role is here.

Don’t forget to install NuGet package Microsoft.ApplicationInsights.RuntimeTelemetry on your worker role and instantiate TelemetryClient at least once on worker role startup.

Bug With StatusMonitor 5.0 Uninstall

| Comments

Once you’ve started using Application Insights it is essential to install Status Monitor. Status Monitor will enable dependencies tracking as mentioned in one of the previous posts. We use Status Monitor to track dependencies for our own internal services. As Brian Harry wrote about Visual Studio Online services “smaller services are better” we have quite a few interconnected services. Knowing that service you depend on became slower or started failing at a glance is very important.

As we monitor our own services with Application Insights - for some of our services we have startup task that installs Status Monitor. Month ago a small bug in Status Monitor was one of the reasons of quite a serious outage.

Status Monitor in a nutshell is just an installer of Application Insights components and UI to see status of monitoring (as name suggests). By itself it doesn’t collect any application telemetry or running any background services. So you may ask - why I’m saying that Status Monitor caused the outage?

And the answer is simple. We are committed to dog food. So we try to run the latest bits of Application Insights SDK and every service restart we are trying to download the latest components. Unfortunately, Status Monitor 5.0 has an issue - when it uninstalls it leaves some registry settings in bad state. Specifically, it make Environment string empty for these three services:

1
2
3
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\IISADMIN
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W3SVC
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\WAS

So after uninstall IIS will try to start and will fail as it doesn’t expect Environment string to be empty. Here is how it surfaced when you run iisreset /start command:

1
The IIS Admin Service or the World Wide Web Publishing Service, or a service dependent on them failed to start.  The service, or dependent services, may had an error during its startup or may be disabled

And these messages you’ll see in Event Log:

1
2
3
4
5
6
7
8
9
10
11
Log Name:      System
Source:        Microsoft-Windows-IIS-IISReset
Date:          2/25/2015 9:08:15 AM
Event ID:      3201
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      sergey-surface
Description:
IIS start command received from user SERGEY-SURFACE\Sergey. The logged data is the status code.
1
2
3
4
5
6
7
8
9
10
11
12
Log Name:      System
Source:        Service Control Manager
Date:          2/25/2015 9:08:15 AM
Event ID:      7000
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      sergey-surface
Description:
The Windows Process Activation Service service failed to start due to the following error: 
The parameter is incorrect.
1
2
3
4
5
6
7
8
9
10
11
12
Log Name:      System
Source:        Service Control Manager
Date:          2/25/2015 9:08:15 AM
Event ID:      7001
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      sergey-surface
Description:
The World Wide Web Publishing Service service depends on the Windows Process Activation Service service which failed to start because of the following error: 
The parameter is incorrect.

Solution is simple - right after uninstall of Status Monitor 5.0 - install the new one or delete “Environment” string from registry keys mentioned above.

There will be some excited features coming in Status Monitor in future and I hope you will never run into issue upgrading it.

Request Name and Url

| Comments

In my previous post on web requests tracking http module I mentioned that Application Insights http module has some smart logic to collect request name. This logic is needed to make meaningful aggregation on UI side. In the screenshot below you see that requests are grouped by request name. Aggregations like number of requests and average execution time for requests were calculated for some pages. And those aggregations completely unusable for requests to “__browserLink”:

Here is how request name calculation logic works today:

  1. ASP.NET MVC support. Request name is calculated as “VERB controller/action”.
  2. ASP.NET MVC Web API support. Following the logic above both requests “/api/movies/” and “/api/movies/5” will be resulted in “GET movies”. So to support Web API request name includes the list of all names of routing parameters in case if “action” parameter wasn’t found. In example above you’ll see requests “GET movies” and “GET movies[id]”.
  3. If routing table is empty or doesn’t have “controller” - HttpRequest.Path will be used as a request name. This property doesn’t include domain name and query string.

Application Insights web SDK will send request name “as is” with regards to letter case. Grouping on UI will be case sensitive so “GET Home/Index” will be counted separately from “GET home/INDEX” even though in many cases they will result in the same controller and action execution. The reason for that is that urls in general are case sensitive (http://www.w3.org/TR/WD-html40-970708/htmlweb.html) and you may want to see if all 404 happened when customer were requesting the page in certain case.

Known issues:

  1. There is no smart request name calculation for attributes-based routing today
  2. Custom implementation of routing is not supported out of the box. You’ll need to implement your own WebOperationNameTelemetryInitializer implementation to override standard behavior.

More on ApplicationInsights.config

| Comments

If you were using my instructions on proxying Application Insights data - please note that format of configuration file has changed and you should not use tag “InProcess” when specifying an endpoint. I updated that post and want to explain how ApplicationInsights.config instantiates objects. This applies to every object you can configure in this configuration file - be it TelemetryInitializer, ContextInitializer, TelemetryModule or Channel.

Main idea behind ApplicationInsights.config file is that this config file should not be required. In ideal world all aspects of monitoring should be coded into your application. That’s why we try to avoid any dependencies on file format or schema for SDK objects.

Every object you can configure in ApplicationInsights.config can define “Type”. It also may have any number of child xml nodes which will be used to initiate corresponding properties of constructed object. For instance the following configuration snippet will construct object of type “ApmTips.Tools.PropertiesContextInitializer, ApmTips.Tools” and assign value “Bar” to property “Foo”. Since this object is defined in ContextInitializers section Application Insights SDK will ensure that class implements “IContextInitializer” interface.

1
2
3
4
5
<ContextInitializers>
  <Add Type="ApmTips.Tools.PropertiesContextInitializer, ApmTips.Tools">
    <Foo>Bar</Foo>
  </Add>
</ContextInitializers>

Corresponding class should look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
namespace ApmTips.Tools
{
    using Microsoft.ApplicationInsights.DataContracts;
    using Microsoft.ApplicationInsights.Extensibility;

    public class PropertiesContextInitializer : IContextInitializer
    {
        public string Foo { get; set; }

        public void Initialize(TelemetryContext context)
        {
            context.Properties["Foo"] = this.Foo;
        }
    }
}

And when you run a program it will add additional property “Foo” with the value “Bar” to every telemetry data item:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
{
  "name":"Microsoft.ApplicationInsights.Request",
  "time":"2015-02-06T17:13:21.1222232+00:00",
  "iKey":"key",
  "tags":{
    ...
    "ai.device.model":"Surface Pro 3",
    "ai.device.machineName":"sergey-surface",
    "ai.operation.name":"GET Home/Index",
    "ai.operation.id":"3518850146076059859"
  },
  "data":{
    "baseType":"RequestData",
    "baseData":{
      "name":"GET Home/Index",
      "startTime":"2015-02-06T17:13:21.1222232+00:00",
      "duration":"00:00:02.5762099",
      "responseCode":"200",
      ...
      "properties":{
        "Foo":"Bar",
      }
    }
  }
}

This will apply to TelemetryChannel node as well. You can override the Type attribute of this node to specify your own channel. “DeveloperMode” and “EndpointAddress” are just public properties of the InProcessTelemetryChannel class that is assumed when Type is not specified.

Developer Mode

| Comments

Application Insights is truly integrated into Visual Studio development experience. It is easy to add telemetry to your project and see how it works on very early stages of development. There should be no surprises in production.

To enable the best development experience in Visual Studio, Application Insights SDK has a special mode called “Developer Mode”. When in Developer Mode Application Insights SDK behavior changes comparing to production environment:

  • you’ll see application insights data items in Debug Output window
  • data items will be sent immediately without buffering (default buffering interval on production is 1 minute)
  • “Track” method of SDK will throw exception if Instrumentation Key is not set (Application Insights SDK wouldn’t throw exceptions in production)

There are also nice integration in Visual Studio - popup window showing that data is flowing to Application Insights and with the direct link to application blade on the portal.

Developer Mode will be turned on automatically in Visual Studio. When you import Application Insights NuGet to your project the following targets file will be added:

1
<Import Project="..\packages\Microsoft.ApplicationInsights.0.12.0-build17386\tools\net40\Microsoft.ApplicationInsights.targets" Condition="Exists('..\packages\Microsoft.ApplicationInsights.0.12.0-build17386\tools\net40\Microsoft.ApplicationInsights.targets')" />

This target will enable Developer Mode by creating the following setting in ApplicationInsights.config file:

1
2
3
<TelemetryChannel>
  <DeveloperMode>true</DeveloperMode>
</TelemetryChannel>

However it will not modify the file you have in source control - it will copy modified file to output directory of the application. And because of configuration file search order SDK will be using this modified configuration file when running in Visual Studio.

You’ll typically deploy Release version of your application to production. So Developer Mode will not be turned on. However if for some reason you want to deploy Debug version - don’t forget to remove this file or just set DeveloperMode to false in ApplicationInsights.config yourself.

Track Dependencies in Console Application

| Comments

Update (11/17/2015): package RuntimeTelemetry was renamed to Microsoft.ApplicationInsights.DependencyCollector and some class names and namespaces were changed correspondingly.

Dependencies tracking in Application Insights is powerful feature that allows to see what SQL and Http calls your application makes. I’ve mentioned that you need to install Status Monitor or Azure WebSites extension to enable it for your web application. I don’t like magic and tools that configures something that I don’t quite understand. I think most of developers and especially devops thinks the same way. Hopefully after this post you can better understand how this feature works and will trust it more.

The main purpose of Status Monitor and Azure WebSites extension is to simplify Application Insights enablement for web applications. When you host your ASP.NET application in IIS or as Azure WebSite it has very predictable structure. So most of enablement steps can be automated. In this post I’ll show how to enable dependencies tracking feature for console application manually so you know what Status Monitor and Azure WebSites extensions automation makes under the hood. You can apply similar steps for any other application type - be it Windows Service, Worker Role or anything else.

Let’s assume you just created a simple console application called “TestDependency.exe” in Visual Studio. As it tests dependencies tracking - I make a simple http call in a Main method of this application:

1
2
3
4
5
6
7
var request = HttpWebRequest.Create("http://bing.com");
var response = request.GetResponse();
using (var s = new StreamReader(response.GetResponseStream()))
{
    Console.WriteLine(s.ReadToEnd().Length);
}
Console.ReadLine();

Now you need to include and configure Application Insights for this application and then enable dependencies tracking feature.

Install Application Insights

Every big feature in Application Insights is implemented in a separate nuget so you can choose what kind of monitoring you want to enable for your application. Dependencies tracking feature implemented in RuntimeTelemetry nuget package. So let’s install it:

1
Install-Package Microsoft.ApplicationInsights.RuntimeTelemetry -Pre

Once installed new telemetry module will appear in ApplicationInsights.config. This telemetry module is responsible for dependencies tracking. We call it remote dependencies module:

1
<Add Type="Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry.RemoteDependencyModule, Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry"/>

Lastly, as it is a console application, you need to set instrumentation key manually:

1
TelemetryConfiguration.Active.InstrumentationKey = "ec126cb1-9adc-4681-9cd4-0fcad33511c9";

When you compile your application plenty of new files will appear in bin/Debug folder. You’ll need to carry these files alongside with your application’s binary when you’ll deploying it. Here are the list of files you’ll need for 0.12 version of Application Insights SDK.

Application Insights configuration:

1
ApplicationInsights.config

Application Insights Core and dependencies:

1
2
3
4
5
Microsoft.ApplicationInsights.dll
Microsoft.Diagnostics.Tracing.EventSource.dll
Microsoft.Threading.Tasks.dll
Microsoft.Threading.Tasks.Extensions.Desktop.dll
Microsoft.Threading.Tasks.Extensions.dll

Files responsible for dependencies tracking:

1
2
3
4
Microsoft.ApplicationInsights.Extensibility.RuntimeTelemetry.dll
Microsoft.ApplicationInsights.Extensions.Intercept_x64.dll
Microsoft.ApplicationInsights.Extensions.Intercept_x86.dll
Microsoft.Diagnostics.Instrumentation.Extensions.Intercept.dll

Small note here - you may notice that RuntimeTelemetry nuget has a dependency to another nuget that is hidden in nuget.org: “Microsoft.Diagnostics.Instrumentation.Extensions.Intercept”. The only reason why it is hidden is that there is no reason to install it by itself, it only has utility features used by other Application Insights nugets.

Enable dependencies tracking feature

Dependencies tracking feature is based on code instrumentation in runtime. Application Insights decorates every call to http or SQL with prefix and postfix callbacks. Timer starts in prefix callback and on postfix - all information regarding dependency call like url and duration being send to Application Insights service. The only way today to notify CLR (common language runtime) to allow code instrumentation is to set certain environment variables before you run your application.

To enable dependencies tracking feature you should set these environment variables:

1
2
3
SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}
SET MicrosoftInstrumentationEngine_Host={CA487940-57D2-10BF-11B2-A3AD5A13CBC0}

These variables tells CLR to load certain COM object as a “profiler”. We call this object Runtime Instrumentation Agent as it’s main purpose is to enable code instrumentation in runtime, without application re-compilation. Settings above will only work on machine where Status Monitor installed as GUIDs points to COM objects that should be registered on computer.

If you are not a big fun of registering COM object (like me) - you can copy content of folder “%ProgramFiles%\Microsoft Application Insights\Runtime Instrumentation Agent” to any other folder, for instance output folder of your application:

Surely you should tell CLR where to look for COM objects by specifying path to corresponding dlls:

1
2
3
4
5
SET COR_ENABLE_PROFILING=1
SET COR_PROFILER={324F817A-7420-4E6D-B3C1-143FBED6D855}
SET COR_PROFILER_PATH="folder"\x86\MicrosoftInstrumentationEngine_x86.dll
SET MicrosoftInstrumentationEngine_Host={CA487940-57D2-10BF-11B2-A3AD5A13CBC0}
SET MicrosoftInstrumentationEngine_HostPATH="folder"\x86\Microsoft.ApplicationInsights.ExtensionsHost_x86.dll

That’s it. Just start console window, set environment variables and run your application. In my case I see this dependency event being sent to Application Insights:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
"ver":1,
"name":"Microsoft.ApplicationInsights.RemoteDependency",
"time":"2015-01-05T05:03:46.4427348+00:00",
"iKey":"ec126cb1-9adc-4681-9cd4-0fcad33511c9",
"device":{"id":"id is a required field for Microsoft.ApplicationInsights.Extensibility.Implementation.DeviceContext"},
"internal":{"sdkVersion":"0.12.0.17386","agentVersion":"0.12.0"},
"data": {
  "type":"Microsoft.ApplicationInsights.RemoteDependencyData",
  "item":{
      "ver":1,
      "name":"http://bing.com/",
      "kind":"Aggregation",
      "value":592,
      "count":1,
      "dependencyKind":"HTTP",
      "success":true,
      "async":true,
      "source":2,
      "properties":{"DeveloperMode":"true"}
  }
}

Looking at this JSON you see that we now send two versions - sdk version (0.12.0.17386) and agent version (0.12.0). Agent version here is a version of Runtime Instrumentation Agent.

Web applications and dependencies tracking

So what exactly Status Monitor and Azure WebSites extension are doing.

Status Monitor:

  1. Installs Runtime Instrumentation Agent binaries to %ProgramFiles%\Microsoft Application Insights\Runtime Instrumentation Agent
  2. Registers binaries as COM objects. COM registration ensures that CLR will pick dll of the correct bittness as IIS may run in either.
  3. Sets environment variables for IIS by setting registry keys HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\W3SVC\Environment and HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\WAS\Environment
  4. Suggest to restart IIS to apply those environment variables

Azure WebSites extension:

  1. Unpack Runtime Instrumentation Agent binaries to extension folder
  2. Detects site bittness to set proper environment variables
  3. Set environment variables for IIS by modifying applicationhost.config

Note, in both cases environment variables are set globally for all applications. So Runtime Instrumentation Agent may work with different versions of Application Insight SDK even if they are loaded in a single process. Basically, the only purpose of it is to enable code injection by SDK. You can enable runtime instrumentation agent for any application and if this application is not using Application Insights - Runtime Instrumentation Agent will do nothing.