APM tips blog

Blog about application monitoring.

WebApi Tracing Powered by ApplicationInsights in Visual Studio 2015 Update 1

| Comments

This blog post was written by Anastasia Baranchenkova

In Visual Studio 2015 Update 1 Application Insights telemetry is available right from the Visual Studio. And you can even get it without providing a valid instrumentation key.

I created a sample application that I describe in details later on. In this application I forward WebApi framework traces to Application Insights (it is a similar thing that SergeyK described in one of his previous posts but it is done differently: by implementing ITraceWriter interface).

Full sample is here.

So if you start debugging web hosted WebApi application even if you do not provide instrumentation key you can see all the traces rights in your Diagnostics Tools window. You can search there, filter and see actual json file that would have been sent if you provided a valid instrumentation key:

For self-hosted WebApi applications you would need to configure ApplicationInsights providing a valid instrumentation key because Diagnostics Tools hub does not currently support this type of application. But you still can see all the telemetry from the VS itself. For that you need to open: View ->Other Windows->Application Insights Search. From there you connect to your Azure Subscription and get back all the telemetry. You can select different time intervals, filter by type or property values and see each telemetry item details:

And now I want to describe in details how you can forward WebApi traces to ApplicationInsigts to get all this beauty.

WebApi web-hosted application

  1. Create WebApi web hosted application. If you have Azure subscription check “Add ApplicationInsights to project”.
  2. If you did not add ApplicationInsights on application creation then add the latest Application Insights for Web nuget through nugget package manager. (In this case you do not have instrumentation key in the ApplicationInsights configuration file and no data will be sent to the portal but you can add it that later and for debugging purposes you have all you need)
  3. Add Microsoft.AspNet.WebApi.Tracing nuget package.
  4. Add ApplicationInsightsTraceWriter.cs

ApplicationInsightsTraceWriter implements System.Web.Http.Tracing.ITraceWriter. Method Trace gets trace message parameters and converts it to ApplicationInsights trace or exception.

1
2
3
4
5
6
7
8
9
internal sealed class ApplicationInsightsTraceWriter : ITraceWriter
{
  public void Trace(HttpRequestMessage request, string category, TraceLevel level, Action<TraceRecord> traceAction)
    {
      message = this.systemDiagnosticsTraceWriter.Format(traceRecord);
      this.client.TrackTrace(message, GetSeverityLevel(level));
  }
}
  1. Add CompositTraceWriter.cs

CompositTraceWriter also implements ITraceWriter and is used in case if some other ITraceWriter is already registered by the application so that ApplicationInsightsTraceWriter does not replace existing but rather is added to the list of trace writers.

  1. Add Extensions.cs

This class adds HttpConfiguration.EnableApplicationInsights extension method that registers ApplicationInsightsTraceWriter.

1
2
writer = new ApplicationInsightsTraceWriter();
configuration.Services.Replace(typeof(ITraceWriter), writer);   
  1. In Global.asax add GlobalConfiguration.Configuration.EnableApplicationInsights();

WebApi self-hosted application

For a self-hosted application you will do almost the same but

  1. Instead of Web nuget package add ApplicationInsights API nuget
  2. Create application insights resource manually through the portal and set instrumentation key programmatically
1
TelemetryConfiguration.Active.InstrumentationKey = “MyKey”;
  1. Call config.EnableApplicationInsights() from Startup.Configuration where config would be an instance of HttpConfiguration that you create in this method for other registration purposes.

I would highly encourage to also read this blog post that explains how and when to use WebApi tracing. The example above demonstrates basic integration points while in the article above you can find detailed information on the best practices of how to use WebApi tracing.

Deploy to Azure With StyleCop

| Comments

I was working on enabling continues deployment from GitHub to Azure WebApp for the sample Glimpse and Application Insights integration application. It is easy to implement this integration. Simplest way is to create “Deploy to Azure” button in your GitHub repository like I explained in this blog post. You can also do it manually:

  1. Open Settings blade of your web app
  2. Click on Continuous deployment
  3. Select External Repository and set your repository URL

Once enabled - KuDu will pull sources from repository, compile it and deploy web application.

This time it didn’t work smoothly for me. I got an error:

1
2
3
4
5
6
7
8
9
10
MSBUILD : error : SA0001 : CoreParser : An exception occurred while parsing the 
file: System.IO.IOException, The specified registry key does not exist.. [D:\home
\site\repository\Glimpse.ApplicationInsights\Glimpse.ApplicationInsights.csproj]
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe" 
"D:\home\site\repository\Glimpse.ApplicationInsights.Sample\
Glimpse.ApplicationInsights.Sample.csproj" /nologo /verbosity:m /t:Build 
/t:pipelinePreDeployCopyAllFilesToOneFolder /p:_PackageTempDir="D:\local\Temp\
8d2f93a7a699626";AutoParameterizationWebConfigConnectionStrings=false;
Configuration=Release;UseSharedCompilation=false 
/p:SolutionDir="D:\home\site\repository\.\\"

My web project doesn’t have StyleCop enabled so the error was quite misleading. The good thing - this error message had original msbuild command. So I opened KuDu debug console using URL: https://<webAppName>.scm.azurewebsites.net/DebugConsole and typed the following command:

1
2
3
4
5
6
7
8
9
10
11
cd LogFiles

"D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe"
        "D:\home\site\repository\Glimpse.ApplicationInsights.Sample\Glimpse.ApplicationInsights.Sample.csproj"
        /nologo
        /verbosity:detailed
        /t:Build
        /t:pipelinePreDeployCopyAllFilesToOneFolder
        /p:_PackageTempDir="D:\local\Temp\8d2f93a7a699626";AutoParameterizationWebConfigConnectionStrings=false;Configuration=Release;UseSharedCompilation=false
        /p:SolutionDir="D:\home\site\repository\.\\"
> buildLog.txt

Note, the command text is different from the original. Specifically, I changed verbosity /verbosity:detailed and redirected output to the file > buildLog.txt. Resulting error message was much easier to troubleshoot:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
Using "StyleCopTask" task from assembly "D:\home\site\packages\StyleCop.MSBuild.4.7.49.1\build\..\tools\StyleCop.dll".
Task "StyleCopTask"
MSBUILD : error : SA0001 : CoreParser : An exception occurred while parsing the file:
System.IO.IOException, The specified registry key does not exist..
[D:\home\site\repository\Glimpse.ApplicationInsights\Glimpse.ApplicationInsights.csproj]
  1 violations encountered.

Done executing task "StyleCopTask" -- FAILED.
Done building target "StyleCop" in project "Glimpse.ApplicationInsights.csproj" -- FAILED.
Done Building Project "D:\home\site\repository\Glimpse.ApplicationInsights\Glimpse.ApplicationInsights.csproj"
        (default targets) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "ResolveProjectReferences" in project
        "Glimpse.ApplicationInsights.Sample.csproj" -- FAILED.

Project that my web application referencing has StyleCop enabled. Also it seems that StyleCop fail to run.

At this point I decided that I don’t need StyleCop when publishing to azure. So I’ve added extra condition into target import AND ('$(_PackageTempDir)' == ''). This condition is the best I come up with to distinguish the build for deployment from regular compilation. Here is corresponding commit and full import statement:

1
2
3
4
5
<Import
        Project="..\..\packages\StyleCop.MSBuild.4.7.49.1\build\StyleCop.MSBuild.Targets"
        Condition="Exists('..\..\packages\StyleCop.MSBuild.4.7.49.1\build\StyleCop.MSBuild.Targets') 
                AND ('$(_PackageTempDir)' == '')"
/>

So I unblocked myself, but this solutions seems hacky. I was thinking of more robust solution with setting special parameter for build using application setting SCM_BUILD_ARGS that will disable StyleCop. See KuDu wiki for details. However I wanted to get to the root cause of why StyleCop needs registry access. So I decided to troubleshoot it further.

I know there is an exception happening in StyleCop and I need its callstack. So I decided to use remote debugger to get it. First, I found github project that will run StyleCop from command line. I found one called StyleCopCmd. I downloaded and compiled it with the proper version of StyleCop.dll. I also added an extra Console.Read in the beggining of it so I’ll have time to attach debugger. This is how I’ll run it from KuDu console:

1
.\StyleCopCmd.exe -s D:\home\site\repository\Glimpse.ApplicationInsights.sln -c -w

I followed instructions to attach remote debugger to StyleCopCmd.exe process. There are caveats:

  1. At first I used wrong credentials and was getting error like this:
  ---------------------------
  Microsoft Visual Studio
  ---------------------------
  Unable to connect to the Microsoft Visual Studio Remote Debugging Monitor named 
  'sitename.scm.azurewebsites.net'. The Microsoft Visual Studio Remote Debugging 
  Monitor (MSVSMON.EXE) does not appear to be running on the remote computer. 
  This may be because a firewall is preventing communication to the remote computer. 
  Please see Help for assistance on configuring remote debugging.
  ---------------------------
  OK   Help   
  ---------------------------

The reason was - I used user name $glimpse-play-ai-2 instead of fully-qualified glimpse-play-ai-2\$glimpse-play-ai-2. You may have the same message with the completely wrong credentials.

  1. When I run Attach to the process from Visual Studio I haven’t seen the process StyleCopCmd.exe. The reason is that this process is SCM (KuDu) process and I needed to specify “sitename.scm.azurewebsites.net” as a Qulifier, not “sitename.azurewebsites.net” in Attach to the Process dialog.
  2. When debugging external code in Visual Studio - make sure to uncheck the box “Just My Code” in debugger options.

I set Visual Studio to stop on every CLR exception and let StyleCopCmd.exe run. It paused on excpetion with the following call stack:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
mscorlib.dll!Microsoft.Win32.RegistryKey.Win32Error(int errorCode, string str) Line 1694        C#
mscorlib.dll!Microsoft.Win32.RegistryKey.CreateSubKeyInternal(string subkey, Microsoft.Win32.RegistryKeyPermissionCheck permissionCheck, object registrySecurityObj, Microsoft.Win32.RegistryOptions registryOptions) Line 409        C#
mscorlib.dll!Microsoft.Win32.RegistryKey.CreateSubKey(string subkey, Microsoft.Win32.RegistryKeyPermissionCheck permissionCheck) Line 297        C#
mscorlib.dll!Microsoft.Win32.RegistryKey.CreateSubKey(string subkey) Line 289        C#
StyleCop.dll!StyleCop.RegistryUtils.CurrentUserRoot.get()        Unknown
StyleCop.dll!StyleCop.RegistryUtils.CUGetValue(string name)        Unknown
StyleCop.dll!StyleCop.StyleCopCore.GetLastUpdateCheckDate()        Unknown
StyleCop.dll!StyleCop.StyleCopCore.CheckForStyleCopUpdate(StyleCop.CodeProject project)        Unknown
StyleCop.dll!StyleCop.StyleCopCore.InitializeProjectForAnalysis(StyleCop.CodeProject project, StyleCop.StyleCopThread.Data data, StyleCop.ResultsCache cache)        Unknown
StyleCop.dll!StyleCop.StyleCopCore.Analyze(System.Collections.Generic.IList<StyleCop.CodeProject> projects, bool ignoreCache, string settingsPath)        Unknown
StyleCop.dll!StyleCop.StyleCopCore.FullAnalyze(System.Collections.Generic.IList<StyleCop.CodeProject> projects)        Unknown
StyleCop.dll!StyleCop.StyleCopConsole.Start(System.Collections.Generic.IList<StyleCop.CodeProject> projects, bool fullAnalyze)        Unknown
StyleCopCmd.exe!StyleCopCmd.Core.StyleCopExecutor.Run() Line 70        C#
StyleCopCmd.exe!StyleCopCmd.Program.Main(string[] args) Line 61        C#

Using disassembler I found that StyleCop tries to set registry key when checks for the latest version:

1
 Registry.CurrentUser.CreateSubKey("Software\\CodePlex\\StyleCop")

So I’ve added this flag to StyleCop settings file to disable this version check. Here is the commit:

1
<BooleanProperty Name="AutoCheckForUpdate">False</BooleanProperty>

And it solved the issue.

Azure Web App infrastructure gives you great flexibility and power deploying and running your web applications. Even though it’s infrastructure has some limitations - it is really easy to troubleshoot issues with all the tools it provides.

You Need This nuget.config

| Comments

Suddenly many projects on my computer stoped compiling - they cannot find NuGet dependencies. After short digging I found that my default NuGet.config file %APPDATA%\NuGet\NuGet.Config was changed. Now it sets packages folder to be ..\..\..\Users\sergkanz\AppData\Roaming\packages. To be absolutely precise the packages folder is set to %APPDATA%\NuGet\..\packages, but Visual Studio will expand the path.

I’m not sure when and how I changed this file. Maybe it is a feature of new NuGet version? Anyway it turns out to be a very good thing. I see two benefits here:

  • For the test projects I share packages and never even copy them from cache. I just point my test projects directly to the cache folder
  • Any project that I share with other people now requires it’s own NuGet.config file. And I forced to create it - no way anybody will have the folder ..\..\..\Users\sergkanz\AppData\Roaming\packages.

Having personalized NuGet.config for the shared project is a good idea. First, you can configure the same packages folder for multiple solutions in one repo. Second, you also set the list of extra package sources you are using and some other settings.

Here is a small NuGet.config that I copy now to the test projects that I want to share with others. Just place it to the solution folder and it will replicate the default $(SolutionDir)\packages behavior:

1
2
3
4
5
6
7
8
9
<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <config>
    <add key="repositoryPath" value=".\packages" />
  </config>
  <packageSources>
    <add key="nuget.org" value="https://www.nuget.org/api/v2/" />
  </packageSources>
</configuration>

Sometimes you need to restart Visual Studio so solution will pick up this configuration.

Application Insights Self-diagnostics

| Comments

This blog post was written by Anastasia Baranchenkova

Each application insights component writes diagnostic information using EventSource.

If you see no data and you already checked that correct instrumentation key is specified you can try to use PerfView to see if there are problems with sending telemetry out (More about collecting Application Insights traces here.

But it may be so that Application Insights is partially functional: channel is working and sending telemetry out but not all events are delivered. For example, you configured custom counter but you do not see it. If counter is configured incorrectly ETW event will be logged and you will actually be able to find Application Insights trace telemetry it in your Search explorer:

Why did you get this even as trace telemetry?

Application Insights Web nuget package adds Diagnostics module in the configuration file by default. This module subscribes to Application Insights diagnostic error events and sends them out to the same Application Insights resource that is used by your application. Diagnostics messages will be sent as trace telemetry and will have “AI:” or “AI Internal:” prefixes.

You can send Application Insights diagnostic messages to a different resource if you provide different instrumentation key for Diagnostics Module in your application insights configuration file:

1
2
3
<Add Type="Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule, Microsoft.ApplicationInsights" >
  <DiagnosticsInstrumentationKey>YOUR_KEY_2</DiagnosticsInstrumentationKey>
</Add>

UPDATE 1/31/16: Recently I’ve been asked several times about other override property that this module has: Severity. Updating this propety can force the diagnostics module to send all the telemetry to the portal and not only errors. I was not thinking that it is very useful for the end user but apparently it is e.g. when you cannot install Fiddler or PerfView on the production box. With this override you can collect verbose traces just by updaing the applicationinsights.config and restarting the application. Important! You do not want this to be enabled by default all the time! It decreases application performance, produces a lot of noice messages that you do not understand and for which you potentially pay money. If you still want to see all the traces in the portal you would add this to your config:

1
2
3
4
<Add Type="Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule, Microsoft.ApplicationInsights" >
  <DiagnosticsInstrumentationKey>YOUR_KEY_2</DiagnosticsInstrumentationKey>
  <Severity>Verbose</Severity>
</Add>

Performance Counters in Non-web Applications

| Comments

This post shows how to collect Performance Counters for the desktop application and features the answer to the questions:

There is no more “TelemetryModules“ collection in “TelemetryConfiguration“ class. where should I store my telemetry modules?

Step 1

Install NuGet package.

1
Install-Package Microsoft.ApplicationInsights.PerfCounterCollector

Step 2

NuGet will create an ApplicationInsights.config file. If you don’t use it (and you probably don’t use it for desktop applications) - remove this file.

Step 3

Define global variable that will live for a lifetime of an application, instantiate it, populate the list of counters and call Initialize method:

1
2
3
4
5
6
7
8
9
10
11
12
13
class Program
{
    private PerformanceCollectorModule perfCollectorModule;

    static void Main(string[] args)
    {
        TelemetryConfiguration.Active.InstrumentationKey = "Foo";

        this.perfCounterCollectorModule = new PerformanceCollectorModule();
        perfCounterCollectorModule.Counters.Add(
          new PerformanceCounterCollectionRequest(
              @"\.NET CLR Memory(??APP_CLR_PROC??)\# GC Handles", "GC Handles"));
        perfCounterCollectorModule.Initialize(TelemetryConfiguration.Active);

Step 4

In order to collect counters for the current process - you should use ??APP_CLR_PROC?? for CLR counters and ??APP_WIN32_PROC?? for windows counters. Typically counter instances will be named after process name. However in case of multiple instances of the process running you will have names like w3wp#3 representing third instance of the w3wp.exe process.

This indeces in instance names will change over time. For example, when process w3wp#2 will finish, w3wp#3 will become w3wp#2. Moreover, instance name for CLR counters is different than for windows counter as CLR counters only count processes that runs .NET code inside.

So PerfCounterCollector will regularly check the mapping between the instance name and process ID using counters: \.NET CLR Memory(*)\Process ID for managed counters and Process(*)\ID Process for windows counters is you are using keywords ??APP_CLR_PROC?? and ??APP_WIN32_PROC?? as instance names.

Step 5

You are all set. Counter will be sent to the portal every minute.

Custom counters will be sent as metrics:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
{
  "name": "Microsoft.ApplicationInsights.foo.Metric",
  "time": "2015-10-07T08:22:20.6783162-07:00",
  "iKey": "Foo",
  "tags": { "ai.internal.sdkVersion": "1.2.0.5639" },
  "data": {
    "baseType": "MetricData",
    "baseData": {
      "ver": 2,
      "metrics": [
        {
          "name": "GC Handles",
          "kind": "Measurement",
          "value": 123
        }
      ],
      "properties": {
        "CounterInstanceName": "TestPerfCounters.vshost",
        "CustomPerfCounter": "true"
      }
    }
  }
}

Standard perofrmance counters will be sent as performance counters:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
{
  "name": "Microsoft.ApplicationInsights.foo.PerformanceCounter",
  "time": "2015-10-07T08:22:20.6464765-07:00",
  "iKey": "Foo",
  "tags": { "ai.internal.sdkVersion": "1.2.0.5639" },
  "data": {
    "baseType": "PerformanceCounterData",
    "baseData": {
      "ver": 2,
      "categoryName": "Process",
      "counterName": "% Processor Time",
      "instanceName": "TestPerfCounters.vshost",
      "value": 20.0318031311035
    }
  }
}

For web applications you can configure the list of performance counters to monitor using configuration file.

Note, that for SDK to collect counters identity application is running under should be part of the group Performance Monitor Users.

Open Source Signing

| Comments

We enabled open source signing for Application Insights SDK on github. Open source signing allows you to build assembly that matches identity of those officially built by Microsoft.

When you import Application Insights NuGet package reference like this will be added to your project. Note that this reference is added to the assembly with the specific public key token:

1
2
3
4
<Reference Include="Microsoft.ApplicationInsights, PublicKeyToken=31bf3856ad364e35, Version=2.0.0.0, Culture=neutral, processorArchitecture=MSIL">
  <HintPath>..\packages\Microsoft.ApplicationInsights.2.0.0.0\lib\net45\Microsoft.ApplicationInsights.dll</HintPath>
  <Private>True</Private>
</Reference>

You may also have some already compiled assemblies that has a reference to Application Insights SDK. Those references would also be on strongly named assembly.

Open source signing allows you to change the code of Application Insights SDK, compile it and replace original assembly for testing. Applicaiton Insights assembly you’ll compile locally will have the same public key token as one compiled by Microsoft. Here is what sn tool will output:

1
2
3
4
5
6
7
8
9
10
11
12
13
>sn -Tp Microsoft.ApplicationInsights.dll

Microsoft (R) .NET Framework Strong Name Utility  Version 4.0.30319.18020
Copyright (c) Microsoft Corporation.  All rights reserved.

Public key (hash algorithm: sha1):
0024000004800000940000000602000000240000525341310004000001000100b5fc90e7027f6
7871e773a8fde8938c81dd402ba65b9201d60593e96c492651e889cc13f1415ebb53fac1131ae
0bd333c5ee6021672d9718ea31a8aebd0da0072f25d87dba6fc90ffd598ed4da35e44c398c454
307e8e33b8426143daec9f596836f97c8f74750e5975c64e2189f45def46b2a2b1247adc3652b
f5c308055da9

Public key token is 31bf3856ad364e35

So you don’t need to change public key token in project file and you don’t need to recompile other assemblies referring Application Insights SDK. You can just replace a file and test your changes.

There are some limitations with open source signing. First, strong name verification will obviously fail:

1
2
3
4
5
6
>sn -vf Microsoft.ApplicationInsights.dll

Microsoft (R) .NET Framework Strong Name Utility  Version 4.0.30319.18020
Copyright (c) Microsoft Corporation.  All rights reserved.

Failed to verify assembly -- Strong name validation failed.

So you may need to disable verification for this public key token (don’t forget to run this command for the correct bittness - x86 or x64):

1
>sn -Vr Microsoft.ApplicationInsights.dll

Next limitation is related to the behavior of ASP.NET infrastracture. Even if you turned strong name verification off on computer, your ASP.NET applicaitons will likely fail with the message like this:

1
2
3
4
5
6
7
A first chance exception of type 'System.IO.FileLoadException' occurred in mscorlib.dll

Additional information: Could not load file or assembly 'Microsoft.ApplicationInsights.dll,
Version=2.0.0.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its
dependencies. Strong name signature could not be verified.  The assembly may have
been tampered with, or it was delay signed but not fully signed with the correct
private key. (Exception from HRESULT: 0x80131045)

The issue explained in details at IIS forum.

When using .NET 4, shadow copying assemblies in an application for which assemblies rarely ever change has improved. In previous versions of ASP.NET, there was often a noticeable delay in application startup time while assemblies were being shadow copied. Now, the framework checks the file date/time of an application’s assemblies and compares that with the file date/time of any shadow copied assemblies. If they are the same, the shadow copying process does not occur. This causes the shadow copying process to kick off only if an assembly has been physically modified.

The process would look something like this for each assembly:
1. Copy assembly from application location to temporary location
2. Open assembly
3. Verify assembly name
4. Validate strong name
5. Compare update to current cached assembly
6. Copy to shadow copy location (if newer)
7. Remove assembly from temporary location

Shadow copying is important if you are modifying assemblies directly in a live application.

But if you want to skip strong name assembly, you must disable shadow copying.

So you need to disable shadow copying for your ASP.NET application in web.config:

1
2
3
<system.web>
  <hostingEnvironment shadowCopyBinAssemblies="false" />
</system.web>

More information on open source signing can be found at corefx documentaiton page.

With open source signing it is much easier to validate changes you may need in Application Insights SDK. We always happy to hear your feedback and accept your contribution!

Nuget Error: An Entry With the Same Key Already Exists

| Comments

UPDATE: Issue got resolved. It turns out that tha list of dependencies is merged by the server and there are some NuGet servers out wild which will merge dependencies for all platforms. Cleaning cache at %userprofile%\.nugget and %localappdata%\nugget and not using those servers solves the problem. Hurrah Open source for resolving issues quickly!

Original post:

Sometimes installing the NuGet you can see the error message An entry with the same key already exists:

1
2
3
4
5
6
7
8
9
PM> Install-Package "Microsoft.ApplicationInsights.DependencyCallstacks" -Source "https://www.myget.org/F/applicationinsights-sdk-labs/" -Pre
Attempting to gather dependencies information for package 'Microsoft.ApplicationInsights.DependencyCallstacks.0.20.0-build14383' with respect to project 'WebApplication3', targeting '.NETFramework,Version=v4.5.2'
Attempting to resolve dependencies for package 'Microsoft.ApplicationInsights.DependencyCallstacks.0.20.0-build14383' with DependencyBehavior 'Lowest'
Install-Package : An entry with the same key already exists.
At line:1 char:1
+ Install-Package "Microsoft.ApplicationInsights.DependencyCallstacks"  ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Install-Package], Exception
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand

Changine the order of installing NuGets sometimes helped so I never tried to troubleshoot it further. Also I found this forum post so I figured it maybe some generic problem.

Yesterday I got this error again. It took me 10 minutes to troubleshoot it. Simple steps:

  • Open windbg (32-bit) and attach to devenv.exe
  • Make it stop on managed exceptions: sxe clr
  • Load sos using command loadby sos clr

Then I printed stack:

1
2
3
4
5
6
7
8
9
0:061> !clrstack
OS Thread Id: 0x3da8 (61)
Child SP       IP Call Site
29abc3e0 76ca3e28 [HelperMethodFrame: 29abc3e0] 
29abc490 727b7d91 System.ThrowHelper.ThrowArgumentException(System.ExceptionResource) [f:\dd\NDP\fx\src\compmod\system\collections\generic\throwhelper.cs @ 63]
29abc4a0 729ef49a System.Collections.Generic.TreeSet`1[[System.Collections.Generic.KeyValuePair`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]], mscorlib]].AddIfNotPresent(System.Collections.Generic.KeyValuePair`2) [f:\dd\NDP\fx\src\compmod\system\collections\generic\sorteddictionary.cs @ 803]
29abc4b0 723b6149 System.Collections.Generic.SortedDictionary`2[[System.__Canon, mscorlib],[System.__Canon, mscorlib]].Add(System.__Canon, System.__Canon) [f:\dd\NDP\fx\src\compmod\system\collections\generic\sorteddictionary.cs @ 167]
29abc4c4 17277f3b NuGet.Resolver.ResolverPackage..ctor(System.String, NuGet.Versioning.NuGetVersion, System.Collections.Generic.IEnumerable`1, Boolean, Boolean)
29abc578 069d15a4 NuGet.Resolver.PackageResolver.Resolve(NuGet.Resolver.PackageResolverContext, System.Threading.CancellationToken)

It’s easy to look at sources as NuGet is open source. Here is line of code that fails. We pass list of dependencies to the constructor of ResolverPackage:

1
resolverPackages.Add(new ResolverPackage(package.Id, package.Version, dependencies, package.Listed, false));

and it in turns add all dependencies into collection:

1
_dependencyIds.Add(dependency.Id, dependency.VersionRange == null ? VersionRange.All : dependency.VersionRange);

So I took all objects on the stack using !dso and found the array of dependencies there. You can see that is consist of 22 dependencies and two of them: 249c9338 and 249c93b8 have the same name:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
0:061> !DumpArray /d 249ca23c
Name:        NuGet.Packaging.Core.PackageDependency[]
MethodTable: 1ecba294
EEClass:     72ce3750
Size:        100(0x64) bytes
Array:       Rank 1, Number of elements 22, Type CLASS
Element Methodtable: 1ecb9d24
[0] 249c9338
[1] 249c9378
[2] 249c93b8
[3] 249c9448
[4] 249c94d8
[5] 249c9568
[6] 249c9604
[7] 249c9694
[8] 249c9724
[9] 249c97b4
[10] 249c9844
[11] 249c98d4
[12] 249c9964
[13] 1d2e5a70
[14] 1d2e5b00
[15] 1d2e5b90
[16] 1d2e5c20
[17] 1d2e5cb0
[18] 1d2e5d40
[19] 1d2e5dd0
[20] 249ca19c
[21] 249ca22c

0:061> !DumpObj /d 249c9338
Name:        NuGet.Packaging.Core.PackageDependency
MethodTable: 1ecb9d24
EEClass:     1ec961e8
Size:        16(0x10) bytes
File:        C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\COMMON7\IDE\EXTENSIONS\Q04IPECQ.ZWQ\NuGet.Packaging.Core.Types.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
73101d7c  4000008        4        System.String  0 instance 1d2dc7f4 _id
11b5bd88  4000009        8 ...ning.VersionRange  0 instance 1d2fdfa8 _versionRange

0:061> !DumpObj /d 1d2dc7f4
Name:        System.String
MethodTable: 73101d7c
EEClass:     72ce3620
Size:        52(0x34) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      Microsoft.Bcl.Async
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
73103cc4  4000243        4         System.Int32  1 instance       19 m_stringLength
731027c0  4000244        8          System.Char  1 instance       4d m_firstChar
73101d7c  4000248       40        System.String  0   shared   static Empty
    >> Domain:Value  0112e5a8:NotInit  <<
      
0:061> !DumpObj /d 249c93b8
Name:        NuGet.Packaging.Core.PackageDependency
MethodTable: 1ecb9d24
EEClass:     1ec961e8
Size:        16(0x10) bytes
File:        C:\PROGRAM FILES (X86)\MICROSOFT VISUAL STUDIO 14.0\COMMON7\IDE\EXTENSIONS\Q04IPECQ.ZWQ\NuGet.Packaging.Core.Types.dll
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
73101d7c  4000008        4        System.String  0 instance 1d2dc8d0 _id
11b5bd88  4000009        8 ...ning.VersionRange  0 instance 1d2fdfe0 _versionRange

0:061> !DumpObj /d 1d2dc8d0
Name:        System.String
MethodTable: 73101d7c
EEClass:     72ce3620
Size:        52(0x34) bytes
File:        C:\WINDOWS\Microsoft.Net\assembly\GAC_32\mscorlib\v4.0_4.0.0.0__b77a5c561934e089\mscorlib.dll
String:      Microsoft.Bcl.Async
Fields:
      MT    Field   Offset                 Type VT     Attr    Value Name
73103cc4  4000243        4         System.Int32  1 instance       19 m_stringLength
731027c0  4000244        8          System.Char  1 instance       4d m_firstChar
73101d7c  4000248       40        System.String  0   shared   static Empty
    >> Domain:Value  0112e5a8:NotInit  <<

Looking at nuspec you can see that Microsoft.Bcl.Async is a dependency for both platforms - Framework 4.0 and Windows 8:

1
2
3
4
5
6
7
8
<group targetFramework=".NETFramework4.0">
<dependency id="Microsoft.Bcl.Async" version="1.0.168" />
<dependency id="Microsoft.Diagnostics.Tracing.EventSource.Redist" version="1.1.24" />
</group>
<group targetFramework=".NETFramework4.5" />
<group targetFramework="WindowsPhone8.0">
<dependency id="Microsoft.Bcl.Async" version="1.0.168" />
</group>

So it seems that NuGet do not distinguish dependencies for the different plaforms while building the list of references. So I filed the issue at GitHub and hope it will be resolved soon.

Workaround is simple if the list od dependencies for the platform is small. Just add -DependencyVersion Ignore when calling Install-Package. You’ll need to install all dependencies manually.

When everything is open source it is very easy to troubleshoot issues. So we open sourcing more code of Application Insights SDK. Now it is server telemetry channel. See this PR.

Operational Insights Agent for SCOM

| Comments

Quite unusual tip today. You may heard of System Center Operations Manager and the fact it provides APM capabilities. Where APM stands for Application Performance Monitoring. You may also heard of Operational Insights which can work as an attach service for Operations Manager.

What you may not know that the latest agent distributed for Operational Insights is not only compatible with System Center, but also brings more APM capabilities than regular System Center SCOM 2012 R2 agent has. It has number of bugfixes, perf improvements and couple features support like monitoring of MVC 5 applicaitons.

You can download this agent here:
- 64-bit windows - 32-bit windows

Another interesting fact about this agent is that looking into the folder %programfiles%\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V7.2.10375.0\x64 you will find the files I mentioned in the post explaining how Application Insights tracks dependencies. So installing this agent and enabling APM will help Application Insights SDK collect reacher informaiton about your applicaiton http and SQL dependencies so you don’t need to install Status Monitor.

More Telemetry Channels

| Comments

Application Insights SDK works on many platforms. You can use it to send telemetry Desktop applicaitons, web services, phone apps. All these platforms has their specifics. Phone and desktop apps typically has as single user and small number of events from every particular instance of applicaiton. Services serves many users the same time and have very high load. Devices can go online and offline very often. Services are typically always online.

Initial versions of Applicaiton Insights SDK attempted to have a single channel that will account for all these differences. However we found that it is not easy to accomodate them in a single channel. So now we have three different channel implementations:

  • InMemoryChannel
  • Persistence channel for devices
  • Windows Server telemetry channel

Sampling that I mentioned in this post mostly applies to the windows server telemetry channel. So I’ve updated that post.

InMemoryChannel

InMemory channel Microsoft.ApplicationInsights.Channel.InMemoryChannel is a lightweight loosy telemetry channel that is used by default. It will batch data and initiate sending every SendingInterval (default is 30 seconds) or when number of items exceeded MaxTelemetryBufferCapacity (default is 500. It also will not retry if it failed to send telemetry.

Package: this channel is part of Applicaiton Insights API package. Sources: InMemoryChannel.cs on github.

Persistence channel for devices

Persistence channel for devices Microsoft.ApplicationInsights.Channel.PersistenceChannel is a channel optimized for devices and mobile apps and works great in offline scenarios. It requires a file storage to persist the data and you should use this constructor to specify folder you want to use for storage.

I already explained how this channel will work for unhandled exceptions. It writes events to disk before attempting to send it. Next time app starts - event will be picked up and send to the cloud. Furthermore, if you are running multiple instances of an applicaiton - all of them will write to the disk, but only one will be sending data to http endpoint. It is controled via global Mutex.

Package: Application Insights Persisted HTTP channel Sources: PersistenceChannel.cs on github

Windows Server telemetry channel

Windows Server telemetry channel Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel is a channel optimized for high volume data delivery. It will send data to endpoint first and only attempt to persist data if endpoint cannot be reached. It is using current user’s local app data folder (%localAppData%\Microsoft\ApplicationInsights) or temp folder (%TMP%).

This channel implements exponential retry intervals and respect Retry-After header set by the server. It solves the problem we call “spiral of death”. Spiral of death happens when endpoint was temporary unavbailable or you hit the throttling limit. Channel starts persisting data if it cannot send it. After some time your throttling limit will be cleared up or connectivity issue will be fixed. So channel will start sending all the new and persisted data. With the big load you may hit throttling limit very easily again. So data will be rejected again. And you’ll start persiting it entering the spiral.

Package: Windows Server telemetry channel Sources: Not public yet.

Summary

Different applications requires different channels. NuGet packages will configure proper channel for you. However if you configure Application Insights manually you need to know which channel is right for you.

Web SDK 1.2.1 Structure Changes

| Comments

This blog post was written by Anastasia Baranchenkova

In version 1.2.1 there were 2 major changes in the structure.

  1. All Web SDK assemblies were renamed:

    • Microsoft.ApplicationInsights.Web.dll was renamed on Microsoft.AI.Web.dll
    • Microsoft.ApplicationInsights.Extensibility.Web.TelemetryChannel.dll was renamed on Microsoft.AI.ServerTelemetryChannel.dll
    • Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.dll was renamed on Microsoft.AI.PerfCounterCollector
    • Microsoft.ApplicationInsights.Extensibility.DependencyCollector.dll was renamed on Microsoft.AI.DependencyCollector.dll
  2. Logic that does not depend on web was moved out from Microsoft.ApplicationInsights.Extensibility.Web to Microsoft.AI.WindowsServer.

Microsoft.AI.WindowsServer.dll is distributed with the new nuget package Application Insights Windows Server. This nuget package can be installed on Worker roles projects, windows services or console applications.

The following telemetry initailizers are part of this assembly (all of them were part of web sdk assembly before):

  • DeviceTelemetryInitializer. This telemetry initailizer fills most of device context properties.
  • DomainNameRoleInstanceTelemetryInitializer. This telemetry initializer sets device context RoleInstance to machine FQDN name.
  • BuildInfoConfigComponentVersionTelemetryInitializer. This telemetry initializer sets component context Version property using buildinfo.config if you have msbuild integration.
  • AzureRoleEnvironmentTelemetryInitializer. This telemetry initailizer sets RoleInstance and RoleInstanceName in case if your application is web or worker role.

The following telemetry modules are part of this assembly:

  • DeveloperModeWithDebuggerAttachedTelemetryModule. This module enables VS F5 experience.
  • UnhandledExceptionTelemetryModule. This new module tracks unhandled exceptions in case if your application is a worker role, windows service or console application.
  • UnobservedExceptionTelemetryModule. This new module tracks task unobserved exceptions.

Additionally this new nuget package changes ApplicationInsights.config file properties so it is copied to the output. Without that in the previous SDK version worker role monitoring did not start till you manually did the same.