Also it was mentioned in Sergey’s blog post that Server telemetry channel stores events on disk if they cannot be sent. The channel would use either current user’s local app data folder or current process temp folder.
I saw several cases when there was a problems with that.
In the first case process did not have access neither to local app data nor to the temp folder. The workaround was to make sure it has it.
Second one was more interesting. There was an application that worked perfectly fine locally but when deployed to a VM it crashed. It appeared that local app data was pointing to an unmapped drive. Unfortunately ApplicationInsights was not catching this type of exception, so it was just crashing without even trying to check temp folder. This bug was fixed in 2.0.0-beta4.
Additionally while fixing this issue we added an ability to specify custom folder either though configuration or in code (if it is not provided or inaccessible old logic is used).
In configuration file you would do it this way (make sure you have correct xml closing tags):
An interesting side note: configuration loading is very flexible. If you see a public property on an object that can be defined in the configuration file (e.g. TelemetryInitializer, TelemetryProcessor or Channel) this property can be set via configuration file. We did not have to change configuration loading logic for this update, we just added a public property to the ServerTelemetryChannel.
This blog post was written by Anastasia Baranchenkova
In Visual Studio 2015 Update 1 Application Insights telemetry is available right from the Visual Studio. And you can even get it without providing a valid instrumentation key.
I created a sample application that I describe in details later on. In this application I forward WebApi framework traces to Application Insights (it is a similar thing that SergeyK described in one of his previous posts but it is done differently: by implementing ITraceWriter interface).
So if you start debugging web hosted WebApi application even if you do not provide instrumentation key you can see all the traces rights in your Diagnostics Tools window. You can search there, filter and see actual json file that would have been sent if you provided a valid instrumentation key:
For self-hosted WebApi applications you would need to configure ApplicationInsights providing a valid instrumentation key because Diagnostics Tools hub does not currently support this type of application. But you still can see all the telemetry from the VS itself. For that you need to open: View ->Other Windows->Application Insights Search. From there you connect to your Azure Subscription and get back all the telemetry. You can select different time intervals, filter by type or property values and see each telemetry item details:
And now I want to describe in details how you can forward WebApi traces to ApplicationInsigts to get all this beauty.
WebApi web-hosted application
Create WebApi web hosted application. If you have Azure subscription check “Add ApplicationInsights to project”.
If you did not add ApplicationInsights on application creation then add the latest Application Insights for Web nuget through nugget package manager. (In this case you do not have instrumentation key in the ApplicationInsights configuration file and no data will be sent to the portal but you can add it that later and for debugging purposes you have all you need)
CompositTraceWriter also implements ITraceWriter and is used in case if some other ITraceWriter is already registered by the application so that ApplicationInsightsTraceWriter does not replace existing but rather is added to the list of trace writers.
Call config.EnableApplicationInsights() from Startup.Configuration where config would be an instance of HttpConfiguration that you create in this method for other registration purposes.
I would highly encourage to also read this blog post that explains how and when to use WebApi tracing. The example above demonstrates basic integration points while in the article above you can find detailed information on the best practices of how to use WebApi tracing.
I was working on enabling continues deployment from GitHub to Azure WebApp for the sample Glimpse and Application Insights integration application. It is easy to implement this integration. Simplest way is to create “Deploy to Azure” button in your GitHub repository like I explained in this blog post. You can also do it manually:
Open Settings blade of your web app
Click on Continuous deployment
Select External Repository and set your repository URL
Once enabled - KuDu will pull sources from repository, compile it and deploy web application.
This time it didn’t work smoothly for me. I got an error:
MSBUILD : error : SA0001 : CoreParser : An exception occurred while parsing the
file: System.IO.IOException, The specified registry key does not exist.. [D:\home
Failed exitCode=1, command="D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe"
Glimpse.ApplicationInsights.Sample.csproj" /nologo /verbosity:m /t:Build
My web project doesn’t have StyleCop enabled so the error was quite misleading. The good thing - this error message had original msbuild command. So I opened KuDu debug console using URL: https://<webAppName>.scm.azurewebsites.net/DebugConsole and typed the following command:
"D:\Program Files (x86)\MSBuild\14.0\Bin\MSBuild.exe""D:\home\site\repository\Glimpse.ApplicationInsights.Sample\Glimpse.ApplicationInsights.Sample.csproj"/nologo/verbosity:detailed/t:Build/t:pipelinePreDeployCopyAllFilesToOneFolder/p:_PackageTempDir="D:\local\Temp\8d2f93a7a699626";AutoParameterizationWebConfigConnectionStrings=false;Configuration=Release;UseSharedCompilation=false/p:SolutionDir="D:\home\site\repository\.\\">buildLog.txt
Note, the command text is different from the original. Specifically, I changed verbosity /verbosity:detailed and redirected output to the file > buildLog.txt. Resulting error message was much easier to troubleshoot:
Using "StyleCopTask" task from assembly "D:\home\site\packages\StyleCop.MSBuild.188.8.131.52\build\..\tools\StyleCop.dll".
Task "StyleCopTask"MSBUILD : error : SA0001 : CoreParser : An exception occurred while parsing the file:
System.IO.IOException, The specified registry key does not exist..
1 violations encountered.
Done executing task "StyleCopTask" -- FAILED.
Done building target "StyleCop" in project "Glimpse.ApplicationInsights.csproj" -- FAILED.
Done Building Project "D:\home\site\repository\Glimpse.ApplicationInsights\Glimpse.ApplicationInsights.csproj" (default targets) -- FAILED.
Done executing task "MSBuild" -- FAILED.
Done building target "ResolveProjectReferences" in project
"Glimpse.ApplicationInsights.Sample.csproj" -- FAILED.
Project that my web application referencing has StyleCop enabled. Also it seems that StyleCop fail to run.
At this point I decided that I don’t need StyleCop when publishing to azure. So I’ve added extra condition into target import AND ('$(_PackageTempDir)' == ''). This condition is the best I come up with to distinguish the build for deployment from regular compilation. Here is corresponding commit and full import statement:
<ImportProject="..\..\packages\StyleCop.MSBuild.184.108.40.206\build\StyleCop.MSBuild.Targets"Condition="Exists('..\..\packages\StyleCop.MSBuild.220.127.116.11\build\StyleCop.MSBuild.Targets') AND ('$(_PackageTempDir)' == '')"/>
So I unblocked myself, but this solutions seems hacky. I was thinking of more robust solution with setting special parameter for build using application setting SCM_BUILD_ARGS that will disable StyleCop. See KuDu wiki for details. However I wanted to get to the root cause of why StyleCop needs registry access. So I decided to troubleshoot it further.
I know there is an exception happening in StyleCop and I need its callstack. So I decided to use remote debugger to get it. First, I found github project that will run StyleCop from command line. I found one called StyleCopCmd. I downloaded and compiled it with the proper version of StyleCop.dll. I also added an extra Console.Read in the beggining of it so I’ll have time to attach debugger. This is how I’ll run it from KuDu console:
I followed instructions to attach remote debugger to StyleCopCmd.exe process. There are caveats:
At first I used wrong credentials and was getting error like this:
Microsoft Visual Studio
Unable to connect to the Microsoft Visual Studio Remote Debugging Monitor named
'sitename.scm.azurewebsites.net'. The Microsoft Visual Studio Remote Debugging
Monitor (MSVSMON.EXE) does not appear to be running on the remote computer.
This may be because a firewall is preventing communication to the remote computer.
Please see Help for assistance on configuring remote debugging.
The reason was - I used user name $glimpse-play-ai-2 instead of fully-qualified glimpse-play-ai-2\$glimpse-play-ai-2. You may have the same message with the completely wrong credentials.
When I run Attach to the process from Visual Studio I haven’t seen the process StyleCopCmd.exe. The reason is that this process is SCM (KuDu) process and I needed to specify “sitename.scm.azurewebsites.net” as a Qulifier, not “sitename.azurewebsites.net” in Attach to the Process dialog.
When debugging external code in Visual Studio - make sure to uncheck the box “Just My Code” in debugger options.
I set Visual Studio to stop on every CLR exception and let StyleCopCmd.exe run. It paused on excpetion with the following call stack:
Azure Web App infrastructure gives you great flexibility and power deploying and running your web applications. Even though it’s infrastructure has some limitations - it is really easy to troubleshoot issues with all the tools it provides.
Suddenly many projects on my computer stoped compiling - they cannot find NuGet dependencies. After short digging I found that my default NuGet.config file %APPDATA%\NuGet\NuGet.Config was changed. Now it sets packages folder to be ..\..\..\Users\sergkanz\AppData\Roaming\packages. To be absolutely precise the packages folder is set to %APPDATA%\NuGet\..\packages, but Visual Studio will expand the path.
I’m not sure when and how I changed this file. Maybe it is a feature of new NuGet version? Anyway it turns out to be a very good thing. I see two benefits here:
For the test projects I share packages and never even copy them from cache. I just point my test projects directly to the cache folder
Any project that I share with other people now requires it’s own NuGet.config file. And I forced to create it - no way anybody will have the folder ..\..\..\Users\sergkanz\AppData\Roaming\packages.
Having personalized NuGet.config for the shared project is a good idea. First, you can configure the same packages folder for multiple solutions in one repo. Second, you also set the list of extra package sources you are using and some other settings.
Here is a small NuGet.config that I copy now to the test projects that I want to share with others. Just place it to the solution folder and it will replicate the default $(SolutionDir)\packages behavior:
This blog post was written by Anastasia Baranchenkova
Each application insights component writes diagnostic information using EventSource.
If you see no data and you already checked that correct instrumentation key is specified you can try to use PerfView to see if there are problems with sending telemetry out (More about collecting Application Insights traces here.
But it may be so that Application Insights is partially functional: channel is working and sending telemetry out but not all events are delivered. For example, you configured custom counter but you do not see it. If counter is configured incorrectly ETW event will be logged and you will actually be able to find Application Insights trace telemetry it in your Search explorer:
Why did you get this even as trace telemetry?
Application Insights Web nuget package adds Diagnostics module in the configuration file by default. This module subscribes to Application Insights diagnostic error events and sends them out to the same Application Insights resource that is used by your application. Diagnostics messages will be sent as trace telemetry and will have “AI:” or “AI Internal:” prefixes.
You can send Application Insights diagnostic messages to a different resource if you provide different instrumentation key for Diagnostics Module in your application insights configuration file:
Recently I’ve been asked several times about other override property that this module has: Severity. Updating this propety can force the diagnostics module to send all the telemetry to the portal and not only errors. I was not thinking that it is very useful for the end user but apparently it is e.g. when you cannot install Fiddler or PerfView on the production box. With this override you can collect verbose traces just by updaing the applicationinsights.config and restarting the application. Important! You do not want this to be enabled by default all the time! It decreases application performance, produces a lot of noice messages that you do not understand and for which you potentially pay money.
If you still want to see all the traces in the portal you would add this to your config:
In order to collect counters for the current process - you should use ??APP_CLR_PROC?? for CLR counters and ??APP_WIN32_PROC?? for windows counters. Typically counter instances will be named after process name. However in case of multiple instances of the process running you will have names like w3wp#3 representing third instance of the w3wp.exe process.
This indeces in instance names will change over time. For example, when process w3wp#2 will finish, w3wp#3 will become w3wp#2. Moreover, instance name for CLR counters is different than for windows counter as CLR counters only count processes that runs .NET code inside.
So PerfCounterCollector will regularly check the mapping between the instance name and process ID using counters: \.NET CLR Memory(*)\Process ID for managed counters and Process(*)\ID Process for windows counters is you are using keywords ??APP_CLR_PROC?? and ??APP_WIN32_PROC?? as instance names.
You are all set. Counter will be sent to the portal every minute.
You may also have some already compiled assemblies that has a reference to Application Insights SDK. Those references would also be on strongly named assembly.
Open source signing allows you to change the code of Application Insights SDK, compile it and replace original assembly for testing. Applicaiton Insights assembly you’ll compile locally will have the same public key token as one compiled by Microsoft. Here is what sn tool will output:
>sn -Tp Microsoft.ApplicationInsights.dll
Microsoft (R) .NET Framework Strong Name Utility Version 4.0.30319.18020
Copyright (c) Microsoft Corporation. All rights reserved.
Public key (hash algorithm: sha1):
Public key token is 31bf3856ad364e35
So you don’t need to change public key token in project file and you don’t need to recompile other assemblies referring Application Insights SDK. You can just replace a file and test your changes.
There are some limitations with open source signing. First, strong name verification will obviously fail:
>sn -vf Microsoft.ApplicationInsights.dll
Microsoft (R) .NET Framework Strong Name Utility Version 4.0.30319.18020
Copyright (c) Microsoft Corporation. All rights reserved.
Failed to verify assembly -- Strong name validation failed.
So you may need to disable verification for this public key token (don’t forget to run this command for the correct bittness - x86 or x64):
>sn -Vr Microsoft.ApplicationInsights.dll
Next limitation is related to the behavior of ASP.NET infrastracture. Even if you turned strong name verification off on computer, your ASP.NET applicaitons will likely fail with the message like this:
A first chance exception of type 'System.IO.FileLoadException' occurred in mscorlib.dll
Additional information: Could not load file or assembly 'Microsoft.ApplicationInsights.dll,
Version=18.104.22.168, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its
dependencies. Strong name signature could not be verified. The assembly may have
been tampered with, or it was delay signed but not fully signed with the correct
private key. (Exception from HRESULT: 0x80131045)
When using .NET 4, shadow copying assemblies in an application for which assemblies rarely ever change has improved. In previous versions of ASP.NET, there was often a noticeable delay in application startup time while assemblies were being shadow copied. Now, the framework checks the file date/time of an application’s assemblies and compares that with the file date/time of any shadow copied assemblies. If they are the same, the shadow copying process does not occur. This causes the shadow copying process to kick off only if an assembly has been physically modified.
The process would look something like this for each assembly: 1. Copy assembly from application location to temporary location 2. Open assembly 3. Verify assembly name 4. Validate strong name 5. Compare update to current cached assembly 6. Copy to shadow copy location (if newer) 7. Remove assembly from temporary location
Shadow copying is important if you are modifying assemblies directly in a live application.
But if you want to skip strong name assembly, you must disable shadow copying.
So you need to disable shadow copying for your ASP.NET application in web.config:
UPDATE: Issue got resolved. It turns out that tha list of dependencies is merged by the server and there are some NuGet servers out wild which will merge dependencies for all platforms. Cleaning cache at %userprofile%\.nugget and %localappdata%\nugget and not using those servers solves the problem. Hurrah Open source for resolving issues quickly!
Sometimes installing the NuGet you can see the error message An entry with the same key already exists:
PM> Install-Package "Microsoft.ApplicationInsights.DependencyCallstacks" -Source "https://www.myget.org/F/applicationinsights-sdk-labs/" -Pre
Attempting to gather dependencies information for package 'Microsoft.ApplicationInsights.DependencyCallstacks.0.20.0-build14383' with respect to project 'WebApplication3', targeting '.NETFramework,Version=v4.5.2'
Attempting to resolve dependencies for package 'Microsoft.ApplicationInsights.DependencyCallstacks.0.20.0-build14383' with DependencyBehavior 'Lowest'
Install-Package : An entry with the same key already exists.
At line:1 char:1
+ Install-Package "Microsoft.ApplicationInsights.DependencyCallstacks" ...
+ CategoryInfo : NotSpecified: (:) [Install-Package], Exception
+ FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PackageManagement.PowerShellCmdlets.InstallPackageCommand
Changine the order of installing NuGets sometimes helped so I never tried to troubleshoot it further. Also I found this forum post so I figured it maybe some generic problem.
Yesterday I got this error again. It took me 10 minutes to troubleshoot it. Simple steps:
Quite unusual tip today. You may heard of System Center Operations Manager and the fact it provides APM capabilities. Where APM stands for Application Performance Monitoring. You may also heard of Operational Insights which can work as an attach service for Operations Manager.
What you may not know that the latest agent distributed for Operational Insights is not only compatible with System Center, but also brings more APM capabilities than regular System Center SCOM 2012 R2 agent has. It has number of bugfixes, perf improvements and couple features support like monitoring of MVC 5 applicaitons.
Another interesting fact about this agent is that looking into the folder %programfiles%\Microsoft Monitoring Agent\Agent\APMDOTNETAgent\V7.2.10375.0\x64 you will find the files I mentioned in the post explaining how Application Insights tracks dependencies. So installing this agent and enabling APM will help Application Insights SDK collect reacher informaiton about your applicaiton http and SQL dependencies so you don’t need to install Status Monitor.
Application Insights SDK works on many platforms. You can use it to send telemetry Desktop applicaitons, web services, phone apps. All these platforms has their specifics. Phone and desktop apps typically has as single user and small number of events from every particular instance of applicaiton. Services serves many users the same time and have very high load. Devices can go online and offline very often. Services are typically always online.
Initial versions of Applicaiton Insights SDK attempted to have a single channel that will account for all these differences. However we found that it is not easy to accomodate them in a single channel. So now we have three different channel implementations:
Persistence channel for devices
Windows Server telemetry channel
Sampling that I mentioned in this post mostly applies to the windows server telemetry channel. So I’ve updated that post.
InMemory channel Microsoft.ApplicationInsights.Channel.InMemoryChannel is a lightweight loosy telemetry channel that is used by default. It will batch data and initiate sending every SendingInterval (default is 30 seconds) or when number of items exceeded MaxTelemetryBufferCapacity (default is 500. It also will not retry if it failed to send telemetry.
Persistence channel for devices Microsoft.ApplicationInsights.Channel.PersistenceChannel is a channel optimized for devices and mobile apps and works great in offline scenarios. It requires a file storage to persist the data and you should use this constructor to specify folder you want to use for storage.
I already explained how this channel will work for unhandled exceptions. It writes events to disk before attempting to send it. Next time app starts - event will be picked up and send to the cloud. Furthermore, if you are running multiple instances of an applicaiton - all of them will write to the disk, but only one will be sending data to http endpoint. It is controled via global Mutex.
Windows Server telemetry channel Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel is a channel optimized for high volume data delivery. It will send data to endpoint first and only attempt to persist data if endpoint cannot be reached. It is using current user’s local app data folder (%localAppData%\Microsoft\ApplicationInsights) or temp folder (%TMP%).
This channel implements exponential retry intervals and respect Retry-After header set by the server. It solves the problem we call “spiral of death”. Spiral of death happens when endpoint was temporary unavbailable or you hit the throttling limit. Channel starts persisting data if it cannot send it. After some time your throttling limit will be cleared up or connectivity issue will be fixed. So channel will start sending all the new and persisted data. With the big load you may hit throttling limit very easily again. So data will be rejected again. And you’ll start persiting it entering the spiral.
Different applications requires different channels. NuGet packages will configure proper channel for you. However if you configure Application Insights manually you need to know which channel is right for you.