Debugging Faulting Application w3wp.exe Crashes

Posted on May 31, 2011 by Paul White

If your websites are going down, due to your application pool failing, or being automatically shut down after a few application errors, then you have a major problem on your hands. In my case the symptom was A process serving the application pool suffered a fatal communication error with the Windows Process Activation Service, in my system logs. If you have each website running in its own application pool then you know which website is going down, but the errors being displayed give you no hints as to the cause. I recently had a website that was randomly going down. Sometimes it would go down every few days, and other times it would only crash once a month. So I finally did some digging for the cause, and found it. If you don't know where to start in debugging your crashed application pools, this might help.

Why Does W3WP.exe Crash?

In my experience crashing happens for only a few reasons. The first is an unhandled Exception which was serious enough to crash the Application Pool which is rare but does happen. The second is a recursive function or a set of functions that continue to pass the ball to one another with nothing stopping them. In my case this was the case. I had two functions that got caught in a loop and continued to run out of control until the server bottle necked due to CPU or memory usage.

How do you know when your Application Pool Crashed?

Easy check your event viewer Application Logs under Windows Logs
Event Viewer with Application Errors Crashed w3wp.exe

The Red Exclamation points are telling you there was an Application Error
But the details of the error don't help us very much. See Below

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2
Faulting module name: KERNELBASE.dll, version: 6.1.7601.17514, time stamp: 0x4ce7c78c
Exception code: 0xe053534f
Fault offset: 0x000000000000a49d
Faulting process id: 0x%9
Faulting application start time: 0x%10
Faulting application path: %11
Faulting module path: %12
Report Id: %13

These errors are followed by a application log from Windows Error Reporting
The details of this message tell us where the system dump was stored. Take note of this, as this will help you find the dump files you need.

Fault bucket , type 0
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 7.5.7601.17514
P3: 4ce7afa2
P4: KERNELBASE.dll
P5: 6.1.7601.17514
P6: 4ce7c78c
P7: e053534f
P8: 000000000000a49d
P9:
P10:

Attached files:

These files may be available here:
C:ProgramDataMicrosoftWindowsWERReportQueueAppCrash_w3wp.exe_624e5315e1074e44338812efe102157b47f6ca8_70c6df40

Analysis symbol:
Rechecking for solution: 0
Report Id: 215432e8-8bba-11e0-8d49-0030482e5b77
Report Status: 4

Check your System Logs

Next we go to our system errors on the event viewer. If you are running websites on your server then your system logs are likely to be very numerous. Every time a Application Pool starts or shuts down it will make an entry in your system logs. This is nothing to be concerned about. Many websites are set to shutdown the application pools after a website hasn't had any visitors for more than 20 minutes. This frees up the memory for other sites that do have traffic.

But if you see any entries with the Yellow Warning this is not good.

fatal communication error with the windows process activation service

In my case it was telling which website was throwing the error and gave me the following message

A process serving application pool 'yourwebsite.com' suffered a fatal communication error with the Windows Process Activation Service. The process id was '10452'. The data field contains the error number.

Retrieving the W3WP.exe Dump Files

The one you want is the biggest one.

WinDbg Download

First if you don't have WinDbg on your system then you need it.
This program will take your dump files and give you the stack trace, which can shed some light on where the problem lies.

Unfortunately Microsoft has recently decided to put WinDbg into some huge 600MB+ ISO package. Forcing you to install the entire package to get just one little program. So after some hard searching I found the raw installers.

Windows WinDbg x86 installer
Windows WinDbg x64 installer

Huge Thanks to RXWEN for keeping these files available

Running WinDbg

In my case since I am running Windows Server 2008 R2 ( only comes in 64 bit ), I installed the 64 bit version of the WinDbg. After you install it, you will find it under Debugging Tools for Windows (x64). Also take note that I am running WinDbg off my production server. You would also download your Dump files to your local machine and try to debug them from there, but to keep things easy in case I ever needed to do some debugging on a Remote Desktop Connection using my cell phone I figured running them off my Production Server was ideal.

How to open WinDbg

After you open the program you will click
File > Open Crash Dump
You will select the dump file which in my case found in the following directory
C:ProgramDataMicrosoftWindowsWERReportQueueAppCrash_w3wp.exe_624e5315e1074e44338812efe102157b47f6ca8_70c6df40

Now you should see a screen like this

WinDbg .loadby sos mscorwks command

In the command line at the bottom of the screen you will type
.loadby sos mscorwks
and press enter. The result should look similar to this

WinDbg .loadby sos mscorwks result

Next we do this
WinDbg !clrstack command

Type the following into the command line
!clrstack
and press enter. You should get something like this showing you the stack trace

WinDbg !clrstack result

What you are looking for is either a single function being called over and over, or maybe 2 or 3 functions that are running in a loop. At the top of the stack I don't see anything so I scroll down a bit.

WinDbg cause of w3wp.exe crashing

and the cause is right before our eyes. In my case I have two custom functions that seem to be passing control back and forth with no end in sight. The first is fixquotes, and the second is blacklistVisitor.

fixquotes is a function that just cleans up strings before they are added to MySQL queries, and blacklistVisitor is a function that I setup to blacklist IPs that were attempting something bad ( like SQL Injection, or Denial of Service Attacks ).

Unfortunately until I had a hacker attempt something bad, I had no idea that these two functions were stuck in a loop. Of course just a single instance wouldn't cause the site to go down. But if the same hacker tried over and over again this could create multiple errors and at this point IIS would assume something is wrong and shut the site down.

Conclusion to finding errors in Crashed Application Pools

Most likely you have a recursive function that is calling itself over and over, or you have several functions that are stuck in a loop. Hoping the steps I went through will help others find errors in their code.

Permalink

211048 Visitors

211048 Views

Categories associated with Debugging Faulting Application w3wp.exe Crashes

Websites

Programming

ASP.NET

Operating Systems

Windows Server 2003

Windows Server 2008

Discussion

neo | Jun 27, 2011 10:03 PM
hello to the author,

i been using the windgb tools x64 on our production server, cause were looking for an answer why the w3wp is crashing, and when i tried to load the 2 gbz file of the memory dump files..the tools says the cannot loadlibary(c:.....net frameworksosfailed..)..what is the problem with this..anyway our production server is windows server 2008 r2 installed with a framework 2.0..

Paul | Jun 27, 2011 10:35 PM
Neo,
You might need to install .NET 4.0 to get that command to run correctly. I recall running into the same problem and installing .NET 4.0 framework with all updates fixed the issue.

Corey | Jul 18, 2011 11:29 AM
Actually neo, if you get that LoadLibrary error you will need to install and run the 32bit version of WinDbg instead.

neo | Jul 20, 2011 9:31 PM
hello to all,

i just run the windbg 32 bit on my windows 7 with framework 4.0 upon loading on the memory dump files with a size of 2 gigz its prompt some error its says again. Could not find the I:Installerwindebug x64 and x86 win 2008 r2 IIS crush dump files checkerappcrushdumpappcrushdumpmemory.dump Dump file, Win32 error On87 the parameter is in correct.

what is the problem with this.

Fabio | Jan 9, 2012 10:07 AM
What if I don't hacve a dump file in my directory ? I have only a WER file, but no big ones! Do I have to enable the dumping somewhere? Thanks

Paul | Jan 9, 2012 11:00 AM
@ Fabio
You need to install DebugDiag that will allow you create a rule that will generate a dump file when one of your application pools crash. I had actually done that before I figured out I needed the WinDbg to actually process my dump files. I am going to try to update this article with details on installing DebugDiag and setting up the system to generate the dumps.

Paul | Jan 9, 2012 11:04 AM
If you need to manually create a dump file, for instances where your Applications are just consuming lots of CPU or Memory, but not crashing, you can do this by opening task manager, and then right click on the process in question and select "Create Dump File". However if this is to debug a application that is crashing then you will still need to install and setup DebugDiag to capture the dump files when the process crashes.

Fabio | Jan 9, 2012 3:17 PM
Thank you, I'll give it a try!

Puneet | Jan 10, 2012 8:19 AM
Thank you so much for this article. It was very helpful. Regards,

Puneet | Jan 10, 2012 4:36 PM
Also, !clrstack -p showed the parameters and their values passed in the methods.

Regards,

Alex | Mar 29, 2012 3:06 PM
for .net 4 read the following to properly .loadby

http://stackoverflow.com/questions/4373683/unable-to-load-sos-in-windbg

Andreas | Apr 3, 2012 8:54 AM
Thank you!

This helped me to find our problem! I was lost before I found this post!

Our issue was a home made url re-writer that at some point got stuck in a loop!

Best regards Andreas

John | Apr 23, 2012 12:06 PM
You have no idea how helpful this post is to me. Thank You!!!!!

-John

Michal | May 10, 2012 5:49 AM
Hello,

We have a very similar problem with an application pool serving a web application on our WS2008 R2 SP1 Server.

A process serving application pool 'EHK' suffered a fatal communication error with the Windows Process Activation Service. The process id was '1164'. The data field contains the error number.

However no dump is created when this error occurs, because the process doesn´t crash, nor DebugDiag catches anything.

This is the only evidence of this error. Nothing else in Application nor System Logs

Any idea how to diagnose and debug this kind of error, please?

Regards,

Paul | May 10, 2012 9:56 AM
@Michal
Windows Server 2008 R2 doesn't create dump files on its own. You have to configure it to do so. I learned this the hard way : )

Visit this page
How to use Debug Diag
This page will show you how to install Debug Diagostics Tools, and how to set them up to force the system to capture a dump file. After you have the dump file come back to this blog for info on how to open and read it.

Michal | May 11, 2012 7:45 AM
@Paul:

Thanks for reply,

This is exactly what I have tried, but still no success :-/ The W3WP process doesn´t crash, so no dump is generated via DebugDiag, and setting performance rule and generating the dump manually doesn´t work either, because the problematic W3WP process doesn´t hang forever - it causes 100% CPU usage for 60s and then dies Trying manual dump didn't bring anything interesting at all.

Simon | May 14, 2012 9:03 AM
Thanks for posting this - Very useful!

MMC | Apr 3, 2013 7:44 PM

Hi there,

When I try to .loadby sos mscorwks it says "unable to find module mscorwrk"

Please let me know what to do.

Thanks

MMC | Apr 3, 2013 7:47 PM

Hi there,

When I try to .loadby sos mscorwks it says "unable to find module mscorwrk"

Please let me know what to do.

Thanks

Paul | Apr 10, 2013 7:33 PM
@ MMC
There is a 32bit and 64bit version of WinDbg. Make sure you are using the right one for your application.

MMC | Apr 11, 2013 7:30 PM
@Paul : I used the right one. is there any other possible reason?

Paul | Apr 11, 2013 7:49 PM
It depends on the framework.

Check this link out to see if these solutions help

MMC | Apr 16, 2013 6:51 PM

@Paul : I tried .loadby sos clr and it worked. now when I give command !clrstack. it says

OS Thread Id: 0x1f80(39)

Unable to walk the managed stack. The current thread is likely not a manged thread. You can run !threads to get a list of managed threads in the process.

Please let me know what's wrong. I created dump file using dubug diagnostic tool. I really want to see stack to find out the issue.

MMC | Apr 18, 2013 10:14 PM

@Paul. Somehow I managed to get it worked. after give .loadby clr and !stackclr I am get the below and I dont understand where the problem lies. can you please help? Thanks in advance

0:005> .loadby sos clr
0:005> !clrstack
*** ERROR: Symbol file could not be found. Defaulted to export symbols for clr.dll -
PDB symbol for clr.dll not loaded
OS Thread Id: 0x17bc (5)
Child SP IP Call Site
010cb614 7579c41f [GCFrame: 010cb614]
010ccfc4 7579c41f [GCFrame: 010ccfc4]
010cd058 7579c41f [InlinedCallFrame: 010cd058]
010cd054 6ee71943 DomainNeutralILStubClass.IL_STUB_PInvoke(System.String, Boolean, Boolean, Boolean, System.Runtime.CompilerServices.StackCrawlMarkHandle, Boolean, System.Runtime.CompilerServices.ObjectHandleOnStack)*** WARNING: Unable to verify checksum for mscorlib.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for mscorlib.ni.dll

010cd058 6ede1244 [InlinedCallFrame: 010cd058] System.RuntimeTypeHandle.GetTypeByName(System.String, Boolean, Boolean, Boolean, System.Runtime.CompilerServices.StackCrawlMarkHandle, Boolean, System.Runtime.CompilerServices.ObjectHandleOnStack)
010cd0b4 6ede1244 System.RuntimeTypeHandle.GetTypeByName(System.String, Boolean, Boolean, Boolean, System.Threading.StackCrawlMark ByRef, Boolean)
010cd0dc 6ede0fb3 System.RuntimeType.GetType(System.String, Boolean, Boolean, Boolean, System.Threading.StackCrawlMark ByRef)
010cd0fc 6ede29f3 System.Type.GetType(System.String)
010cd10c 6e49df7d System.CodeDom.Compiler.CompilerInfo.get_IsCodeDomProviderTypeValid()*** WARNING: Unable to verify checksum for System.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.ni.dll

010cd114 6bad029e System.Web.Compilation.CompilationUtil.GetRecompilationHash(System.Web.Configuration.CompilationSection)*** WARNING: Unable to verify checksum for System.Web.ni.dll
*** ERROR: Module load completed but symbols could not be loaded for System.Web.ni.dll

010cd1f8 6bb15fc4 System.Web.Configuration.CompilationSection.get_RecompilationHash()
010cd22c 6bacf553 System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate2(System.Web.Compilation.StandardDiskBuildResultCache)
010cd27c 6bacf144 System.Web.Compilation.BuildManager.CheckTopLevelFilesUpToDate(System.Web.Compilation.StandardDiskBuildResultCache)
010cd2a8 6bacee53 System.Web.Compilation.BuildManager.RegularAppRuntimeModeInitialize()
010cd2bc 6bace706 System.Web.Compilation.BuildManager.Initialize()
010cd2d0 6bb172ec System.Web.Compilation.BuildManager.InitializeBuildManager()
010cd308 6baca043 System.Web.HttpRuntime.HostingInit(System.Web.Hosting.HostingEnvironmentFlags, System.Security.Policy.PolicyLevel, System.Exception)
010cd360 6bac9e1c System.Web.HttpRuntime.InitializeHostingFeatures(System.Web.Hosting.HostingEnvironmentFlags, System.Security.Policy.PolicyLevel, System.Exception)
010cd374 6bac8716 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel, System.Exception)
010cd3bc 6bac8489 System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
010cd67c 6faa21bb [GCFrame: 010cd67c]
010cd660 6faa21bb [GCFrame: 010cd660]
010cd808 6faa21bb [ContextTransitionFrame: 010cd808]
010cd958 6faa21bb [GCFrame: 010cd958]
010cd93c 6faa21bb [GCFrame: 010cd93c]
010cebcc 6faa21bb [CustomGCFrame: 010cebcc]
010cebe0 6faa21bb [CustomGCFrame: 010cebe0]
010cebf4 6faa21bb [CustomGCFrame: 010cebf4]
010cec08 6faa21bb [CustomGCFrame: 010cec08]
010cec1c 6faa21bb [CustomGCFrame: 010cec1c]
010cec30 6faa21bb [CustomGCFrame: 010cec30]
010cec44 6faa21bb [CustomGCFrame: 010cec44]
010cf018 6faa21bb [TPMethodFrame: 010cf018] System.Web.Hosting.HostingEnvironment.Initialize(System.Web.Hosting.ApplicationManager, System.Web.Hosting.IApplicationHost, System.Web.Configuration.IConfigMapPathFactory, System.Web.Hosting.HostingEnvironmentParameters, System.Security.Policy.PolicyLevel)
010cf038 6bac79d8 System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
010cf228 6bac66ef System.Web.Hosting.ApplicationManager.CreateAppDomainWithHostingEnvironmentAndReportErrors(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
010cf264 6bac664b System.Web.Hosting.ApplicationManager.GetAppDomainWithHostingEnvironment(System.String, System.Web.Hosting.IApplicationHost, System.Web.Hosting.HostingEnvironmentParameters)
010cf2ac 6bac6595 System.Web.Hosting.ApplicationManager.CreateObjectInternal(System.String, System.Type, System.Web.Hosting.IApplicationHost, Boolean, System.Web.Hosting.HostingEnvironmentParameters)
010cf2d4 6bac1e49 System.Web.Hosting.ProcessHost.StartApplication(System.String, System.String, System.Object ByRef)
010cf320 6c165950 DomainNeutralILStubClass.IL_STUB_COMtoCLR(IntPtr, IntPtr, IntPtr)
010cf56c 6faa25a1 [GCFrame: 010cf56c]
010cf6c4 6faa25a1 [ComMethodFrame: 010cf6c4]

Paul | Apr 18, 2013 11:00 PM
@MMC

Is that the entire output? I don't see anything that looks like a recursive function call.

MMC | Apr 21, 2013 9:34 PM

@Paul

Yes that's the entire output. I dont think it's a recursive function do you think is it anything else?

Todd | Aug 5, 2016 3:45 PM

I'm having this exact problems (same error codes) but receive no application log from Windows Error Reporting. I also looked in the C:ProgramDataMicrosoftWindowsWERReportArchive as show in the example..but no files.

Do I need to turn something on the generate logs?

Todd | Aug 5, 2016 4:24 PM

I'm trying DebugDiag. Did read down far enough.

KGS | Nov 23, 2016 12:06 PM

@Paul, great article, thank you for taking the time to put this together. We are experiencing this exact problem. Everything you describe, in sequence. Sometimes we go days without, other times it will fault many times in a row (in a matter of minutes). Followed your instructions, however, when I browse to the AppCrash folder, the only file present is Report.wer. None of the other files you show (most notably, the main dump file) are not there. Can you think of any reason why this would be the case? Does something need to be enabled? Thank you in advance for any guidance you can provide.

Ram | Jul 26, 2017 6:03 AM

Hi Paul,

I am curious to finding out how to simulate such w3wp crashes? Please throw some light.

Regards,
Ram

Paul | Jul 26, 2017 9:02 AM

Hi Ram,

Interesting question. The reason I would debug the W3WP, is because I don't know of the reason of the crash. In my case the issue was a infinite loop that only happend under certain criteria. If you are looking to simulate a crash, you could try to create an opperation that would do something similar, which would consume all your system's resources.

name
Email	Needed to confirm comment, but not made public.
Website