How to identify what is leaking memory in a .NET application

A web api in production was leaking memory, lots of it. It was consuming over 27 gigs of memory where the usual consumption is around 200 Mbs.

I used this approach to pinpoint what was leaking since the memory leak was replicated again on a staging environment which I had access to. I needed to act quickly before the process chokes the server and becomes in need of a reboot.

  1. Create a process dump file. This can be done through task manager. The task manager you get from ctr+alt+del is a 64 bit version, this will generate a 64 bit dump. To generate a 32 bit dump use the 32 bit task manager (C:\Windows\SysWOW64\taskmgr.exe)TaskManager
  2. Download and install windbg.exe. The latest version of this can be found in the Windows 8.1 SDK. Make sure to use the 64 bit version when debugging 64 bit applications.
  3. Open the dump file in windbg.
  4. Go to File -> Symbol Path and add the debug symbols of your application. I have added the bin/debug directory as the symbol path.
  5. Open the command window (if not already open) from View -> Command.
  6. Type the following commands:
    1. .loadby sos clr (this loads the sos.dll to help in debugging managed applications)
    2. .symfix (automatically points to the Microsoft symbol store)
    3. !dumpheap (this may take some time to run depending on the dump size)

The last command will show you all the objects on the managed heaps, it will also show you the number of objects created for each type and the size of these instances. This is very helpful in narrowing the search radius for the leaky code. The dump looks something like this.

dump

The first column represents the memory address. The second column is the number of instances in memory. The third column is the size these instances are taking up.

How my brain categorizes bugs

As developers we spend most of our time debugging code we wrote (or others wrote). The debugging process is time consuming, frustrating and a pain in the neck but very rewarding when you solve a tough bug. I take bug fixing very seriously and tend to get attached a lot to the code, so much in fact that my brain automatically categorizes bugs. Here are the categories as proposed by my brain:

  • The Business Bug: These kind of bugs are related to the business or logic your are implementing, if you are familiar with the implemented logic this should be an easy bug to fix.
  • The “Time Bomb” Bug: These bugs are hidden away under layers of code and logic, they have been there for an unknown amount of time waiting for the appropriate moment to pop up. The appropriate time for this bug to show up is always the most inappropriate time for the developer.
  • The Pregnant Bug: These bugs are mean !!! Solving a bug of this type will trigger the bug to go into labor and give birth to other bugs.
  • The “Chain Reaction” Bug: These are my favorite bugs, they are the opposite of “The Pregnant Bug”. Solving a “Chain reaction” bug will automatically solve other bugs along the way.
  • The “Peek-a-boo” Bug: These bugs are shy, they will show themselves briefly and then hide away. Sometimes these bugs are a one time occurrence, but most of the time you will have to play hide and seek with the bug.
  • The Environment Bug: These bugs are related to the environment where the code is developed or deployed. These bug often come as a relief to the developer since there is nothing wrong with the code. But beware these kind of bugs can turn out to be tough suckers to solve.
  • The Hybrid Bug:  Most of my bugs fall into this category. You can mix and match any of the above categories to produce a new category. For example you can have a “Business Time Bomb” bug or a “Pregnant Environment” bug.

These are the categories that I can squeeze out of my brain now. If your brain assigns your bugs to other categories please share them Smile