I was trying to data bind a list of colors to a ComboBox and have the SelectedValue property bound to a particular color. The frustrating thing was that even though there was a color matching the “selected” color in the list, the ComboBox never had a selection when first displayed.

Eventually I discovered that my “selected” color was white (all color components at 255) while the list’s equivalent was Color.White. Careful reading of the Color.Equals documentation tells me that I was silly to think that white is white or black is black:

Black and FromArgb(0,0,0) are not considered equal, since Black is a named color and FromArgb(0,0,0) is not.

The path with the problem was something like this: \\qa\tests\۩۩۩۩۩

 The unusual character there is the Arabic symbol “Place of Sajdah” (U+06E9). For some reason, this path didn’t work correctly.

Ultimately, I tracked the problem down to a bit of code that was trying to test whether the path ended with a separator character, and if not, add one. It looked like this.

if (!path.EndsWith(Path.DirectorySeparatorChar.ToString()))
{
    path += Path.DirectorySeparatorChar;
}

This code wasn’t working as expected, so when the path was combined (using string concatenation) with a file name, the path separator was missing, which lead to the final symptom described by the defect. There are a couple of problems with this code.

1. System.IO.Path.Combine is the right way to merge paths where possible — you shouldn’t have to worry about whether there is a trailing slash or not
2. The simplest form of EndsWith (and StartsWith) doesn’t work as expected with roughly 20% of Unicode characters

Michael Kaplan explains that this is because many Unicode characters don’t have entries in the Windows sorting weight tables, and are therefore effectively invisible for some comparisons.

Surprisingly, @"\\qa\tests\۩۩۩۩۩".EndsWith(@"\") evaluates to true. All of the Place of Sajdah characters are “invisible” in terms of weight, and once you realize that, then obviously the string ends with a backslash. Also surprising is that all strings start with “۩”. That is, "Cheese".StartsWith("۩") evaluates to true. The string.Equals method and operator don’t have this problem, though, because they compare lengths before ever getting to the part where the Place of Sajdah characters would be thrown out of “۩۩۩Monkey۩۩۩” and thus become equal to “Monkey”.

The way to fix the problem is to use either StringComparison.Ordinal or StringComparison.OrdinalIgnoreCase in the call to StartsWith or EndsWith. So the code snippet from above would work as expected when modified as follows:

if (!path.EndsWith(Path.DirectorySeparatorChar.ToString(), StringComparison.OrdinalIgnoreCase))
{
    path += Path.DirectorySeparatorChar;
}

My first approach was to try to profile the process’ memory usage with the Visual Studio 2005 profiler, but after wading through the report for a while, I was stymied by a minor obstacle: I didn’t know what the heck I was doing. I didn’t have a good sense of what the memory in a .NET process ought to look like, or what was unusual. I found some useful information in an article by Tim Anderson, but I still was lost.

I thought maybe a dedicated profiler might steer me along better, so I downloaded a trial of Red Gate’s ANTS profiler. After spending some time with it, I found a few little memory issues (did you know that the default Windows Forms icon is something like 10K compared to a blank icon at 1K?), and some potential bigger issues (we have a background thread that holds a large data structure in memory even when the thread is paused). The epiphany came when I realized that I was looking at a process with 100 MB in private bytes, but the entire managed heap was less than 1 MB. The leak wasn’t even in the managed heap.

I wondered if there might be some way to use the unmanaged debug heap, and _CrtDumpMemoryLeaks from a C# application. I briefly considered pinvoke, but quickly ran into complications with loading the C runtime libraries, since they have to be in the side-by-side directory and only loaded from an application with a valid manifest, etc. etc.

Wayne Nelson, my boss, reminded me of a module that makes heavy use of unmanaged code that we’ve suspected of leaking for a while, and we came up with the idea of exercising that code in the presence of performance counters. It was pretty easy to write a little console application that referenced that module, invoking its main call path in a loop of 100,000 iterations.

Then I opened Performance Monitor and added a couple of counters: One for the process private bytes, and the other for the total bytes in all managed heaps in the process. I also scaled the private bytes to match the scaling of the managed heaps. Here is the resulting graph, where the blue line is private bytes, and the yellow line is managed bytes:

It’s obvious that the leak is, in fact, in unmanaged memory. The first jump is due to loading in some native DLLs, but the steady climb from there to the next plateu represents the 100,000 loop iterations.

Having the graph as a tool, it became a matter of doing a conceptual binary search through the code. I would comment out large chunks until the private bytes line ran flat. Then I knew that the leak was in the commented-out code. In this case it became almost reductio ad absurdum. All that was left was a pinvoke call:

int result = NativeMethods.StgOpenStorageOnILockBytes(lockBytes,
  IntPtr.Zero,
  this.readOnly ? PropertyReadFlags : PropertyWriteFlags,
  IntPtr.Zero,
  0,
  out storage);

The ILockBytes being passed in was my own C# implementation that wrapped a System.IO.Stream instance, so to get a little closer to absurdum, I made it a zero byte Stream and still had the leak. In other words, I was trying to open structured storage on a zero byte stream, which obviously failed, but still caused a memory leak. The only things left in play were my pinvoke signature, and my ILockBytes implementation, which didn’t allocate any unmanaged memory.

The call to StgOpenStorageOnILockBytes did call my ILockBytes Stat method:

public void Stat(ref System.Runtime.InteropServices.ComTypes.STATSTG pstatstg, int grfStatFlag)
{
  pstatstg.cbSize = stream.Length;
  pstatstg.type = (int)STGTY.LOCKBYTES;
}

It seemed pretty harmless.

A hint came from the ILockBytes::Stat documentation. If grfStatFlag is STATFLAG_NONAME, then the Stat implementation shouldn’t allocate the STATSTG::pwcsName member. Checking in the debugger, grfStatFlag was, in fact, STATFLAG_NONAME. But I wasn’t allocating any memory or setting pwcsName anyway.

Then I realized that the STATSTG structure was coming from unmanaged code, and as such code is wont, it was completely uninitialized. The pwcsName member was coming through pointing to garbage. I changed my Stat implementation to set pwcsName to null, and my Performance Monitor graph went sublimely flat. My theory is that upon returning from the Stat method, the marshaler was taking the never initialized garbage string and allocating a copy with CoTaskMemAlloc. But since the caller had specified STATFLAG_NONAME, it was not expecting to need to free any memory. For good measure, I initialized all of the STATSTG members.

While this was a pretty strange leak, the strategy for finding it can be applied to other more mundane leaks as well.

1. Write a small program that repeatedly calls any suspect code (anything that calls unmanaged code).
2. Set up a Performance Monitor session that graphs both private bytes and the managed heaps to see if you’ve found the general leaky area.
3. Start trimming the code and rerunning the test program until the leak disappears.
4. Figure out what could possibly leak in the located code.

The exception was thrown from line 5 in the code below.

1
2
3
4
5
6
7
8
9
10

public static FileTypeHandlerCatalog Load(string fileName) 
{ 
using (StreamReader reader = new StreamReader(fileName)) 
    {         
        XmlSerializer serializer = new XmlSerializer(typeof(FileTypeHandlerCatalog));
        FileTypeHandlerCatalog catalog = (FileTypeHandlerCatalog)serializer.Deserialize(reader); 
        catalog.Validate(); 
        return catalog; 
    } 
}

The exception message wasn’t terribly helpful either since it was obviously disgorging  implementation details from somewhere deep in the XmlSerializer class:

System.ArgumentException: Item has already been added. Key in dictionary: 'path' Key being added: 'path'

The stack trace led me to the System.CodeDom.Compiler.Executor.ExecWaitWithCaptureUnimpersonated method, which contains the following bit of code:

StringDictionary sd = new StringDictionary(); 
foreach (DictionaryEntry entry in Environment.GetEnvironmentVariables()) 
{ 
  sd.Add((string) entry.Key, (string) entry.Value); 
}

I guess it is copying the environment variables so that it can add a new one related to security. The problem is that when running under Ant, for some reason there were three different PATH environment variables with an assortment of cases: PATH, Path and path. Looking at the StringDictionary.Add implementation, the problem became clear. In line 7, it converts the key to lower case in order to add it to the hash table, resulting in duplicate keys.

1
2
3
4
5
6
7
8

public virtual void Add(string key, string value) 
{ 
    if (key == null) 
    { 
        throw new ArgumentNullException("key"); 
    }     
    this.contents.Add(key.ToLower(CultureInfo.InvariantCulture), value); 
}

So it appears to be a bug in the Executor class — it doesn’t expect to encounter multiple keys in the environment variable dictionary that differ only by case.

I solved the problem by pregenerating the serialization assembly using sgen, which is generally a good idea anyway.