Over engineered build systems from Hell

While I was at Autodesk years ago we went through various build systems. When I first started the build was written in Perl. Dependencies were specified using a .csv file. I had no idea how it worked, nor did I care how it worked since I was interested in other things during those years. The build could routinely take an hour and 45 minutes, and was mind-numbingly boring since we usually had to do it everyday. And if you were unlucky multiple times per day. Even worse In fact on the build servers the build would routinely take almost 4 hours!

What a horrible tax.

Later on, a hot-shot programmer came along and rewrote the build system to use ruby and rake. It was supposed to be faster, which it kind of was. But the complexity was just as bad, and no one knew ruby nor how rake worked. Then the developer left the company, leaving behind a black box. So that silo of information was gone, and no one knew how the build system worked. It took 2 developers about a year and a half or so to learn the build system well. To really be able to work on it at the level the original developer had done.

To be sure there were problems with the build. It still took a long time to build the product. Somewhere near an hour. About the only thing we did gain was the ability to do incremental builds.

But modifying the build was the main problem. The build, written in ruby, completely reinvented the wheel on so many different areas. To understand this better, you have to understand that the product at the time was built using Microsoft tools because it was based solely on the Microsoft platform. Thus the source project files were in a build language that Microsoft created. That build language was built into visual studio and was called MSBuild. Instead of using Microsoft tools to create the build, ruby and rake were used instead. Instead of using Microsoft tools to parse the xml project files, a ruby gem was used. Instead of using anything from Microsoft to help with the build, everything was re-invented from scratch. Parsing visual studio .vcproj (and eventually .vcxproj) files was done tediously and laboriously and mind numbingly using rake and some xml gem. Talk about recreating the wheel! For instance, lots and lots of code was written to duplicate a simple function Microsoft already offered. Though a Microsoft API call that retrieved a fully instantiated object with full properties all intact.

Copying files to the build directory was another disaster too. It would take around 10 to 12 minutes to copy 7000~14000 files. It originally was somewhere near 7000 files, but grew over time. All written in ruby code that no one knew how to debug except to put print statements in.

Another problem was adding build properties. If you wanted to add a build property (a key value pair), you had to add it to multiple places in the ruby build, knowing exactly what to modify (in duplicate) and such. It was horrible.

Mixing ruby with MSBuild was like mixing iron and clay. They don’t mix well at all. It was like a ruby straight jacket that hindered the build and visual studio upon which it was based.

There had to be a better way.

Eventually when the frustrations with the build boiled over, I learned MSBuild and figured out how to build max without ruby. It took over a year, from when I first got a working prototype, to get something into the main branch of max. Simply due to bureaucratic inertia. There are lots of people in positions of power who simply say no before learning about a subject. Which was something all too common there. The first liberation was freeing the list of things to get built from ruby. Yes the list of DLL’s and EXE’s to get built was specified in some arcane ruby syntax somewhere. The first thing was moving that list to a democratic XML file. Now any build language could parse it and find out what to build. The second thing was moving the list of files to copy to an XML file. Now any build system could know what files to copy as well.

Once those two things were in place, it was time to put in the build system that I originally came up with (during a particular Christmas break).

It was written in pure XML, with one MSBuild extension that was written using C#. All the tools were native to visual studio, and what you built on the command line was what was built in visual studio. They both used the same properties (using native property sheets) and built in the same way.

What’s more I found that using native MSBuild tools to copy those 7000+ files to build now was incredibly fast. In fact, while once debugging through that old ruby code responsible for copying, I found the source of the 10 minute copy task. Or why it took 10 minutes. It was using an N factorial algorithm! So given directory A with B thru Z subdirectories, it would iterate through all directories n! times. Each directory was not parsed once, but N! times according to the amount of sub-directories that existed. It was an unholy mess that proves that re-inventing the wheel is usually a disaster waiting to happen. Now back to the improvement: With the new msbuild copy mechanism it took 20 seconds to copy all those files. 20 seconds versus 10 minutes was a big improvement.

Incremental builds also greatly improved. Go ahead and build your product from scratch. Now don’t change a thing and rebuild. If you have a smart build system, it should just take a few seconds and nothing will happen. The build system will be smart enough to essentially report that nothing changed and therefore it did no work. My new build did just that in a matter of seconds. The old build system used to take about 5 minutes to do that (And it still did work anyways…).

Speaking of performance. The time to compile and link the actual files didn’t change much, because that was always in visual studio’s corner and not ruby. The improvement in performance came from the copying actions that now took 20 seconds. Also noticeable was the shorter time involved from when the build started to when the first CPP file was getting compiled. In ruby/rake it took quite a few minutes. In the new build it took a few seconds. Eventually when I got a new SSD hard-drive, I was able to get the build down to 22 minutes on my machine.

The build at Century Software

Later on I moved to Century Software, a local company to where I live. That was such a fun place. Anyways their build system for their windows product was written in Make! Yes Make the original, ancient build system that you can’t even find documentation for anymore. I mean literally, I found (I think) one page of documentation somewhere on some professors lecture notes. The docs were  horrible. Make implemented here was so old it built one C file at a time. No multi-threading, no parallel builds nothing. Slow was the operative word here. That and a incomprehensible built output that was so verbose it was almost impossible to comprehend. The only good thing about it was that it immediately stopped on the first error.

So eventually I rebuilt that using MSBuild too. It took me a few months in my spare time. No bureaucratic inertia, no one telling me no. I just worked on it in my spare time and eventually I had a complete and fully functioning system for the tinyterm product. This build was the best I’ve ever done, with zero duplication, extremely small project files and a build that was very fast. It went from 45 minutes to a minute and a half.

When writing a product, the build system should be done using the tools that the platform provides. There should be no transmogrifying the code, or the build scripts (like openssl) before doing the build. When writing ruby on rails use rake for your build process’s. When targeting Microsoft platforms use msbuild. Using java, then use maven. Data that must be shared between build platforms should be in xml so that anything can parse them. And most important of all, distrust must go, and developers and managers must have an open mind to new things. Otherwise the development process’s will take so long, and be so costly, and exert such a tax that new features will suffer, bugs will not get fixed and customers will not be served and they will take their money elsewhere.

Difference between Debug and Release Builds


I wrote this document way back in 2010, which I found sitting on my computer. I wrote it for folks in our QA department. But I thought it good enough to share with other folks since it describes some basic things. So here is the document:

In a debug build of max, uninitialized memory is initialized with not garbage, but a certain pattern used to help catch bugs: 0xcccccccc.

For instance:

0x0017EBD8 cccccccc
0x0017EBDC 00000000
0x0017EBE0 00000000
0x0017EBE4 00000000
0x0017EBE8 00000000
0x0017EBEC 00000302
0x0017EBF0 cccccccc
0x0017EBF4 3ed6d0e0

Uninitialized memory is marked with this pattern to help catch bugs. For any pointer dereferencing this value will cause an access violation and crash the application.

Case Study

The animation reaction manager displays a curve control. That curve control can of course display many curves. Each curve can contain many key points.


And here are the points:


The problem is manifested when you insert a point into the middle of the curve. You should expect this:


But instead, in a debug build you get this:


The problem is not the curve control, but actually the reactor controller. The reactor controller goes back and alters the points after the point was correctly inserted. (A bug was fixed by me btw) This reactor controller does its dirty business after the UI widget has already correctly displayed its data. This highlights two bugs:

1. Why is the reactor controller modifying the UI control when the UI was already correctly displayed?

2. Why is the reactor controller modifying the point with such a massive value?

The point where the first point gets altered is in the Reactor::UpdateCurves function in the following file:

In this code:

CurvePoint pt = curve->GetPoint(0,i);
// pt here is correct
switch (type)
pt.p = Point2(masterState->fvalue, slaveStates[i].fstate);

DbgAssert(pt.p.y > -1000.0); // assert fails since pt is incorrect
curve->SetPoint(0,i, &pt, FALSE, TRUE);

The value for slaveStates[i].fstate is the following floating point number:


Where did this value come from?

Well examining it in the debugger’s memory shows this:

memory address of some memory:

0x0017EBD8 cccccccc

As you can see, a debug build take uninitialized memory and fills it with 0xcccccccc. This is to help catch errors. Now if you take that value and interpret it as an integer you get:

0x0017EBD8 -858993460

And if you interpret it as unsigned integer you get:

0x0017EBD8 3435973836

And if you interpret those bits as a floating point number you get:

0x0017EBD8 -1.0737418e+008

That is scientific notation for -1.0737418 x 10^8, or simply -1.0737418 x 100,000,000 or rather, -107374180.0. Or basically negative 107 million.

So this explains why the value of the first point is such a massive negative number. Solving this bug is not the point of this document, but is another story altogether.

In every single Debug build (On Windows platforms), all over the world, all uninitialized variables (on windows platforms) will always be 0xcccccccc. That number is always rendered as the same floating point value shown above. It is always the same with NO EXCEPTIONS.

That means if you want predictability in reproducing bugs you go with a debug build. If you want random behavior you go with a release build:

Here is the same work-flow in a release build:

Max 2010 (Release build):


The inserted point has a completely random y position (thankfully not as big as -107 million!). Also the first point is assigned to a random value. What is the random value? It is whatever the value was the last time memory was used there in that memory location. That memory could be clean, or it could be leftover from anything else. It could be memory that was last used 2 seconds ago, or 2 hours ago, or 2 days ago.

So again, if you see bugs with random behavior, it’s most likely an uninitialized variable. If you want to narrow down the repro steps, and actually solve it, use a debug build.

Solving STL assertion: ‘Assertion failed: vector iterators incompatible’ when calling std::vector::clear

Recently here at work we ran across a problem in the STL code that result from calling std::vector<>::clear(). The problem was that calling clear on a vector threw a debug message, which in our case, crashed the application. The callstack look like this:

msvcp100d.dll!std::_Debug_message() Line 13 C++
gw_objio.dle!std::_Vector_const_iterator<std::_Vector_val<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > > > >::_Compat() Line 239 C++
gw_objio.dle!std::_Vector_const_iterator<std::_Vector_val<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > > > >::operator==() Line 203 C++
gw_objio.dle!std::_Vector_const_iterator<std::_Vector_val<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > > > >::operator!=() Line 208 C++
gw_objio.dle!std::vector<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > > >::erase() Line 1194 C++
gw_objio.dle!std::vector<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> >,std::allocator<std::basic_string<wchar_t,std::char_traits<wchar_t>,std::allocator<wchar_t> > > >::clear() Line 1218 C++

Where the debug message says:

c:program files (x86)microsoft visual studio 10.0vcincludevector(238) : Assertion failed: vector iterators incompatible

If you look at the code for vector<>::clear it is really very simple:

void clear()
{ // erase all
erase(begin(), end());

So what was the problem?

For our particular case, the memory for the std::vector was allocated via LocalAlloc by some programmer years ago. This problem is not immediately apparent since in this particular case, LocalAlloc was allocating memory for a typedef struct that happened to contain the vector (Among others). Now, since the vector is NOT plain old data (POD) it’s constructor was not called, and when the memory was freed it’s internal pointers were left dangling. So when the time came to use them (i.e. clear), things blew up.

Not fun to debug.

I take it this (perhaps) once used to work originally. But perhaps with the switch to the Microsoft Visual C 10.0 compiler and it’s attendant change in the STL, it was enough for this problem to bubble to the surface.

Point of the story:

1. We are in the 20th century now. If you are programming in C++ in the 20th century, don’t use Paleolithic era API’s to allocate memory for non POD data.

2. Don’t mix and match memory allocation routines. If using C++ use new/delete especially for non-POD data.

Solving linker error ‘LNK2022: metadata operation failed’

If you are compiling managed C++ projects, and ever encountered this idiotic linker error:

1>xxx.obj : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (_PROPSHEETPAGEW): (0x020001f8).

1>xxx.obj : error LNK2022: metadata operation failed (8013118D) : Inconsistent layout information in duplicated types (_PROPSHEETPAGEA): (0x02000206).

than this post is for you.

For some reason in our code, this linker error always comes in pairs. No matter, in our code base it usually manifests itself after rearranging header files in a file that is compiled with /clr compiler switch.

The problem is caused by windows.h being included after other header files:

#pragma unmanaged
#include “file.h”
#include “otherfile.h”
#include <windows.h>

#pragma managed

The solution is to make sure that windows.h is included before anything else.

#pragma unmanaged
#include <windows.h>
#include “file.h”
#include “otherfile.h”

#pragma managed


Solving Link error 1112

Yesterday I brushed off some old code I had not touched in quite a while. It was a layer manager plugin for Autodesk 3dsmax that I had written years ago. I don’t think I had even opened any project files for this project in the last 12 months. So I re-acquainted with the code, made some changes to the files, and went to recompile it. Now this project compiles to both 32 and 64 bit targets. So I randomly chose a 64 bit target, and built the sucker.

Imagine my shock when I encountered a linker error stating:

module machine type ‘X86’ conflicts with target machine type ‘x64’

I knew enough about this in the past, and never had a problem before. Like I said before, I hadn’t touched the code in long time. What’s more, is my code was under source control!

I opened up the project settings and double and triple checked my Linker Target Machine settings. Indeed it was properly set at: MachineX64. Perfect!

Still the problem persisted. Then I got worried that the library files I was linking in were corrupted, and somehow 32 bit libraries were intermixed with 64 bit libraries. I didn’t want to download the entire 3dsMax SDK again, in the off chance this hypothesis was true. So I researched a solution to check what type of libraries I had!

So I used the utility DumpBin.exe that is found here:

"C:Program Files (x86)Microsoft Visual Studio 9.0VCbindumpbin.exe"

And by the way, in windows 7 I did a search a few times for ‘dumpbin’, which failed every time. Turns out you have to search for dumpbin.exe. Go figure.

Anyways, I used this tool to dump out the functions in my lib file. Then I was able to inspect the functions and see what platform there were compiled for.

For instance, notice the ‘Machine’ entry in a function I dumped out using dumpbin.exe:

Version      : 0
Machine      : 8664 (x64)
TimeDateStamp: 49B974BD Thu Mar 12 14:46:53 2009
SizeOfData   : 0000002D
DLL name     : maxutil.dll
Symbol name  : ??0Path@Util@MaxSDK@@QEAA@PEBD@Z (public: __cdecl MaxSDK::Util::Path::Path(char const *))
Type         : code
Name type    : name
Hint         : 18
Name         : ??0Path@Util@MaxSDK@@QEAA@PEBD@Z

Notice the second line contains x64, a dead give away this is a function compiled for an x64 target platform. So the problem was, my project linked in a lot of libraries from the 3dsmax sdk. The 3dsmax sdk contains about 49 library files. How was I supposed to inspect every function from every library file?

Sounds like a job for a script:

@echo off
call "C:Program Files (x86)Microsoft Visual Studio 10.0VCvcvarsall.bat"

for %%f in (*.lib) do dumpbin.exe -headers %%f | findstr /c:"Machine      :" >> _lib-exports.txt"

@echo on

So I put the above script into a batch file, and placed the batch file in the directory that contained all my library files. I iterated through all the library files, dumping out the header information. This was piped to find string, which searched for the machine information. I then appended that to a log file which I could inspect at my leisure. The log file contained 13642 lines looking like this:

Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)

So that was it. I opened the file in my favorite text editor, and a search for x86 turned up nothing. It was x64 all the way down.

So now what was I supposed to do? I was stuck. Let me review all I had done.

  1. My target machine linker setting was properly set.
  2. My build configuration setting was properly calling ‘x64’ and not win32.
  3. All the libraries I was importing were actually 64 bit libraries.

All appeared lost and hopeless until I ran across a tidbit on an MSDN forum.

Here the gentleman answered:

If you have left the ide at the default settings, then it will actually use the x86_amd64 compiler only. This is set under tools->options->projects and solutions->vc++ directories. Under amd64 executables the directory should be $(VCInstallDir)binx86_amd64 followed by $(VCInstallDir)bin.

This method works for all versions of windows so its a default setting. If you want to though you can change the x86_amd64 to just amd64.

So I looked at the executable settings for my x64 build configurations in VS 2008. It looked like this:


I changed it to this by adding the x86_amd64 directory:


And everything now works. My project now compiles and links just fine.

So I am left to surmise that some external add-on to visual studio hosed my executable tools settings for 64 bit configurations. I had off and on over the last year been using bullseye, which messes around with the settings in the project executable directories. So Perhaps that is what messed up that setting in visual studio. But now it’s fixed, and I am very happy.

Pointer Truncation

Pointer truncation is an evil that all developers of native C++ code need to worry about. This is simply storing a wide piece of memory in a narrower bit of memory. This can become an issue on applications that are ported from 32 bit to 64 bit. In 32 bit applications, pointers are of course 32 bits wide. So it is no big deal to do something like this:

int* foo = new int[45];
int bad = (int)foo;

However in 64 bit applications, that is bad. Since that pointer could be using the high bits of it’s memory address, and when cast or truncated to a 32 bit int, it will lose information. The high bits of a pointer are the bits that pointer to regions higher than 0xFFFFFFFF. For instance a 32 bit pointer has 8 digits (in hexadecimal notation) and can look like this:


But a 64 bit pointer has 16 digits (in hex) and can look like this:


The bits above that are 1 are the high bits. Since a 64 bit app can address more than 2^32 bits of memory, these bits can get used if the 64 bit app requires lots of memory. There is a way to force pointers to be assigned to the higher regions of memory without the expense of allocating all that memory. That way is to use VirtualAlloc in the Windows API, and reserve (but not commit) the lower 4 Gigs of memory. 

The results of a random debugging session involving the code below, shows that the high bits are completely lost when stored in the variable bad.

// a pointer, which is 64 bits wide on a 64 bit platform
int* foo = new int[45];
// this will truncate the memory pointer
int bad = (int)foo; // PURE Evil! Integers are 32 bits wide

Notice how the integer loses the leading value from the pointer (circled) as shown in the debugger watch window below: image

If the coder really gets adventurous they can try converting that integer back to a pointer again, with especially disastrous results:

// a pointer, which is 64 bits wide on a 64 bit platform
int* foo = new int[45];
// this will truncate the memory pointer
int bad = (int)foo; // PURE Evil! Integers are 32 bits wide

// an accident waiting to happen
int* worse = (int*)bad; // catastrophic!
// worse now looks like this: 0xffffffffa0040140
// now the high bits of 'worse' are filled with 0xff, which automatically
// makes it point to invalid regions of memory
// Any accessing of this pointer will result in an access violation.

which shows this in the debug watch window:


Notice how the high bits of the pointer worse are now pointing to regions of memory that the windows kernal has flagged as off limits. Any attempt to do anything with this pointer will result in an access violation, and if not correctly handled will result in the termination of the application.

One good way to find these is to call VirtualAlloc with very big values when your application begins. (I’m simplifying greatly here, trust me). This will reserve large chunks of memory below 4 Gigs. So that by the time your application makes actual memory allocations, all or most of the pointers will be above the 4 Gigs boundary mark. Then you have to test your application quite thoroughly to flush out these hot spots.

Well you might ask, should not the compiler aid you in this situation? Not the MS VC++ 10.0 compiler. Even at warning level 4, nothing is emitted to warn about the truncation.

Also Code Analysis will not help as it not enabled on 64 bit builds. i.e.

1>—— Build started: Project: VirtualMemTest, Configuration: Debug x64 ——
1>cl : Command line warning D9040: ignoring option ‘/analyze’; Code Analysis warnings are not available in this edition of the compiler

And on 32 bit builds of the same code code analysis still emits no warning.

3ds max API Linker Dependencies

Part of making a plug-in for 3ds max is to specify the correct libraries in your  c++ project settings. 3ds max ships with 38 libraries. It is not easy to know which one of these 38 to link against if you include a header file.

Therefore I have put together a spreadsheet containing the linker dependencies for all the entities in the 3ds max SDK.

The link to the excel spread sheet below contains all the entities in the SDK, and which library file they link against.

Note that there are entities that don’t link against anything. This is for the following reasons:


1. If the class contains all inline methods.
2. If the class contains methods that are  all abstract, and it has no member variables. (i.e. a pure interface)
3. If the class contains an implementation in 3dsmax.exe itself, and thus needs no library to link against.

The 3dsmax.exe file itself is quite large, and contains a lot of implementations for entities declared in the max SDK.

Again, I hope this helps, and if you have any questions feel free to comment, and or email me a question.

How to Write Native C++ Debugger Visualizers in Visual Studio for Complicated Types


This explains how to change how the visual studio native debugger displays data for very complicated types. It explains how to change this:


to this:


In other words, it greatly simplifies displaying native code in the debugger window.

This is not a tutorial on how to modify or customize autoexp.dat in general. This is a very specific tutorial on how to custom the debugger variable display for types that are very complicated and involve deep aggregation or inheritance.


Last year, I was trying to debug a particular problem and found it difficult to view a particular data type in the visual studio debugger. This particular data type was way over engineered and  complicated just for the sake of being complicated. It was a class that simply held a string and did path operations on the string. But it held the string in a complicated morass of class hierarchies that made displaying the actual string very difficult in the visual studio native debugger. So I embarked on a task to display the data I wanted in the debugger windows. This is possible by modifying the file:

“C:Program Files (x86)Microsoft Visual Studio 9.0Common7PackagesDebuggerautoexp.dat”

I have created a demo class hierarchy that is needlessly complex and demonstrates the problem that you have to drill down deep in the VS debugger to see the data. I also included code to run it:

  1. // NativeVisualizerTest.cpp : Defines the entry point for the console application.
  2. //
  3. #include “stdafx.h”
  4. #include <string>
  5. #include <memory> // for autoptr
  6. // The inner string, taken from std::string
  7. typedef std::basic_string<TCHAR> StupidString;
  8. // Another string that makes debugging that much harder
  9. class SpecialString    : public StupidString
  10. {
  11. public:
  12. SpecialString()     {}
  13. SpecialString(TCHAR* s)    : StupidString(s)    {}
  14. ~SpecialString()    {}
  15. };
  16. // Another class that wraps the SpecialString
  17. class PathPrivateInternal
  18. {
  19. public:
  20. PathPrivateInternal() {}
  21. PathPrivateInternal(TCHAR* s)
  22. : mString(s) {}
  23. ~PathPrivateInternal(){}
  24. private:
  25. SpecialString mString;
  26. };
  27. // The final class that holds a PathPrivateInternal
  28. // Visualizing this crazy class in the debugger will NOT be fun!
  29. class Path
  30. {
  31. public:
  32. Path()
  33. : mPtr(new PathPrivateInternal())
  34. { }
  35. Path(TCHAR* s)
  36. : mPtr(new PathPrivateInternal(s))
  37. { }
  38. ~Path()    { }
  39. private:
  40. std::auto_ptr<PathPrivateInternal> mPtr;
  41. };
  42. int _tmain(int argc, _TCHAR* argv[])
  43. {
  44. StupidString        str(_T(“I am a foo bar”));
  45. SpecialString      strB(_T(“I am a foo bar”));
  46. PathPrivateInternal ppi(_T(“I am a foo bar”));
  47. Path                  p(_T(“I am a foo bar”));
  48. return 0;
  49. }

So in this example, I have three concrete classes, a smart pointer (i.e. std::autoptr), and a typedef that have to be deciphered in order to display the string I really want to see. This is really a yucky mess: all for a class to simply hold a string and perform path like operations on it.


If I put a breakpoint in line 54, and add p to the watch window, I will see this in the debugger:


Notice how many sub tree’s I had to open in the debugger in order to see the string? !! Yuck !!

Unraveling the Gordian knot

The trick to displaying this in the debugger is to write a visualizer for each different class in this hierarchy. First you start from the inner class, and work towards the outer class. In this example the inner class is std::basic_string<TCHAR> and the outer string is Path.

The following list shows the order we need to write visualizers:




In the case of StupidString we are in luck. autoexp.dat already contains a visualizer for std::basic_string that displays the data nicely.


But the other three types do not display helpful information in the preview window.


Most of the work to visualize all the data types takes place here in SpecialString. The class SpecialString isn’t a mere typedef, so there is a little more work to get it to display in the debugger. The trick is to treat it like a std::basic_string. So find the section in autoexp.dat that displays std::basic_string and plagiarize that code like crazy. Except don’t copy the children section.

Here is the ANSI string version of std::basic_string

  1. ;——————————————————————————
  2. ;  std::string/basic_string
  3. ;——————————————————————————
  4. std::basic_string<char,*>{
  5. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  6. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  7. children
  8. (
  9. #if(($e._Myres) < ($e._BUF_SIZE))
  10. (
  11. #([actual members]: [$e,!] , #array( expr: $e._Bx._Buf[$i], size: $e._Mysize))
  12. )
  13. #else
  14. (
  15. #([actual members]: [$e,!],  #array( expr: $e._Bx._Ptr[$i], size: $e._Mysize))
  16. )
  17. )
  18. }

In this code, all that is needed is the “preview” code in line 5. Now there is a nice gotcha here. In order to properly display the type for SpecialString this program has to be compiled for non Unicode. That is because the visualizer for std::basic_string plagiarized from above is for <char> types. If I want to compile for Unicode, I need to plagiarize the following code (some stuff omitted for clarities sake):

  1. std::basic_string<unsigned short,*>|std::basic_string<wchar_t,*>{
  2. preview
  3. (
  4. #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,su] )
  5. #else ( [$e._Bx._Ptr,su] )
  6. )
  7. stringview
  8. (
  9. #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sub] )
  10. #else ( [$e._Bx._Ptr,sub] )
  11. )
  12. }

The stringview part allows you to display your data Text Visualizer dialog. Copy that to the SpecialString visualizer too:

  1. SpecialString{
  2. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  3. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  4. }


Now the debugger properly displays the string in the watch windows…


Notice now that the last two classes PathPrivateInternal and Path both now show some useful information in their preview windows. This is good, but not the final solution.


Now to visualize class PathPrivateInternal, will take just a little work.

  1. PathPrivateInternal{
  2. preview (
  3. [$e.mString]
  4. )
  5. }

Notice I only had to specify the member variable mString. There was no need to specify a type to render the data as, since the debugger already knows that mString is of type SpecialString. And the debugger already knows how render that. Here are the results:



The last part is to specify the visualizer for class Path.

  1. Path{
  2. preview (
  3. [$e.mPtr]
  4. )
  5. }

Which displays this:


However notice that it rendered the text as auto_ptr “I am a foo bar”. This is because of the visualizer for auto_ptr, which is also included in autoexp.dat:

  1. ;——————————————————————————
  2. ;  std::auto_ptr
  3. ;——————————————————————————
  4. std::auto_ptr<*>{
  5. preview
  6. (
  7. #(
  8. “auto_ptr “,
  9. (*(($T1 *)$e._Myptr))
  10. )
  11. )
  12. children
  13. (
  14. #(
  15. ptr: (*(($T1 *)$e._Myptr))
  16. )
  17. )
  18. }

Notice the hard coded string in the preview section. Also notice that the template type in the auto_ptr declarations is a star (*) meaning use this visualizer for all type’s that auto_ptr is using. There are a few things that need to be done to fix this. First, simply specialize the visualizer for when auto_ptr is holding a type of PathPrivateInternal.

  1. std::auto_ptr<PathPrivateInternal>{
  2. preview
  3. (
  4. [$e._Myptr]
  5. )
  6. }

Now, in the autoexp.dat file there are three sections, listed in order:

  1. [AutoExpand]
  2. [Visualizer]
  3. [hresult]

If the custom visualizer for std::auto_ptr<PathPrivateInternal> is placed after std::auto_ptr<*> then this custom visualizer will get ignored. This is because the debugger uses the first visualizer that it finds in the autoexp.dat file that satisfies the type criteria. And the star (*) template type unfortunately accepts all types including the std::auto_ptr<PathPrivateInternal>. Therefore put all custom visualizers at the beginning of the Visualizer section, NOT the end. After doing that, you will get the results shown below:


Here is another view with the Path type expanded:



Here is the final listing of all the visualizers:

  1. ;——————————————————————————
  2. ;  My Custom Types
  3. ;——————————————————————————
  4. SpecialString{
  5. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  6. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  7. }
  8. PathPrivateInternal{
  9. preview (
  10. [$e.mString]
  11. )
  12. }
  13. std::auto_ptr<PathPrivateInternal>{
  14. preview
  15. (
  16. [$e._Myptr]
  17. )
  18. }
  19. Path{
  20. preview (
  21. [$e.mPtr]
  22. )
  23. }

And remember these main points:

  • Start with inner nested types and work your way outwards
  • Copy from STL types when needed.
  • Put all custom visualizers at the beginning of the [Visualizer] section in autoexp.dat
  • Use stringview to display text in the Text Visualizer if needed.
  • Use specialized template types to display special cases if needed.

How to Instrument a Unit Test DLL for performance profiling in VS 2008

This tutorial will show how to instrument a unit test DLL for performance profiling. Visual Studio will allow you to do performance profiling on individual tests (i.e. functions) in a test suite. This can be done in the user interface (i.e. UI or IDE). However if you have a unit test DLL that contains lots of unit tests or test methods, it’s apparently not possible to do performance profiling in the IDE.

Here was my original question:

 Question on MSDN forums

I have a MS Unit Test DLL written in C# that targets a C++/CLI managed assembly. I have roughly 60 unit tests in that unit test dll. What I would like to do is run all the unit tests under the performance profiler, in the IDE. That is using the performance explorer to run a performance session for all the unit tests in my test DLL. I know how to do this from the command line, but I feel that is a last resort. I’d like to do this in the IDE if possible. It is possible to create a performance session for one unit test in the unit test DLL. The steps are listed here: ms-help://MS.VSCC.v90/MS.MSDNQTR.v90.en/dv_vsetlt01/html/4cc514d2-47bd-4f1c-a039-5ffae7c68bf1.htm i.e. Right click over a test result, and select "Create Performance Session". Whereupon the Performance Explorer will show a new session based off of one unit test. That’s great, but I’ve got 60 unit tests in my DLL. I don’t want to have to create 60 performance sessions.


Build the code that you will be testing. In my case it was the following DLL’s (Assuming a common root folder)

DLL name


..ManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dl The DLL with the code I wanted to test
..UnitTestbinx86DebugUnitTest.dll This DLL contains my unit tests

Then just to play it safe, run all the unit tests (This can be done in the IDE) in UnitTest.dll. It pays to be paranoid like this since if you want to detect bugs early and swiftly.

Now the strategy for profiling an entire DLL is to:

  • Instrument the DLL. This means to inject performance related commands into the code. Don’t worry it’s perfectly safe.
    • Turn on performance profiling via a global command.
      • Run your unit tests. Here is where the real work gets done.
    • Turn off performance profiling via a global command. (A results file gets saved somewhere).
  • Un-Instrument the DLL. This means restoring it to it’s original state. (i.e. Don’t ship a DLL with performance profiling enabled).

Instrument the Binaries

I instrument my DLL’s using the following batch script called: Instrument_ON.bat

  1. @echo off
  3. @echo Instrumenting Binary
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools\"
  6. %VS90TEAMTOOLS%\VSInstr.exe ..\ManagedAPI\Win32\Debug\UnderstandWrapper.ManagedAPI.dll
  7. %VS90TEAMTOOLS%\VSInstr.exe ..\UnitTest\bin\x86\Debug\UnitTest.dll
  8. pause
  9. @echo on

The results of instrumenting it looks like this:

Instrumenting Binary
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll –> F:CodePlexUnderstandAPIManagedManagedAPIWin32De
Original file backed up to F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header flags have been updated to reflect this.
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll –> F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll
Original file backed up to F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.
Press any key to continue . . .

Turn on Monitor

I then turn on performance profiling using the following batch script called: Performance_ON.bat

  1. @echo off
  3. @echo Turning ON performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VsPerfCmd /start:trace /output:ManagedAPI.vsp
  6. pause
  7. @echo on

The results of this batch scripts looks like this:

Turning ON performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Press any key to continue . . .

The results of this script is it starts another process that will monitor any application that happens to be instrumented with performance profiling, or code coverage profiling. In our case it started a process called VsPerfMon.exe:


Run Unit Tests

I then run all the unit tests in my DLL using another batch script. Now I have two build configurations (debug and release) so I have two batch scripts to run those: Run_Tests_Debug.bat and Run_Tests_Release.bat. Here is one of them:

  1. @echo off
  2. color 16
  3. @echo Running Unit Tests
  4. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\VsPerfCLREnv" /traceon
  5. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat";
  7. mstest /testcontainer:%CD%\UnitTest\bin\x86\Debug\UnitTest.dll > results.txt
  9. pause
  10. @echo on

Notice there the actual call to mstest.exe is what runs the unit tests. Notice that I piped the results of running the actual unit tests into a text file (i.e. results.txt). I did this because it gives me a more permanent record of how my unit tests did.

The mstest.exe file resides on my computer at:

"C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEMSTest.exe"

The output, to the console window, of this script on my looks like the following:

Running Unit Tests
Enabling VSPerf Trace Profiling of managed applications (excluding allocation profiling).

Current Profiling Environment variables are:
Setting environment for using Microsoft Visual Studio 2008 x86 tools.
Press any key to continue . . .

Noticed I redirected the standard output (standard out) to a text file. The results.txt file looks like this:

Microsoft (R) Test Execution Command Line Tool Version 9.0.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll…
Starting execution…

Results               Top Level Tests
——-               —————
Passed                UnitTest.DatabaseTest.DatabaseTest_GetAllEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_GetFileEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntity
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntitybyReference
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntityUnique
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupFile
Passed                UnitTest.DatabaseTest.DatabaseTest_Misc
Passed                UnitTest.DatabaseTest.DatabaseTest_OpenClose
Passed                UnitTest.Derived.ClassTypeTest.TestClassNests
Passed                UnitTest.Derived.ClassTypeTest.TestClassType
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeDerivedFrom
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeEnums
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeFields
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeInheritance
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeTemplates
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeUnions
Passed                UnitTest.Derived.ClassTypeTest.TestGetDefinedMethods
Passed                UnitTest.Derived.ClassTypeTest.TestGetMethods
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Enumerators
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Type
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_base_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_classes_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntityClassTypes
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Constructor
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_LexerStuff
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_A
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_C
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_D
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_E
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_F
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_MemberBlank
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Overrides
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Pure
Passed                UnitTest.EntityTest.EntityTest_Basic
Passed                UnitTest.EntityTest.EntityTest_Comments
Passed                UnitTest.EntityTest.EntityTest_Comments_Enums
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_Comments_Typedefs
Passed                UnitTest.EntityTest.EntityTest_FileReference
Passed                UnitTest.EntityTest.EntityTest_LineNumbers
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_UnresolvedMethods
Passed                UnitTest.EntityTest.EntityTest_Refs
Passed                UnitTest.EntityTest.EntityTest_Types
Passed                UnitTest.KindTest.KindTest_Basic
Passed                UnitTest.LexerTest.Lexer_Test
Passed                UnitTest.LexerTest.Lexer_Test_GetLexeme
Passed                UnitTest.LexerTest.Lexer_Test_Next
Passed                UnitTest.LexerTest.Lexer_Test_Previous
Passed                UnitTest.LexerTest.Lexer_Test_Ref
Passed                UnitTest.ReferenceTest.ReferencesTest_Basic
Passed                UnitTest.ReferenceTest.ReferencesTest_Copying
58/58 test(s) Passed

Test Run Completed.
  Passed  58
  Total   58
Results file:      F:CodePlexUnderstandAPIManagedTestResultsChris Johnson_CHRISJOHNSON-PC 2010-02-17 23_38_54.trx
Run Configuration: Default Run Configuration

Feels good to see all the unit tests pass. But now I have to save out the results of the performance profiling. And to do this, I now have to turn off the VsPerfMon.exe process that is running, and was running when I ran my unit tests.

Turn off Monitor

I have a batch script that turns off monitoring and saves out a results file containing all the profiling goodness. Performance_OFF.bat

  1. @echo off
  3. @echo Turning off performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VSPerfCmd /shutdown
  7. pause
  8. @echo on

The results of this batch script looks like this:

Turning off performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Shutting down the Profile Monitor
Press any key to continue . . .

Now a new file is saved out to my computer:


Notice the name: ManagedAPI.vsp. This is the same name I passed in when I turned on the Performance Monitor. Remember this?

from: Performance_ON.bat

VsPerfCmd /start:trace /output:ManagedAPI.vsp

Now at this point, you can do two things:

  1. Examine your performance results. This is done by double clicking the ManagedAPI.vsp file. This will just launch visual studio, and display the results of your tests:
  2. Or Uninstrument your DLL’s. It’s never a good idea to ship DLL’s with instrumentation in them. 🙂

Un-Instrument the Binaries

I have a script that does this for me: Instrument_OFF.bat

  1. @echo off
  3. @echo Removing Instrumented Files
  5. pushd ..\ManagedAPI\Win32\Debug
  6. if exist UnderstandWrapper.ManagedAPI.instr.pdb (
  7.     del UnderstandWrapper.ManagedAPI.instr.pdb
  8.     del UnderstandWrapper.ManagedAPI.dll
  9.     rename UnderstandWrapper.ManagedAPI.dll.orig UnderstandWrapper.ManagedAPI.dll
  10. )
  11. popd
  13. pushd ..\UnitTest\bin\x86\Debug
  14. if exist UnitTest.instr.pdb (
  15.     del UnitTest.instr.pdb
  16.     del UnitTest.dll
  17.     rename UnitTest.dll.orig UnitTest.dll
  18. )
  19. popd
  21. pause
  22. @echo on

When I run this, I get the following results in the command window:

Removing Instrumented Files
Press any key to continue . . .

However, sometimes I do this only after I reviewed my performance results. Since the performance results display in visual studio may be dependent on the instrumented binaries to properly display its data.

View Results

Now that you have your .vsp file it is time to open it. The file ManagedAPI.vsp shows the following in visual studio:


From there you can analyze the results and find your performance bottlenecks.

_CrtCheckMemory() ignores small heap corruptions

The common runtime libraries have some useful debugging tools contained in crtdbg.h. In that header file is a useful method called _CrtCheckMemory(). This function is used to validate the debug heap. According to MSDN:

The _CrtCheckMemory function validates memory allocated by the debug heap manager by verifying the underlying base heap and inspecting every memory block. If an error or memory inconsistency is encountered in the underlying base heap, the debug header information, or the overwrite buffers, _CrtCheckMemory generates a debug report with information describing the error condition. When _DEBUG is not defined, calls to _CrtCheckMemory are removed during preprocessing.

So it seems like a good idea to put these in your application to check for memory corruptions.

But I found a problem, or a limitation in using this. This method seems to ignore small heap corruptions.

For example, If I wanted to overwrite an array by 5 elements, I would do the following:

void corruptMemory()
    const int ARRAY_SIZE = 500;
    int* iparray = new int[ARRAY_SIZE]; // a memory leak
    int offset = 5;
    int badIndex = ARRAY_SIZE + offset;
    iparray[badIndex] = 0xdeadbeaf; // heap corruption

While experimenting with this, I found the _CrtCheckMemory() method will not catch this over-run:

void bar()
    _ASSERTE(_CrtCheckMemory()); // this should trigger the assert

While running the above code, I found that the _CrtCheckMemory() remains blissfully ignorant of the memory corruption, returns TRUE to the assert macro, and nothing happens!

Only when I set the variable offset to a value of 6 or higher will _CrtCheckMemory will find and detect something. Using the above code, this will manifest itself (in a debug build) in two assert message boxes:

The first assert:


And the assert arising from the failure condition returned from _CrtCheckMemory to _ASSERTE:


Of course when I set the offset to a big enough value, say 89, I can corrupt something really important and screw up the CRT, and also crash the application on exit.

I have a project that demonstrates his. The project also demonstrates how to dump memory leaks to the output window at the end of program execution.

I’ll try to put the link here: