How to Write Native C++ Debugger Visualizers in Visual Studio for Complicated Types

Introduction

This explains how to change how the visual studio native debugger displays data for very complicated types. It explains how to change this:

debugger_before

to this:

after

In other words, it greatly simplifies displaying native code in the debugger window.

This is not a tutorial on how to modify or customize autoexp.dat in general. This is a very specific tutorial on how to custom the debugger variable display for types that are very complicated and involve deep aggregation or inheritance.

Problem

Last year, I was trying to debug a particular problem and found it difficult to view a particular data type in the visual studio debugger. This particular data type was way over engineered and  complicated just for the sake of being complicated. It was a class that simply held a string and did path operations on the string. But it held the string in a complicated morass of class hierarchies that made displaying the actual string very difficult in the visual studio native debugger. So I embarked on a task to display the data I wanted in the debugger windows. This is possible by modifying the file:

“C:Program Files (x86)Microsoft Visual Studio 9.0Common7PackagesDebuggerautoexp.dat”

I have created a demo class hierarchy that is needlessly complex and demonstrates the problem that you have to drill down deep in the VS debugger to see the data. I also included code to run it:

  1. // NativeVisualizerTest.cpp : Defines the entry point for the console application.
  2. //
  3. #include “stdafx.h”
  4. #include <string>
  5. #include <memory> // for autoptr
  6. // The inner string, taken from std::string
  7. typedef std::basic_string<TCHAR> StupidString;
  8. // Another string that makes debugging that much harder
  9. class SpecialString    : public StupidString
  10. {
  11. public:
  12. SpecialString()     {}
  13. SpecialString(TCHAR* s)    : StupidString(s)    {}
  14. ~SpecialString()    {}
  15. };
  16. // Another class that wraps the SpecialString
  17. class PathPrivateInternal
  18. {
  19. public:
  20. PathPrivateInternal() {}
  21. PathPrivateInternal(TCHAR* s)
  22. : mString(s) {}
  23. ~PathPrivateInternal(){}
  24. private:
  25. SpecialString mString;
  26. };
  27. // The final class that holds a PathPrivateInternal
  28. // Visualizing this crazy class in the debugger will NOT be fun!
  29. class Path
  30. {
  31. public:
  32. Path()
  33. : mPtr(new PathPrivateInternal())
  34. { }
  35. Path(TCHAR* s)
  36. : mPtr(new PathPrivateInternal(s))
  37. { }
  38. ~Path()    { }
  39. private:
  40. std::auto_ptr<PathPrivateInternal> mPtr;
  41. };
  42. int _tmain(int argc, _TCHAR* argv[])
  43. {
  44. StupidString        str(_T(“I am a foo bar”));
  45. SpecialString      strB(_T(“I am a foo bar”));
  46. PathPrivateInternal ppi(_T(“I am a foo bar”));
  47. Path                  p(_T(“I am a foo bar”));
  48. return 0;
  49. }

So in this example, I have three concrete classes, a smart pointer (i.e. std::autoptr), and a typedef that have to be deciphered in order to display the string I really want to see. This is really a yucky mess: all for a class to simply hold a string and perform path like operations on it.

Demonstration

If I put a breakpoint in line 54, and add p to the watch window, I will see this in the debugger:

debugger_before

Notice how many sub tree’s I had to open in the debugger in order to see the string? !! Yuck !!

Unraveling the Gordian knot

The trick to displaying this in the debugger is to write a visualizer for each different class in this hierarchy. First you start from the inner class, and work towards the outer class. In this example the inner class is std::basic_string<TCHAR> and the outer string is Path.

The following list shows the order we need to write visualizers:

Type

StupidString
SpecialString
PathPrivateInternal
std::autoptr
Path

StupidString

In the case of StupidString we are in luck. autoexp.dat already contains a visualizer for std::basic_string that displays the data nicely.

image

But the other three types do not display helpful information in the preview window.

SpecialString

Most of the work to visualize all the data types takes place here in SpecialString. The class SpecialString isn’t a mere typedef, so there is a little more work to get it to display in the debugger. The trick is to treat it like a std::basic_string. So find the section in autoexp.dat that displays std::basic_string and plagiarize that code like crazy. Except don’t copy the children section.

Here is the ANSI string version of std::basic_string

  1. ;——————————————————————————
  2. ;  std::string/basic_string
  3. ;——————————————————————————
  4. std::basic_string<char,*>{
  5. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  6. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  7. children
  8. (
  9. #if(($e._Myres) < ($e._BUF_SIZE))
  10. (
  11. #([actual members]: [$e,!] , #array( expr: $e._Bx._Buf[$i], size: $e._Mysize))
  12. )
  13. #else
  14. (
  15. #([actual members]: [$e,!],  #array( expr: $e._Bx._Ptr[$i], size: $e._Mysize))
  16. )
  17. )
  18. }

In this code, all that is needed is the “preview” code in line 5. Now there is a nice gotcha here. In order to properly display the type for SpecialString this program has to be compiled for non Unicode. That is because the visualizer for std::basic_string plagiarized from above is for <char> types. If I want to compile for Unicode, I need to plagiarize the following code (some stuff omitted for clarities sake):

  1. std::basic_string<unsigned short,*>|std::basic_string<wchar_t,*>{
  2. preview
  3. (
  4. #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,su] )
  5. #else ( [$e._Bx._Ptr,su] )
  6. )
  7. stringview
  8. (
  9. #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sub] )
  10. #else ( [$e._Bx._Ptr,sub] )
  11. )
  12. }

The stringview part allows you to display your data Text Visualizer dialog. Copy that to the SpecialString visualizer too:

  1. SpecialString{
  2. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  3. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  4. }

image

Now the debugger properly displays the string in the watch windows…

image

Notice now that the last two classes PathPrivateInternal and Path both now show some useful information in their preview windows. This is good, but not the final solution.

PathPrivateInternal

Now to visualize class PathPrivateInternal, will take just a little work.

  1. PathPrivateInternal{
  2. preview (
  3. [$e.mString]
  4. )
  5. }

Notice I only had to specify the member variable mString. There was no need to specify a type to render the data as, since the debugger already knows that mString is of type SpecialString. And the debugger already knows how render that. Here are the results:

debugger_middle

Path

The last part is to specify the visualizer for class Path.

  1. Path{
  2. preview (
  3. [$e.mPtr]
  4. )
  5. }

Which displays this:

image

However notice that it rendered the text as auto_ptr “I am a foo bar”. This is because of the visualizer for auto_ptr, which is also included in autoexp.dat:

  1. ;——————————————————————————
  2. ;  std::auto_ptr
  3. ;——————————————————————————
  4. std::auto_ptr<*>{
  5. preview
  6. (
  7. #(
  8. “auto_ptr “,
  9. (*(($T1 *)$e._Myptr))
  10. )
  11. )
  12. children
  13. (
  14. #(
  15. ptr: (*(($T1 *)$e._Myptr))
  16. )
  17. )
  18. }

Notice the hard coded string in the preview section. Also notice that the template type in the auto_ptr declarations is a star (*) meaning use this visualizer for all type’s that auto_ptr is using. There are a few things that need to be done to fix this. First, simply specialize the visualizer for when auto_ptr is holding a type of PathPrivateInternal.

  1. std::auto_ptr<PathPrivateInternal>{
  2. preview
  3. (
  4. [$e._Myptr]
  5. )
  6. }

Now, in the autoexp.dat file there are three sections, listed in order:

  1. [AutoExpand]
  2. [Visualizer]
  3. [hresult]

If the custom visualizer for std::auto_ptr<PathPrivateInternal> is placed after std::auto_ptr<*> then this custom visualizer will get ignored. This is because the debugger uses the first visualizer that it finds in the autoexp.dat file that satisfies the type criteria. And the star (*) template type unfortunately accepts all types including the std::auto_ptr<PathPrivateInternal>. Therefore put all custom visualizers at the beginning of the Visualizer section, NOT the end. After doing that, you will get the results shown below:

image

Here is another view with the Path type expanded:

image

Summary

Here is the final listing of all the visualizers:

  1. ;——————————————————————————
  2. ;  My Custom Types
  3. ;——————————————————————————
  4. SpecialString{
  5. preview        ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,s]) #else ( [$e._Bx._Ptr,s]))
  6. stringview    ( #if(($e._Myres) < ($e._BUF_SIZE)) ( [$e._Bx._Buf,sb]) #else ( [$e._Bx._Ptr,sb]))
  7. }
  8. PathPrivateInternal{
  9. preview (
  10. [$e.mString]
  11. )
  12. }
  13. std::auto_ptr<PathPrivateInternal>{
  14. preview
  15. (
  16. [$e._Myptr]
  17. )
  18. }
  19. Path{
  20. preview (
  21. [$e.mPtr]
  22. )
  23. }

And remember these main points:

  • Start with inner nested types and work your way outwards
  • Copy from STL types when needed.
  • Put all custom visualizers at the beginning of the [Visualizer] section in autoexp.dat
  • Use stringview to display text in the Text Visualizer if needed.
  • Use specialized template types to display special cases if needed.

How to Instrument a Unit Test DLL for performance profiling in VS 2008

This tutorial will show how to instrument a unit test DLL for performance profiling. Visual Studio will allow you to do performance profiling on individual tests (i.e. functions) in a test suite. This can be done in the user interface (i.e. UI or IDE). However if you have a unit test DLL that contains lots of unit tests or test methods, it’s apparently not possible to do performance profiling in the IDE.

Here was my original question:

 Question on MSDN forums

I have a MS Unit Test DLL written in C# that targets a C++/CLI managed assembly. I have roughly 60 unit tests in that unit test dll. What I would like to do is run all the unit tests under the performance profiler, in the IDE. That is using the performance explorer to run a performance session for all the unit tests in my test DLL. I know how to do this from the command line, but I feel that is a last resort. I’d like to do this in the IDE if possible. It is possible to create a performance session for one unit test in the unit test DLL. The steps are listed here: ms-help://MS.VSCC.v90/MS.MSDNQTR.v90.en/dv_vsetlt01/html/4cc514d2-47bd-4f1c-a039-5ffae7c68bf1.htm i.e. Right click over a test result, and select "Create Performance Session". Whereupon the Performance Explorer will show a new session based off of one unit test. That’s great, but I’ve got 60 unit tests in my DLL. I don’t want to have to create 60 performance sessions.

Preparation

Build the code that you will be testing. In my case it was the following DLL’s (Assuming a common root folder)

DLL name

Description

..ManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dl The DLL with the code I wanted to test
..UnitTestbinx86DebugUnitTest.dll This DLL contains my unit tests

Then just to play it safe, run all the unit tests (This can be done in the IDE) in UnitTest.dll. It pays to be paranoid like this since if you want to detect bugs early and swiftly.

Now the strategy for profiling an entire DLL is to:

  • Instrument the DLL. This means to inject performance related commands into the code. Don’t worry it’s perfectly safe.
    • Turn on performance profiling via a global command.
      • Run your unit tests. Here is where the real work gets done.
    • Turn off performance profiling via a global command. (A results file gets saved somewhere).
  • Un-Instrument the DLL. This means restoring it to it’s original state. (i.e. Don’t ship a DLL with performance profiling enabled).

Instrument the Binaries

I instrument my DLL’s using the following batch script called: Instrument_ON.bat

  1. @echo off
  2.  
  3. @echo Instrumenting Binary
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools\"
  5.  
  6. %VS90TEAMTOOLS%\VSInstr.exe ..\ManagedAPI\Win32\Debug\UnderstandWrapper.ManagedAPI.dll
  7. %VS90TEAMTOOLS%\VSInstr.exe ..\UnitTest\bin\x86\Debug\UnitTest.dll
  8. pause
  9. @echo on

The results of instrumenting it looks like this:

Instrumenting Binary
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll –> F:CodePlexUnderstandAPIManagedManagedAPIWin32De
bugUnderstandWrapper.ManagedAPI.dll
Original file backed up to F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header flags have been updated to reflect this.
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll –> F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll
Original file backed up to F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.
Press any key to continue . . .

Turn on Monitor

I then turn on performance profiling using the following batch script called: Performance_ON.bat

  1. @echo off
  2.  
  3. @echo Turning ON performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VsPerfCmd /start:trace /output:ManagedAPI.vsp
  6. pause
  7. @echo on

The results of this batch scripts looks like this:

Turning ON performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Press any key to continue . . .

The results of this script is it starts another process that will monitor any application that happens to be instrumented with performance profiling, or code coverage profiling. In our case it started a process called VsPerfMon.exe:

image

Run Unit Tests

I then run all the unit tests in my DLL using another batch script. Now I have two build configurations (debug and release) so I have two batch scripts to run those: Run_Tests_Debug.bat and Run_Tests_Release.bat. Here is one of them:

  1. @echo off
  2. color 16
  3. @echo Running Unit Tests
  4. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\VsPerfCLREnv" /traceon
  5. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat";
  6.  
  7. mstest /testcontainer:%CD%\UnitTest\bin\x86\Debug\UnitTest.dll > results.txt
  8.  
  9. pause
  10. @echo on

Notice there the actual call to mstest.exe is what runs the unit tests. Notice that I piped the results of running the actual unit tests into a text file (i.e. results.txt). I did this because it gives me a more permanent record of how my unit tests did.

The mstest.exe file resides on my computer at:

"C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEMSTest.exe"

The output, to the console window, of this script on my looks like the following:

Running Unit Tests
Enabling VSPerf Trace Profiling of managed applications (excluding allocation profiling).

Current Profiling Environment variables are:
COR_ENABLE_PROFILING=1
COR_PROFILER={6468ec6c-94bd-40d3-bd93-4414565dafbf}
COR_LINE_PROFILING=0
COR_GC_PROFILING=0
Setting environment for using Microsoft Visual Studio 2008 x86 tools.
Press any key to continue . . .

Noticed I redirected the standard output (standard out) to a text file. The results.txt file looks like this:

Microsoft (R) Test Execution Command Line Tool Version 9.0.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll…
Starting execution…

Results               Top Level Tests
——-               —————
Passed                UnitTest.DatabaseTest.DatabaseTest_GetAllEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_GetFileEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntity
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntitybyReference
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntityUnique
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupFile
Passed                UnitTest.DatabaseTest.DatabaseTest_Misc
Passed                UnitTest.DatabaseTest.DatabaseTest_OpenClose
Passed                UnitTest.Derived.ClassTypeTest.TestClassNests
Passed                UnitTest.Derived.ClassTypeTest.TestClassType
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeDerivedFrom
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeEnums
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeFields
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeInheritance
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeTemplates
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeUnions
Passed                UnitTest.Derived.ClassTypeTest.TestGetDefinedMethods
Passed                UnitTest.Derived.ClassTypeTest.TestGetMethods
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Enumerators
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Type
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_base_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_classes_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntityClassTypes
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Constructor
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_LexerStuff
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_A
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_C
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_D
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_E
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_F
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_MemberBlank
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Overrides
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Pure
Passed                UnitTest.EntityTest.EntityTest_Basic
Passed                UnitTest.EntityTest.EntityTest_Comments
Passed                UnitTest.EntityTest.EntityTest_Comments_Enums
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_Comments_Typedefs
Passed                UnitTest.EntityTest.EntityTest_FileReference
Passed                UnitTest.EntityTest.EntityTest_LineNumbers
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_UnresolvedMethods
Passed                UnitTest.EntityTest.EntityTest_Refs
Passed                UnitTest.EntityTest.EntityTest_Types
Passed                UnitTest.KindTest.KindTest_Basic
Passed                UnitTest.LexerTest.Lexer_Test
Passed                UnitTest.LexerTest.Lexer_Test_GetLexeme
Passed                UnitTest.LexerTest.Lexer_Test_Next
Passed                UnitTest.LexerTest.Lexer_Test_Previous
Passed                UnitTest.LexerTest.Lexer_Test_Ref
Passed                UnitTest.ReferenceTest.ReferencesTest_Basic
Passed                UnitTest.ReferenceTest.ReferencesTest_Copying
58/58 test(s) Passed

Summary
——-
Test Run Completed.
  Passed  58
  ———-
  Total   58
Results file:      F:CodePlexUnderstandAPIManagedTestResultsChris Johnson_CHRISJOHNSON-PC 2010-02-17 23_38_54.trx
Run Configuration: Default Run Configuration

Feels good to see all the unit tests pass. But now I have to save out the results of the performance profiling. And to do this, I now have to turn off the VsPerfMon.exe process that is running, and was running when I ran my unit tests.

Turn off Monitor

I have a batch script that turns off monitoring and saves out a results file containing all the profiling goodness. Performance_OFF.bat

  1. @echo off
  2.  
  3. @echo Turning off performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VSPerfCmd /shutdown
  6.  
  7. pause
  8. @echo on

The results of this batch script looks like this:

Turning off performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Shutting down the Profile Monitor
————————————————————
Press any key to continue . . .

Now a new file is saved out to my computer:

F:CodePlexUnderstandAPIManagedPerformanceManagedAPI.vsp

Notice the name: ManagedAPI.vsp. This is the same name I passed in when I turned on the Performance Monitor. Remember this?

from: Performance_ON.bat

VsPerfCmd /start:trace /output:ManagedAPI.vsp

Now at this point, you can do two things:

  1. Examine your performance results. This is done by double clicking the ManagedAPI.vsp file. This will just launch visual studio, and display the results of your tests:
  2. Or Uninstrument your DLL’s. It’s never a good idea to ship DLL’s with instrumentation in them. 🙂

Un-Instrument the Binaries

I have a script that does this for me: Instrument_OFF.bat

  1. @echo off
  2.  
  3. @echo Removing Instrumented Files
  4.  
  5. pushd ..\ManagedAPI\Win32\Debug
  6. if exist UnderstandWrapper.ManagedAPI.instr.pdb (
  7.     del UnderstandWrapper.ManagedAPI.instr.pdb
  8.     del UnderstandWrapper.ManagedAPI.dll
  9.     rename UnderstandWrapper.ManagedAPI.dll.orig UnderstandWrapper.ManagedAPI.dll
  10. )
  11. popd
  12.  
  13. pushd ..\UnitTest\bin\x86\Debug
  14. if exist UnitTest.instr.pdb (
  15.     del UnitTest.instr.pdb
  16.     del UnitTest.dll
  17.     rename UnitTest.dll.orig UnitTest.dll
  18. )
  19. popd
  20.  
  21. pause
  22. @echo on

When I run this, I get the following results in the command window:

Removing Instrumented Files
Press any key to continue . . .

However, sometimes I do this only after I reviewed my performance results. Since the performance results display in visual studio may be dependent on the instrumented binaries to properly display its data.

View Results

Now that you have your .vsp file it is time to open it. The file ManagedAPI.vsp shows the following in visual studio:

image

From there you can analyze the results and find your performance bottlenecks.