This tutorial will show how to instrument a unit test DLL for performance profiling. Visual Studio will allow you to do performance profiling on individual tests (i.e. functions) in a test suite. This can be done in the user interface (i.e. UI or IDE). However if you have a unit test DLL that contains lots of unit tests or test methods, it’s apparently not possible to do performance profiling in the IDE.
Here was my original question:
I have a MS Unit Test DLL written in C# that targets a C++/CLI managed assembly. I have roughly 60 unit tests in that unit test dll. What I would like to do is run all the unit tests under the performance profiler, in the IDE. That is using the performance explorer to run a performance session for all the unit tests in my test DLL. I know how to do this from the command line, but I feel that is a last resort. I’d like to do this in the IDE if possible. It is possible to create a performance session for one unit test in the unit test DLL. The steps are listed here: ms-help://MS.VSCC.v90/MS.MSDNQTR.v90.en/dv_vsetlt01/html/4cc514d2-47bd-4f1c-a039-5ffae7c68bf1.htm i.e. Right click over a test result, and select "Create Performance Session". Whereupon the Performance Explorer will show a new session based off of one unit test. That’s great, but I’ve got 60 unit tests in my DLL. I don’t want to have to create 60 performance sessions.
Preparation
Build the code that you will be testing. In my case it was the following DLL’s (Assuming a common root folder)
|
DLL name |
Description |
| ..ManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dl | The DLL with the code I wanted to test |
| ..UnitTestbinx86DebugUnitTest.dll | This DLL contains my unit tests |
Then just to play it safe, run all the unit tests (This can be done in the IDE) in UnitTest.dll. It pays to be paranoid like this since if you want to detect bugs early and swiftly.
Now the strategy for profiling an entire DLL is to:
- Instrument the DLL. This means to inject performance related commands into the code. Don’t worry it’s perfectly safe.
- Turn on performance profiling via a global command.
- Run your unit tests. Here is where the real work gets done.
- Turn off performance profiling via a global command. (A results file gets saved somewhere).
- Turn on performance profiling via a global command.
- Un-Instrument the DLL. This means restoring it to it’s original state. (i.e. Don’t ship a DLL with performance profiling enabled).
Instrument the Binaries
I instrument my DLL’s using the following batch script called: Instrument_ON.bat
- @echo off
- @echo Instrumenting Binary
- set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools\"
- %VS90TEAMTOOLS%\VSInstr.exe ..\ManagedAPI\Win32\Debug\UnderstandWrapper.ManagedAPI.dll
- %VS90TEAMTOOLS%\VSInstr.exe ..\UnitTest\bin\x86\Debug\UnitTest.dll
- pause
- @echo on
The results of instrumenting it looks like this:
Instrumenting Binary
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.File to Process:
F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll –> F:CodePlexUnderstandAPIManagedManagedAPIWin32De
bugUnderstandWrapper.ManagedAPI.dll
Original file backed up to F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.origSuccessfully instrumented file F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process. The CLR header flags have been updated to reflect this.
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.File to Process:
F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll –> F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll
Original file backed up to F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.origSuccessfully instrumented file F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.
Press any key to continue . . .
Turn on Monitor
I then turn on performance profiling using the following batch script called: Performance_ON.bat
- @echo off
- @echo Turning ON performance coverage session recorder
- set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
- %VS90TEAMTOOLS%\VsPerfCmd /start:trace /output:ManagedAPI.vsp
- pause
- @echo on
The results of this batch scripts looks like this:
Turning ON performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.Press any key to continue . . .
The results of this script is it starts another process that will monitor any application that happens to be instrumented with performance profiling, or code coverage profiling. In our case it started a process called VsPerfMon.exe:
Run Unit Tests
I then run all the unit tests in my DLL using another batch script. Now I have two build configurations (debug and release) so I have two batch scripts to run those: Run_Tests_Debug.bat and Run_Tests_Release.bat. Here is one of them:
- @echo off
- color 16
- @echo Running Unit Tests
- call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\VsPerfCLREnv" /traceon
- call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat";
- mstest /testcontainer:%CD%\UnitTest\bin\x86\Debug\UnitTest.dll > results.txt
- pause
- @echo on
Notice there the actual call to mstest.exe is what runs the unit tests. Notice that I piped the results of running the actual unit tests into a text file (i.e. results.txt). I did this because it gives me a more permanent record of how my unit tests did.
The mstest.exe file resides on my computer at:
"C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEMSTest.exe"
The output, to the console window, of this script on my looks like the following:
Running Unit Tests
Enabling VSPerf Trace Profiling of managed applications (excluding allocation profiling).Current Profiling Environment variables are:
COR_ENABLE_PROFILING=1
COR_PROFILER={6468ec6c-94bd-40d3-bd93-4414565dafbf}
COR_LINE_PROFILING=0
COR_GC_PROFILING=0
Setting environment for using Microsoft Visual Studio 2008 x86 tools.
Press any key to continue . . .
Noticed I redirected the standard output (standard out) to a text file. The results.txt file looks like this:
Microsoft (R) Test Execution Command Line Tool Version 9.0.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll…
Starting execution…Results Top Level Tests
——- —————
Passed UnitTest.DatabaseTest.DatabaseTest_GetAllEntities
Passed UnitTest.DatabaseTest.DatabaseTest_GetFileEntities
Passed UnitTest.DatabaseTest.DatabaseTest_LookupEntity
Passed UnitTest.DatabaseTest.DatabaseTest_LookupEntitybyReference
Passed UnitTest.DatabaseTest.DatabaseTest_LookupEntityUnique
Passed UnitTest.DatabaseTest.DatabaseTest_LookupFile
Passed UnitTest.DatabaseTest.DatabaseTest_Misc
Passed UnitTest.DatabaseTest.DatabaseTest_OpenClose
Passed UnitTest.Derived.ClassTypeTest.TestClassNests
Passed UnitTest.Derived.ClassTypeTest.TestClassType
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeDerivedFrom
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeEnums
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeFields
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeInheritance
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeTemplates
Passed UnitTest.Derived.ClassTypeTest.TestClassTypeUnions
Passed UnitTest.Derived.ClassTypeTest.TestGetDefinedMethods
Passed UnitTest.Derived.ClassTypeTest.TestGetMethods
Passed UnitTest.Derived.EnumTypeTest.EnumTest_Enumerators
Passed UnitTest.Derived.EnumTypeTest.EnumTest_Type
Passed UnitTest.Derived.FileEntityTest.TestFileEntity
Passed UnitTest.Derived.FileEntityTest.TestFileEntity_base_h
Passed UnitTest.Derived.FileEntityTest.TestFileEntity_classes_h
Passed UnitTest.Derived.FileEntityTest.TestFileEntityClassTypes
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Constructor
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Inline
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Inline_B
Passed UnitTest.Derived.MethodTypeTest.MethodTest_LexerStuff
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_A
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_B
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_C
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_D
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_E
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Member_F
Passed UnitTest.Derived.MethodTypeTest.MethodTest_MemberBlank
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Overrides
Passed UnitTest.Derived.MethodTypeTest.MethodTest_Pure
Passed UnitTest.EntityTest.EntityTest_Basic
Passed UnitTest.EntityTest.EntityTest_Comments
Passed UnitTest.EntityTest.EntityTest_Comments_Enums
Passed UnitTest.EntityTest.EntityTest_Comments_FunctionDeclarations
Passed UnitTest.EntityTest.EntityTest_Comments_FunctionDefinitions
Passed UnitTest.EntityTest.EntityTest_Comments_Typedefs
Passed UnitTest.EntityTest.EntityTest_FileReference
Passed UnitTest.EntityTest.EntityTest_LineNumbers
Passed UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDeclarations
Passed UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDefinitions
Passed UnitTest.EntityTest.EntityTest_LineNumbers_UnresolvedMethods
Passed UnitTest.EntityTest.EntityTest_Refs
Passed UnitTest.EntityTest.EntityTest_Types
Passed UnitTest.KindTest.KindTest_Basic
Passed UnitTest.LexerTest.Lexer_Test
Passed UnitTest.LexerTest.Lexer_Test_GetLexeme
Passed UnitTest.LexerTest.Lexer_Test_Next
Passed UnitTest.LexerTest.Lexer_Test_Previous
Passed UnitTest.LexerTest.Lexer_Test_Ref
Passed UnitTest.ReferenceTest.ReferencesTest_Basic
Passed UnitTest.ReferenceTest.ReferencesTest_Copying
58/58 test(s) PassedSummary
——-
Test Run Completed.
Passed 58
———-
Total 58
Results file: F:CodePlexUnderstandAPIManagedTestResultsChris Johnson_CHRISJOHNSON-PC 2010-02-17 23_38_54.trx
Run Configuration: Default Run Configuration
Feels good to see all the unit tests pass. But now I have to save out the results of the performance profiling. And to do this, I now have to turn off the VsPerfMon.exe process that is running, and was running when I ran my unit tests.
Turn off Monitor
I have a batch script that turns off monitoring and saves out a results file containing all the profiling goodness. Performance_OFF.bat
- @echo off
- @echo Turning off performance coverage session recorder
- set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
- %VS90TEAMTOOLS%\VSPerfCmd /shutdown
- pause
- @echo on
The results of this batch script looks like this:
Turning off performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.Shutting down the Profile Monitor
————————————————————
Press any key to continue . . .
Now a new file is saved out to my computer:
F:CodePlexUnderstandAPIManagedPerformanceManagedAPI.vsp
Notice the name: ManagedAPI.vsp. This is the same name I passed in when I turned on the Performance Monitor. Remember this?
from: Performance_ON.bat
…
VsPerfCmd /start:trace /output:ManagedAPI.vsp
Now at this point, you can do two things:
- Examine your performance results. This is done by double clicking the ManagedAPI.vsp file. This will just launch visual studio, and display the results of your tests:
- Or Uninstrument your DLL’s. It’s never a good idea to ship DLL’s with instrumentation in them. 🙂
Un-Instrument the Binaries
I have a script that does this for me: Instrument_OFF.bat
- @echo off
- @echo Removing Instrumented Files
- pushd ..\ManagedAPI\Win32\Debug
- if exist UnderstandWrapper.ManagedAPI.instr.pdb (
- del UnderstandWrapper.ManagedAPI.instr.pdb
- del UnderstandWrapper.ManagedAPI.dll
- rename UnderstandWrapper.ManagedAPI.dll.orig UnderstandWrapper.ManagedAPI.dll
- )
- popd
- pushd ..\UnitTest\bin\x86\Debug
- if exist UnitTest.instr.pdb (
- del UnitTest.instr.pdb
- del UnitTest.dll
- rename UnitTest.dll.orig UnitTest.dll
- )
- popd
- pause
- @echo on
When I run this, I get the following results in the command window:
Removing Instrumented Files
Press any key to continue . . .
However, sometimes I do this only after I reviewed my performance results. Since the performance results display in visual studio may be dependent on the instrumented binaries to properly display its data.
View Results
Now that you have your .vsp file it is time to open it. The file ManagedAPI.vsp shows the following in visual studio:
From there you can analyze the results and find your performance bottlenecks.
Yet Another Programming Blog 