Over engineered build systems from Hell

While I was at Autodesk years ago we went through various build systems. When I first started the build was written in Perl. Dependencies were specified using a .csv file. I had no idea how it worked, nor did I care how it worked since I was interested in other things during those years. The build could routinely take an hour and 45 minutes, and was mind-numbingly boring since we usually had to do it everyday. And if you were unlucky multiple times per day. Even worse In fact on the build servers the build would routinely take almost 4 hours!

What a horrible tax.

Later on, a hot-shot programmer came along and rewrote the build system to use ruby and rake. It was supposed to be faster, which it kind of was. But the complexity was just as bad, and no one knew ruby nor how rake worked. Then the developer left the company, leaving behind a black box. So that silo of information was gone, and no one knew how the build system worked. It took 2 developers about a year and a half or so to learn the build system well. To really be able to work on it at the level the original developer had done.

To be sure there were problems with the build. It still took a long time to build the product. Somewhere near an hour. About the only thing we did gain was the ability to do incremental builds.

But modifying the build was the main problem. The build, written in ruby, completely reinvented the wheel on so many different areas. To understand this better, you have to understand that the product at the time was built using Microsoft tools because it was based solely on the Microsoft platform. Thus the source project files were in a build language that Microsoft created. That build language was built into visual studio and was called MSBuild. So instead of using Microsoft tools to create the build, ruby and rake were used instead. So instead of using Microsoft tools to parse the xml project files, a ruby gem was used. So instead of using anything from Microsoft to help with the build, everything was re-invented from scratch. Parsing visual studio .vcproj (and eventually .vcxproj) files was done tediously and laboriously and mind numbingly using rake and some xml gem. Talk about recreating the wheel! So much code written to duplicate a simple function call to a Microsoft function could have retrieved a fully instantiated object with full properties all intact.

Copying files to the build directory was another disaster too. It would take around 10 to 12 minutes to copy 7000~14000 files. It originally was somewhere near 7000 files, but grew over time. All written in ruby code that no one knew how to debug except to put print statements in.

Another problem was adding build properties. If you wanted to add a build property (a key value pair), you had to add it to multiple places in the ruby build, knowing exactly what to modify (in duplicate) and such. It was horrible.

Mixing ruby with MSBuild was like mixing iron and clay. They don’t mix well at all. It was alike a ruby straight jacket that hindered the build and visual studio upon which it was based.

There had to be a better way.

Eventually when the frustrations with the build boiled over, I learned MSBuild and figured out how to build max without ruby. It took over a year, from when I first got a working prototype, to get something into the main branch of max. Simply due to bureaucratic inertia. There are lots of people in positions of power who simply say no before learning about a subject. Which was something all too common there. The first liberation was freeing the list of things to get built from ruby. Yes the list of DLL’s and EXE’s to get built was specified in some arcane ruby syntax somewhere. The first thing was moving that list to a democratic XML file. Now any build language could parse it and find out what to build. The second thing was moving the list of files to copy to an XML file. Now any build system could know what files to copy as well.

Once those two things were in place, it was time to put in the build system that I originally came up with during a particular Christmas break.

It was written in pure XML, with one MSBuild extension that was written using C#. All the tools were native to visual studio, and what you built on the command line was what was built in visual studio. They both used the same properties (using native property sheets) and built in the same way.

What’s more I found that using native MSBuild tools to copy those 7000+ files to build now was incredibly fast. In fact, I while once debugging through that old ruby code responsible for copying, I found the source of the 10 minute copy task. Or why it took 10 minutes. It was using an N factorial algorithm! So given directory A with B thru Z subdirectories, it would iterate through all directories n! times. Each directory was not parsed once, but N! times according to the amount of sub-directories that existed. It was an unholy mess that proves that re-inventing the wheel is usually a disaster waiting to happen. Now back to the improvement: With the new msbuild copy mechanism it took 20 seconds to copy all those files. 20 seconds versus 10 minutes was a big improvement.

Incremental builds also greatly improved. Go ahead and build your product from scratch. Now don’t change a thing and rebuild. If you have a smart build system, it should just take a few seconds and nothing will happen. The build system will be smart enough to essentially report that nothing changed and therefore it did no work. My new build did just that in a matter of seconds. The old build system used to take about 5 minutes to do that (And it still did work anyways…).

Speaking of performance. The time to compile and link the actual files didn’t change much, because that was always in visual studio’s corner and not ruby. The improvement in performance came from the copying actions that now took 20 seconds. Also noticeable was the shorter time involved from when the build started to when the first CPP file was getting compiled. In ruby/rake it took quite a few minutes. In the new build it took a few seconds. Eventually when I got a new SSD hard-drive, I was able to get the build down to 22 minutes on my machine.

Better yet was the removal of duplication of anything. Everything was native and iron was mixed with iron and clay was mixed with clay. Sort of….

One developer, (who we called doctor No, because he said no to everything good), holding on to the fantasy that max would be multi-platform someday would not let ruby go. So there were in essence two build systems that could do the same thing. In fact he wanted an option to invoke one build system from another! So I had to put in an option to invoke msbuild from ruby/rake! This was hobbling msbuild with an old clunker. Kind of like buying a new car and towing the old one around everywhere you go. Yes extremely stupid and frustrating.

Which goes to show that old ways of thinking die hard, or don’t die at all.

The build at Century Software

Later on I moved to Century Software, a local company to where I live. That was such a fun place. Anyways their build system for their windows product was written in Make! Yes Make the original, ancient build system that you can’t even find documentation for anymore. I mean literally, I found (I think) one page of documentation somewhere on some professors lecture notes. The docs were  horrible. Make implemented here was so old it built one C file at a time. No multi-threading, no parallel builds nothing. Slow was the operative word here. That and a incomprehensible built output that was so verbose it was almost impossible to comprehend. The only good thing about it was that it immediately stopped on the first error.

So eventually I rebuilt that using MSBuild too. It took me a few months in my spare time. No bureaucratic inertia, no one telling me no. I just worked on it in my spare time and eventually I had a complete and fully functioning system for the tinyterm product. This build was the best I’ve ever done, with zero duplication, extremely small project files and a build that was very fast. It went from 45 minutes to a minute and a half.

When writing a product, the build system should be done using the tools that the platform provides. There should be no transmogrifying the code, or the build scripts (like openssl) before doing the build. When writing ruby on rails use rake for your build process’s. When targeting Microsoft platforms use msbuild. Using java, then use maven. Data that must be shared between build platforms should be in xml so that anything can parse them. And most important of all, distrust must go, and developers and managers must have an open mind to new things. Otherwise the development process’s will take so long, and be so costly, and exert such a tax that new features will suffer, bugs will not get fixed and customers will not be served and they will take their money elsewhere.

Advertisements

How to Instrument a Unit Test DLL for performance profiling in VS 2008

This tutorial will show how to instrument a unit test DLL for performance profiling. Visual Studio will allow you to do performance profiling on individual tests (i.e. functions) in a test suite. This can be done in the user interface (i.e. UI or IDE). However if you have a unit test DLL that contains lots of unit tests or test methods, it’s apparently not possible to do performance profiling in the IDE.

Here was my original question:

 Question on MSDN forums

I have a MS Unit Test DLL written in C# that targets a C++/CLI managed assembly. I have roughly 60 unit tests in that unit test dll. What I would like to do is run all the unit tests under the performance profiler, in the IDE. That is using the performance explorer to run a performance session for all the unit tests in my test DLL. I know how to do this from the command line, but I feel that is a last resort. I’d like to do this in the IDE if possible. It is possible to create a performance session for one unit test in the unit test DLL. The steps are listed here: ms-help://MS.VSCC.v90/MS.MSDNQTR.v90.en/dv_vsetlt01/html/4cc514d2-47bd-4f1c-a039-5ffae7c68bf1.htm i.e. Right click over a test result, and select "Create Performance Session". Whereupon the Performance Explorer will show a new session based off of one unit test. That’s great, but I’ve got 60 unit tests in my DLL. I don’t want to have to create 60 performance sessions.

Preparation

Build the code that you will be testing. In my case it was the following DLL’s (Assuming a common root folder)

DLL name

Description

..ManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dl The DLL with the code I wanted to test
..UnitTestbinx86DebugUnitTest.dll This DLL contains my unit tests

Then just to play it safe, run all the unit tests (This can be done in the IDE) in UnitTest.dll. It pays to be paranoid like this since if you want to detect bugs early and swiftly.

Now the strategy for profiling an entire DLL is to:

  • Instrument the DLL. This means to inject performance related commands into the code. Don’t worry it’s perfectly safe.
    • Turn on performance profiling via a global command.
      • Run your unit tests. Here is where the real work gets done.
    • Turn off performance profiling via a global command. (A results file gets saved somewhere).
  • Un-Instrument the DLL. This means restoring it to it’s original state. (i.e. Don’t ship a DLL with performance profiling enabled).

Instrument the Binaries

I instrument my DLL’s using the following batch script called: Instrument_ON.bat

  1. @echo off
  2.  
  3. @echo Instrumenting Binary
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools\"
  5.  
  6. %VS90TEAMTOOLS%\VSInstr.exe ..\ManagedAPI\Win32\Debug\UnderstandWrapper.ManagedAPI.dll
  7. %VS90TEAMTOOLS%\VSInstr.exe ..\UnitTest\bin\x86\Debug\UnitTest.dll
  8. pause
  9. @echo on

The results of instrumenting it looks like this:

Instrumenting Binary
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll –> F:CodePlexUnderstandAPIManagedManagedAPIWin32De
bugUnderstandWrapper.ManagedAPI.dll
Original file backed up to F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedManagedAPIWin32DebugUnderstandWrapper.ManagedAPI.dll.
Warning VSP2013 : Instrumenting this image requires it to run as a 32-bit process.  The CLR header flags have been updated to reflect this.
Microsoft (R) VSInstr Post-Link Instrumentation 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

File to Process:
   F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll –> F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll
Original file backed up to F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.orig

Successfully instrumented file F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll.
Press any key to continue . . .

Turn on Monitor

I then turn on performance profiling using the following batch script called: Performance_ON.bat

  1. @echo off
  2.  
  3. @echo Turning ON performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VsPerfCmd /start:trace /output:ManagedAPI.vsp
  6. pause
  7. @echo on

The results of this batch scripts looks like this:

Turning ON performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Press any key to continue . . .

The results of this script is it starts another process that will monitor any application that happens to be instrumented with performance profiling, or code coverage profiling. In our case it started a process called VsPerfMon.exe:

image

Run Unit Tests

I then run all the unit tests in my DLL using another batch script. Now I have two build configurations (debug and release) so I have two batch scripts to run those: Run_Tests_Debug.bat and Run_Tests_Release.bat. Here is one of them:

  1. @echo off
  2. color 16
  3. @echo Running Unit Tests
  4. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\Team Tools\Performance Tools\VsPerfCLREnv" /traceon
  5. call "C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\vcvarsall.bat";
  6.  
  7. mstest /testcontainer:%CD%\UnitTest\bin\x86\Debug\UnitTest.dll > results.txt
  8.  
  9. pause
  10. @echo on

Notice there the actual call to mstest.exe is what runs the unit tests. Notice that I piped the results of running the actual unit tests into a text file (i.e. results.txt). I did this because it gives me a more permanent record of how my unit tests did.

The mstest.exe file resides on my computer at:

"C:Program Files (x86)Microsoft Visual Studio 9.0Common7IDEMSTest.exe"

The output, to the console window, of this script on my looks like the following:

Running Unit Tests
Enabling VSPerf Trace Profiling of managed applications (excluding allocation profiling).

Current Profiling Environment variables are:
COR_ENABLE_PROFILING=1
COR_PROFILER={6468ec6c-94bd-40d3-bd93-4414565dafbf}
COR_LINE_PROFILING=0
COR_GC_PROFILING=0
Setting environment for using Microsoft Visual Studio 2008 x86 tools.
Press any key to continue . . .

Noticed I redirected the standard output (standard out) to a text file. The results.txt file looks like this:

Microsoft (R) Test Execution Command Line Tool Version 9.0.30729.1
Copyright (c) Microsoft Corporation. All rights reserved.
Loading F:CodePlexUnderstandAPIManagedUnitTestbinx86DebugUnitTest.dll…
Starting execution…

Results               Top Level Tests
——-               —————
Passed                UnitTest.DatabaseTest.DatabaseTest_GetAllEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_GetFileEntities
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntity
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntitybyReference
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupEntityUnique
Passed                UnitTest.DatabaseTest.DatabaseTest_LookupFile
Passed                UnitTest.DatabaseTest.DatabaseTest_Misc
Passed                UnitTest.DatabaseTest.DatabaseTest_OpenClose
Passed                UnitTest.Derived.ClassTypeTest.TestClassNests
Passed                UnitTest.Derived.ClassTypeTest.TestClassType
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeDerivedFrom
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeEnums
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeFields
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeInheritance
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeTemplates
Passed                UnitTest.Derived.ClassTypeTest.TestClassTypeUnions
Passed                UnitTest.Derived.ClassTypeTest.TestGetDefinedMethods
Passed                UnitTest.Derived.ClassTypeTest.TestGetMethods
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Enumerators
Passed                UnitTest.Derived.EnumTypeTest.EnumTest_Type
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_base_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntity_classes_h
Passed                UnitTest.Derived.FileEntityTest.TestFileEntityClassTypes
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Constructor
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Inline_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_LexerStuff
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_A
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_B
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_C
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_D
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_E
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Member_F
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_MemberBlank
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Overrides
Passed                UnitTest.Derived.MethodTypeTest.MethodTest_Pure
Passed                UnitTest.EntityTest.EntityTest_Basic
Passed                UnitTest.EntityTest.EntityTest_Comments
Passed                UnitTest.EntityTest.EntityTest_Comments_Enums
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_Comments_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_Comments_Typedefs
Passed                UnitTest.EntityTest.EntityTest_FileReference
Passed                UnitTest.EntityTest.EntityTest_LineNumbers
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDeclarations
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_FunctionDefinitions
Passed                UnitTest.EntityTest.EntityTest_LineNumbers_UnresolvedMethods
Passed                UnitTest.EntityTest.EntityTest_Refs
Passed                UnitTest.EntityTest.EntityTest_Types
Passed                UnitTest.KindTest.KindTest_Basic
Passed                UnitTest.LexerTest.Lexer_Test
Passed                UnitTest.LexerTest.Lexer_Test_GetLexeme
Passed                UnitTest.LexerTest.Lexer_Test_Next
Passed                UnitTest.LexerTest.Lexer_Test_Previous
Passed                UnitTest.LexerTest.Lexer_Test_Ref
Passed                UnitTest.ReferenceTest.ReferencesTest_Basic
Passed                UnitTest.ReferenceTest.ReferencesTest_Copying
58/58 test(s) Passed

Summary
——-
Test Run Completed.
  Passed  58
  ———-
  Total   58
Results file:      F:CodePlexUnderstandAPIManagedTestResultsChris Johnson_CHRISJOHNSON-PC 2010-02-17 23_38_54.trx
Run Configuration: Default Run Configuration

Feels good to see all the unit tests pass. But now I have to save out the results of the performance profiling. And to do this, I now have to turn off the VsPerfMon.exe process that is running, and was running when I ran my unit tests.

Turn off Monitor

I have a batch script that turns off monitoring and saves out a results file containing all the profiling goodness. Performance_OFF.bat

  1. @echo off
  2.  
  3. @echo Turning off performance coverage session recorder
  4. set VS90TEAMTOOLS="%VS90COMNTOOLS%..\..\Team Tools\Performance Tools"
  5. %VS90TEAMTOOLS%\VSPerfCmd /shutdown
  6.  
  7. pause
  8. @echo on

The results of this batch script looks like this:

Turning off performance coverage session recorder
Microsoft (R) VSPerf Command Version 9.0.30729 x86
Copyright (C) Microsoft Corp. All rights reserved.

Shutting down the Profile Monitor
————————————————————
Press any key to continue . . .

Now a new file is saved out to my computer:

F:CodePlexUnderstandAPIManagedPerformanceManagedAPI.vsp

Notice the name: ManagedAPI.vsp. This is the same name I passed in when I turned on the Performance Monitor. Remember this?

from: Performance_ON.bat

VsPerfCmd /start:trace /output:ManagedAPI.vsp

Now at this point, you can do two things:

  1. Examine your performance results. This is done by double clicking the ManagedAPI.vsp file. This will just launch visual studio, and display the results of your tests:
  2. Or Uninstrument your DLL’s. It’s never a good idea to ship DLL’s with instrumentation in them. 🙂

Un-Instrument the Binaries

I have a script that does this for me: Instrument_OFF.bat

  1. @echo off
  2.  
  3. @echo Removing Instrumented Files
  4.  
  5. pushd ..\ManagedAPI\Win32\Debug
  6. if exist UnderstandWrapper.ManagedAPI.instr.pdb (
  7.     del UnderstandWrapper.ManagedAPI.instr.pdb
  8.     del UnderstandWrapper.ManagedAPI.dll
  9.     rename UnderstandWrapper.ManagedAPI.dll.orig UnderstandWrapper.ManagedAPI.dll
  10. )
  11. popd
  12.  
  13. pushd ..\UnitTest\bin\x86\Debug
  14. if exist UnitTest.instr.pdb (
  15.     del UnitTest.instr.pdb
  16.     del UnitTest.dll
  17.     rename UnitTest.dll.orig UnitTest.dll
  18. )
  19. popd
  20.  
  21. pause
  22. @echo on

When I run this, I get the following results in the command window:

Removing Instrumented Files
Press any key to continue . . .

However, sometimes I do this only after I reviewed my performance results. Since the performance results display in visual studio may be dependent on the instrumented binaries to properly display its data.

View Results

Now that you have your .vsp file it is time to open it. The file ManagedAPI.vsp shows the following in visual studio:

image

From there you can analyze the results and find your performance bottlenecks.