Check out all open files from Perforce

So I found myself editing a LOT of files the other day. My task was to fix up some new file names – That is replacing the old file names (i.e. #include foo.h to #include foonew.h) with the new ones. The problem was the source code contains over 10,000 files. My searches were finding upwards of 90 files per file substitution, and all were under version control. In other words, they were locked for editing. Visual Studio thankfully, will edit the files in memory, but you (me!) as a user needed to check out the files so I could save the edits. And I need to make those files writable by checking them out from our version control system.

So how did I make the task of opening the files for editing easier? (i.e. The task of checking files out from my version control system: Perforce)

First attempt
I have defined in visual studio a call to an external tool (perforce) that I can use to check out a file from visual studio. In visual studio I created a new external tool, pointed it to the Perforce command line client (p4.exe), and gave it a argument of:

edit $(ItemPath)

where $(ItemPath) is the full filename of the currently selected open document, or the path to the vcproj file (Depending on what is selected). So in the end you get a command line call that looks like this:

c:program filesPerforcep4 edit C:foo.h

and viola, the file becomes writable, and you can proceed with your work. But… there is a problem with this simple approach. It doesn’t always work!

Why?

In short, because the windows operating system is case insensitive to file names, and Perforce (If it is hosted on a Linux server like ours is) is case sensitive. So if a file name with incorrect casing is sent to Perforce via the ‘edit’ command, Perforce will refuse to check out the file. The only alternative is to then manually find the offending file in Perforce and check it out manually. Not a fun task when navigating through thousands and thousands of files. Not fun at all!

What is happening is that the filename that visual studio gives you through the $(ItemPath) symbol is constructed from the xml content in the project file (i.e. .vcproj), and not from the real file name itself. The file paths in the vcproj file can be entirely in the wrong case and still work, because the windows OS is case insensitive.

So my fix was to write a Ruby script which would fix up the file name to it’s real casing, and then pass that to perforce to check out the file. Well actually it was two Ruby scripts.
The first script is an extension to the pathname class, and simply is used to determine the real casing given a string for a file name. This uses a brute force approach, but is the only way of that I know how to get the real file name.
The second script calls the first script, and is the one you call from the ‘external tools’ option in visual studio.

#------------------------
#pathname_extensions.rb
#------------------------
require 'pathname'

class Pathname
  def self.get_real_case path
    fullPath = Pathname.new(path)
    parts = []

    # Split up the path into parts
    fullPath.descend do |item|
      parts.push item
    end

    # Split up the path into parts
    components = []
    fullPath.each_filename do |item|
      components.push item
    end

    corrected_filename = parts[0].to_s
    index = 0
    parts.each do |path|
      #if its a directory, then list the subdirectories in it.
      if path.directory? then
        #p path
        subs = path.entries
        looking_for = components[index]
        #puts 'Looking for: ' + looking_for
        subs.each do |sub|
          if (sub.to_s.downcase == looking_for.downcase) then
            #puts 'found: ' + sub
            corrected_filename = File.join(corrected_filename, sub.to_s)
            break
          end
        end
        #puts ''
      end
      index += 1
    end
    corrected_filename.gsub("/","")
  end
end

Which is used by this ruby script:

#------------------------
#perforce_checkout.rb
#------------------------
$:.unshift File.dirname(__FILE__)
require 'pathname_extensions'

puts '-------------------------------------------------'
puts 'Perforce Checkout'
puts '-------------------------------------------------'

# Get the file name passed in to the script
full_file_name = ARGV[0]
puts 'Attempt Checkout: ' + full_file_name

corrected_filename = Pathname.get_real_case(full_file_name)
puts 'Real file name  : ' + corrected_filename

if File.exists?(corrected_filename) then
  perforce_command = 'p4 edit ' + corrected_filename
  puts ''
  # Try to check out from perforce
  system perforce_command
else
  puts 'File not found: ' + corrected_filename
end

Therefore I just create my external tool in visual studio by calling the Ruby.exe app, and pass in to that two parameters:

1. The name of the checkout script: perforce_checkout.rb
2. The name of the file from visual studio: $(ItemPath)

Like this:

Tool for checking out a file from Perforce
Tool for checking out a file from Perforce

Working on Multiple Files

So this script works great on giving me the real case for a file name. But what about if I want to check out 50 files all at once? Do I have to execute this tool 50 times in visual studio? Heaven forbid!

So I finally dived in and learned how to write a macro in visual studio that will automatically call my Ruby Script for me.

This macro iterates through all the open documents in visual studio, and if they are read only, it will attempt to call my ruby script. It looks like this:

Imports System
Imports EnvDTE
Imports EnvDTE80
Imports EnvDTE90
Imports System.Diagnostics
Imports System.IO

Public Module SourceControl
    Sub CheckoutAllOpenDocuments()
        'Set some global variables
        Dim RubyPath = "E:AppsRuby1_86binruby.exe"
        Dim RubyScript = "E:QEperforce_checkout.rb"
        ChDir(Path.GetDirectoryName(RubyScript))
        ' Prepare the status bar
        Dim status As EnvDTE.StatusBar = DTE.StatusBar
        Dim index As Integer = 0
        Dim count As Integer = DTE.Documents.Count
        ' Iterate over all open documents in the IDE
        For Each document As EnvDTE.Document In DTE.Documents
            Debug.Print("Document: {0}", document.FullName)
            Dim info As FileInfo = New FileInfo(document.FullName)
            ' Only attempt to check out the file, if it is read only
            If info.IsReadOnly Then
                ' Construct the command string to pass to the ruby script
                Dim commandString = RubyPath + " " + RubyScript + " " + document.FullName
                ' Update the status bar with the file name, so we have some indication of
                ' what is happening
                status.Progress(True, document.FullName, index, count)
                index += 1
                ' Call the perforce checkout script.
                ' Pass in -1 for the timeout parameter, otherwise it doesn't work on multiple files
                Shell("cmd /c """ + commandString + """", AppWinStyle.Hide, True, -1)
                ' This works too
                'Shell(commandString, AppWinStyle.Hide, True, -1)
            End If
        Next document
        status.Progress(False)
    End Sub
End Module
Advertisements

Understand 4 C++ Introduction

Not just another source code analyzer

During the last 3 years, I have been using a program called Understand 4 c++. It is made by a company out of St. George Utah called Scientific Toolworks. This product is their only piece of software as far as I know, but it is one of my favorites.

This product is a source code analyzer. It parses native code (version 2.0 does managed code too), and builds a database that contains all sorts of meta-data about your code. It doesn’t do lint style scrutinization out of the box, but it gives you all sorts of metrics about your code. For instance, this software can take a class, and tell you:

  1. Each place it was instantiated. This includes the file and line where it was used.
  2. How many methods the class has.
  3. The types of each method.
  4. If any member methods override a base class method.
  5. How many member variables the class has.
  6. A listing of base classes.

This same type of scrutinization can be extended to all types of ‘entities’ in the c/c++ language. For instance you can look at all sorts of meta-data associated with a function. For instance, given a function, you can determine:

  1. The number and type of it’s local variables.
  2. The type of return value of the function.
  3. How many local variables are declared in the body of the function.
  4. Which functions the function calls.
  5. Who calls this function.
  6. Etc..

API – Reflection for native code

Some of the features that are exposed in the UI of the product are limited compared to what the API exposes. The API is the closest thing there is to reflection for native code that I know of. Reflection for managed code is a really handy tool, that I’m sure C/C++ people have been wanting for a long time. But it only works for source code that is already compiled and that is currently executing. This tool allows you to statically ‘reflect’ over an arbitrary source code base. Powerful? Absolutely. I consider this tool indispensable when working with, and maintaining a huge source code base made of mostly native code. (I need to mention (again) that this tool and API also analyzes C# code too).

All this talk about the API, and I haven’t yet stated what languages the API is in. The API consists of one header file written in C. You need to combine this is with a help file that offers a list of entity types and reference types. If you don’t have the help file, the key to the API’s power will be forever locked to you. Anyways, they also offer a Perl API too. I wrote a lot of tools in the past targeting the perl API, but those tools are difficult to maintain due to perl’s complexities, and limitations.

The C API appears deceptively simple, and indeed it is. However the documentation is very basic and doesn’t explain a few important concepts. It looks presently as if it hasn’t been updated in years.

I therefore needed a better way to tap into the power of the API. Perl was out of the question: I was trying as fast as possible to forget any perl I ever knew. C was pretty low level, but not easy to work with. I wanted to write in C#!! But the makers of Understand 4 c++ didn’t offer a managed C# API. So what did I do?

I wrote a managed API myself.

A Managed API

During the previous year, I had learned to use C++ to write managed code for Microsoft .NET. Microsoft calls this language C++/CLI. So I took the Understand C API, and wrapped it in a C++ managed API that targets the Microsoft .NET framework.

After a few months of late night programming(And a long… Christmas vacation) , lots of frustrations, and lots of emails to product support, I finally got the managed API settled down to the point that it was usable. I wrote the whole thing in visual studio team developer edition, using unit tests. Doing so, I was able to achieve around 90% code coverage.

The end result was, that I can use C# to write my understand tools in. Sweet! I used that API to write some fun tools with. Eventually discovered a deep, obscure bug in the understand API. I reported it to them, and they promptly fixed it, and posted an update. (Chalk up one point for agile development practices, and a penalty for all other software makers who sit on their critical bug fixes for months on end). The people at Scientific Toolworks even did a light code review of the API. However, in the end, the entire project was mine, with no little to no input from anyone, except product support.

I currently plan on posting the managed API to the internet for free, in the hopes that someone will use it, benefit from it, and perhaps help make it better. The people at Scientific Tool works have also expressed an interest to post the API on one of their websites as well.

Example Code

Using the managed API, I am able to write code that looks like this. Here this method searches an array of methods, looking for a match based off of the symbolic method name.

private MethodType[] CompareFunctionNames(MethodType method, MethodType[] baseClassMethods)
{
	List result = new List();
	if (baseClassMethods != null)
	{
		foreach (MethodType baseMethod in baseClassMethods)
		{
			if (baseMethod.NameShort == method.NameShort)
			{
				result.Add(baseMethod);
			}
		}
	}
	
	return result.ToArray();
}

This code below gets all methods from a class definition and iterates over each method searching for ones with no parameters.

private void ScrutinizeClassForFunctions(ClassType classEnt, FileEntity ownerFile)
{
	MethodType[] methods = classEnt.GetMethods();
	if (methods != null)
	{
		foreach (MethodType method in methods)
		{
			//Ignore Methods that have no parameters
			if ("()" != method.GetParameters(false))
			{
				// Ignore the constructor and destructor methods since that is just noise!
				String constructorName = classEnt.NameShort;
				String destructorName = "~" + classEnt.NameShort;
				if ((method.NameShort != constructorName) && (method.NameShort != destructorName))
				{
					// Check if the list of parent classes contains a function with a same name 
					// as the supplied function entity
					MethodSearchResult misMatches = FunctionMatch(method, classEnt, ownerFile);
					misMatches.ReportResults();
				}
			}
		}
	}
}

So this API allows you to quickly and easily write custom tools to scrutinize and analyze your source code.