Solving Link error 1112

Yesterday I brushed off some old code I had not touched in quite a while. It was a layer manager plugin for Autodesk 3dsmax that I had written years ago. I don’t think I had even opened any project files for this project in the last 12 months. So I re-acquainted with the code, made some changes to the files, and went to recompile it. Now this project compiles to both 32 and 64 bit targets. So I randomly chose a 64 bit target, and built the sucker.

Imagine my shock when I encountered a linker error stating:

module machine type ‘X86’ conflicts with target machine type ‘x64’

I knew enough about this in the past, and never had a problem before. Like I said before, I hadn’t touched the code in long time. What’s more, is my code was under source control!

I opened up the project settings and double and triple checked my Linker Target Machine settings. Indeed it was properly set at: MachineX64. Perfect!

Still the problem persisted. Then I got worried that the library files I was linking in were corrupted, and somehow 32 bit libraries were intermixed with 64 bit libraries. I didn’t want to download the entire 3dsMax SDK again, in the off chance this hypothesis was true. So I researched a solution to check what type of libraries I had!

So I used the utility DumpBin.exe that is found here:

"C:Program Files (x86)Microsoft Visual Studio 9.0VCbindumpbin.exe"

And by the way, in windows 7 I did a search a few times for ‘dumpbin’, which failed every time. Turns out you have to search for dumpbin.exe. Go figure.

Anyways, I used this tool to dump out the functions in my lib file. Then I was able to inspect the functions and see what platform there were compiled for.

For instance, notice the ‘Machine’ entry in a function I dumped out using dumpbin.exe:

Version      : 0
Machine      : 8664 (x64)
TimeDateStamp: 49B974BD Thu Mar 12 14:46:53 2009
SizeOfData   : 0000002D
DLL name     : maxutil.dll
Symbol name  : ??0Path@Util@MaxSDK@@QEAA@PEBD@Z (public: __cdecl MaxSDK::Util::Path::Path(char const *))
Type         : code
Name type    : name
Hint         : 18
Name         : ??0Path@Util@MaxSDK@@QEAA@PEBD@Z

Notice the second line contains x64, a dead give away this is a function compiled for an x64 target platform. So the problem was, my project linked in a lot of libraries from the 3dsmax sdk. The 3dsmax sdk contains about 49 library files. How was I supposed to inspect every function from every library file?

Sounds like a job for a script:

@echo off
call "C:Program Files (x86)Microsoft Visual Studio 10.0VCvcvarsall.bat"

for %%f in (*.lib) do dumpbin.exe -headers %%f | findstr /c:"Machine      :" >> _lib-exports.txt"

pause
@echo on

So I put the above script into a batch file, and placed the batch file in the directory that contained all my library files. I iterated through all the library files, dumping out the header information. This was piped to find string, which searched for the machine information. I then appended that to a log file which I could inspect at my leisure. The log file contained 13642 lines looking like this:

Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)
Machine      : 8664 (x64)

So that was it. I opened the file in my favorite text editor, and a search for x86 turned up nothing. It was x64 all the way down.

So now what was I supposed to do? I was stuck. Let me review all I had done.

1. My target machine linker setting was properly set.
2. My build configuration setting was properly calling ‘x64’ and not win32.
3. All the libraries I was importing were actually 64 bit libraries.

All appeared lost and hopeless until I ran across a tidbit on an MSDN forum.

Here the gentleman answered:

If you have left the ide at the default settings, then it will actually use the x86_amd64 compiler only. This is set under tools->options->projects and solutions->vc++ directories. Under amd64 executables the directory should be $(VCInstallDir)binx86_amd64 followed by $(VCInstallDir)bin.

This method works for all versions of windows so its a default setting. If you want to though you can change the x86_amd64 to just amd64.

So I looked at the executable settings for my x64 build configurations in VS 2008. It looked like this:

$(VCInstallDir)bin
$(WindowsSdkDir)bin
…

I changed it to this by adding the x86_amd64 directory:

$(VCInstallDir)binx86_amd64
$(VCInstallDir)bin
$(WindowsSdkDir)bin
…

And everything now works. My project now compiles and links just fine.

So I am left to surmise that some external add-on to visual studio hosed my executable tools settings for 64 bit configurations. I had off and on over the last year been using bullseye, which messes around with the settings in the project executable directories. So Perhaps that is what messed up that setting in visual studio. But now it’s fixed, and I am very happy.