Pointer Truncation

Pointer truncation is an evil that all developers of native C++ code need to worry about. This is simply storing a wide piece of memory in a narrower bit of memory. This can become an issue on applications that are ported from 32 bit to 64 bit. In 32 bit applications, pointers are of course 32 bits wide. So it is no big deal to do something like this:

int* foo = new int[45];
int bad = (int)foo;

However in 64 bit applications, that is bad. Since that pointer could be using the high bits of it’s memory address, and when cast or truncated to a 32 bit int, it will lose information. The high bits of a pointer are the bits that pointer to regions higher than 0xFFFFFFFF. For instance a 32 bit pointer has 8 digits (in hexadecimal notation) and can look like this:

0x12345678

But a 64 bit pointer has 16 digits (in hex) and can look like this:

0x1111111122222222

The bits above that are 1 are the high bits. Since a 64 bit app can address more than 2^32 bits of memory, these bits can get used if the 64 bit app requires lots of memory. There is a way to force pointers to be assigned to the higher regions of memory without the expense of allocating all that memory. That way is to use VirtualAlloc in the Windows API, and reserve (but not commit) the lower 4 Gigs of memory. 

The results of a random debugging session involving the code below, shows that the high bits are completely lost when stored in the variable bad.

// a pointer, which is 64 bits wide on a 64 bit platform
int* foo = new int[45];
// this will truncate the memory pointer
int bad = (int)foo; // PURE Evil! Integers are 32 bits wide

Notice how the integer loses the leading value from the pointer (circled) as shown in the debugger watch window below: image

If the coder really gets adventurous they can try converting that integer back to a pointer again, with especially disastrous results:

// a pointer, which is 64 bits wide on a 64 bit platform
int* foo = new int[45];
// this will truncate the memory pointer
int bad = (int)foo; // PURE Evil! Integers are 32 bits wide

// an accident waiting to happen
int* worse = (int*)bad; // catastrophic!
// worse now looks like this: 0xffffffffa0040140
// now the high bits of 'worse' are filled with 0xff, which automatically
// makes it point to invalid regions of memory
// Any accessing of this pointer will result in an access violation.

which shows this in the debug watch window:

image

Notice how the high bits of the pointer worse are now pointing to regions of memory that the windows kernal has flagged as off limits. Any attempt to do anything with this pointer will result in an access violation, and if not correctly handled will result in the termination of the application.

One good way to find these is to call VirtualAlloc with very big values when your application begins. (I’m simplifying greatly here, trust me). This will reserve large chunks of memory below 4 Gigs. So that by the time your application makes actual memory allocations, all or most of the pointers will be above the 4 Gigs boundary mark. Then you have to test your application quite thoroughly to flush out these hot spots.

Well you might ask, should not the compiler aid you in this situation? Not the MS VC++ 10.0 compiler. Even at warning level 4, nothing is emitted to warn about the truncation.

Also Code Analysis will not help as it not enabled on 64 bit builds. i.e.

1>—— Build started: Project: VirtualMemTest, Configuration: Debug x64 ——
1>cl : Command line warning D9040: ignoring option ‘/analyze’; Code Analysis warnings are not available in this edition of the compiler

And on 32 bit builds of the same code code analysis still emits no warning.