Lockup in call DialogBoxIndirectParam at call to NtUserCallHwndParamLock

One day, while running some of our automated test suites, I suddenly got a lockup in while running the following script: fat_rendering_exposurecontrol.ms.

I had seen this before last year. But it was a problem that would come and go at whim. Even recompiling would make it go away.

The root problem was caused by a ‘less than’ operator failing. I’m not really interested in why this is failing, but it helps explain the rest of the problem.

Anyways, it all starts with a call to an STL min function like this:

std::min( a, b )

Eventually the less than operator devolves down to this grotesque mess:

c:Program Files (x86)Microsoft Visual Studio 9.0VCincludexutility [line 264]

template<class
_Ty1, class
_Ty2> inline

bool
__CLRCALL_OR_CDECL
_Debug_lt(const
_Ty1& _Left, const
_Ty2& _Right,

const
wchar_t *_Where, unsigned
int
_Line)

{    // test if _Left < _Right and operator< is strict weak ordering

if (!(_Left < _Right))

return (false);

else
if (_Right < _Left)

_DEBUG_ERROR2(“invalid operator<“, _Where, _Line); <– here

return (true);

}

_Right was 3814.9946

_Left was 6405.5967

_Left < _Right    is false
!(_Left < _Right)    is true

Therefore the code had absolutely no business hitting the else clause of the if statement and calling _DEBUG_ERROR2 . Weird!! Perhaps it’s a compiler error… But anyways…

The error eventually reports an assert error to the Debug CRT. Which calls our own debug CRT report hook. There-upon we simply pop up an assert. The problem arises when the assert dialog doesn’t appear, but apparently locks on something.

The call stack for the assert:

>    user32.dll!NtUserCallHwndParamLock()     <– That locks up here:

user32.dll!InternalDialogBox()

user32.dll!DialogBoxIndirectParamAorW()

user32.dll!DialogBoxIndirectParamW()

BugslayerUtil.dll!JWnd::Dialog() Line 53    C++

BugslayerUtil.dll!JModalDlg::DoModal() Line 72    C++

BugslayerUtil.dll!PopTheFancyAssertion() Line 474    C++

BugslayerUtil.dll!RealSuperAssertion() Line 281    C++

BugslayerUtil.dll!SuperAssertionA() Line 566    C++

maxutil.dll!assert1() Line 237    C++ <– here we pop up a normal assert.

3dsmax.exe!CrashHandler::Imp::RTLReportHook() Line 1725    C++

msvcr90d.dll!_VCrtDbgReportW() Line 576    C

msvcr90d.dll!_CrtDbgReportWV() Line 242    C++

msvcr90d.dll!_CrtDbgReportW(…) Line 42    C++

msvcp90d.dll!std::_Debug_message() Line 22    C++ <– less than operator flubs it here.

dltonerep.dlu!std::_Debug_lt<float,float>() Line 265    C++

dltonerep.dlu!std::min<float>() Line 3399    C++

So I’m wondering how a call to

::DialogBoxIndirectParam

could lock up?

The NtUserCallHwndParamLock function assembly is this:

NtUserCallHwndParamLock:

00000000778DB7C0 mov r10,rcx

00000000778DB7C3 mov eax,1027h

00000000778DB7C8 syscall

00000000778DB7CA ret

00000000778DB7CB nop

00000000778DB7CC nop

00000000778DB7CD nop

00000000778DB7CE nop

00000000778DB7CF nop

It seems that I cannot step through assembly code because I get this error:

The dialog box states: “Unable to step. The process has been soft broken.”

[Edit]

Eventually I solved this by eliminating the use of the std::min and std::max. I simply wrote my own functions to return a minimum and maximum value, and that solved everything.

As for the lockup, well, it could have been because of some contention over resources, as I was trying to pop up a dialog, and perhaps the CRT was trying to as well.

Advertisements

Getting the Individual Symbols for the OS

Sometimes while debugging you may come across a call stack that looks like this:

As a developer you should never debug without symbols for the OS. Every call stack in max contains modules from the OS. Without debug symbols the call stacks will be incomplete and debugging will become harder. For every stack frame that has no symbols, the debugger will insert an annoying message under it saying ‘Frames below may be incorrect and/or missing, no symbols loaded for <module>’. Missing symbols are so detrimental to debugging that the debugger may sometimes provide completely bogus names for stack frames.

There is a simple remedy to quickly, easily and simply getting individuals symbols for the operating system. If you want to get a whole bunch of symbols at once, then there are other ways to do that.

Just Right click and click on the module in the call stack and select ‘Load Symbols From’ > ‘Microsoft Symbol Servers’

This will download the symbol for the one selected module in your stack. Then all you then do is wait while Microsoft downloads a PDB to a known location on your computer. It will then rebuild your call stack for you and then you get this: