Lockup in call DialogBoxIndirectParam at call to NtUserCallHwndParamLock

One day, while running some of our automated test suites, I suddenly got a lockup in while running the following script: fat_rendering_exposurecontrol.ms.

I had seen this before last year. But it was a problem that would come and go at whim. Even recompiling would make it go away.

The root problem was caused by a ‘less than’ operator failing. I’m not really interested in why this is failing, but it helps explain the rest of the problem.

Anyways, it all starts with a call to an STL min function like this:

std::min( a, b )

Eventually the less than operator devolves down to this grotesque mess:

c:Program Files (x86)Microsoft Visual Studio 9.0VCincludexutility [line 264]

template<class
_Ty1, class
_Ty2> inline

bool
__CLRCALL_OR_CDECL
_Debug_lt(const
_Ty1& _Left, const
_Ty2& _Right,

const
wchar_t *_Where, unsigned
int
_Line)

{    // test if _Left < _Right and operator< is strict weak ordering

if (!(_Left < _Right))

return (false);

else
if (_Right < _Left)

_DEBUG_ERROR2(“invalid operator<“, _Where, _Line); <– here

return (true);

}

_Right was 3814.9946

_Left was 6405.5967

_Left < _Right    is false
!(_Left < _Right)    is true

Therefore the code had absolutely no business hitting the else clause of the if statement and calling _DEBUG_ERROR2 . Weird!! Perhaps it’s a compiler error… But anyways…

The error eventually reports an assert error to the Debug CRT. Which calls our own debug CRT report hook. There-upon we simply pop up an assert. The problem arises when the assert dialog doesn’t appear, but apparently locks on something.

The call stack for the assert:

>    user32.dll!NtUserCallHwndParamLock()     <– That locks up here:

user32.dll!InternalDialogBox()

user32.dll!DialogBoxIndirectParamAorW()

user32.dll!DialogBoxIndirectParamW()

BugslayerUtil.dll!JWnd::Dialog() Line 53    C++

BugslayerUtil.dll!JModalDlg::DoModal() Line 72    C++

BugslayerUtil.dll!PopTheFancyAssertion() Line 474    C++

BugslayerUtil.dll!RealSuperAssertion() Line 281    C++

BugslayerUtil.dll!SuperAssertionA() Line 566    C++

maxutil.dll!assert1() Line 237    C++ <– here we pop up a normal assert.

3dsmax.exe!CrashHandler::Imp::RTLReportHook() Line 1725    C++

msvcr90d.dll!_VCrtDbgReportW() Line 576    C

msvcr90d.dll!_CrtDbgReportWV() Line 242    C++

msvcr90d.dll!_CrtDbgReportW(…) Line 42    C++

msvcp90d.dll!std::_Debug_message() Line 22    C++ <– less than operator flubs it here.

dltonerep.dlu!std::_Debug_lt<float,float>() Line 265    C++

dltonerep.dlu!std::min<float>() Line 3399    C++

So I’m wondering how a call to

::DialogBoxIndirectParam

could lock up?

The NtUserCallHwndParamLock function assembly is this:

NtUserCallHwndParamLock:

00000000778DB7C0 mov r10,rcx

00000000778DB7C3 mov eax,1027h

00000000778DB7C8 syscall

00000000778DB7CA ret

00000000778DB7CB nop

00000000778DB7CC nop

00000000778DB7CD nop

00000000778DB7CE nop

00000000778DB7CF nop

It seems that I cannot step through assembly code because I get this error:

The dialog box states: “Unable to step. The process has been soft broken.”

[Edit]

Eventually I solved this by eliminating the use of the std::min and std::max. I simply wrote my own functions to return a minimum and maximum value, and that solved everything.

As for the lockup, well, it could have been because of some contention over resources, as I was trying to pop up a dialog, and perhaps the CRT was trying to as well.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s