Problem with Qt compilation under Windows 64-bit

This short post will show you how to resolve following error during Qt compilation on 64-bit windows:

Creating library ..\..\..\..\lib\QtWebKitd4.lib and object ..\..\..\..\lib\Qt
WebKitd4.exp
PluginViewWin.obj : error LNK2019: unresolved external symbol _HBeginPaint refer
enced in function "private: static struct HDC__ * __cdecl WebCore::PluginView::h
ookedBeginPaint(struct HWND__ *,struct tagPAINTSTRUCT *)" (?hookedBeginPaint@Plu
ginView@WebCore@@CAPEAUHDC__@@PEAUHWND__@@PEAUtagP AINTSTRUCT@@@Z)
PluginViewWin.obj : error LNK2019: unresolved external symbol _HEndPaint referen
ced in function "private: static int __cdecl WebCore::PluginView::hookedEndPaint
(struct HWND__ *,struct tagPAINTSTRUCT const *)" (?hookedEndPaint@PluginView@Web
Core@@CAHPEAUHWND__@@PEBUtagPAINTSTRUCT@@@Z)
..\..\..\..\lib\QtWebKitd4.dll : fatal error LNK1120: 2 unresolved externals
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\
VC\BIN\x86_amd64\link.EXE"' : return code '0x460'
Stop.
NMAKE : fatal error U1077: '"C:\Program Files (x86)\Microsoft Visual Studio 9.0\
VC\BIN\nmake.exe"' : return code '0x2'

This error is caused by bug in QMAKE_HOST variable used in WebCore.pro file. This variable is tested to determine destination platform (x86 or x64).
This variable contains x86 value also on 64-bit builds instead of x86_64 value. This value is tested on the following line in a WebCore.pro file:

win32:!win32-g++*:contains(QMAKE_HOST.arch, x86_64):{

If you want to fix this error, simply replace QMAKE_HOST by QMAKE_TARGET variable. Updated line should look like this:

win32:!win32-g++*:contains(QMAKE_TARGET.arch, x86_64):{

After this update all required asm defines are correctly included and build of WebKit will be successful.

Update for this error

In newer versions of Qt there is already applied fix the error with QMAKE_HOST / QMAKE_TARGET, but also there is a new problem ;-(.

The Problem is in the following “if” definition:

   if(win32-msvc2005|win32-msvc2008):equals(TEMPLATE_PREFIX, "vc") {
        SOURCES += \
            plugins/win/PaintHooks.asm
    }

When you compile your project for MSVC 2010, .asm file isn’t included. It’s necessary to add win32-msvc2010 to this definition. So updated WebCore.pro for Qt 4.8 will look like this:

 if(win32-msvc2005|win32-msvc2008|win32-msvc2010):equals(TEMPLATE_PREFIX, "vc") {
        SOURCES += \
            plugins/win/PaintHooks.asm
    }

Next problem solution

Next problem which I found during compilation is wrongly set QMAKE_TARGET.arg. For unspecified reason, when I compile 64bit library in 64bit system in 64bit VS command line, this variable has value x86 instead of x86_64.
The problem is in the qmake project.cpp file. QMAKE trying to determine 64/32 bit version based on the %PATH% configuration. Qmake search for VS-PATH\bin\amd64 of \bin\x86_amd64 string. Unfortunately there is a bug in concating searched string. This is how looks original project.cpp file at line 3177:

QString vcBin64 = qgetenv("VCINSTALLDIR").append("\\bin\\amd64");
QString vcBinX86_64 = qgetenv("VCINSTALLDIR").append("\\bin\\x86_amd64");

The problem is, that VCINSTALLDIR contains \ at the end of the string, and then we append \ again. So, result is:

QString vcBin64 = qgetenv("VCINSTALLDIR");
if ( vcBin64.at( vcBin64.size()-1 ) != '\\' )
  vcBin64.append("\\");
vcBin64.append("bin\\amd64");

QString vcBinX86_64 = qgetenv("VCINSTALLDIR");
if ( vcBinX86_64.at( vcBinX86_64.size()-1 ) != '\\' )
  vcBinX86_64.append("\\");
vcBinX86_64.append("bin\\x86_amd64");

Now, QMake correctly determine 32/64 bit compilation and then correctly include PaintHooks.asm and your Qt WebCore will be sucesfully compiled.