Category: Linux

Integration of Google breakpad on any platform is really challenge. I didn’t figure out how to do it by using recommended ways. So I did it my way 😉

Create your own Qt project

As first thing I did when integrating Qt and Breakpad is creation of my own .pri file. I manually add all required files from Breakpad one-by-one, starting with src/platform/exception_handler continuing with all files included in previous ones. The complete .pri file you will find at end of this post.

After that I created simple CCrashHandler (inspired by qt-breakpad project) class which register exception handler. It’s necessary to implement this handler for each platform separately because each platform have different parameters in google_breakpad::ExceptionHandler() constructor. CrashHandler source code you can find also at end of this post.

Using Breakpad on Linux:

Article about Linux integration.

Notes Linux

• GCC 4.6.3
• Ubuntu 12.04.1 32/64bit

Get and Compile Breakpad

#get breakpad latest version
svn checkout http://google-breakpad.googlecode.com/svn/trunk/ google-breakpad-read-only

#compile all breakpad tools for extracting symbol files
./configure
make
sudo make install

#when compilation is ready, you should have installed file dump_syms from original directory /src/tools/linux/dump_syms in your /usr/bin
dump_syms

Generate symbol file and test debug info

Now how to generate and use symbol file:

#generate .sym file
dump_syms ./Application > Application.sym

#store sym file in the correct location
#this step is necessary. Without that minidump_stackwalk tool doesn't work

head -n1 Application.sym
#Result: MODULE Linux x86_64 6EDC6ACDB282125843FD59DA9C81BD830 Application

mkdir -p ./symbols/Application/6EDC6ACDB282125843FD59DA9C81BD830
mv Application.sym ./symbols/Application/6EDC6ACDB282125843FD59DA9C81BD830

#show stack with using minidump_stackwalk tool
minidump_stackwalk ./crash.dmp ./symbols

Result of minidump_stackwalk tool can look like this:

Use additional tools

There is script written by Mozzila corp which simplifies extracting and storing .sym file:
http://mxr.mozilla.org/mozilla-central/source/toolkit/crashreporter/tools/symbolstore.py?raw=1

Now with this script, you can do only one step instead of all described above.

python /path/to/script/symbolstore.py /usr/local/bin/dump_syms ./symbol-storage ./Application

This command correctly creates folder Application/uuid in path symbol-storage and copy generated .sym file inside.

Using Breakpad on MacOS:

To integrate Breakpad and Qt on MacOS I used the same article as for Linux because I’m using gcc and linux-like development toolchain. For native XCode integration you have to probably use this article.

Notes MacOS

• GCC 4.6.3
• Qt creator toolchain
• Mac OS 10.7.3 64bit
• Don’t using any of XCode tool chaing

Do almost all steps like on Linux integration. There is one exception, because you can’t compile Breakpad tools using linux-like configure&make. You have to build these tools with XCode.

Compile google_breakpad/src/tools/mac/dump_syms/dump_syms.xcodeproj with XCode

Before I managed to successfully compile dump_syms in XCode, I have to update Architectures Debug/Release to 64-bit Intel and BaseSDK to “Latest Mac OS X”. I also fix all issues displayed in the left part of XCode window.

Next step is change compilation type from Debug to Release. After a short searching I found a way in “Schema->Edit schema->Build configuration”.

To be honest, I was not sure what I did, but it worked ;-). Maybe there is some easier way how to compile dump_syms in xcode, but I don’t know how. This was my first contact with XCode ;-).

The application is compiled to path “/Users/User/Library/Developer/Xcode/DerivedData/dump_syms-bjvr/Build/Products/Release/dump_syms”. Now copy dump_syms to safe location from where you will use it.

The rest steps are same like in the Linux.

Using Breakpad on Windows

Notes Windows

Article about Windows integration.

• Compile with Visual Studio 2010
• Post-mortem debugging is done also via VS2010
• Google breakpad under windows generates common .dmp files which can by simply load directly to VS

Generate symbol files and test debug info

You have to choices for Windows. First option is use standard .PDB and .DMP files mechanism in VS. Second is use the same way how you can did it for Linux and Mac.

First way: Let compiler generate .PDB file. Store this files and use them later in VS together with your crash dump.

Second way: Use dump_sys.exe. This file is located in the google breakpad directory tree: google-breakpad\src\tools\windows\binaries\dump_syms.exe .

dump_syms.exe AtomixDevelopment.exe >AtomixDevelopment.sym

And also like under Linux, you can use Python script symbolstore.py to extract symbols to correct directory:

python.exe symbolstore.py binaries\dump_syms.exe .\Symbols AtomixDevelopment.exe

Minidump stackwalk for windows:
http://code.google.com/p/pcxfirefox/source/browse/trunk/betterpgo/talos/talos/breakpad/win32/minidump_stackwalk.exe?r=19

The best thing at end

The most amazing thing I found about dump_syms is their platform independence. If you correctly extract symbol file on each platform and store it in one location, you can analyze any crashdump on one location. No matter what is source platform, you will analyze .dmp file by using minidump_stackwalk ./crash.dmp ./symbols.

How to include CrashHandler directly to your project

This is what I get when I follow all includes from .h and .cpp files starting with exception_handler.h for all platforms. With small modification you can include it to your project and your application will be fully breakpad-able ;-).

# ---------- HEADER -----------------------------------------------------------
OTHERS   += $$PWD/axCrashHandler.pri
HEADERS += $$PWD/CrashHandler.h
SOURCES += $$PWD/CrashHandler.cpp

BREAKPAD_PATH=$$EXTERNAL_LIBRARIES_PATH/breakpad-qt
INCLUDEPATH += $$BREAKPAD_PATH/src

OSMAC {
  HEADERS += $$BREAKPAD_PATH/src/client/mac/handler/exception_handler.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/crash_generation/crash_generation_client.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/crash_generation/crash_generation_server.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/crash_generation/client_info.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/handler/minidump_generator.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/handler/dynamic_images.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/handler/breakpad_nlist_64.h
  HEADERS += $$BREAKPAD_PATH/src/client/mac/handler/mach_vm_compat.h
  HEADERS += $$BREAKPAD_PATH/src/client/minidump_file_writer.h
  HEADERS += $$BREAKPAD_PATH/src/client/minidump_file_writer-inl.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/macho_utilities.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/byteswap.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/MachIPC.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/scoped_task_suspend-inl.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/file_id.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/macho_id.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/macho_walker.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/macho_utilities.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/bootstrap_compat.h
  HEADERS += $$BREAKPAD_PATH/src/common/mac/string_utilities.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/linux_libc_support.h
  HEADERS += $$BREAKPAD_PATH/src/common/string_conversion.h
  HEADERS += $$BREAKPAD_PATH/src/common/md5.h
  HEADERS += $$BREAKPAD_PATH/src/common/memory.h
  HEADERS += $$BREAKPAD_PATH/src/common/using_std_string.h
  HEADERS += $$BREAKPAD_PATH/src/common/convert_UTF.h
  HEADERS += $$BREAKPAD_PATH/src/processor/scoped_ptr.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_exception_mac.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/breakpad_types.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_format.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_size.h
  HEADERS += $$BREAKPAD_PATH/src/third_party/lss/linux_syscall_support.h

  SOURCES += $$BREAKPAD_PATH/src/client/mac/handler/exception_handler.cc
  SOURCES += $$BREAKPAD_PATH/src/client/mac/crash_generation/crash_generation_client.cc
  SOURCES += $$BREAKPAD_PATH/src/client/mac/crash_generation/crash_generation_server.cc
  SOURCES += $$BREAKPAD_PATH/src/client/mac/handler/minidump_generator.cc
  SOURCES += $$BREAKPAD_PATH/src/client/mac/handler/dynamic_images.cc
  SOURCES += $$BREAKPAD_PATH/src/client/mac/handler/breakpad_nlist_64.cc
  SOURCES += $$BREAKPAD_PATH/src/client/minidump_file_writer.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/macho_id.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/macho_walker.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/macho_utilities.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/string_utilities.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/file_id.cc
  SOURCES += $$BREAKPAD_PATH/src/common/mac/MachIPC.mm
  SOURCES += $$BREAKPAD_PATH/src/common/mac/bootstrap_compat.cc
  SOURCES += $$BREAKPAD_PATH/src/common/md5.cc
  SOURCES += $$BREAKPAD_PATH/src/common/string_conversion.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/linux_libc_support.cc
  SOURCES += $$BREAKPAD_PATH/src/common/convert_UTF.c
  LIBS += /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
  LIBS += /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
  #breakpad app need debug info inside binaries
  QMAKE_CXXFLAGS+=-g
}

OSLIN {
  HEADERS += $$BREAKPAD_PATH/src/client/linux/handler/exception_handler.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/crash_generation/crash_generation_client.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/handler/minidump_descriptor.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/minidump_writer/minidump_writer.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/minidump_writer/line_reader.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/minidump_writer/linux_dumper.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/minidump_writer/linux_ptrace_dumper.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/minidump_writer/directory_reader.h
  HEADERS += $$BREAKPAD_PATH/src/client/linux/log/log.h
  HEADERS += $$BREAKPAD_PATH/src/client/minidump_file_writer-inl.h
  HEADERS += $$BREAKPAD_PATH/src/client/minidump_file_writer.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/linux_libc_support.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/eintr_wrapper.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/ignore_ret.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/file_id.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/memory_mapped_file.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/safe_readlink.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/guid_creator.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/elfutils.h
  HEADERS += $$BREAKPAD_PATH/src/common/linux/elfutils-inl.h
  HEADERS += $$BREAKPAD_PATH/src/common/using_std_string.h
  HEADERS += $$BREAKPAD_PATH/src/common/memory.h
  HEADERS += $$BREAKPAD_PATH/src/common/basictypes.h
  HEADERS += $$BREAKPAD_PATH/src/common/memory_range.h
  HEADERS += $$BREAKPAD_PATH/src/common/string_conversion.h
  HEADERS += $$BREAKPAD_PATH/src/common/convert_UTF.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_format.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_size.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/breakpad_types.h
  HEADERS += $$BREAKPAD_PATH/src/processor/scoped_ptr.h
  HEADERS += $$BREAKPAD_PATH/src/third_party/lss/linux_syscall_support.h
  SOURCES += $$BREAKPAD_PATH/src/client/linux/crash_generation/crash_generation_client.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/handler/exception_handler.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/handler/minidump_descriptor.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/minidump_writer/minidump_writer.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/minidump_writer/linux_dumper.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/minidump_writer/linux_ptrace_dumper.cc
  SOURCES += $$BREAKPAD_PATH/src/client/linux/log/log.cc
  SOURCES += $$BREAKPAD_PATH/src/client/minidump_file_writer.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/linux_libc_support.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/file_id.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/memory_mapped_file.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/safe_readlink.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/guid_creator.cc
  SOURCES += $$BREAKPAD_PATH/src/common/linux/elfutils.cc
  SOURCES += $$BREAKPAD_PATH/src/common/string_conversion.cc
  SOURCES += $$BREAKPAD_PATH/src/common/convert_UTF.c
  #breakpad app need debug info inside binaries
  QMAKE_CXXFLAGS+=-g
}

OSWIN {
  BREAKPAD_PATH=q:/Applications/breakpad-qt/third-party/latest-breakpad
  INCLUDEPATH += $$BREAKPAD_PATH/src
  HEADERS += $$BREAKPAD_PATH/src/common/windows/string_utils-inl.h
  HEADERS += $$BREAKPAD_PATH/src/common/windows/guid_string.h
  HEADERS += $$BREAKPAD_PATH/src/client/windows/handler/exception_handler.h
  HEADERS += $$BREAKPAD_PATH/src/client/windows/common/ipc_protocol.h
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/minidump_format.h 
  HEADERS += $$BREAKPAD_PATH/src/google_breakpad/common/breakpad_types.h 
  HEADERS += $$BREAKPAD_PATH/src/client/windows/crash_generation/crash_generation_client.h 
  HEADERS += $$BREAKPAD_PATH/src/processor/scoped_ptr.h 

  SOURCES += $$BREAKPAD_PATH/src/client/windows/handler/exception_handler.cc
  SOURCES += $$BREAKPAD_PATH/src/common/windows/string_utils.cc
  SOURCES += $$BREAKPAD_PATH/src/common/windows/guid_string.cc
  SOURCES += $$BREAKPAD_PATH/src/client/windows/crash_generation/crash_generation_client.cc 
}

And this is the minimal implementation of your new CrashHandler. My ownd crash handler is much more sophisticate to perform reporting, sending crash dump etc. But for purposes of this article this is what you need (And also I don’t want to share my whole know-how here ;-)))

CrashHandler.h

#pragma once
#include <QtCore/QString>

namespace Atomix
{
	class CrashHandlerPrivate;
	class CrashHandler
	{
	public:
		static CrashHandler* instance();
    void Init(const QString&  reportPath);

		void setReportCrashesToSystem(bool report);
		bool writeMinidump();

	private:
		CrashHandler();
		~CrashHandler();
		Q_DISABLE_COPY(CrashHandler)
		CrashHandlerPrivate* d;
	};
}

CrashHandler.cpp

#include "CrashHandler.h"
#include <QtCore/QDir>
#include <QtCore/QProcess>
#include <QtCore/QCoreApplication>
#include <QString>

#if defined(Q_OS_MAC)
#include "client/mac/handler/exception_handler.h"
#elif defined(Q_OS_LINUX)
#include "client/linux/handler/exception_handler.h"
#elif defined(Q_OS_WIN32)
#include "client/windows/handler/exception_handler.h"
#endif

namespace Atomix
{
	/************************************************************************/
	/* CrashHandlerPrivate                                                  */
	/************************************************************************/
	class CrashHandlerPrivate
	{
	public:
		CrashHandlerPrivate()
		{
			pHandler = NULL;
		}

		~CrashHandlerPrivate()
		{
			delete pHandler;
		}

		void InitCrashHandler(const QString& dumpPath);
		static google_breakpad::ExceptionHandler* pHandler;
		static bool bReportCrashesToSystem;
	};

	google_breakpad::ExceptionHandler* CrashHandlerPrivate::pHandler = NULL;
	bool CrashHandlerPrivate::bReportCrashesToSystem = false;

	/************************************************************************/
	/* DumpCallback                                                         */
	/************************************************************************/
#if defined(Q_OS_WIN32)
	bool DumpCallback(const wchar_t* _dump_dir,const wchar_t* _minidump_id,void* context,EXCEPTION_POINTERS* exinfo,MDRawAssertionInfo* assertion,bool success)
#elif defined(Q_OS_LINUX)
	bool DumpCallback(const google_breakpad::MinidumpDescriptor &md,void *context, bool success)
#elif defined(Q_OS_MAC)
	bool DumpCallback(const char* _dump_dir,const char* _minidump_id,void *context, bool success)
#endif
	{
		Q_UNUSED(context);
#if defined(Q_OS_WIN32)
		Q_UNUSED(_dump_dir);
		Q_UNUSED(_minidump_id);
		Q_UNUSED(assertion);
		Q_UNUSED(exinfo);
#endif
		qDebug("BreakpadQt crash");

		/*
		NO STACK USE, NO HEAP USE THERE !!!
		Creating QString's, using qDebug, etc. - everything is crash-unfriendly.
		*/
		return CrashHandlerPrivate::bReportCrashesToSystem ? success : true;
	}

	void CrashHandlerPrivate::InitCrashHandler(const QString& dumpPath)
	{
		if ( pHandler != NULL )
			return;

#if defined(Q_OS_WIN32)
		std::wstring pathAsStr = (const wchar_t*)dumpPath.utf16();
		pHandler = new google_breakpad::ExceptionHandler(
			pathAsStr,
			/*FilterCallback*/ 0,
			DumpCallback,
			/*context*/
			0,
			true
			);
#elif defined(Q_OS_LINUX)
		std::string pathAsStr = dumpPath.toStdString();
		google_breakpad::MinidumpDescriptor md(pathAsStr);
		pHandler = new google_breakpad::ExceptionHandler(
			md,
			/*FilterCallback*/ 0,
			DumpCallback,
			/*context*/ 0,
			true,
			-1
			);
#elif defined(Q_OS_MAC)
		std::string pathAsStr = dumpPath.toStdString();
		pHandler = new google_breakpad::ExceptionHandler(
			pathAsStr,
			/*FilterCallback*/ 0,
			DumpCallback,
			/*context*/
			0,
			true,
			NULL
			);
#endif
	}

	/************************************************************************/
	/* CrashHandler                                                         */
	/************************************************************************/
	CrashHandler* CrashHandler::instance()
	{
		static CrashHandler globalHandler;
		return &globalHandler;
	}

	CrashHandler::CrashHandler()
	{
		d = new CrashHandlerPrivate();
	}

	CrashHandler::~CrashHandler()
	{
		delete d;
	}

	void CrashHandler::setReportCrashesToSystem(bool report)
	{
		d->bReportCrashesToSystem = report;
	}

	bool CrashHandler::writeMinidump()
	{
		bool res = d->pHandler->WriteMinidump();
		if (res) {
			qDebug("BreakpadQt: writeMinidump() successed.");
		} else {
			qWarning("BreakpadQt: writeMinidump() failed.");
		}
		return res;
	}

	void CrashHandler::Init( const QString& reportPath )
	{
		d->InitCrashHandler(reportPath);
	}
}

Main.cpp

And here is how to use it:

#include "axCore/axCrashHandler/CrashHandler.h"
#include <QDebug>
#include <QCoreApplication>
#include <iostream>

int buggyFunc()
{
	delete reinterpret_cast<QString*>(0xFEE1DEAD);
	return 0;
}

int main(int argc, char * argv[])
{
	qDebug() << "App start";
	QCoreApplication app(argc, argv);

#if defined(Q_OS_WIN32)
	Atomix::CrashHandler::instance()->Init("c:\\dump");
#elif defined(Q_OS_LINUX)
	Atomix::CrashHandler::instance()->Init("/Users/dev/dump");
#elif defined(Q_OS_MAC)
	Atomix::CrashHandler::instance()->Init("/Users/User/dump");
#endif
	
	qDebug() << "CrashHandlerSet";
	buggyFunc();
	return 0;
}