How to parse Apache Log and insert data to MySQL database

#!/bin/bash

echo Generating new data from log
echo "#generated sql file" > data.sql

echo "truncate table log_data;" >> data.sql
echo "START TRANSACTION;" >> data.sql

FIND_PATTERN="\
\(\S*\)\s*\
\(\S*\)\s*\
\(\S*\s*\S*\)\s*\
[\x5B]\([^\x5D]*\)[\x5D]\s*\
\"\([^\"]*\)\"\s*\
\(\S*\)\s*\
\(\S*\)\s*\
\"\([^\"]*\)\"\s*\
\"\([^\"]*\)\"\s*\
";

REPLACE_PATTERN="\
INSERT INTO log_data \
(domain,ip,other,date,url,status,size,referral,clientstring) \
VALUES \
('\1','\2','\3','\4','\5','\6','\7','\8','\9');";

APACHE_FILE="other_vhosts_access.log";

if [ -f $APACHE_FILE.size ];
then
   ALREADY_PROCEED=`cat $APACHE_FILE.size`
   echo "Already proceed size ($APACHE_FILE): "$ALREADY_PROCEED;
else
   ALREADY_PROCEED=0;
   echo "New log file $APACHE_FILE";
fi

FILTER_PATTERN="\.css|\.ico|\.png|\.jpg|\.txt|\.js|\.gif|\.swf"

#echo $FIND_PATTERN"/"$REPLACE_PATTERN;

cat $APACHE_FILE |
tail -c +$ALREADY_PROCEED | #preskocime jiz zpracovane radky
#head -5000 | #debug
#head   -1128863 | #tail -2  |
#tail -n +1177103 | head -3 |
#sed -r "s/(\(.*\))//g" | #debug - odebere z clientstring (...)
#sed  "s/^[^+]*//g" | #debug - odebere pocatek stringu az do GET
#sed -r "s/Mozilla|Gecko|Firefox//g" | #debug odebere tyhle klicovy slova
sed -r "s/^(subdomain|www)\.//"   | #odebereme prefix subdomain. a www.
sed -r "s/^([^:]*)(:[0-9]*)/\1/"  | #odebereme port
sed -r -n "/^(orm-|inventic).*/p" | #exportujem jen nektere domeny
sed "s/\([\x5c][\x22]\)//g"       | #odebrani \" retezcu
sed "s/\([']\)//g"                | #odebrani apostrof ktery delaj pak bordel v$
sed -r "s/ \"GET ([\x2F].*) HTTP[\x2F][0-9.]*/ \"\1/" | #dame pryc GET a HTTP1.1
#sed -r -n  "/$FILTER_PATTERN/p" #| debug- vypise filtrovane .css/.png/...
sed -r "/$FILTER_PATTERN/d"       | #vyfiltrujeme pryc .css, .png atd
#sed  "/^$FIND_PATTERN$/d"        | #debug - toto lze pouzit na vypis radku kte$
sed -n "/^$FIND_PATTERN$/p"       | #vypiseme vsechny radky ktere jsou validni
sed "s/$FIND_PATTERN/$REPLACE_PATTERN/" >> data.sql

echo "COMMIT;" >> data.sql

#ulozime velikost zpracovaneho logu
stat -c"%s" $APACHE_FILE > $APACHE_FILE.size
#vlozime data do Mysql
echo Putting data to MySQL
mysql \
  --user=root \
  --password=1234 \
  --host=192.168.0.10 \
  --database=server_log < data.sql

SandyBridge

CPU

http://www.alza.cz/intel-core-i7-2600k-quad-core-sandy-bridge-d201345.htm

http://www.alfacomp.cz/php/product.php?eid=1051400830984YJ12YG

http://interlink.tsbohemia.cz/intel-core-i7-2600k-3-40ghz-8mb-box-lga1155-sandy-bridge-s-integrovanou-grafikou-odemceny-nasobic_d116932.html

http://www.czechcomputer.cz/product.jsp?artno=85286

http://www.mironet.cz/intel-core-i72600k-quad-core–34-ghz–8mb-cache–1155–box–overclocking+dp117723/

MB

http://interlink.tsbohemia.cz/komponenty-it-zakladni-desky-desktopove-intel-bez-vga-socket-1155_c10856.html

http://www.alfacomp.cz/php/product.php?eid=10514008M0984YC12H4

http://www.czechcomputer.cz/product.jsp?artno=84646

http://www.alza.cz/asus-p8p67-pro-d202981.htm

RAM

http://interlink.tsbohemia.cz/dimm-ddr3-4096mb-2000-dc-kit-2×2048-transcend-9-9-9-24-axeram_d94573.html

http://interlink.tsbohemia.cz/dimm-ddr3-4096mb-2400-dc-kit-2×2048-transcend-9-11-9-28-axeram_d109507.html

http://www.alza.cz/a-data-4gb-kit-ddr3-1600mhz-cl7-xpg-series-d214745.htm

MONITOR

http://www.alza.cz/lcd-monitory/24-a-vetsi-lcd/18843375.htm#CatId=18843375&limit=&prod=1396,1287,&pn=&Pg=0


Collection of links about ORM, Datamapper and design patterns

Domain model

http://www.slideshare.net/weierophinney/architecting-your-models

ORM design patterns

http://martinfowler.com/eaaCatalog/mapper.html
http://martinfowler.com/eaaCatalog/dataMapper.html

DataMapper

http://stackoverflow.com/questions/207758/data-mapper-pattern
http://stackoverflow.com/questions/762252/fowler-data-mapper-object-creation
http://stackoverflow.com/questions/4658340/should-data-mapper-reference-domain-model
http://stackoverflow.com/questions/1977684/should-a-finder-method-be-part-of-the-data-mapper-or-part-of-the-domain-class
http://stackoverflow.com/questions/3738687/using-the-data-mapper-pattern-should-the-entities-domain-objects-know-about-th

Anemic domain model

http://www.martinfowler.com/bliki/AnemicDomainModel.html (Anemic domain model – why huge service layer  is bad)
http://stackoverflow.com/questions/258534/anemic-domain-model-pros-cons

Data Mapper best practices

http://stackoverflow.com/questions/4465237/doctrine2-best-practice-should-entities-use-services
http://stackoverflow.com/questions/4108291/using-entitymanager-inside-doctrine-2-0-entities/4109458#4109458

Doctrine2 articles

http://www.doctrine-project.org/docs/orm/2.0/en/cookbook/aggregate-fields.html

Qt programming tips

How to configure mkspec for paralel builds on Windows

Edit qmake.conf file in Qt\mkspecs\win32-msvcXXXX. Add -MP2 / -MP4 to the following defines:

QMAKE_CFLAGS_RELEASE    = -O2 -MD -MP4
QMAKE_CFLAGS_RELEASE_WITH_DEBUGINFO += -O2 -MD -Zi -MP4
QMAKE_CFLAGS_DEBUG      = -Zi -MDd -MP4

Mount error(12): Cannot allocate memory

Problem

When mounting some windows samba drives, “mount error(12)” error can occurred. Problem is in windows samba server settings. Full linux error message:

mount error(12): Cannot allocate memory
Refer to the mount.cifs(8) manual page (e.g.man mount.cifs)

and error message in windows event log:

Source: srv
Event ID: 2017
Level: Error
The server was unable to allocate from the system nonpaged pool because the server reached the configured limit for nonpaged pool allocations.

Solution

Update following registry:

HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\LargeSystemCache = 1
HKLM\SYSTEM\CurrentControlSet\Services\LanmanServer\Parameters\Size = 3

After you update these values, restart your computer.

More about this issue:

http://alan.lamielle.net

How to deploy Qt application on MacOs

Missing dylib libraries

If your application using some externals dynamic libraries (called .dylib in MacOs system), you have to provide all these libraries. The simplest way is to provide these libraries into the Application.app directory.

Find out which libraries are used

Qt provide really useful tool called “otool” which serves to lots of purpose. One of them is to detect which shared libraries are required by our application. For this purpose otool has parameter -L. The usage is following:

otool -L ./Test1.app/Contents/MacOS/Test1

Output could looks like this:

./Test1:
        libboost_iostreams.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_thread.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_date_time.dylib (compatibility version 0.0.0, current version 0.0.0)
        libboost_regex.dylib (compatibility version 0.0.0, current version 0.0.0)
        /Users/dev/dev/SharedLibraries/libiconv/lib/libiconv.2.dylib (compatibility version 8.0.0, current version 8.0.0)
        /Users/dev/dev/ExternalLibraries/../SharedLibraries/libxml/lib/libxml2.2.dylib (compatibility version 10.0.0, current version 10.8.0)
        /Users/dev/dev/ExternalLibraries/../SharedLibraries/libxslt/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.26.0)
        /Users/dev/dev/ExternalLibraries/../SharedLibraries/libxslt/lib/libexslt.0.dylib (compatibility version 9.0.0, current version 9.15.0)
        /Users/dev/dev/ExternalLibraries/yaml-cpp-0.2.5/build/libyaml-cpp.0.2.dylib (compatibility version 0.2.0, current version 0.2.5)
        QtGui.framework/Versions/4/QtGui (compatibility version 4.7.0, current version 4.7.0)
        QtCore.framework/Versions/4/QtCore (compatibility version 4.7.0, current version 4.7.0)
        /usr/lib/libstdc++.6.dylib (compatibility version 7.0.0, current version 7.9.0)
        /usr/lib/libgcc_s.1.dylib (compatibility version 1.0.0, current version 625.0.0)
        /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 125.2.0)

Output from “otool” is simmilar to linux tool “ldd”.

The Mac Deployment Tool

Because manual copying and configuring all shared libraries to our application could be difficult, Qt have deployment tool which do all work for us. Tool is called “macdeployqt” and can be found in /Developer/Tools/Qt/macdeployqt.

 /Developer/Tools/Qt/macdeployqt ./GeneratedFilesMacOs/Debug/Test1.app

Note: Every time when you call macdeployqt tool, project have to be build from scratch. But is sufficient when whole result directory is removed and linked again:

rm -rf ./GeneratedFilesMacOs/Debug/Test1.app/
make

g++ template method problem

Quick post about problem which I already solved, but think that could be handy for someone else.

I have following code:

template<class T>
struct Test
{
	T val;
	template <class T2>
	void DoSomething( T2 &obj )
	{
		obj.FindByType<int>();
	}
};

When compiling under Visual studio, everything is ok. But when try the same code snippet under g++, have following error:

test.cpp: In member function ‘void Test<T>::DoSomething(T2&)’:
test.cpp:48: error: expected primary-expression before ‘int’
test.cpp:48: error: expected ‘;’ before ‘int’

This is because compiler doesn’t known FindByType method, because T2 is templated argument. The solution which I found is in adding keyword template before FindByType method name. So updated source code will look like this:

template<class T>
struct Test
{
	T val;
	template <class T2>
	void DoSomething( T2 &obj )
	{
		obj.template FindByType<int>();
	}
};

After this update, code will be compiled correctly under both compilers.

Git tips

Show files status

git status #show all files
git statis -u -s #show untracked in short listing

Commit with message

git commit -m"text"

Recursive add files by mask to git

git add *.mask -A

Linux-recursive remove some files

find -iname '*mask' | xargs rm

How to create global .gitignore file

git config --global core.excludesfile /file-path/.gitignore-global

TENG – c++ templating engine

Project site: http://teng.sourceforge.net/?page=home

External project documentation: http://teng.olmik.net/

Latest TENG source code: http://teng.cvs.sourceforge.net/teng/ (Note: Source code referenced from main site isn’t latest! )

How to compile TENG on windows

Compilation under Windows is possible only using MinGW and with few modifications in TENG code (because there is few glitch which didn’t meet c++ standards). If you want more information or updated TENG version, please let me know.

How to install and configure git with ssh key

  • Download git client depends on your OS platform
  • Open your .ppk key in PuTTY Key generator and export key to OpenSSH key:
  • Content of this OpenSSH key store to file ~/.ssh/id_dsa (copy exported key to ~/.ssh folder and rename it to id_dsa)
  • Now change security permissions on this file to 400 (chmod 400 ~/.ssh/id_dsa)
  • Now cal ssh-add to load id_dsa key

And it’s done. Now you can use your git. For example clone some repository:

git clone ssh://[email protected]/~/repository/example.git

Automatic ssh-key loading during startup:

TODO