Friday, January 25, 2013

Configuring Jenkins CI to use multiple TYPO3 versions as build target for Extbase extensions


If you develop TYPO3 extbase extensions, you should know the concepts of test driven development (TDD). Covering as much as possible of your code with tests ensures, that new code or code changes don't break the functionality of your extension. In bigger projects, where several developers work together on an Extbase extension, it is helpfull to use a continuous integration server to automate testing and to ensure code quality.

If the Extbase extension must be compatible with different TYPO3 versions, you have to ensure this during the development process. In this situation, it can be helpful to run automated tests with different TYPO3 versions as build target.

In this article, I will show how to configure a single job in Jenkins CI to use multiple TYPO3 versions as a build target for  running Extbase extension tests.

Before you can start to configure the job in Jenkins CI, you must configure different TYPO3 installations on the server where Jenkins CI is located. This is necessary, because running phpunit tests against an Extbase extension requires the extension to be installed on an working TYPO3 installation.

Configuring the TYPO3 installations

TYPO3 basic setup
On the Jenkins CI Server, you create 3 new virtual hosts. Each virtual host contains a TYPO3 installation with a different version of TYPO3 (4.5.x, 4.7.x and 6.0.x). To keep the TYPO3 installation maintainable, you can symlink each TYPO3 source to a central place on the server, so you easily can update TYPO3 build versions (e.g. version 4.5.20 to 4.5.21)

After the TYPO3 setup, each TYPO3 installation can be opened locally by a hostname like the following
  • http://typo3-45.build.jenkins.local/
  • http://typo3-47.build.jenkins.local/
  • http://typo3-60.build.jenkins.local/
Setting permissions
Next you have to login to each TYPO3 installation and update some settings in the install tool.

Set the file mode mask to:
[BE][fileCreateMask] = 0666 
[BE][folderCreateMask] = 0777

This is necessary for TYPO3 6.0, because there seems to be some problems with local permissions in typo3temp/ (Uncaught TYPO3 Exception: #1294586098: Lock file could not be created) when running the tests through Jenkins CI. 

Installing and configuring the TYPO3 extension "phpunit"
In order to run automated tests in a TYPO3 installation, you have to install the extension "phpunit" from TER. Be sure to get a compatible version of phpunit for each TYPO3 version you use. 

After the extension "phpunit" has been installed, you must create a backend user named "_cli_phpunit". Just create the user and set a random password. 

Make sure, that you can at least execute the tests which came with "phpunit" successfully. This step only ensures, that tests actually can be run successfully.

Setting up the job in Jenkins

The next step is to configure the job in Jenkins. First of all, we create an empty job in Jenkins.



The next step is to create an ANT Build File (built.xml), which contains all settings for  the job. Below is an example ANT Build File for the job described in this article.

<project name="TYPO3-Extbase-Extension" default="build" basedir=".">
        <property name="output" location="${basedir}/build/"/>
        <property file="build.properties" />

        <target name="init">
                <mkdir dir="${output}"/>
                <mkdir dir="${output}/phpcs/"/>
        </target>

        <target name="build" depends="init, typo3-45-unittests, typo3-47-unittests, typo3-60-unittests, phpcs, phpmd, phpcpd">
        </target>

        <target name="typo3-45-unittests">
                <exec executable="php" failonerror="true">
                        <arg path="${typo3path-45}/typo3/cli_dispatch.phpsh" />
                        <arg value="phpunit" />
                        <arg path="${basedir}/src/Tests" />
                </exec>
        </target>

        <target name="typo3-47-unittests">
                <exec executable="php" failonerror="true">
                        <arg path="${typo3path-47}/typo3/cli_dispatch.phpsh" />
                        <arg value="phpunit" />
                        <arg path="${basedir}/src/Tests" />
                </exec>
        </target>

        <target name="typo3-60-unittests">
                <exec executable="php" failonerror="true">
                        <arg path="${typo3path-60}/typo3/cli_dispatch.phpsh" />
                        <arg value="phpunit" />
                        <arg path="${basedir}/src/Tests" />
                </exec>
        </target>

        <target name="phpcs">
                <exec executable="phpcs">
                        <arg line="--report=checkstyle
                                --report-file=${output}/phpcs/checkstyle.xml
                                --standard=/var/lib/jenkins/external_libraries/PHP_CodeSniffer/TYPO3/ruleset.xml
                                ${basedir}" />
                </exec>
        </target>

        <target name="phpmd">
                <exec executable="phpmd">
                        <arg line=" . xml codesize,unusedcode,naming,design --reportfile ${output}/messdetector.xml --exclude Tests/" />
                </exec>
        </target>

        <target name="phpcpd">
                <exec executable="phpcpd">
                        <arg line=" --log-pmd ${output}/phpcpd.xml ." />
                </exec>
        </target>
</project>

The ANT Build File also requires a configuration file (built.properties) with some properties.

php=/usr/bin/php5
typo3path-45=/var/www/path-to-your-typo3-45-root
typo3path-47=/var/www/path-to-your-typo3-47-root
typo3path-60=/var/www/path-to-your-typo3-60-root

After both files have been created, copy them to the /workspace directory of your job. At his point, the workspace directory should not exist and you have to create it manually. Make sure you set the correct permissions, so the jenkins user has read-write permissions to that directory.

The next thing you have to do is to configure the repository, where Jenkins CI checks out the TYPO3 Extbase extension. In this article, the Extbase extension is located in a GIT Repository. Configure the Repository URL, the Branch and set the "Local subdirectory for repo (optional)" to "src". Below is a screenshot of the settings.



Now you have to configure the ANT Build File. Just insert the path to the Build File as shown below.



The ANT Build File above also contains some settings for phpcs, phpmd and phpcpd. If you want to use Checkstyle, PHP messdetection and PHP duplicate code detection, you can configure those settings in the Post-Build Actions as shown on the screenshot below.



Now your job is ready to run the first time. Actually it will fail, since the configuration process is not finished yet, but it is required to run the job, so the Extbase extension is checked out from the GIT repository.

Installing the Extbase extension and running the job

After running the job the first time, you should have the folder /src inside the jobs workspace directory. This folder contains the extension, which we now symlink to the /typo3conf/ext directory of each TYPO3 installation we created earlier. Make sure, that the name of the symlink is identical to the extension key.

The symlink ensures, that updates to the extension automatically are available in all TYPO3 installations.

Now you have to login to each TYPO3 installation and install the Extbase extension with the extension manager.

When the Extbase extension is installed, you are ready to run the job and will (hopefully) see, that your Extbase extension´s tests will run on all configured TYPO3 versions.



You can also configure the job only to use a special TYPO3 version as build target as shown below.

The above example will only run tests against TYPO3 version 4.5.

Additional notes

During the development process of an extension, the table-structure may change due to new or changed fields. Those changes are not automatically updated in the local TYPO3 installations, where the extension is installed, so you have to make sure, that you update the extension´s table structure by using the extension manager in each TYPO3 installation, if you made changes to the extension´s table structure.



Thursday, January 3, 2013

Logwatch filter for ModSecurity 2

I often use ModSecurity 2 and the OWASP ModSecurity Core Rule Set (CRS) to protect a website from potential attacks. ModSecurity 2 is able to write blocked attacks to a audit logfile, so you actually can see, which Core Rule and which data matched the attack that has been blocked. The logfile can also help you to analyze false positives, so you can modify the CRS to your needs.

As a server admin, you regulary should check the logfiles of your server. One tool to help you analyzing your server´s logfiles is Logwatch, which can send reports by e-mail with a summary of the logfile analysis. Sadly Logwatch was´nt able to analyze ModSecurity 2 audit logfiles and I could´nt find a filter for Logwatch, which fullfilled my needs.

So I wrote a filter for Logwatch, which analyzes a ModSecurity 2 audit logfile for blocked attacks and collects those information for a given time period as a report. The report is seperated by vhost, so you can have a quick overview on which attacks have been blocked on which vhost. Also the reports contains a top 10 summary of blocked IP addresses.

Here is a sample output from the filter:
--------------------- ModSecurity2 (mod_security2) Begin ------------------------

ATTACKS BLOCKED ON VHOSTS:

subdomain.domain.tld - 2 time(s)
[ip: xxx.xxx.xxx.xxx] [id: 981231 ] [msg: SQL Comment Sequence Detected.]  - 1 time(s)
[ip: xxx.xxx.xxx.xxx] [id: 981231 ] [msg: SQL Comment Sequence Detected.]  - 1 time(s)

www.site.tld - 1 time(s)
[ip: xxx.xxx.xxx.xxx] [id: 990012 ] [msg: Rogue web site crawler]  - 1 time(s)
[ip: xxx.xxx.xxx.xx] [id: 981318 ] [msg: SQL Injection Attack: Common Injection Testing Detected]  - 5 time(s)
[ip: xxx.xxx.xxx.xx] [id: 950901 ] [msg: SQL Injection Attack: SQL Tautology Detected.]  - 2 time(s)

www.anothersite.tld - 1 time(s)
[ip: xxx.xxx.xxx.xxx] [id: 958291 ] [msg: Range: field exists and begins with 0.]  - 1 time(s)

TOP 10 BLOCKED IPS:
xxx.xxx.xxx.xxx - 2 time(s)
xx.xxx.xxx.xxx - 1 time(s)
xxx.xxx.xx.xx - 1 time(s)
xxx.xxx.xxx.xx - 1 time(s)
xxx.xxx.xxx.xxx - 1 time(s)

---------------------- ModSecurity2 (mod_security2) End -------------------------

The filter has been tested with ModSecurity 2 version 2.6.0 (CRS 2.2.0) and version 2.7.1 (CRS 2.2.6)

I published the Logwatch filter for Mod Security 2 on Github, so feel free to submit change requests or bug reports.

TYPO3 - get resulting SQL from Extbase query

When you develop a TYPO3 Extbase extension, you may sometimes wonder, why a query does not return the exprected results. In this situation, it would be great if you can get the resultig SQL from TYPO3.

Actually, there are some solutions on the internet for this problem. Sadly, the one I found here does not work for me in TYPO3 6.0. The code snippet found here pointed me to the right direction. There is only one problem with the snippet, since it does not contain the whole SQL query (some parameters are filled with a questionmark).

So here is hopefully a complete solution on how to output the resulting SQL from an Extbase query.

Attention: The following code changes should only be done for development purposes on development systems.

  1. Open the file "/typo3/sysext/extbase/Classes/Persistence/Generic/Storage/Typo3DbBackend.php"
  2. Navigate to the function "GetObjectDataByQuery()"
  3. Go to the line next to "$this->replacePlaceholders($sql, $parameters, $tableName);"
  4. Insert the following code 

var_dump($sql);

This should output the resulting SQL query. Please notice, that this change can result in more than just one SQL query output, since now every query processed by Extbase is printed out.

If anyone has another solution for this problem, feel free to drop me a message.

Happy debugging :-)