Standalone unit- and functional tests for ExtBase extensions in TYPO3 6.2

The TYPO3 core team did a lot of work in the last weeks to simplify unit- and functional testing in TYPO3, so now everything about testing TYPO3 core and TYPO3 extensions has become much more clear and straightlined. In this article I will describe how I switched the unit- and functional tests in one of my TYPO3 extensions from cli_dispatch to standalone phpunit tests.

Initial situation

I am working an an event management and registration extension for TYPO3. Since I always work with unit- and functional tests and a CI environment, I've setup my IDE PHPStorm to execute unit- and functional tests as described in my old blogpost and I also integrated the extension in Travis CI and Scrutinizer CI. Since the extension only should be compatible with TYPO3 6.2+, I already used the new functional tests introduced in TYPO3 6.2 instead of the functional testing framework which comes with the TYPO3 extension phpunit.

Standalone unit tests for PHPStorm

Helmut Hummel wrote an excellent article about how to execute TYPO3 Unit tests in PHPStorm. I've setup my unit tests as descibed in Helmut's article, but with one difference. Instead of configuring the unit test bootstrap file as described in step 5, I created a PHPUnit configuration file, which holds all PHPUnit configuration and which later on also will be used on Travis CI for test execution.

Unit tests use an alternative configuration file

The configuration file for PHPUnit is like shown below.

<phpunit
        backupGlobals="false"
        backupStaticAttributes="false"
        bootstrap="../../../../../typo3/sysext/core/Build/UnitTestsBootstrap.php"
        colors="true"
        convertErrorsToExceptions="true"
        convertWarningsToExceptions="true"
        forceCoversAnnotation="false"
        processIsolation="false"
        stopOnError="false"
        stopOnFailure="false"
        stopOnIncomplete="false"
        stopOnSkipped="false"
        verbose="false">

    <testsuites>
        <testsuite name="EXT:sf_event mgt">
            <directory>../Unit/</directory>
        </testsuite>
    </testsuites>
</phpunit>

After I had setup everything, I executed my unit tests, which actually did not run. With my old setup and within the TYPO3 extension PHPUnit, the tests executed successfully. After some hours of debugging, I fixed the failing tests. I had to do more mocking inside my tests, so they really could run independent from a working TYPO3 installation.

Standalone functional tests for PHPStorm

The configuration for functional tests in PHPStorm is similar to the unit test setup with some small differences. You must switch on PHPUnit process isolation and you must configure database settings, if you do not run the tests in an working TYPO3 CMS installation. Documentation about functional tests in TYPO3 can be found in the TYPO3 wiki about functional testing.

Since I want to run the functional tests without a working TYPO3 CMS installation and also on Travis CI, I configured a PHPUnit configuration file and added environment variables for database access to the PHPStorm Run/Debug configuration.

Environment variables for functional tests

The PHPUnit configuration file is like shown below:

<phpunit
        backupGlobals="false"
        backupStaticAttributes="false"
        bootstrap="../../../../../typo3/sysext/core/Build/FunctionalTestsBootstrap.php"
        colors="true"
        convertErrorsToExceptions="true"
        convertWarningsToExceptions="true"
        forceCoversAnnotation="false"
        processIsolation="true"
        stopOnError="false"
        stopOnFailure="false"
        stopOnIncomplete="false"
        stopOnSkipped="false"
        verbose="false">

    <testsuites>
        <testsuite name="EXT:sf_event mgt">
            <directory>../Functional/</directory>
        </testsuite>
    </testsuites>
</phpunit>

Note, that I've set processIsolation to true.

After I had setup the functional test configuration, my functional tests also did not work. Again this was caused incomplete mocking and also by direct usage of the ExtBase objectManager in my tests. After I completed the mocking and removed all direct usages of the objectmanager in the tests, the functional tests were executed successfully in PHPStorm.

Unit and functional tests on Travis CI and code analysis on Scrutinizer CI

My TYPO3 extension was already integrated in Travis CI and Scrutinizer CI. Since I switched from test execution through cli_dispatch to standalone unit test execution, I had to modify my Travis CI configuration to execute tests directly through phpunit.

Both unit- and functional tests can be executed on commandline with the following commands:

phpunit -c typo3conf/ext/sf_event_mgt/Build/UnitTests.xml
phpunit -c typo3conf/ext/sf_event_mgt/Build/FunctionalTests.xml

Please note, that the test execution as shown above only works if the commands are executed from within a directory with a full TYPO3 CMS directory structure. In case of the functional tests, you must also set environment variables for Database access.

To get the test execution working on Jenkins CI, you just have to clone the TYPO3 core and create the TYPO3 directory structure (fileadmin, uploads and typo3conf/ext) and finally move your extension to typo3conf/ext/. After that, you're ready to execute both unit- and functional tests on Jenkins CI.

My final .travis.yml file is like shown below:

language: php
php:
  - 5.3
  - 5.4

env:
  - DB=mysql TYPO3_BRANCH=master COVERAGE=0

matrix:
  include:
    - php: 5.5
      env: DB=mysql TYPO3_BRANCH=master COVERAGE=1

notifications:
  email:
    - derhansen@gmail.com

before_script:
  - cd ..
  - git clone --single-branch --branch $TYPO3_BRANCH --depth 1 https://github.com/TYPO3/TYPO3.CMS.git typo3_core
  - mv typo3_core/* .
  - sudo apt-get install parallel
  - composer self-update
  - composer install
  - mkdir -p uploads typo3temp typo3conf/ext
  - mv sf_event_mgt typo3conf/ext/

script:
  - >
    if [[ "$COVERAGE" == "0" ]]; then
      echo;
      echo "Running unit tests";
      ./bin/phpunit --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/UnitTests.xml
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      echo "Running unit tests";
      ./bin/phpunit --coverage-clover=coverage.clover --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/UnitTests.xml
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      export typo3DatabaseName="typo3";
      export typo3DatabaseHost="localhost";
      export typo3DatabaseUsername="root";
      export typo3DatabasePassword="";
      find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' | parallel --gnu 'echo; echo "Running functional test suite {}"; ./bin/phpunit --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/FunctionalTests.xml {}'
    fi
  - >
    if [[ "$COVERAGE" == "1" ]]; then
      echo;
      echo "Uploading code coverage results";
      wget https://scrutinizer-ci.com/ocular.phar
      cp -R typo3conf/ext/sf_event_mgt/.git .
      php ocular.phar code-coverage:upload --format=php-clover coverage.clover
    fi

In order to get functional tests to execute successfully, you must provide TYPO3 database credentials and the TYPO3 database name as shown in the config above.

The Travis CI configuration file already includes code coverage analysis through Scrutinizer CI. In the last part of the configuration, the code coverage results are uploaded to Scrutinizer.

Merging unit- and functional test code coverage data

[EDIT 28.06.2014] It seems I have missed one thing regarding code coverage. With cli_dispatch test execution, both unit- and functional tests run at the same time resulting in one file with code coverage data (if code coverage is switched on). With standalone unit- and functional tests, tests are seperated in at least 2 runs. The config for Travis CI I posted above just handles code coverage for unit tests and not for funtional tests. Also, the Travis CI config runs functional tests in parallel to speed up functional test execution. To include code coverage data from both unit- and functional tests, I changed to Travis CI config as following for the functional test execution.

find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' | parallel --gnu 'echo; echo "Running functional test suite {}"; ./bin/phpunit --coverage-clover={}functionaltest-coverage.clover --colors -c typo3conf/ext/sf_event_mgt/Tests/Build/FunctionalTests.xml {}'

With this change, phpunit creates one file with code coverage data for each functional test suite. Keep in mind, that this will really slow down functional test execution.

And for the upload of code coverage data to Scrutinizer CI I changed the following:

find . -wholename '*/typo3conf/ext/sf_event_mgt/Tests/Functional/*Test.php' -exec php ocular.phar code-coverage:upload --format=php-clover {}functionaltest-coverage.clover \;

This uploads each file with code coverage data for functional tests to scrutinizer CI. The Scrutinizer CI configuration must now be changed as shown below.

external_code_coverage:
    timeout: 700
    runs: 3

With this change, Scrutinizer CI waits for 3 code coverage data uploads and merges them, so finally the code analysis end up with code coverage data from all unit- and functional test. There is one downside with this settings. The number of runs must be changed manually depending on the amount of test suites with code coverage data you have in your project. But finally, Scrutinizer CI shows the same code coverage as PHPStorm.

Code coverage in PHPStorm and Scrutinizer CI both show 92%

Conclusion

The switch from cli_dispatch to standalone test execution for ExtBase extensions is'nt really complicated. Test setup for unit- and functional tests has become much more easy and clear now and I think (and hope) that standalone unit- and functional tests will reduce the barriers for TYPO3 extension developers to actually create TYPO3 Extension with good test coverage.

I would like to thank all people in the TYPO3 community who made standalone tests possible. I really appreciate the work that has been done.

TYPO3 - How to prevent empty FlexForm values from getting saved

When you create an ExtBase extension you can use a FlexForm to enable the user to change some of the TypoScript settings you define for the extension directly in the plugin settings. As an example, you could assume that your ExtBase extension has the following TypoScript settings:

plugin.tx_myext {
  settings {
    myFirstSetting = A value
    mySecondSetting = Another value
  }
}

If you add a FlexForm to your extension with two input fields, where the user can change both "myFirstSetting" and "mySecondSetting", then you will be able to access the user configured settings in your extension controllers through an array which is available in $this->settings. But - if the user just configures one of the two settings resets an already set setting in FlexForm to an empty value, then the second setting will be saved with an empty value and you have to merge the TypoScript and FlexForm settings manually in your controller and manually control, which setting overrides each other.

There has been a lot of discussion about this problem in the TYPO3 mailinglist and on forge and as far as I could find out, a solution is'nt implemented very easy, since you have to decide for each individual field, if you want to let an empty Flexform setting override a predefined TypoScript setting. Georg Ringer did a lot of work on handling this override-behaviour in his great News-Extension, where you define a TypoScript setting named "overrideFlexformSettingsIfEmpty", which contains all fields which should be overridden by TypoScript values, if the FlexForm contains empty values.

There is nothing wrong with the solution show in the News-Extension and in the ExtBase Wiki, but I took a different approach and tried to find out if it is possible to prevent empty FlexForm values from getting saved to the database when the plugin configuration is saved, so you don't have to do the merge/override manually.

Outgoing from the example described above the table tt_content contains a record for the ExtBase plugin, which holds the FlexForm configuration as XML. For the case, that the user just configured one of the two settings by FlexForm, the content in the field pi_flexform contains the following:

<?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<T3FlexForms>
    <data>
        <sheet index="sDEF">
            <language index="lDEF">
                <field index="settings.myFirstSetting">
                    <value index="vDEF">A value</value>
                </field>
                <field index="settings.mySecondSetting">
                    <value index="vDEF"></value>
                </field>
            </language>
        </sheet>
    </data>
</T3FlexForms>

You will notice, that the setting mySecondSetting contains an empty value. This is the field I want to remove from the XML when saving the plugin configuration, since the setting should be overridden by the default value configured in TypoScript.

To do so, I used the TYPO3 core hook processDatamap_postProcessFieldArray, which allows me to modify the values being written to the database.

In ext_localconf.php I define the class to use for the DataHandler hook.

// DataHandler hook
$GLOBALS['TYPO3_CONF_VARS']['SC_OPTIONS']['t3lib/class.t3lib_tcemain.php']['processDatamapClass']['NAMESPACE.' . $_EXTKEY] = 'NAMESPACE\MyExt\Hooks\DataHandlerHooks';

Next I created the class containing the new processDatamap_postProcessFieldArray function. The contents of the file is as following.

<?php
namespace NAMESPACE\MyExt\Hooks;

/***************************************************************
 *
 *  Copyright notice
 *
 *  (c) 2014 Torben Hansen <derhansen@gmail.com>
 *
 *  All rights reserved
 *
 *  This script is part of the TYPO3 project. The TYPO3 project is
 *  free software; you can redistribute it and/or modify
 *  it under the terms of the GNU General Public License as published by
 *  the Free Software Foundation; either version 3 of the License, or
 *  (at your option) any later version.
 *
 *  The GNU General Public License can be found at
 *  http://www.gnu.org/copyleft/gpl.html.
 *
 *  This script is distributed in the hope that it will be useful,
 *  but WITHOUT ANY WARRANTY; without even the implied warranty of
 *  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
 *  GNU General Public License for more details.
 *
 *  This copyright notice MUST APPEAR in all copies of the script!
 ***************************************************************/

use TYPO3\CMS\Core\Utility\GeneralUtility;

/**
 * Hooks for DataHandler
 */
class DataHandlerHooks {

 /**
  * Checks if the fields defined in $checkFields are set in the data-array of pi_flexform. If a field is
  * present and contains an empty value, the field is unset.
  *
  * Structure of the checkFields array:
  *
  * array('sheet' => array('field1', 'field2'));
  *
  * @param string $status
  * @param string $table
  * @param string $id
  * @param array $fieldArray
  * @param \TYPO3\CMS\Core\DataHandling\DataHandler $reference
  *
  * @return void
  */
 public function processDatamap_postProcessFieldArray($status, $table, $id, &$fieldArray, &$reference) {
  if ($table === 'tt_content' && $status == 'update' && isset($fieldArray['pi_flexform'])) {
   $checkFields = array(
    'sDEF' => array(
     'settings.myFirstSetting',
     'settings.mySecondSetting'
    ),
   );

   $flexformData =  GeneralUtility::xml2array($fieldArray['pi_flexform']);

   foreach ($checkFields as $sheet => $fields) {
    foreach($fields as $field) {
     if (isset($flexformData['data'][$sheet]['lDEF'][$field]['vDEF']) &&
      $flexformData['data'][$sheet]['lDEF'][$field]['vDEF'] === '') {
      unset($flexformData['data'][$sheet]['lDEF'][$field]);
     }
    }

    // If remaining sheet does not contain fields, then remove the sheet
    if (isset($flexformData['data'][$sheet]['lDEF']) && $flexformData['data'][$sheet]['lDEF'] === array()) {
     unset($flexformData['data'][$sheet]);
    }
   }

   /** @var \TYPO3\CMS\Core\Configuration\FlexForm\FlexFormTools $flexFormTools */
   $flexFormTools = GeneralUtility::makeInstance('TYPO3\\CMS\\Core\\Configuration\\FlexForm\\FlexFormTools');
   $fieldArray['pi_flexform'] = $flexFormTools->flexArray2Xml($flexformData, TRUE);
  }
 }

}

In the function I have defined an array which holds sheet- and fieldnames, which should be removed if empty. Next I iterate through all sheets and fields configured in $checkFields to check, if they really are empty and if so, I just remove them from the array.

You may notice, that I converted the given XML structure to an array, so I can easily iterate through all sheets/fields. When everything is finished, I use the TYPO3 FlexFormTools to convert the array back to FlexForm XML.

After implementing the new function by the hook, the resulting FlexForm does not contain fields with empty values as defined in my function and I can now use $this->settings in my controller (or retrieving the settings through the configurationManager) without the need to manually merge/override TypoScript and FlexForm settings.

Please note, that I just tested the above with TYPO3 6.2.