Monday, December 2, 2013

Selenium tests on a Windows 8.1 machine with Internet Explorer 11

Today I tried to add an Windows 8.1 machine to my Selenium Grid and ran into some strange problems. I installed a new Windows 8.1 (32 bit) machine as a selenium node for my selenium grid. After the installation of Windows 8.1 was completed, I configured all browsers (Internet Explorer, Chrome, Firefox and Opera) and the necessary Webdrivers on the machine. Then I connected the machine to the Selenium grid and started some tests. everything seemed to run fine until I started the tests on Internet Explorer 11.

When running the tests, the browser window opened the requested URL as expected but then nothing more happened. The window did'nt even close after the test, so there must be something wrong. In the Selenium log on the machine I could see, that selenium was unable to find an element, which definitely was present on the website. After some debugging and search in the internet I found out, that Internet Explorer 11 on Windows 8.1 by default has some security settings, which must be disabled in case to run Selenium tests on the machine.

  • Disable the Protection Mode for all zones
  • Disable "Enhanced Protection Mode"

After I disabled both settings (directly in the "Internet Options") the tests were executed as expected.

Also make sure to use 32 bit versions of both the Internet Explorer driver and the operating system. I tried to get things running with the 64 bit version of Windows 8.1, but the test execution on Internet Explorer was extremely slow (it took about 4 seconds to type a character), so I ended up using the 32 bit version of Windows 8.1.

Monday, November 18, 2013

Howto debug a TYPO3 ExtBase extension with PhpStorm and XDebug


In this article I will show you step by step, how simple it is to setup PhpStorm and XDebug to debug a TYPO3 ExtBase extension. Of course you can also debug every other TYPO3 component with this setup.

Setup 

Before you can start with debugging, you need to setup XDebug, a debugging extension for you webbrowser and your PhpStorm project.

XDebug Setup
First you need to install XDebug on you system and add the following lines to your xdebug.ini (on Ubuntu 12.04 the file is located at /etc/php5/conf.d/xdebug.ini) configuration file.

xdebug.max_nesting_level = 500

xdebug.idekey = PHPSTORM
xdebug.remote_host = 127.0.0.1
xdebug.remote_enable = 1
xdebug.remote_port = 9000
xdebug.remote_handler = dbgp
xdebug.remote_mode = req

Make sure you restart your webserver after you have added updated the XDebug configuration file.

Browser Setup
Next you need to install a debug extension for you webbrowser. Since I use Google Chrome, I install the Xdebug helper, which is open source (on GitHub) and available on the Google Chrome Web Store.

PHPStorm Setup
Finally configure the server in you PhpStorm project settings like shown below.

Server Configuration

There are 2 important things to keep in mind.
  1. The host must not contain http:// 
  2. Symlinked files/directories must be mapped manually. On the screenshot above I map the typo3_src folder and the project-folder to to real location on my filesystem.
That's it - now you are ready to debug your TYPO3 ExtBase extension in PhpStorm.


Start debugging
Now it's time to start the debug session.

1. Enable your browser debug extension for the website you want to debug. In Chrome you just click on the debug icon.


2. Enable PhpStorm to listen to debug sessions by clicking on the icon shown below.


3. Set a breakpoint in your controller


4. Open the website on the server. PhpStorm now catches the incoming debug session and stops at the first breakpoint.


As you see on the screenshot above, PhpStorm stops at the breakpoint which has been set in step 3.

Happy debugging!


Troubleshooting

1. PhpStorm claims port 9000 is busy.
I had this error on an Ubuntu 12.04 development machine. Here php-fpm was installed and was listening on port 9000, so it was conflicting with XDebug. to resolve this problem I changed to port for php-fpm to listen to port 9001 instead by editing the configuration file /etc/php5/fpm/pool/www.conf

2. Debugger does'nt stop at breakpoint
When the debugger does'nt stop at the breakpoint, then it could be, that there is something wrong with the path mapping. I resolved this by adding xdebug_break(); to my code which resulted PhpStorm in stopping at the line with xdebug_break() and I could fix the path mappings at this point manually.


Tuesday, November 12, 2013

How to use content element layouts with gridelements

I often use the content element layout field in TYPO3 to add some selective CSS styling to content elements. For example, if the editor inserts a divider content element on a TYPO3 page, she/he can adjust the color of the divider by using the content element layout field like shown below.



I just define the available layouts in the page TS of the root page.

TCEFORM.tt_content.layout {
  addItems.60 = Color: Brown
  addItems.61 = Color: Gray
}

Then I use the following Typoscript to add an extra css class to the divider (e.g. brown and gray).

temp < tt_content.div
tt_content.div = CASE
tt_content.div {
  key.field = layout
  default < temp
  
  60 = COA
  60 < temp
  60.wrap = <div class="divider brown">|</div>

  61 = COA
  61 < temp
  61.wrap = <div class="divider gray">|</div>
}

Normally I globally use the defined content element layouts on other content elements too (e.g. headers), so the editor always uses the same field in TYPO3 to change the colors of content elements. You can also use the shown technique with gridelements, so the user can change the css styling of the gridelement without the need to use a flexform configuration. I just reuse the previously defined content element layout in the Typoscript of my gridelement as shown below by using a CASE on the wrap.

# ID of gridelement 
1 <  lib.gridelements.defaultGridSetup
1 {
  columns {
    # column 1
    0 < .default
    0.wrap = <div class="column1">|</div>

    # column 2
    1 < .default
    1.wrap = <div class="column2">|</div>
  }
  wrap.cObject = CASE
  wrap.cObject {
    key.field = layout
    default = TEXT
    default.value = <div class="mydefaultclass">|</div>

    60 = TEXT
    60.value = <div class="mydefaultclass brown">|</div>

    61 = TEXT
    61.value = <div class="mydefaultclass gray">|</div>
  }
}

Now the gridelement uses the TYPO3 content layout field to add an extra CSS class to its wrap.

Monday, October 28, 2013

MySQL errno: 24 after upgrade from Debian Squeeze to Wheezy

Today I upgraded one of my Debian Squeeze servers to Debian Wheezy. During the upgrade, MySQL came up with some error messages like shown below:

[ERROR] /usr/sbin/mysqld: Can't open file: './database/table1.frm' (errno: 24)
[ERROR] /usr/sbin/mysqld: Can't open file: './database/table2.frm' (errno: 24)
[ERROR] /usr/sbin/mysqld: Can't open file: './database/table3.frm' (errno: 24)
[ERROR] /usr/sbin/mysqld: Can't open file: './database/table4.frm' (errno: 24)

After some research I found out, that the cause of the problem could be, that the mysql process has too many open files. I then added the following to the my.cnf and restarted the mysql service.

[mysqld]
open_files_limit = 10000

You may adjust the limit to a value of your choice. The default limit for Debian systems is 1024, so I would recommend to set this the limit to a value greater than 1024.

After adjusting the open files limit, the mysql server was running fine again.

Tuesday, October 22, 2013

TYPO3 - PHPUnit Testing Framework tests for repositories with storagePids

When I develop a TYPO3 ExtBase extension, I often use the Testing Framework shipped with the TYPO3 extension PHPUnit. With the Testing Framework it is possible to create real testrecords in the TYPO3 database and use those records in your tests to verify everything works as expected.

In a project I faced the situation, that one of my repository methods did not find the records created with the Testing Framework. Well, after some hours of debugging I finally found the solution.

The problem

I created a new TYPO3 ExtBase extension with some tables, a controller and a view. Really nothing special about that. I then created a function in my repository which searches for some records in the database with a configurable limit. 


/**
 * Returns the latest records. 
 *
 * @param int $limit The Limit
 * @return array|\TYPO3\CMS\Extbase\Persistence\QueryResultInterface
 */
public function findLatest($limit) {
 $query = $this->createQuery();
 $query->setOrderings(array('datecreated' => QueryInterface::ORDER_DESCENDING));
 $query->setLimit((integer)$limit);
 $records = $query->execute();
 return $records;
}

Then I started to create some tests for the newly created function in the repository. One test was as following.


/**
 * Test if findLatest returns expected amout of records if limit given
 *
 * @test
 * @return void
 */
public function findLatestReturnsLimitedResult() {
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));

 $this->createTestRecords();
 $this->assertEquals(2, $this->myRepository->findLatest(2)->count());
}

In the test above, 3 records with the Testing Framework are created and I'll use the function findLatest with a given limit of 2 to test, if really 2 records are returned.

Well, actually findLatest did return 0 records. I then created some real records in the TYPO3 backend and verified, that findLatest actually works as expected, so the problem must be somewhere in my test or in the TYPO3 setup.

So I started to debug my test and first of all, I verified, that the Testing Framework really created 3 records. Yes, 3 records were created, all having the pid 0. Ok then, since the testrecords where created correctly, there seem to be something wrong with the search. I'll then debugged the resulting SQL Query from the repository function findLatest and found out, that the records were searched in pid 1. OK, so the problem seems to have something to do with the storagePids.

ExtBase, tests and storagePid(s)

When you create an ExtBase extension, the Extension Builder automatically creates some TS for you where you can set the storagePid(s) for your extension.


persistence {
  # cat=plugin.tx_myext//a; type=string; label=Default storage PID
  storagePid =
}

This storagePid can be used to set the location of your records in the TYPO3 backend. Generally every TYPO3 Extension you create with the Extension Builder makes use of the storagePid(s) and overrides them, if a startingPoint is set in the plugin.

When you run your tests in TYPO3 backend (or through your IDE), ExtBase uses the backendConfigurationManager to get the ExtBase framework configuration. In the backendConfigurationManager class, there is a function called getDefaultStoragePid(), which returns the local function getCurrentPageId(). Let's have a look at this function.

protected function getCurrentPageId() {
 $pageId = (integer) \TYPO3\CMS\Core\Utility\GeneralUtility::_GP('id');
 if ($pageId > 0) {
  return $pageId;
 }
 // get current site root
 $rootPages = $GLOBALS['TYPO3_DB']->exec_SELECTgetRows('uid', 'pages', 'deleted=0 AND hidden=0 AND is_siteroot=1', '', '', '1');
 if (count($rootPages) > 0) {
  return $rootPages[0]['uid'];
 }
 // get root template
 $rootTemplates = $GLOBALS['TYPO3_DB']->exec_SELECTgetRows('pid', 'sys_template', 'deleted=0 AND hidden=0 AND root=1', '', '', '1');
 if (count($rootTemplates) > 0) {
  return $rootTemplates[0]['pid'];
 }
 // fallback
 return self::DEFAULT_BACKEND_STORAGE_PID;
}

So, if your TYPO3 installation where you develop your ExtBase extension has a root page, then this page ID is returned. Or if you have no root page but a TS Template which is marked as root template, then the page ID of that page is returned.

The returned page ID is used in the queryFactory class to set the storagePid(s), which leads to the situation I described earlier.

Solution / Workarounds

So how do we tell TYPO3 to search for our test records in pid 0 during tests? On solution is to set the storagePid for the ExtBase Framework to 0 like shown below.

config.tx_extbase {
  persistence {
    storagePid = 0
  }
}

This solution has some problems, since it can be overridden by other extensions and also it requires manual configuration when setting up your testing environment.

You could also just disable the check for storagePid(s) (with setRespectStoragePage(FALSE)) in your tests, but I think this solution is'nt really good, since your tests then also may find records you created in the TYPO3 backend during development.

If you want to keep your tests independent, I would recommend to set the storagePid(s) directly in the setup of the tests.

/**
 * Set up for test
 *
 * @return void
 */
public function setUp() {
 $this->testingFramework = new \Tx_Phpunit_Framework('tx_myext');
 $this->fixture = $this->objectManager->get('Derhansen\\MyExt\\Domain\\Repository\\MyRepository');

 /** @var $querySettings Typo3QuerySettings */
 $querySettings = $this->objectManager->get('TYPO3\\CMS\\Extbase\\Persistence\\Generic\\Typo3QuerySettings');
 $querySettings->setStoragePageIds(array(0));
 $this->fixture->setDefaultQuerySettings($querySettings);
}

The code above sets the storage pid to 0 and now the tests should find the records created with the testing framework and tests should not interfere with other records created manually.

Sunday, September 15, 2013

Blocking brute force attacks against the TYPO3 backend login using ModSecurity

Since the last few weeks I have noticed a constantly growing amount of failed TYPO3 backend login attempts on websites I administer. It seems I'm not alone with this problem, since a few other TYPO3 users are reporting the same issues. The method always seems to be the same - someone is doing a brute force attack against the password of the TYPO3 backend user "admin". The attack seems to be automated, takes several hours and comes from various IP addresses and from different countries in the world.

If you do not have any TYPO3 backend account with the username "admin", you could just ignore the case that someone is trying to hack your TYPO3 installation. Well, since I have configured a [BE][warning_email_addr] my mailbox gets flooded with thousands of emails about failed TYPO3 login attempts.

But what is the best way to handle and block those attacks? When I faced the first attack, I just denied the IP address of the attacker into the local .htaccess file of the TYPO3 installation. This method is implemented very quick, but always requires manual steps and is only valid for a given IP address. In the austrian TYPO3 forum I have read about a TYPO3 extension, which extends the TYPO3 backend login and is able to automatically blacklist IP addresses for failed login attempts. This idea seems to be good in general, but when you administer just more than one TYPO3 installation, the installation and configuration process of the extension can be very time consuming. I finally ended up with the solution, that once again ModSecurity seems to be the best way to handle this kind of attacks to TYPO3. During research I found an article about brute force protection using ModSecurity. Sadly the rules did'nt fit out of the box into my setup, so I had to create my own ruleset.

Below follows a ModSecurity ruleset, which blocks the IP address of an attacker for 10 minutes, if there were more than 5 failed TYPO3 authentication attempts from that IP address. Please note, that the ruleset only was tested with ModSecurity 2.7.x together with the OWASP ModSecurity Core Rule Set.

<IfModule mod_security2.c>

    #
    # TYPO3 backend login brute force protection
    #
    <Location "/typo3/index.php">
        # Turn on security engine (if disabled)
        SecRuleEngine On

        # Deny IP address if too many TYPO3 authentication failures
        SecRule IP:bf_block "@eq 1" \
        "id:'5000102', \
        msg:'IP address temporarily blocked due to too many TYPO3 authentication failures', \
        phase:2, \
        log, \
        deny"

        # Check for TYPO3 authentication failure and increment counter
        SecRule RESPONSE_BODY "LOGIN_ERROR### begin" \
        "id:'5000100', \
        msg:'Failed TYPO3 authentication attempt', \
        phase:5, \
        t:none, \
        setvar:IP.bf_counter=+1, \
        expirevar:IP.bf_counter=3600, \
        nolog, \
        pass"

        # Check for too many failures from a single IP address
        SecRule IP:bf_counter "@gt 5" \
        "id:'5000101', \
        msg:'Too many TYPO3 authentication failures from IP address', \
        phase:5, \
        t:none, \
        setvar:IP.bf_block, \
        setvar:!IP.bf_counter, \
        expirevar:IP.bf_block=600, \
        log, \
        pass"

    </Location>

</ifModule>

The ruleset creates a counter (IP.bf_counter) for each IP address, when a failed TYPO3 login is monitored. A failed login is identified by the string "LOGIN_ERROR### begin" in the response body. I don't check for the occourence of the string "Your login attempt did not succeed", since some TYPO3 installations may have a e.g. german or danish backend login and within this the resulting error message will be translated and the rule would not match.

I've decided not to log each failed login attempt, since it fills up the ModSecurity audit log with the complete response from the TYPO3 backend login. If the counter is greater than 5, the counter is unset and a new variable (IP.bf_block) with a lifetime of 600 seconds is defined. Depending on this counter, the IP address gets blocked with a 403 error.

If you have blocked your own IP address during to too many failed login attempts, you have to unset the counter variable by modifying your ruleset. This is not very practical, so another possibility to unset the variable is to simply delete the file /tmp/ip.pag - this is where ModSecurity stores all variables in a default setup. Note, that deleting the file will remove all variables set by ModSecurity. Also I don't know, if deleting the file will result in any other unwanted side-effects, so be carefull with that.

Besides using ModSecurity to protect TYPO3 websites against known and unknown attacks, I also recommend using a Yubikey to add two-factor authentication to TYPO3 admin accounts.


Wednesday, August 7, 2013

TYPO3 Error "Sorry, but an error occurred while connecting to the server." in Extension Manager

After the latest security updates for some TYPO3 extensions I was about to update those extension in a TYPO3 installation. After opening the extension manager, TYPO3 (version 4.7) did show the following error message:

Connection Problem
Sorry, but an error occurred while connecting to the server. Please check your network connection.

First I thought, there was a problem with some TYPO3 cache files, but after cleaning the typo3temp folder, the problem still occured. I then looked into to Webservers Error log and found the following logentry.

PHP Fatal error:  Cannot unset string offsets in /path/to/typo3/sysext/em/classes/tools/class.tx_em_tools.php on line 382

After some research I finally found the solution to the problem, which is the result of a bug in the TYPO3 Extension Manager, which has been fixed for some days ago.

I manually opened the file "ext_emconf.php " of every installed TYPO3 extension and checked, that the constraints-array was correct. In my case it was powermail which had a wrong entry in "ext_emconf.php".

I had to change:
'conflicts' => '',
to this:
'conflicts' => array(),

After I fixed the wrong entry, the extension manager worked fine again.