TYPO3 - Using Fluid StandaloneView to render localized templates in a scheduler task (part 2)

Today I found, that the technique I described in my blogpost about rendering localized Fluid templates in a scheduler task does not work as expected. As long as you want to switch the language used to render the templates only one time, then you're fine. But as soon as you want to switch the language several times (e.g. sending multiple localized e-mails in one request), then you experience that only the first language switch is respected.

The root cause for this is the TYPO3 LocalizationUtility, which includes the static translate() method that is used return translated language labels from XLF/XML language files. The LocalizationUtility is not designed to handle multiple language switches in one request, so at this point I'm stuck.

I order to keep things simple for the integrator (use one e-mail template with language labels to send out localized e-mails in a scheduler task), I decided to create an own viewHelper which uses a modified version of the LocalizationUtility. The modified version of the LocalizationUtility does not contain any static variables or methods and can be used with dependency injection. You can find the code in this GitHub repository.

Usage

In my Fluid StandaloneView templates I now use my own translate viewHelper as shown below.

The viewHelper uses the LocalizationService (which is the TYPO3 LocalizationUtility with some small modifications - e.g. removed all "static" declarations). As a result of this, all functionality of the original viewHelper / TranslationUtility are remained (e.g. overwriting language labels with TypoScript)

I extended the original demo extension from my first blogpost so it makes use of the new viewHelper / LocalizationService. The Extension now includes a form, which renders multiple Fluid StandaloneViews in one request and the language is switched for each individual StandaloneView (see result-section in the screenshot below)

The demo extension also includes a command controller, which includes a command that also renders multiple standaloneViews in one request (see screenshot below)

If I don't find any major problems, I will make use of this technique to send out multilingual e-mails in a scheduler task in my Event Management Extension.

Final notice

The technique shown should only be used in the backend context of TYPO3 when you want to render multilingual Fluid StandaloneViews in one request. I'm not very happy with the approach of "just" taking some code from the TYPO3 core and adapting it to my needs, since this is not always a clean solution and it may include some drawbacks.

TYPO3 - Using Fluid StandaloneView to render localized templates in a scheduler task (part 1)

tl;dr: If you want to use Fluid StandaloneView to render a template in a given language from the backend context (e.g. scheduler task), make sure you set the language in $GLOBALS['BE_USER']->uc['lang']

If you want to switch the language several times in one request, please read on in part 2 of this blogpost.

Problem description

For my Event Management TYPO3 Extension I am developing a feature, where I need to send out localized e-mails to users. The e-mail content is created using a Fluid StandanloneView in a TYPO3 scheduler task. Actually, this doesn't work out of the box.

In TYPO3 you can use Fluid StandaloneView to render HTML based content, which e.g. can be added to body field of an HTML e-mail. The TYPO3 Wiki contains some code snippets on how this can be processed. If you use the provided code snippets in your ExtBase extension, it works fine for localized content as long as you use it in the frontend context of TYPO3 (e.g. website user requests an action). 

When you use Fluid StandaloneView from the backend context to generate localized content, the output differs from what you would expect. When you are logged in as a TYPO3 backend user, then the localized content is rendered with the language the backend user has chosen in the user setup. In addition, if you render a Fluid StandaloneView from a commandController, then the language is ignored completely.

Solution

After digging into the problem, I found a simple way to control the language which is used during the rendering process of the Fluid StandaloneView. For the backend context, you just need to set a language (ISO2 code, lowercase) in the $GLOBALS['BE_USER']->uc['lang'] setting. Below follows a code example which shows how this is done (see line 12).



Note, that you also have to make sure, that the Fluid StandaloneView knows, from which extension the localizations get loaded. This is set in line 24.

The shown method works in both TYPO3 6.2 and the current TYPO3 master. I provided a little demo extension, which I used for testing purposes. It contains a backend module and a command controller which renders a Fluid StandaloneView with a given language.

Technical background

The f:translate viewHelper uses the TYPO3 LocalizationUtility to render localized content. For the frontend context, all labels are translated depending on the language set in $GLOBALS['TSFE']->config['config']['language']. For the backend context, the language set in $GLOBALS['BE_USER']->uc['lang'] is used.

TYPO3 6.2 - How to create a custom header layout and keep the alignment field working

tl;dr: When you add custom headers layouts to TYPO3, always keep in mind to retain original functionality of the alignment select field as shown at the end of this blogpost (solution 1 or solution 2), so editors don't get confused.


If you use CSS Styled Content in TYPO3, there are many tutorials on the internet showing how to configure custom headers. So why do I write another article about this you may ask. Well, in a project I "just" needed to create a custom header layout, where the user could use the TYPO3 "alignment"-field to select the alignment of the custom header. The resulting HTML should include the class "custom_class" like shown below:

<h1 class="custom_class">TYPO3 Content Management System</h1>

After some time of unsuccessful research, I finally ended up with the solutions shown below (at the end of the blogpost).

The standard solution (not recommended)

Besides many non-working or outdated solutions (e.g. {register:headerStyle} has been deprecated with TYPO3 4.7 I guess) you may find solutions which instruct you to do as following:

1. Define a custom header style in Page TSConfig.

TCEFORM.tt_content.header_layout {
    addItems.10 = Header with custom layout
}

2. Add TypoScript to inherit the header style from lib.stdheader.10.1 and finally overwrite the dataWrap attribute.

lib.stdheader.10.10 < lib.stdheader.10.1
lib.stdheader.10.10 {
  dataWrap = <h1 class="custom_class">|</h1>
}

What is wrong with this solution? Well, did you ever try to set the header alignment for a custom header, which is configured as shown above?

It does'nt work! The editor may get confused and claim, that TYPO3 is not working correct, because the alignment selection in the backend doesn't affect the frontend.

Why is the alignment not working?

In order to remain the alignment that is set by CSS Styled Content, we need to understand how CSS Styled Content technically sets it. When the editor sets the alignment of a header to "Center", the resulting HTML header tag will include the CSS class "csc-header-alignment-center", which controls the header alignment in the frontend.

What happens "under the hood" is as following: A look into the TypoScript of CSS Styled Content from TYPO3 6.2.14 shows, that header alignments are loaded into a register called headerClass, which is used in the dataWrap attribute to set all classes for the resulting header. And this is the reason, why the example above does not work as expected. Overwriting the dataWrap attribute removes the alignment set by CSS Styled Content.

Solution 1 - extending the headerClass register

As we now know, that we need to include the register headerClass in the dataWrap attribute to keep the alignment, we just extend the value of the headerClass register.

1. Define a custom header style in Page TSConfig.

TCEFORM.tt_content.header_layout {
    addItems.10 = Header with custom layout
}

2. Load the original register into a new register (headerClassFor10) and add a new TEXT content object to the existing COA (content object array)

lib.stdheader.3.headerClassFor10 < lib.stdheader.3.headerClass
lib.stdheader.3.headerClassFor10.cObject {
    5 = TEXT
    5.noTrimWrap = |custom_class ||
}
lib.stdheader.10.10 < lib.stdheader.10.1
lib.stdheader.10.10.dataWrap = <h1{register:headerClassFor10}>|</h1>

The resulting header tag now includes the CSS classes set by CSS Styled Content and the new class "custom_class".

<h1 class="custom_class csc-header-alignment-center">TYPO3 Content Management System</h1>

If you have several custom header layouts, make sure, that you define a new register for each custom header layout (e.g. headerClassFor10 and headerClassFor11 in case you have defined header 10 and 11).

Solution 2 - extending the dataWrap with a span-tag

If you don't like solution 1, you can use the solution shown below, which just adds a HTML span tag inside the header tag (which is valid for both HTML4 and HTML5). This time, I just show you how the TypoScript for the custom header should look like (you've already seen the Page TSConfig 2 times).

lib.stdheader.10.10 < lib.stdheader.10.1
lib.stdheader.10.10 {
  dataWrap = <h1{register:headerClass}><span class="custom_class">|</span></h1>
}

The resulting HTML output for this header is as following:

<h1 class="csc-header-alignment-center"><span class="custom_class">TYPO3 Content Management System</span></h1>

Faster builds of my TYPO3 extensions on Travis CI

tl;dr If your Travis CI project was created before 01.01.2015 and you think your builds must run faster, switch to container based builds by adding sudo: false and configuring a cache for composer dependencies in your .travis.yml file.

Nearly all my Open Source TYPO3 extensions are built on Travis CI. This runs really fine and I am very happy with the service Travis CI offers for free if your project is Open Source. What I did'nt know was, that Travic CI also offers the possibility to use container based builds (saw it first in this commit for the TYPO3 master). This feature was announced in the end of 2014 and has some advantages over the Travis CI standard infrastructure. Container based builds do have more resources (2 cores vs. 1.5 cores and 4 GB Ram vs. 3 GB ram), better network capacity and can use a cache for dependencies (e.g. composer dependencies).

My extension sf_event_mgt has a lot of functional tests and since I also collect code coverage data, builds did run very long (15-30 minutes, worst result was more than 48 minutes) on the Travis CI standard infrastructure. Since my Travis CI project was created before the 1st of january 2015, my builds were automatically built using the old infrastructure. I therefore tried to use the container based infrastructure to see if my builds would run faster.

In order to force your build to run on the container based infrastructure, you just have to add the following to your .travis.yml file.

sudo: false

cache:
  directories:
    - $HOME/.composer/cache

The sudo: false option tells Travis CI to use the container based infrastructure and the cache configuration sets the cache-directory, which will persist during builds.

When I started the first build, I was really amazed about how fast the builds started and how much faster builds were completed. There was only one problem, which showed up in the logs.

  - Installing  symfony/yaml  ( v2.7.1 )
    Downloading:  connection...      Failed to download symfony/yaml from dist: Could not authenticate against github.com 
    Now trying to download from source 
  - Installing symfony/yaml  ( v2.7.1 )
    Cloning 9808e75c609a14f6db02f70fccf4ca4aab53c160

In order to fix the failing download of dependencies from github.com, you must create an OAUTH token for your GitHub repository and add it to the .travis.yml. I therefore created an OAUTH token (which just has read access to my public repositoried) in my GitHub account. Next I added the OATH token to the .travis.yml file. As the token should and must be kept secret, Travis CI offers the possibility to encrypt sensitive data. For my TYPO3 extension sf_event_mgt I did as shown below:

sudo gem install travis
travis encrypt GITHUB_COMPOSER_AUTH=my-github-oauth-token -r derhansen/sf_event_mgt

Next I added the resulting encrypted key to my .travis.yml and configured composer to use the GitHub OAUTH token if available.

env:
  global:
    secure: my-encrypted-key

before_script:
  - if [ "$GITHUB_COMPOSER_AUTH" ]; then composer config -g github-oauth.github.com $GITHUB_COMPOSER_AUTH; fi
  - composer install

After restarting the build, the error did'nt show up in the logs of my Travis CI build any more. My complete configuration can be found in this gist.

Finally, the switch to the container based builds seems to have reduced the average build time by about 50% and seems to run more stable than before.

TYPO3 7.3 - Extending the backend login form by using the new backend login form API

Back in january 2015 I added support for TYPO3 7.1 to the YubiKey TYPO3 extension and raised the extension compatibility level to be compatible with TYPO3 version 6.2.0 up to 7.99.99. When TYPO3 7.2 was released, I found out, that the YubiKey extension did'nt work any more with the new TYPO3 version, because the TYPO3 backend login has been rewritten. I debugged into the issue and found out, that the new backend login did not respect the template set in $TBE_STYLES['htmlTemplates'][$tmplPath].

It seemed I was not the only one having this problem, as another user already reported the same issue on forge. So with TYPO3 7.2+, you can't use $TBE_STYLES['htmlTemplates'][$tmplPath] to add own fields to the backend login form, since it is not supported any more.

In this article I describe, how the new backend login form API is integrated in the OpenID extension and I also describe, how I extended the username/passwors login provider to add an additional field to the default backend login form.

The new login form API in TYPO3 7.3

In order to get the missing functionality of extending the backend login form back, the TYPO3 core developers added the backend login form API and also did a lot of great work refactoring the backend login. The backend login form API uses login providers to enable extension developers to add an own backend login form for e.g. an own authentication service. As an example, lets have a look at how the OpenID authentication service is integrated.

The OpenID authentication service registers a new login provider by adding the following code in the file ext_localconf.php


New login providers must add a new key to the array $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['backend']['loginProviders']. The new key must be an unix timestamp (unique for all login providers). The following array has the following keys:

• provider: The name of the login provider
• sorting: Defines the sorting order in case of there are multiple login providers
• icon-class: A class which defines the icon for the login provider
• label: the label for the login provider 

Having the OpenID authentication service installed, the login provider of the OpenID extension adds a link to the backend login (see screenshot below).

TYPO3 7.3 backend login form showing the OpenID link

When you follow the "Login with OpenID" link, you come to an own login form showing the OpenID field (screenshot below).

TYPO3 7.3 OpenID login form

From there, you have the possibility to switch back to the username and password login. This is, because also the username and password login has an own login provider.

Lets have a look at the contents of the login provider class.

As you can see, there is not much magic there. The OpenidLoginProvider implements the LoginProviderInterface and has only one method (render), which is called in the LoginController of the TYPO3 backend login form. In case of the OpenID extension, you see that setTemplatePathAndFilename is used to set the Fluid template for the OpenID login form.

The template is kept really simple. It uses the layout "Login" (this is required) and has a section called "loginFormFields" (also required), which contains the OpenID login field.

Integration of the backend login form API to the YubiKey extension

For the YubiKey extension, I could have used the same technique as the OpenID extension does, but since some TYPO3 installations may require each user to use YubiKey two-factor authentication, it would be not very user-friendly to let a user manually switch to the YubiKey backend login provider when opening the TYPO3 backend login form.

In order to add the YubiKey field to the backend login, I extended the existing username and password login provider with my own login form. In the ext_localconf.php, I added the following code.

Next I created the YubiKey login provider, which calls the parent login provider (username and password) and sets the path to the Fluid template which contains the both the username and password fields and the YubiKey login field.

Note, that I do not implement the LoginProviderInterface, but extend the UsernamePasswordLoginProvider.

Finally, after the YubiKey extension is installed and activated, the TYPO3 backend login now again has an additional field for the YubiKey two factor authentication (screenshot below)

TYPO3 7.3 backend login with the additional YubiKey field

As TYPO3 7.3 has been released yesterday, extension authors ,who created own authentication providers for the TYPO3 backend, can now start to implement the new backend login API to their extensions.

The latest version 1.1.0 of the TYPO3 YubiKey extension uses the new backend login form API and is compatible with TYPO3 6.2 and TYPO3 7.x (except 7.2)

Laravel 5 and queues using redis - get amount scheduled jobs in queue

I'm currently digging into Laravel 5 and am using the queue component to put time consuming tasks in a queue, so they can be processed in the background. For performance reasons, I switched from using the database queue driver to use the redis queue driver. My application has a dashboard, which shows the amount of scheduled jobs for each queue in the queue system. Using the database queue driver, I used the following code:

Using the redis queue driver, Laravel creates a list for each queue on the redis server, so you can easily use the functions from the predis package, to interact with the redis server. To get the amount of jobs in a queue with the redis queue driver, I use the code as shown below.


I also needed the possibility to remove all jobs in a specific queue at once in my application. To do so, I used the code shown below.

Upgrading to Redmine 3.0.1 on Debian Wheezy

Today I wanted to upgrade my Redmine installation from Redmine 2.6.x to the new version Redmine 3.0.1. Since I already have processed many Redmine updates I thought this update would be easy as always, but certainly I ran into an unexpected problem. After I followed all steps from the upgrade manual I restarted the apache2 process, so mod_passenger reloads the Redmine application. After that, Redmine did'nt start and showed the following error message:

undefined method `page_cache_directory' for ActionController::Base:Class

After doing some Google search, I found this article on stackoverflow, which pointed me to the right direction. Actually, Debian Wheezy comes with an pretty old version of Phusion Passenger (3.0.x), so I performed the following steps and installed a newer version (4.0.x) of Phusion Passenger from Wheezy Backports.

First I added Wheezy Backports to my apt sources file /etc/apt/sources.list

deb http://http.debian.net/debian wheezy-backports main

Next I used apt-get update to update the local apt repositories and finally I installed Phusion Passenger from the Wheezy Backports with the following command

apt-get -t wheezy-backports install "libapache2-mod-passenger"

After this, I restarted the apache2 process and Redmine 3.0.1 started successfully.

New version of sf_event_mgt released

The new version 1.0.0 of sf_event_mgt has been released. I decided not to increase the version number to 0.5.4  but to start with 1.0.0 now, as the extension runs stable in some projects I maintain and also there is no need to keep the extension in alpha/beta state any more. The new version contains some small bugfixes in the template files and also comes with some nice new features.

The double-opt in feature for the confirmation e-mail can now be configured to be optional (settings.registration.autoConfirmation). If the new setting is activated, registrations by participants will automatically be confirmed.

For locations it is not possible to add latitude, longitude and a description. This can be useful, if the location should be shown on a map.

One feature, that has been requested by multiple users is the iCalendar download. For events, it is now possible to add a link, which downloads the event details as an iCalendar file. To keep things flexible, the iCalendar file is rendered through Fluid, so it is also possible to add own fields to the iCalendar file.

The last new major feature is the possibility to create a registration for multiple participants at once. There is a maximum amount of registrations, which can be configured for each event, so a user is not able to book all available places for an event at once.

All new features are also described in the manual of the extension. Additional details for this release can be found here.

Thanks to all users, who gave me feedback end new ideas for the extension!

If you find a bug and want to request a feature, please use the issue tracker on GitHub

TYPO3 6.2 - Problems with GIFBUILDER and filenames containing umlauts

After migrating a TYPO3 4.5 website to TYPO3 6.2 I had some problems in a third party extension, which resulted in the following exception being thrown.

#1: PHP Warning: imagecreatefromgif(typo3temp/_processed_/csm_t%C3%A4st.gif): failed to open stream: No such file or directory in /path-to-typo3/typo3_src-6.2.9/typo3/sysext/core/Classes/Imaging/GraphicalFunctions.php line 2873 

I did some debugging and found out, that the exception was thrown, because the extension used the result of an IMG_RESOURCE object (which in this case is the path to an resized image) for further TypoScript image processing. The special thing about it was, that the exception was only thrown for images where the filename contained umlauts.

Generally, I never use umlauts in filenames and if TYPO3 is not configured to use an utf-8 to store filenames, TYPO3 automatically converts umlauts (e.g. "ä" to "ae"). The setup in this TYPO3 installation was different and in the install tool the following settings were present.

[SYS][UTF8filesystem] = 1
[SYS][systemLocale] = de_DE.utf8

The code in the third party extension was fine for TYPO3 4.5, since resized and temporary images did not include the original filename. For TYPO3 6.2, this is different in some situations and resized/temporary images may get "csm_" as prefix followed by the original filename. This can result in some strange behaviour with images not being rendered as intended. I will demonstrate this below.

My test setup uses TYPO3 6.2.9, PHP 5.5 and I have configured [SYS][UTF8filesystem] = 1 and [SYS][systemLocale] = de_DE.utf8.

First I uploaded a file named "test.jpg" and assigned it through TypoScript to the frontend output as shown below.

lib.test = IMAGE
lib.test.file = fileadmin/gfx/test.jpg

There is nothing special about this and as you may expect, the image was successfully rendered in the frontend.

Next I uploaded a file names "täst.jpg" and did the same as shown above.

lib.test = IMAGE
lib.test.file = fileadmin/gfx/täst.jpg

Also here, the image was rendered successfully in the frontend. A look in the sourcecode did show, that the umlaut image filename has been urlencoded.

<img alt="" border="0" height="250" src="fileadmin/gfx/t%C3%A4st.jpg" width="250" />

This all works fine for simple TypoScript IMAGE objects. Now lets get our good, old friend GIFBUILDER into the team :-)

The following TypoScript is kept simple for demostration purposes.

lib.test = IMAGE
lib.test {
  file = GIFBUILDER
  file {
    XY = 100,100
    10 = IMAGE
    10.file = fileadmin/gfx/test.jpg
    20 = TEXT
    20.text = Test
    20.fontColor = #000000
    20.fontFile = fileadmin/vera.ttf
    20.offset = 10,20
  }
}

The output of the TypoScript above is the original image "test.jpg", where the text "Test" is rendered on top of the image.

As the last step, I replace the image "test.jpg" with the image "täst.jpg".

lib.test = IMAGE
lib.test {
  file = GIFBUILDER
  file {
    XY = 100,100
    10 = IMAGE
    10.file = fileadmin/gfx/täst.jpg
    20 = TEXT
    20.text = Test
    20.fontColor = #000000
    20.fontFile = fileadmin/vera.ttf
    20.offset = 10,20
  }
}

What would you expect to be rendered? I guess, the same result as for the previous example. Actually, the final image is just a GIF file, which has a white background and shows the text "Test". The image with the umlaut is not processed.

So why does TYPO3 not render the image, when umlauts are used in the filename? The main reason for this behaviour is located in GIFBUILDER, which uses image path that is rawurlencoded.

1. When GIFBUILDER starts to render the final image, it iterates through all TypoScript objects inside it. For IMAGE objects, the GIFBUILDER uses it's own function getResource($file, $fileArray) to return a GIFBUILDER image.

case 'IMAGE':
 $fileInfo = $this->getResource($conf['file'], $conf['file.']);
 if ($fileInfo) {
  $this->combinedFileNames[] = preg_replace('/\\.[[:alnum:]]+$/', '', basename($fileInfo[3]));
  $this->setup[$theKey . '.']['file'] = $fileInfo[3];

2. The function getResource uses the the function getImgResource from the contentObjectRenderer to create get the (resized/scaled) image resource for the file.

public function getResource($file, $fileArray) {
 if (!GeneralUtility::inList($this->imageFileExt, $fileArray['ext'])) {
  $fileArray['ext'] = $this->gifExtension;
 }
 /** @var \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer $cObj */
 $cObj = GeneralUtility::makeInstance('TYPO3\\CMS\\Frontend\\ContentObject\\ContentObjectRenderer');
 $cObj->start($this->data);
 return $cObj->getImgResource($file, $fileArray);
}

3. The function getImgResource resizes/scales the given image, saves the processed image as /fileadmin/_processed_/csm_täst_9f3d74f15e.gif and returns an array including a lot of information (see below) about the processed file.

if ($processedFileObject->isProcessed() && !isset($GLOBALS['TSFE']->tmpl->fileCache[$hash])) {
 $GLOBALS['TSFE']->tmpl->fileCache[$hash] = array(
  0 => $processedFileObject->getProperty('width'),
  1 => $processedFileObject->getProperty('height'),
  2 => $processedFileObject->getExtension(),
  3 => $processedFileObject->getPublicUrl(),
  'origFile' => $fileObject->getPublicUrl(),
  'origFile_mtime' => $fileObject->getModificationTime(),
  // This is needed by \TYPO3\CMS\Frontend\Imaging\GifBuilder,
  // in order for the setup-array to create a unique filename hash.
  'originalFile' => $fileObject,
  'processedFile' => $processedFileObject,
  'fileCacheHash' => $hash
 );
}
$imageResource = $GLOBALS['TSFE']->tmpl->fileCache[$hash];

4. The interesting part here is $processedFileObject->getPublicUrl(), which returns the public URL for the processed FAL image.

public function getPublicUrl($identifier) {
 $publicUrl = NULL;
 if ($this->baseUri !== NULL) {
  $uriParts = explode('/', ltrim($identifier, '/'));
  $uriParts = array_map('rawurlencode', $uriParts);
  $identifier = implode('/', $uriParts);
  $publicUrl = $this->baseUri . $identifier;
 }
 return $publicUrl;
}

Array_map is used on all $uriParts with rawurlencode() to ensure the path to the image is encoded correctly. For my testfile "täst.jpg", the function returns "fileadmin/_processed_/csm_t%C3%A4st_9f3d74f15e.gif".

Now going back to step 1 where the GIFBUILDER iterates through all TypoScript objects you will see, that $fileInfo[3] is used as a path to the image and this is what results in the GIFBUILDER object not being rendered correctly. As the filename is urlencoded, it can't be used by GIFBUILDER for futher processing which in this case is rendering the text "Test" on it.

Conclusion

The problem only appears under rare curcumstances (using [SYS][UTF8filesystem] = 1), and I believe that not many people are affected of it, since TYPO3 by default has disabled this feature.  It seems, this is a bug in TYPO3, which actually was reported by Peter Niederlag nearly at the same time I struggled with the problem. I'll join the discussion and am pretty sure that a bugfix will be available soon.

Update 29.01.2015: The bugfix for the problem described above has been merged today for TYPO3 6.2 and current master.