Saturday, May 30, 2015

Laravel 5 and queues using redis - get amount scheduled jobs in queue

I'm currently digging into Laravel 5 and am using the queue component to put time consuming tasks in a queue, so they can be processed in the background. For performance reasons, I switched from using the database queue driver to use the redis queue driver. My application has a dashboard, which shows the amount of scheduled jobs for each queue in the queue system. Using the database queue driver, I used the following code:

Using the redis queue driver, Laravel creates a list for each queue on the redis server, so you can easily use the functions from the predis package, to interact with the redis server. To get the amount of jobs in a queue with the redis queue driver, I use the code as shown below.


I also needed the possibility to remove all jobs in a specific queue at once in my application. To do so, I used the code shown below.

Tuesday, March 31, 2015

Upgrading to Redmine 3.0.1 on Debian Wheezy

Today I wanted to upgrade my Redmine installation from Redmine 2.6.x to the new version Redmine 3.0.1. Since I already have processed many Redmine updates I thought this update would be easy as always, but certainly I ran into an unexpected problem. After I followed all steps from the upgrade manual I restarted the apache2 process, so mod_passenger reloads the Redmine application. After that, Redmine did'nt start and showed the following error message:

undefined method `page_cache_directory' for ActionController::Base:Class

After doing some Google search, I found this article on stackoverflow, which pointed me to the right direction. Actually, Debian Wheezy comes with an pretty old version of Phusion Passenger (3.0.x), so I performed the following steps and installed a newer version (4.0.x) of Phusion Passenger from Wheezy Backports.

First I added Wheezy Backports to my apt sources file /etc/apt/sources.list
deb http://http.debian.net/debian wheezy-backports main

Next I used apt-get update to update the local apt repositories and finally I installed Phusion Passenger from the Wheezy Backports with the following command

apt-get -t wheezy-backports install "libapache2-mod-passenger"

After this, I restarted the apache2 process and Redmine 3.0.1 started successfully.

Tuesday, March 17, 2015

New version of sf_event_mgt released

The new version 1.0.0 of sf_event_mgt has been released. I decided not to increase the version number to 0.5.4  but to start with 1.0.0 now, as the extension runs stable in some projects I maintain and also there is no need to keep the extension in alpha/beta state any more. The new version contains some small bugfixes in the template files and also comes with some nice new features.

The double-opt in feature for the confirmation e-mail can now be configured to be optional (settings.registration.autoConfirmation). If the new setting is activated, registrations by participants will automatically be confirmed.

For locations it is not possible to add latitude, longitude and a description. This can be useful, if the location should be shown on a map.

One feature, that has been requested by multiple users is the iCalendar download. For events, it is now possible to add a link, which downloads the event details as an iCalendar file. To keep things flexible, the iCalendar file is rendered through Fluid, so it is also possible to add own fields to the iCalendar file.

The last new major feature is the possibility to create a registration for multiple participants at once. There is a maximum amount of registrations, which can be configured for each event, so a user is not able to book all available places for an event at once.

All new features are also described in the manual of the extension. Additional details for this release can be found here.

Thanks to all users, who gave me feedback end new ideas for the extension!

If you find a bug and want to request a feature, please use the issue tracker on GitHub

Saturday, January 17, 2015

TYPO3 6.2 - Problems with GIFBUILDER and filenames containing umlauts

After migrating a TYPO3 4.5 website to TYPO3 6.2 I had some problems in a third party extension, which resulted in the following exception being thrown.

#1: PHP Warning: imagecreatefromgif(typo3temp/_processed_/csm_t%C3%A4st.gif): failed to open stream: No such file or directory in /path-to-typo3/typo3_src-6.2.9/typo3/sysext/core/Classes/Imaging/GraphicalFunctions.php line 2873 

I did some debugging and found out, that the exception was thrown, because the extension used the result of an IMG_RESOURCE object (which in this case is the path to an resized image) for further TypoScript image processing. The special thing about it was, that the exception was only thrown for images where the filename contained umlauts.

Generally, I never use umlauts in filenames and if TYPO3 is not configured to use an utf-8 to store filenames, TYPO3 automatically converts umlauts (e.g. "ä" to "ae"). The setup in this TYPO3 installation was different and in the install tool the following settings were present.

[SYS][UTF8filesystem] = 1
[SYS][systemLocale] = de_DE.utf8

The code in the third party extension was fine for TYPO3 4.5, since resized and temporary images did not include the original filename. For TYPO3 6.2, this is different in some situations and resized/temporary images may get "csm_" as prefix followed by the original filename. This can result in some strange behaviour with images not being rendered as intended. I will demonstrate this below.

My test setup uses TYPO3 6.2.9, PHP 5.5 and I have configured [SYS][UTF8filesystem] = 1 and [SYS][systemLocale] = de_DE.utf8.

First I uploaded a file named "test.jpg" and assigned it through TypoScript to the frontend output as shown below.

lib.test = IMAGE
lib.test.file = fileadmin/gfx/test.jpg

There is nothing special about this and as you may expect, the image was successfully rendered in the frontend.

Next I uploaded a file names "täst.jpg" and did the same as shown above.

lib.test = IMAGE
lib.test.file = fileadmin/gfx/täst.jpg

Also here, the image was rendered successfully in the frontend. A look in the sourcecode did show, that the umlaut image filename has been urlencoded.

<img alt="" border="0" height="250" src="fileadmin/gfx/t%C3%A4st.jpg" width="250" />

This all works fine for simple TypoScript IMAGE objects. Now lets get our good, old friend GIFBUILDER into the team :-)

The following TypoScript is kept simple for demostration purposes.

lib.test = IMAGE
lib.test {
  file = GIFBUILDER
  file {
    XY = 100,100
    10 = IMAGE
    10.file = fileadmin/gfx/test.jpg
    20 = TEXT
    20.text = Test
    20.fontColor = #000000
    20.fontFile = fileadmin/vera.ttf
    20.offset = 10,20
  }
}

The output of the TypoScript above is the original image "test.jpg", where the text "Test" is rendered on top of the image.

As the last step, I replace the image "test.jpg" with the image "täst.jpg".

lib.test = IMAGE
lib.test {
  file = GIFBUILDER
  file {
    XY = 100,100
    10 = IMAGE
    10.file = fileadmin/gfx/täst.jpg
    20 = TEXT
    20.text = Test
    20.fontColor = #000000
    20.fontFile = fileadmin/vera.ttf
    20.offset = 10,20
  }
}

What would you expect to be rendered? I guess, the same result as for the previous example. Actually, the final image is just a GIF file, which has a white background and shows the text "Test". The image with the umlaut is not processed.

So why does TYPO3 not render the image, when umlauts are used in the filename? The main reason for this behaviour is located in GIFBUILDER, which uses image path that is rawurlencoded.

1. When GIFBUILDER starts to render the final image, it iterates through all TypoScript objects inside it. For IMAGE objects, the GIFBUILDER uses it's own function getResource($file, $fileArray) to return a GIFBUILDER image.

case 'IMAGE':
 $fileInfo = $this->getResource($conf['file'], $conf['file.']);
 if ($fileInfo) {
  $this->combinedFileNames[] = preg_replace('/\\.[[:alnum:]]+$/', '', basename($fileInfo[3]));
  $this->setup[$theKey . '.']['file'] = $fileInfo[3];

2. The function getResource uses the the function getImgResource from the contentObjectRenderer to create get the (resized/scaled) image resource for the file.

public function getResource($file, $fileArray) {
 if (!GeneralUtility::inList($this->imageFileExt, $fileArray['ext'])) {
  $fileArray['ext'] = $this->gifExtension;
 }
 /** @var \TYPO3\CMS\Frontend\ContentObject\ContentObjectRenderer $cObj */
 $cObj = GeneralUtility::makeInstance('TYPO3\\CMS\\Frontend\\ContentObject\\ContentObjectRenderer');
 $cObj->start($this->data);
 return $cObj->getImgResource($file, $fileArray);
}

3. The function getImgResource resizes/scales the given image, saves the processed image as /fileadmin/_processed_/csm_täst_9f3d74f15e.gif and returns an array including a lot of information (see below) about the processed file.

if ($processedFileObject->isProcessed() && !isset($GLOBALS['TSFE']->tmpl->fileCache[$hash])) {
 $GLOBALS['TSFE']->tmpl->fileCache[$hash] = array(
  0 => $processedFileObject->getProperty('width'),
  1 => $processedFileObject->getProperty('height'),
  2 => $processedFileObject->getExtension(),
  3 => $processedFileObject->getPublicUrl(),
  'origFile' => $fileObject->getPublicUrl(),
  'origFile_mtime' => $fileObject->getModificationTime(),
  // This is needed by \TYPO3\CMS\Frontend\Imaging\GifBuilder,
  // in order for the setup-array to create a unique filename hash.
  'originalFile' => $fileObject,
  'processedFile' => $processedFileObject,
  'fileCacheHash' => $hash
 );
}
$imageResource = $GLOBALS['TSFE']->tmpl->fileCache[$hash];

4. The interesting part here is $processedFileObject->getPublicUrl(), which returns the public URL for the processed FAL image.

public function getPublicUrl($identifier) {
 $publicUrl = NULL;
 if ($this->baseUri !== NULL) {
  $uriParts = explode('/', ltrim($identifier, '/'));
  $uriParts = array_map('rawurlencode', $uriParts);
  $identifier = implode('/', $uriParts);
  $publicUrl = $this->baseUri . $identifier;
 }
 return $publicUrl;
}

Array_map is used on all $uriParts with rawurlencode() to ensure the path to the image is encoded correctly. For my testfile "täst.jpg", the function returns "fileadmin/_processed_/csm_t%C3%A4st_9f3d74f15e.gif".

Now going back to step 1 where the GIFBUILDER iterates through all TypoScript objects you will see, that $fileInfo[3] is used as a path to the image and this is what results in the GIFBUILDER object not being rendered correctly. As the filename is urlencoded, it can't be used by GIFBUILDER for futher processing which in this case is rendering the text "Test" on it.

Conclusion

The problem only appears under rare curcumstances (using [SYS][UTF8filesystem] = 1), and I believe that not many people are affected of it, since TYPO3 by default has disabled this feature.  It seems, this is a bug in TYPO3, which actually was reported by Peter Niederlag nearly at the same time I struggled with the problem. I'll join the discussion and am pretty sure that a bugfix will be available soon.

Update 29.01.2015: The bugfix for the problem described above has been merged today for TYPO3 6.2 and current master.

Tuesday, December 9, 2014

TYPO3 6.2 - Still some problems with newsletter image rendering

I've done a lot of TYPO3 4.5 to TYPO3 6.2 migrations the last weeks and struggled with some strange behaviour of TYPO3 6.2 in combination with newsletter pages and the TYPO3 Extensions direct_mail and direct_mail_subscription.

I'm pretty sure, some people might run into the same problems during the usage of TYPO3 6.2 and direct_mail / direct_mail_subscription, so I'll describe how I solved the most common problems in this article.

Update 01.01.2015

The first two problems are solved now and you can use renderMethod = table in TYPO3 6.2 and 7.x directly with css_styled_content.

Newsletter pages not rendering images


In order to display HTML newsletters in various e-mail clients it is recommended to output the page layout in old-school HTML table design. In order to do so, I use the following TypoScript for images.


tt_content.image.20.renderMethod = table

This renders image-output in table design and enables the editor to set alignments for images in newsletters. In TYPO3 6.0 and 6.1 I had some major problems with using css_styled_content with the renderMethod shown above, so I used to include the css_styled_content configuration for TYPO3 v4.7 as described in my former blogpost.

This all worked fine for TYPO3 sites migrated from 6.x to TYPO3 6.2. With TYPO3 sites, created directly with TYPO3 6.2, this did'nt work any more and images were just not rendered. I spent several hours of debugging to find out, that a setting in the LocalConfiguration.php was the cause for this. Sites, that have been migrated from TYPO3 6.x to TYPO3 6.2 had [FE][activateContentAdapter] enabled in the Install tool.

After I enabled the [FE][activateContentAdapter] for the directly created TYPO3 6.2 sites, images were rendered correctly with the css_styled_content (v4.7) configuration.


Anyway, the install tool shows a warning, that setting activateContentAdapter is slow, so this only seems to be a temporary solution.

In my opinion a better solution would be to use css_styled_content which comes with TYPO3 6.2 and deactivate [FE][activateContentAdapter], but at the time of writing, there is a bug (see problem below) preventing me from using that option.

Image rendering buggy using TYPO3 6.2 css styled content and renderMethod = table


When I tried to debug my former problem with images not being rendered in page output, I also tried to use css_styled_content from TYPO3 6.2 directly. This resulted in the images being rendered incorrectly (see screenshots below).

3 TYPO3 Logos in TYPO3 backend - each with an individial label in the image

The output using css_styled_content with renderMethod = table results in the following output.

Resulting image output shows 4 images and only the last image is rendered


This seems to be a problem in TYPO3 6.2 and there is also an issue on Forge for this one. I will try to create created a patch, so both the linked and related issues can be closed.

Newsletter not sent to recipient


The last problem I spent some time with is, that newly subscribed recipients did not receive the sent newsletter. As I only send out HTML newsletters, I used to set the following TypoScript in the configuration for direct_mail_subscription


plugin.feadmin.dmailsubscription {
  create {
    overrideValues.module_sys_dmail_html = 1
  }
}

It seems, that this does not work with the latest versions of direct_mail_subscription, so newly created records in tt_address did not contain the nescecary flag. I tried several other approaches including setting hidden input fields for the field mod_sys_dmail_html to the registration form and setting TCAdefaults, but all approaches did'nt work. At least the issue is known, so a fix for this problem may be available soon.

To work around this issue, I set the default value for the field mod_sys_dmail_html to 1 directly in the database by using the following SQL


ALTER TABLE `tt_address` CHANGE `module_sys_dmail_html` `module_sys_dmail_html` tinyint(3) unsigned NOT NULL DEFAULT '1';

Make sure, that this setting may be reverted, if you use the database analyzer in the install tool and apply the original default value for the field.

Friday, November 7, 2014

TYPO3 CMS - Event management extension with registration option

Today I've published the first release of my latest extension "Event management and registration" (sf_event_mgt) for TYPO3 CMS. The extension enables TYPO3 editors to manage events in the TYPO3 backend and can show a list- and a detail-view for upcoming, past or all events in the frontend. The list-view can be extended by individual layouts as known from the TYPO3 news extension (tx_news), so it should for example be possible to create a nice image slider for all upcoming top-events.

As events also could require a registration process (e.g. if the number of participants is limited), I added a registration option, so users can register for an event. TYPO3 editors can assign a maximun value for participants to each individual event, so it can not be over-booked. The registration contains a double opt in process, so users must confirm, that they actually have registered for the event. To ensure, that new registrations are processed in time by the registering user, a validity for confirmation-links can be configured.

Event record in TYPO3 backend

The extension also comes with a backend module, which enables TYPO3 editors to get an overview of all available events. Also the backend module contains a CSV-Export option for events where registration is enabled, so a list of registered participants easily can be exported.

Event backend administration module
The backend module also contains a notification module, where TYPO3 editors can send e-mail notifications to all participants of an event (e.g. if the event has been cancelled). Notification templates can be configured individually with some lines of TypoScript and with a Fluid template.

Notification module to send notifications to participants
Finally, the extension has some nice features, which are more technical, but makes the extension very flexible and extendible. Of course, the extension comes with a documentation ReST format which explains all features in detail.

I've tried to keep the code quality at a high level, respected the TYPO3 CGL as much as possible and also "Event management and registration" is fully covered by tests (which does not mean it is bug-free).

The extension is available on TER and bugs as well as feature requests can be reported on GitHub, where also the source code of the extension is hosted.



Saturday, October 25, 2014

TYPO3 Neos - Set a dynamic sender/recipient for a mailform

I've been starting using TYPO3 Neos for some weeks now and I am really impressed. In so many places you just see how much high quality work, time, effort and love has been put into the product by the TYPO3 Neos team - thumbs up!

In one of my first projects I created a simple contact form like shown in the TYPO3 Neos Integrators Cookbook. There was only one thing I had some problem with and this was to set the sender e-mail address of the form dynamically, so the recipient of an e-mail form just can press the "reply" button in the e-mail app to directly send an answer to the person who filled out the contact form.

I really could'nt find a solution for my problem on the internet, so after digging into the code of the TYPO3.Form EmailFinisher, I finally found some help in the description of the class. You can use submitted form fields as placeholders for every option of the EmailFinisher like shown below.


finishers:
  -
    identifier: 'TYPO3.Form:Email'
    options:
      templatePathAndFilename: resource://Vendor.Site/Private/Templates/Email/Contact.txt
      subject: Contact from website
      recipientAddress: office@example.net
      recipientName: 'Office of Company'
      senderAddress: '{email}'
      senderName: '{name}'
      format: plaintext

With this configuration in the Yaml for the contact form, the sender e-mail address and the sender name will be used from the submitted from data. Of course you can also set the recipient e-mail address dynamically the same way.

Keep in mind, that setting the sender e-mail address like shown may cause some problems with local spamfilters. To prevent potential problems, you could whitelist the webservers IP address directly in the spamfilter.