How Extbase determines the storagePid setting for a plugin

When you search for the terms "extbase storagePid" in your favorite search engine, you will find several blogs, forum-posts, mailing list entries and code examples on how to set the storagePid for the plugin of an Extbase extension.  Many examples refer to TypoScript persistence settings for the extension itself (plugin.tx_myext.persistence.storagePid), which is good, as long as you do not directly set persistence settings for a plugin like shown in the example.

Example

Let us assume, that you have a TYPO3 Extbase extension, that has configured a storagePid for a plugin in TypoScript as shown below.

plugin.tx_myext_myplugin {
 persistence {
  storagePid = {$plugin.tx_myext_myplugin.persistence.storagePid}
 }
}

Now you want the editor to be able to set the storagePid from the plugin settings. Since a TYPO3 plugin by default has a configuration setting called Record Storage Page, you use this configuration setting and select a page.

Surprisingly, the Plugin will still use the setting configured in TypoScript. But why?

Extbase storagePid determination

Extbase uses the configurationManager to determine the storagePid. When you create an Extbase extension which should be configurable by users, it is important to understand how and in which order Extbase determines/overrides the storagePid for a plugin.

Step 1 - Default storagePid

Extbase uses the default storagePid which is 0

Step 2 - Extbase persistence settings

If Extbase has configured a storagePid in config.tx_extbase.persistence.storagePid, then Extbase will use this storagePid. (note, in the code, this happens before before step 1)

Step 3 - Extension/Plugin configuration from TypoScript

Extbase fetches the configuration for the plugin from TypoScript persistence settings in plugin.tx_myext.persistence.storagePid and merges it with plugin.tx_myext_myplugin.persistence.storagePid - The plugin settings override the extension settings. If the resulting storagePid is set (or empty), Extbase will use the configured value as storagePid

Step 4 - Override storagePid from starting point setting

If the plugin uses the starting point (Record Storage Page) and one/multiple pages are selected as starting point, Extbase will now use this as storagePid.

Step 5 - Override storagePid from Plugin settings

Now Extbase fetches the configuration for the plugin from TypoScript from plugin.tx_myext_myplugin.persistence.storagePid and if set (or empty), Extbase now uses this as storagePid.

Step 6 - Override storagePid from Flexform

Finally Extbase checks, if the plugin has configured a flexform and if so, it merges all settings, persistence and view elements from the flexform with the current Extbase plugin configuration. The merge is done recursive and settings from TypoScript will be overridden by the flexform settings. So if you have configured persistence.storagePid in your flexform, this will be used as the storagePid.

Make sure you read the important note below, since there is something to keep in mind when working with empty flexform values.

Step 7 - Expand storagePid if recursive setting is set

If persistence.recursive is set in plugin.tx_myext_myplugin or in your flexform, Extbase will expand all configured storagePids recursive by the configured depth. If set, the final storagePid will contain a list of page uids for all subfolders of the original storagePid.


It is important to know, that each determination step will override the storagePid setting from the previous step. So for our example from above, the starting point setting gets overridden by the Plugin TypoScript setting.

Also keep in mind, that as soon as you use plugin.tx_myext_myplugin.persistence.storagePid, the persistence settings for the extension plugin.tx_myext.persistence.storagePid and also the record storage page settings will get overridden.

StoragePid when creating new records

When you have set the storagePid (which in fact may contain several page uids), Extbase will always use the first uid when creating a new record. So if you have configured  plugin.tx_myext_myplugin.persistence.storagePid = 10,11,12 then new records created from the plugin will be saved to the page with the uid 10.

In order to override this behaviour, you can either add a property to your domain object which contains the page uid - e.g. getPid() - or you can override the storagePid for new records by TypoScript with the newRecordStoragePid setting as shown below:

plugin.tx_myext {
 persistence {
  storagePid = {$plugin.tx_myext.persistence.storagePid}
  classes {
   Vendor\Myext\Domain\Model\Mymodel {
    newRecordStoragePid = 16
   }
  }
 }
}

Setting the storagePid from flexform

To set the storagePid from flexform, you can use the code snippet below. It is an extract from a flexform XML structure which contains a field for the storagePid and another field for the recursive setting.

<persistence.storagePid>
    <TCEforms>
        <exclude>1</exclude>
        <label>Storage PID</label>
        <config>
            <type>group</type>
            <internal_type>db</internal_type>
            <allowed>pages</allowed>
            <size>3</size>
            <maxitems>99</maxitems>
            <minitems>0</minitems>
            <show_thumbs>1</show_thumbs>
            <wizards>
                <suggest>
                    <type>suggest</type>
                </suggest>
            </wizards>
        </config>
    </TCEforms>
</persistence.storagePid>

<persistence.recursive>
    <TCEforms>
        <label>LLL:EXT:lang/locallang_general.xlf:LGL.recursive</label>
        <config>
            <type>select</type>
            <items type="array">
                <numIndex index="1" type="array">
                    <numIndex index="0">LLL:EXT:news/Resources/Private/Language/locallang_be.xlf:flexforms_general.recursive.I.inherit</numIndex>
                    <numIndex index="1"></numIndex>
                </numIndex>
                <numIndex index="2" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.0</numIndex>
                    <numIndex index="1">0</numIndex>
                </numIndex>
                <numIndex index="3" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.1</numIndex>
                    <numIndex index="1">1</numIndex>
                </numIndex>
                <numIndex index="4" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.2</numIndex>
                    <numIndex index="1">2</numIndex>
                </numIndex>
                <numIndex index="5" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.3</numIndex>
                    <numIndex index="1">3</numIndex>
                </numIndex>
                <numIndex index="6" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.4</numIndex>
                    <numIndex index="1">4</numIndex>
                </numIndex>
                <numIndex index="7" type="array">
                    <numIndex index="0">LLL:EXT:frontend/Resources/Private/Language/locallang_ttc.xlf:recursive.I.5</numIndex>
                    <numIndex index="1">250</numIndex>
                </numIndex>
            </items>
        </config>
    </TCEforms>
</persistence.recursive>

It is important, that the field naming of the fields is equal to the TypoScript naming. Since Extbase will merge all flexform fields to the plugin configuration, persistence.storagePid and persistence.recursive from the flexform will override the TypoScript settings.

Important note:
If you have set a storagePid by flexform and delete it afterwards, then the final storagePid will be empty. The reason for this is the field pi_flexform in tt_content table for the plugin, which will contain an empty value for persistence.storagePid

<field index="persistence.storagePid">
  <value index="vDEF"></value>
</field>

This empty value actually overrides all other storagePid settings. You can avoid this behavior by using the processDatamap_postProcessFieldArray hook to unset empty field values in flexform if needed

Conclusion

As a result of this blogpost, I would recommend to use plugin.tx_myext.persistence.storagePid and the record storage page option to set the storagePid of a plugin, since it works out of the box and does not have any things to keep mind.

TYPO3 - PHPUnit Testing Framework tests for repositories with storagePids

When I develop a TYPO3 ExtBase extension, I often use the Testing Framework shipped with the TYPO3 extension PHPUnit. With the Testing Framework it is possible to create real testrecords in the TYPO3 database and use those records in your tests to verify everything works as expected.

In a project I faced the situation, that one of my repository methods did not find the records created with the Testing Framework. Well, after some hours of debugging I finally found the solution.

The problem

I created a new TYPO3 ExtBase extension with some tables, a controller and a view. Really nothing special about that. I then created a function in my repository which searches for some records in the database with a configurable limit. 

/**
 * Returns the latest records. 
 *
 * @param int $limit The Limit
 * @return array|\TYPO3\CMS\Extbase\Persistence\QueryResultInterface
 */
public function findLatest($limit) {
 $query = $this->createQuery();
 $query->setOrderings(array('datecreated' => QueryInterface::ORDER_DESCENDING));
 $query->setLimit((integer)$limit);
 $records = $query->execute();
 return $records;
}

Then I started to create some tests for the newly created function in the repository. One test was as following.

/**
 * Test if findLatest returns expected amout of records if limit given
 *
 * @test
 * @return void
 */
public function findLatestReturnsLimitedResult() {
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));
 $this->testingFramework->createRecord('tx_myext_domain_model_table', array('datecreated' => time()));

 $this->createTestRecords();
 $this->assertEquals(2, $this->myRepository->findLatest(2)->count());
}

In the test above, 3 records with the Testing Framework are created and I'll use the function findLatest with a given limit of 2 to test, if really 2 records are returned.

Well, actually findLatest did return 0 records. I then created some real records in the TYPO3 backend and verified, that findLatest actually works as expected, so the problem must be somewhere in my test or in the TYPO3 setup.

So I started to debug my test and first of all, I verified, that the Testing Framework really created 3 records. Yes, 3 records were created, all having the pid 0. Ok then, since the testrecords where created correctly, there seem to be something wrong with the search. I'll then debugged the resulting SQL Query from the repository function findLatest and found out, that the records were searched in pid 1. OK, so the problem seems to have something to do with the storagePids.

ExtBase, tests and storagePid(s)

When you create an ExtBase extension, the Extension Builder automatically creates some TS for you where you can set the storagePid(s) for your extension.

persistence {
  # cat=plugin.tx_myext//a; type=string; label=Default storage PID
  storagePid =
}

This storagePid can be used to set the location of your records in the TYPO3 backend. Generally every TYPO3 Extension you create with the Extension Builder makes use of the storagePid(s) and overrides them, if a startingPoint is set in the plugin.

When you run your tests in TYPO3 backend (or through your IDE), ExtBase uses the backendConfigurationManager to get the ExtBase framework configuration. In the backendConfigurationManager class, there is a function called getDefaultStoragePid(), which returns the local function getCurrentPageId(). Let's have a look at this function.

protected function getCurrentPageId() {
 $pageId = (integer) \TYPO3\CMS\Core\Utility\GeneralUtility::_GP('id');
 if ($pageId > 0) {
  return $pageId;
 }
 // get current site root
 $rootPages = $GLOBALS['TYPO3_DB']->exec_SELECTgetRows('uid', 'pages', 'deleted=0 AND hidden=0 AND is_siteroot=1', '', '', '1');
 if (count($rootPages) > 0) {
  return $rootPages[0]['uid'];
 }
 // get root template
 $rootTemplates = $GLOBALS['TYPO3_DB']->exec_SELECTgetRows('pid', 'sys_template', 'deleted=0 AND hidden=0 AND root=1', '', '', '1');
 if (count($rootTemplates) > 0) {
  return $rootTemplates[0]['pid'];
 }
 // fallback
 return self::DEFAULT_BACKEND_STORAGE_PID;
}

So, if your TYPO3 installation where you develop your ExtBase extension has a root page, then this page ID is returned. Or if you have no root page but a TS Template which is marked as root template, then the page ID of that page is returned.

The returned page ID is used in the queryFactory class to set the storagePid(s), which leads to the situation I described earlier.

Solution / Workarounds

So how do we tell TYPO3 to search for our test records in pid 0 during tests? On solution is to set the storagePid for the ExtBase Framework to 0 like shown below.

config.tx_extbase {
  persistence {
    storagePid = 0
  }
}

This solution has some problems, since it can be overridden by other extensions and also it requires manual configuration when setting up your testing environment.

You could also just disable the check for storagePid(s) (with setRespectStoragePage(FALSE)) in your tests, but I think this solution is'nt really good, since your tests then also may find records you created in the TYPO3 backend during development.

If you want to keep your tests independent, I would recommend to set the storagePid(s) directly in the setup of the tests.

/**
 * Set up for test
 *
 * @return void
 */
public function setUp() {
 $this->testingFramework = new \Tx_Phpunit_Framework('tx_myext');
 $this->fixture = $this->objectManager->get('Derhansen\\MyExt\\Domain\\Repository\\MyRepository');

 /** @var $querySettings Typo3QuerySettings */
 $querySettings = $this->objectManager->get('TYPO3\\CMS\\Extbase\\Persistence\\Generic\\Typo3QuerySettings');
 $querySettings->setStoragePageIds(array(0));
 $this->fixture->setDefaultQuerySettings($querySettings);
}

The code above sets the storage pid to 0 and now the tests should find the records created with the testing framework and tests should not interfere with other records created manually.

Mapping anderer Tabellen in Extbase

Neulich stand ich vor der eigentlich recht simplen Aufgabe aus einer Extbase Extension heraus auf die erzeugten Daten einer anderen Extbase Extension zuzugreifen. Im Web gibt es auch einige Tutorials, welche beschreiben wie man dieses macht, aber fast alle zeigen, wie man auf die Tabellen fe_users und fe_groups zugreift, welche beide schon als Domain-Model bei Extbase mitgeliefert werden. Ein funktionierendes Praxisbeispiel für den Zugriff auf eigene Datenstrukturen konnte ich leider nicht finden.

In diesem Beitrag zeige ich deshalb, wie man aus einer Extbase Extension heraus auf die Daten einer anderen Extbase Extensions zugreifen kann.

Ausgangssituation:

In einer TYPO3 Installation wurden 2 Extbase Extensions (testext1 und testext2) eingerichtet, welche jeweils eigene Repositories und Storagefolder haben. Die Typoscript Constants der Extensions sind wie folgt eingerichtet:

plugin.tx_testext1 {
  persistence {
    storagePid = 5
  }
}

plugin.tx_testext2 {
  persistence {
    storagePid = 6
  }
}

Die Datensätze von testext1 werden also im Sysordner mit der Pid 5 abgelegt und die Datensätze von testext2 im Sysordner mit der Pid 6. In testext1 gibt es ein Domainmodel mit dem Namen Testext1Table und in testext2 gibt es ein Domainmodel mit dem Namen Testext2Table.

Ziel ist es nun, aus einem Controller in testext2 auf die angelegten Daten von testext1 zuzugreifen.

Vorgehensweise:

Als erstes muss in testext2 ein Domainmodel angelegt werden, welches das Domainmodel von textext1 extended. Also wird in textext2\Classes\Domain\Model\ die Datei Testext1Table.php angelegt.

class Tx_Testext2_Domain_Model_Testext1Table extends Tx_Testext1_Domain_Model_Testext1Table {

}

Somit wurde der testext2 schon mal das Domainmodel aus testext1 bekannt gemacht. Damit wir auf die Daten aus testext1 zugreifen können, wird noch ein entsprechendes Repository benötigt. Dieses wird in  textext2\Classes\Domain\Repository\Testext1TableRepository.php definiert.

class Tx_Testext2_Domain_Repository_Testext1TableRepository extends Tx_Extbase_Persistence_Repository {

}

Nun kennt die Extension testext2 das Domainmodel und hat auch ein entsprechendes Repository. Das neue Repository wird in einem Controller von testext2 per Dependency Injection implementiert.

/**
 * testext1TableRepository
 *
 * @var Tx_Testext2_Domain_Repository_Testext1TableRepository
 */
protected $testext1TableRepository;

/**
 * injectTestext1TableRepository
 *
 * @param Tx_Testext2_Domain_Repository_Testext1TableRepository $testext1TableRepository
 * @return void
 */
public function injectTestext1TableRepositoryRepository(Tx_Testext2_Domain_Repository_Testext1TableRepository $testext1TableRepository) {
 $this->testext1TableRepository = $testext1TableRepository;
}

Wer an dieser Stelle jetzt schon erste Ergebnisse aus dem implementierten Repository erwartet, wird enttäuscht und bekommt beim Versuch Daten aus dem Repository abzufragen/anzuzeigen eine Exception vom Typ Tx_Extbase_Persistence_Storage_Exception_SqlError, welche besagt, dass eine Tabelle nicht existiert.

Die Extension testext2 weiß also anscheinend noch gar nicht, aus welcher Tabelle die Daten des neuen Repositories abgefragt werden sollen. Um dieses einzurichten, muss man im Typoscript Setup von testext2 ein Mapping einrichten.

config.tx_extbase {
 persistence {
    classes { 
        Tx_Testext2_Domain_Model_Testext1Table {
            mapping.tableName = tx_testext1_domain_model_testext1table
        }
    }
 }
}

An dieser Stelle ist es wichtig, dass das Mapping direkt auf den Tabellennamen in der Datenbank stattfindet.

Auch jetzt erhalten wir über das neue Repository in testext2 noch keine Daten, da die testext2 noch nicht weiß, in welcher StoragePid die Datensätze von testext1 liegen. Hierzu ergänzt man lediglich die Typoscript Constants von testext2 wie folgt.

plugin.tx_testext2 {
  persistence {
    storagePid = 6,5
  }
}

Die StoragePid von testext1 wurde hinter der StoragePid von testext2 angehängt. Nun kann das neue Repository in testext2 angesprochen und die erwarteten Daten erfolgreich zurückgeliefert werden.