Neos for large content webpages

trsh · January 2, 2017, 3:16pm

The idea how to Edit content with Neos is very cool, but confuses me, when I think of large content (articles, news, etc.). For example, imagine I have a page with tons of news- I generate 20 at day, so it’s 7300 in a year. To work with this, an advanced list with sorting by date, search, paging, etc is for sure required.

What’s the recommended way to get this working with Neos?

mficzel · January 2, 2017, 8:56pm

Hello Janis,

the pure amount of nodes is not the problem and you can use all the filter or search functions you have in the frontend directly in the backend. That results in a very direct way of searching and editing that actually helps editors understand the site and its content even better.

If you have lots of nodes you will have to take some precautions nevertheless:

Make sure you use elastic-search for flow-queries that read lots of nodes or even parts of your site
Have Redis-Cache available for routing- and fusion/ts2-caches since the default file-cache-backend will not invalidate huge numbers of nodes (see https://gist.github.com/aertmann/8e3eb6736e9498c7144b)
Help editors to navigate the node-tree since having more than 20 nodes on one level makes that quite hard.
3.1. Cluster the nodes. A structures for year/month/day/news_title.html might work good in your case but that depends pretty much on your project.
3.2. Another way is to hide the news from the navigate structure at all … in that case you have to find a way to create them for your editors.

Regards, Martin

trsh · January 3, 2017, 7:18am

you can use all the filter or search functions you have in the frontend directly in the backend

I disagree. Front-end goes always for simplicity- Like some single button “Show 10 older news”, where in back-end Editors need advanced tools for surfing trough existing content, moving it to archives, etc.

Help editors to navigate the node-tree since having more than 20 nodes on one level makes that quite hard.

Yes. And regarding what I wrote above, nodes is not good choice for large content, like news. Adding Redis or Elastic for me is an overkill, where simple paging can solve “to much data” problem.

How about an plugin, and handle stuff in it? Or other advice.

mficzel · January 3, 2017, 11:27am

That is absolutely true but a good navigation in the frontend is still a must. It is not a bad idea to make this the primary navigation in the backend aswell in addition to the navigate-component. That gives the editors a good feeling how visible the content they are editing really is.

Providing special views in the backend or maybe some extra search-options is possible and often seen in larger projects.

Avoiding nodes completely is no problem. You can create flow-domain objects here with a strict data-model and a specific backend-module for editing. Be aware that you loose the easy inline-editing of Neos in that case. Be careful with that decision since as long as your domain-data represents content nodes are usually not a bad idea.

If you simply want to page trough your nodes you do’nt need elastic. But if you want to execute complex queries keep in mind that elastic can help if you run into problems there. It is no problem to add that later if you ever need it.

Redis is a bit different. If you have lots of cache entries you will need it or at least you will need a cache-backend other than the default file-backend. The point where you should think about that is when editors mention slow behavior after changing something. That is usually the cache-invalidation which is slow for the file-backend.

Regards, Martin

trsh · January 3, 2017, 1:44pm

Providing special views in the backend or maybe some extra search-options is possible and often seen in larger projects.

How I do that in Neos? Any examples?

Be aware that you loose the easy inline-editing of Neos in that case.

Why? Can I combine, reuse inline editing components?

mficzel · January 3, 2017, 7:25pm

You just show/hide some parts if you are in backend-context:

In fusion/ts2 @if.isInBackend = ${node.context.inBackend}
In fluid <f:if condition="{neos:rendering.inBackend()}">

Inline editing in Neos always edits nodes. Everything else may be possible but is way beyond the supported use-cases and i do’nt know working examples. If you want the neos-editing experience simply use nodes. If the navigate component does not work for your documents hide them in there and provide another way for your editors to navigate and create the documents.

If you fear working with nodes will harm the performance just test it. I doubt this will be a real problem if you are only listing, paginating and displaying document-nodes without complex queries.

Regards, Martin

aertmann · January 4, 2017, 5:05pm

Admittedly there’s no really good way to handle this in the UI, however you can make use of some workarounds to achieve it.

E.g. you can hide news articles from the page tree and make it possible to find and create them in another way. You can make a create new button in your frontend news article list (IIRC @dimaip has an example of that). To find existing articles, the frontend can be used to navigate/search for them. Alternatively hidden node types will still be searchable in the tree.

There’s a new UI in the making where the navigate tree should be able to handle this amount of nodes better.

dimaip · January 4, 2017, 5:24pm

Hey yes so we have ~10K news nodes under the same parent on psmb.ru.

We hide them from the tree. To navigate them, editors use all the same ways the website visitors have: list, search, tags etc.
If I had to add additional ways to navigate that data for editors only, I would just add that as an extra UI element that is rendered only in BE (i.e. in the same way as we did CreateNodeButton). But I would think twice: if editors needed it, maybe website visitors would benefit from those extra navigation options too, perhaps as a optional UI/

To create news nodes we use this: https://github.com/psmb/Psmb.CreateNodeButton

P.S.: here’s the whole source code of our website https://github.com/psmb/PsmbDistr

trsh · January 5, 2017, 11:12am

I’n checking your project. As a Neos noob (beginner) I do not understand one thing. You have a Node named news. What magic creates data storage and it’s fields for storing the data?

trsh · January 5, 2017, 11:13am

And big thanks to all of you, for the input.

dimaip · January 5, 2017, 12:28pm

I store all news nodes under /site/a/, which was originally created manually. Editors create new nodes with CreateNodeButton. It’s automatically configured to create new nodes under that storage node: https://github.com/psmb/PsmbDistr/blob/a451f5d2af26c21731cdb71ba2cd9643bf03175f/Packages/Sites/Sfi.Site/Resources/Private/TypoScript/Objects/Body.ts2#L22

trsh · January 5, 2017, 12:49pm

My question was- What magic populates the Database.? Like news have different non standard fields, like, startDate, endData, title etc. How is the Custom node information stored on DB level? When? How?

trsh · January 5, 2017, 12:51pm

I see tables like xxx_news_related_yy. When, how is this stuff created? manually?

dimaip · January 5, 2017, 12:52pm

Eh where do you see those tables?

News are stored inside Neos content repository. Content repository stores properties in one blob field, json encoded.

trsh · January 6, 2017, 9:14am

Eh where do you see those tables?

github.com

psmb/PsmbDistr/blob/a451f5d2af26c21731cdb71ba2cd9643bf03175f/Packages/Application/Psmb.PsmbImport/Classes/Psmb/PsmbImport/DataProvider/NewsDataProvider.php

<?php
namespace Psmb\PsmbImport\DataProvider;

use Ttree\ContentRepositoryImporter\DataProvider\DataProvider;
use Ttree\ContentRepositoryImporter\DataType\StringValue;

class NewsDataProvider extends DataProvider {
	/**
	 * @return array
	 */
	public function fetch() {
		$result = [];
		$query = $this->createQuery()
		->select('*')
		->from('tx_news_domain_model_news', 'n')
		->where('n.hidden=0 AND n.deleted=0')
		->orderBy('n.datetime');
		$statement = $query->execute();
		while ($record = $statement->fetch()) {
			$date = new \DateTime();

This file has been truncated. show original

github.com

psmb/PsmbDistr/blob/a451f5d2af26c21731cdb71ba2cd9643bf03175f/Packages/Application/Psmb.PsmbImport/Classes/Psmb/PsmbImport/DataProvider/NewsRelationDataProvider.php

<?php
namespace Psmb\PsmbImport\DataProvider;

use Ttree\ContentRepositoryImporter\DataProvider\DataProvider;
use Ttree\ContentRepositoryImporter\DataType\StringValue;

class NewsRelationDataProvider extends DataProvider {
	/**
	 * @return array
	 */
	public function fetch() {
		$result = [];
		$query = $this->createQuery()
		->select('*')
		->from('tx_news_domain_model_news_related_mm', 'r')
		->orderBy('r.sorting');
		$statement = $query->execute();
		while ($record = $statement->fetch()) {
			$result[] = [
				'__label' => $record['uid_local'] . ' - ' . $record['uid_foreign'],

This file has been truncated. show original

trsh · January 6, 2017, 9:15am

Here, see above

dimaip · January 6, 2017, 10:05am

Ehh that’s just a data migration script from our old TYPO3 website…

trsh · January 6, 2017, 10:07am

SO all data regarding news is stored in a blob? How can I do order by Date for example?

dimaip · January 6, 2017, 10:10am

Via ElasticSearch index. E.g.: https://github.com/psmb/PsmbDistr/blob/master/Packages/Sites/Sfi.Site/Resources/Private/TypoScript/Objects/AnnouncementsList.ts2#L4

Neos is totally useless for larger websites without ElasticSearch integration. Luckily it’s quite easy to do.

Btw, if you want to ask a lot of quick questions about my setup, you can just ask me on our Slack, if you want to. Just ping me by my user name (dimaip).