Cr_default_p_neos_asset_usage

jan82 · September 26, 2024, 7:35am

Hello

I am using Neos 9 beta 13 and I have today with

./flow content:createVariantsRecursively

created my nodes in three other dimensions. Now unfortunately I have a cr_default_p_neos_asset_usage table with around 10 gigabytes of data. I’ve tried a few things: contentstream:prune and then a cr:projectionreplayall, but the table blows up again. The system now reacts extremely slowly when publishing the workspace. Does anyone have an idea what I can do?

Greetings,

Jan

lubitz · September 26, 2024, 10:38am

Hi Jan,

thanks for reporting this. This is very helpful for us.

We are already in the process of rewriting the asset usage storage, which should™ also address your issue.

github.com/neos/neos-development-collection

FEATURE: AssetUsage as CatchUpHook

neos:9.0 ← dlubitz:90/feature/asset-usage-again

opened 06:27PM - 23 Sep 24 UTC

dlubitz

+3840 -974

This is an overhaul of the latest "asset usage" implementation which where using… a dedicated projection to keep track of asset usages. As this was not handling all necessary events we tried to improve this, but got stuck due to the need of a lot of information of the graph, workspaces, etc (see https://github.com/neos/neos-development-collection/pull/5109). Therfore we decided to try a new approach to use a CatchUpHook instead, which enables us to access all needed information without re-implementing the whole graph logic. **General approach** The package stores the usage of an asset per node and property in a dimension space point. It only stores the usage in the highest/base workspace (usually "live") as long as there are no changes in the same property in a lower/dependent workspace (e.g. user workspace). This allows to get easily an answer to the following questions: 1) Which nodes do I need to edit, to remove all usages for a safe asset removal (Neos Asset Usage/Media Browser) 2) Which cache entries do I need to flush on a change to an asset (this requires an additional traversal over all dependent workspaces). **CatchUpHook** The CatchUpHook is keeping track of all changes to node properties referencing an asset. On replay the CatchUpHook is not handling the events for performance reasons. **Indexing Command** The indexing command fills up the index initially. It can also be used to rebuild the index after projection replays or imports of events. ``` ./flow assetusage:index --contentrepository default ``` Fixes: #4808 #5084

This is currently in the review process and will make it hopefully into the next beta release.

May I ask you for patience until the next beta release?

If there is there any chance to get a dump of your event stream, I could check in depth if this issue will still persist after the overhaul.

Best
Denny

jan82 · September 26, 2024, 12:59pm

Thank you for your work. I sent you a link to a SQL dump. I help myself right now by exporting and importing again

lubitz · September 26, 2024, 1:19pm

Great, I will have a look into this.

jan82 · September 26, 2024, 3:01pm

Even if I import from an export, there are a lot of entries created. But much less:

jan82 · September 26, 2024, 3:05pm

And that happens after the first login:

jan82 · September 26, 2024, 3:28pm

And when I run

./flow contentgraphintegrity:runviolationdetection

I get:

Hierarchy relation {“position”:128,“contentstreamid”:“9ac41adf-835f-4a93-9774-403cebba594b”,“dimensionspacepointhash”:“5797e6cfaca8c2b6d12c0fbb5c753642”,“parentnodeanchor”:32,“childnodeanchor”:33,“subtreetags”:“{}”} is disconnected.

Subgraph defined by content stream a0682a3e-7a8f-4b70-bdb9-9aeef55b84e5 and dimension space point {“language”:“fr”} is cyclic for node aggregates 9530b0a6-391a-4f69-b59f-be3e6a63c0d6,4656bc1a-cbff-4742-a37d-5e7b0727bb3d,019bceb1-6cc6-43a1-aeae-2f3bcd19a40d,d5ee101f-acdc-421d-9b56-824e2ea0af48,afd2322e-60d6-450a-ad83-e234e804fafc,8f84f268-0c59-410d-8afe-2c6c5f77545c,db697162-b712-41d2-82f9-562e7d56d7ed

and

Destination node aggregate 11e33631-2145-4874-a0d9-a6c024058dfb does not cover any dimension space points the source ea20c788-84b2-4bea-a94c-679c1c2daa01 does in content stream 9ac41adf-835f-4a93-9774-403cebba594b

These are only examples…

lubitz · October 2, 2024, 9:23am

Hi @jan82,

I had a look into your data and as I said this issue with the asset usage table will be solved with the next beta release. We completly changed the way how the asset usages are stored, which is more efficent.

Regarding your disconnected nodes, I would suggest to create a new topic, as this seems to be unrelated to the asset usage.

Best
Denny

jan82 · October 3, 2024, 6:35am

Thank you!

jan82 · October 14, 2024, 7:30pm

I’ve now installed beta 14. However, I’m facing two problems:

I cannot excecute the nodemigration 20240924095647 - my system doesn’t find any migrations. The list is empty an trying to run the migrations leads to an “Error on applying node migrations:
Specified version is not available.”
When opening the website I get a:

Exception #1258721059 in line 62 of /home/httpdocs/Data/Temporary/Development/Cache/Code/Flow_Object_Classes/Neos_Flow_Security_Aspect_PolicyEnforcementAspect.php: The security context contained no tokens which could be authenticated.
Evaluated following 1 privilege target(s):
“Neos.Neos:AllControllerActions”: ABSTAIN
(0 granted, 0 denied, 1 abstained)

35 Neos\Flow\Security\Authorization\Interceptor\PolicyEnforcement_Original::invoke()
34 Neos\Flow\Security\Aspect\PolicyEnforcementAspect_Original::enforcePolicy(Neos\Flow\Aop\JoinPoint)
33 Neos\Flow\Aop\Advice\AroundAdvice::invoke(Neos\Flow\Aop\JoinPoint)
32 Neos\Flow\Aop\Advice\AdviceChain::proceed(Neos\Flow\Aop\JoinPoint)
31 Neos\Flow\Mvc\Controller\StandardController::indexAction()
30 Neos\Flow\Mvc\Controller\ActionController_Original::callActionMethod(Neos\Flow\Mvc\ActionRequest, Neos\Flow\Mvc\Controller\Arguments, GuzzleHttp\Psr7\Response)
29 Neos\Flow\Mvc\Controller\ActionController_Original::processRequest(Neos\Flow\Mvc\ActionRequest)
28 Neos\Flow\Mvc\Dispatcher_Original::initiateDispatchLoop(Neos\Flow\Mvc\ActionRequest)
27 Neos\Flow\Mvc\Dispatcher_Original::dispatch(Neos\Flow\Mvc\ActionRequest)
26 Neos\Flow\Mvc\DispatchMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
25 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
24 Neos\Flow\Http\Middleware\SecurityEntryPointMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
23 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
22 Neos\Flow\Http\Middleware\RequestBodyParsingMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
21 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
20 Neos\Flow\Mvc\FlashMessage\FlashMessageMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
19 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
18 Neos\Flow\Http\Middleware\PoweredByMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
17 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
16 Neos\Flow\Mvc\Routing\RoutingMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
15 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
14 Neos\Neos\FrontendRouting\SiteDetection\SiteDetectionMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
13 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
12 Neos\FluidAdaptor\Core\Widget\AjaxWidgetMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
11 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
10 Neos\Flow\Http\Middleware\SessionMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
9 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
8 Neos\Flow\Http\Middleware\MethodOverrideMiddleware::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
7 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
6 Neos\Flow\Http\Middleware\TrustedProxiesMiddleware_Original::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
5 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
4 Neos\Flow\Http\Middleware\StandardsComplianceMiddleware::process(GuzzleHttp\Psr7\ServerRequest, Neos\Flow\Http\Middleware\MiddlewaresChain)
3 Neos\Flow\Http\Middleware\MiddlewaresChain::handle(GuzzleHttp\Psr7\ServerRequest)
2 Neos\Flow\Http\RequestHandler::handleRequest()
1 Neos\Flow\Core\Bootstrap::run()

Any ideas?

jan82 · October 14, 2024, 8:18pm

Okay, I found out that it was a path issue. After copying the the Migration file to the correct folder (Migrations/ContentRepository/Version20240924095647.yaml with “s” - I used the path from the yaml-file here: Migrating dimension config [v9] - Content Dimensions - Content Repository - Guide - Neos Docs) it worked. Now I can log into the backend, but my frontend is still not working:

Fatal error: Uncaught TypeError: Neos\Neos\Fusion\Helper\NodeHelper_Original::nearestContentCollection(): Argument #2 ($nodePath) must be of type string, null given in Data/Temporary/Development/Cache/Code/Flow_Object_Classes/Neos_Neos_Fusion_Helper_NodeHelper.php:56 Stack trace: #0 [internal function]: Neos\Neos\Fusion\Helper\NodeHelper_Original->nearestContentCollection(Object(Neos\ContentRepository\Core\Projection\ContentGraph\Node), NULL)

jan82 · October 14, 2024, 8:37pm

Okay, the migration didn’t succeed - although there was no error. I’ve still only one Neos.Neos:Site (empty dimension), although I’ve 4 configured dimensions.

I now tried the following:

created a new language dimension
create a new migration to this new empty dimension ( MoveDimensionSpacePoint accoring to 20240924093814.yaml)
new dimension showed up in “cr_default_p_graph_dimensionspacepoints”, but Site-Nodes was not assigned
UpdateRootNodeAggregateDimensions didn’t multiplicate the Site-Node’s, too

I’ll go to bed now - maybe I’ll have a good idea tomorrow

jan82 · October 15, 2024, 4:49am

It’s my footer. I configured in fusion:

     content = Neos.Neos:ContentCollection {
         nodePath = ${q(site).children('footer').property('_path')}
         attributes.class = 'content'
     }

and in the template:

<div @if={props.content} class="footer-information__entry">
{props.content}
</div>

This makes the complete page crash. There is a content collection at site.footer. I cannot even make a Debug Output of props.content without crashing. No idea what’s wrong there, but it works with a static footer.

However, the whole beta14 feels to be much more responsive. Thank you for your hard work.

jan82 · October 15, 2024, 7:07am

Is it now possible to remove the old asset usage table?

Marc · October 16, 2024, 10:33am

Hi:) Thanks for experimenting already with the new beta^^

Im currently still writing the beta14 release notes they should appear any minute here:https://discuss.neos.io/t/neos-9-beta-14-release/6648/1

Regarding your ContentCollection i can say:

A special cases is fetching the path to render a content collection which can be simplified:

content = Neos.Neos:ContentCollection {
-   nodePath = ${q(site).children('footer').property('_path')}
+   @context.node = ${q(site).children('footer').get(0)}
}

Is it now possible to remove the old asset usage table?

yes

Regarding your missing S in

Migrations /ContentRepository

is there anything we can fix with our documentation?

However, the whole beta14 feels to be much more responsive. Thank you for your hard work.

thank you so much we are glad to hear that!!!

jan82 · October 16, 2024, 10:59am

Hi Marc

That works. Thank you.

This missing “s” mislead me…

Jan