Cr_default_p_neos_asset_usage

jan82 · September 26, 2024, 7:35am

Hello

I am using Neos 9 beta 13 and I have today with

./flow content:createVariantsRecursively

created my nodes in three other dimensions. Now unfortunately I have a cr_default_p_neos_asset_usage table with around 10 gigabytes of data. I’ve tried a few things: contentstream:prune and then a cr:projectionreplayall, but the table blows up again. The system now reacts extremely slowly when publishing the workspace. Does anyone have an idea what I can do?

Greetings,

Jan

lubitz · September 26, 2024, 10:38am

Hi Jan,

thanks for reporting this. This is very helpful for us.

We are already in the process of rewriting the asset usage storage, which should™ also address your issue.

github.com/neos/neos-development-collection

FEATURE: AssetUsage as CatchUpHook

neos:9.0 ← dlubitz:90/feature/asset-usage-again

opened 06:27PM - 23 Sep 24 UTC

dlubitz

+3840 -974

This is an overhaul of the latest "asset usage" implementation which where using… a dedicated projection to keep track of asset usages. As this was not handling all necessary events we tried to improve this, but got stuck due to the need of a lot of information of the graph, workspaces, etc (see https://github.com/neos/neos-development-collection/pull/5109). Therfore we decided to try a new approach to use a CatchUpHook instead, which enables us to access all needed information without re-implementing the whole graph logic. **General approach** The package stores the usage of an asset per node and property in a dimension space point. It only stores the usage in the highest/base workspace (usually "live") as long as there are no changes in the same property in a lower/dependent workspace (e.g. user workspace). This allows to get easily an answer to the following questions: 1) Which nodes do I need to edit, to remove all usages for a safe asset removal (Neos Asset Usage/Media Browser) 2) Which cache entries do I need to flush on a change to an asset (this requires an additional traversal over all dependent workspaces). **CatchUpHook** The CatchUpHook is keeping track of all changes to node properties referencing an asset. On replay the CatchUpHook is not handling the events for performance reasons. **Indexing Command** The indexing command fills up the index initially. It can also be used to rebuild the index after projection replays or imports of events. ``` ./flow assetusage:index --contentrepository default ``` Fixes: #4808 #5084

This is currently in the review process and will make it hopefully into the next beta release.

May I ask you for patience until the next beta release?

If there is there any chance to get a dump of your event stream, I could check in depth if this issue will still persist after the overhaul.

Best
Denny

jan82 · September 26, 2024, 12:59pm

Thank you for your work. I sent you a link to a SQL dump. I help myself right now by exporting and importing again

lubitz · September 26, 2024, 1:19pm

Great, I will have a look into this.

jan82 · September 26, 2024, 3:01pm

Even if I import from an export, there are a lot of entries created. But much less:

jan82 · September 26, 2024, 3:05pm

And that happens after the first login:

jan82 · September 26, 2024, 3:28pm

And when I run

./flow contentgraphintegrity:runviolationdetection

I get:

Hierarchy relation {“position”:128,“contentstreamid”:“9ac41adf-835f-4a93-9774-403cebba594b”,“dimensionspacepointhash”:“5797e6cfaca8c2b6d12c0fbb5c753642”,“parentnodeanchor”:32,“childnodeanchor”:33,“subtreetags”:“{}”} is disconnected.

Subgraph defined by content stream a0682a3e-7a8f-4b70-bdb9-9aeef55b84e5 and dimension space point {“language”:“fr”} is cyclic for node aggregates 9530b0a6-391a-4f69-b59f-be3e6a63c0d6,4656bc1a-cbff-4742-a37d-5e7b0727bb3d,019bceb1-6cc6-43a1-aeae-2f3bcd19a40d,d5ee101f-acdc-421d-9b56-824e2ea0af48,afd2322e-60d6-450a-ad83-e234e804fafc,8f84f268-0c59-410d-8afe-2c6c5f77545c,db697162-b712-41d2-82f9-562e7d56d7ed

and

Destination node aggregate 11e33631-2145-4874-a0d9-a6c024058dfb does not cover any dimension space points the source ea20c788-84b2-4bea-a94c-679c1c2daa01 does in content stream 9ac41adf-835f-4a93-9774-403cebba594b

These are only examples…

lubitz · October 2, 2024, 9:23am

Hi @jan82,

I had a look into your data and as I said this issue with the asset usage table will be solved with the next beta release. We completly changed the way how the asset usages are stored, which is more efficent.

Regarding your disconnected nodes, I would suggest to create a new topic, as this seems to be unrelated to the asset usage.

Best
Denny

jan82 · October 3, 2024, 6:35am

Thank you!