TL;DR: we welcome you to join us every Friday at 9.30am 11am CET to discuss about the Event Sourced Content Repository (or just listen, if you prefer!)
As discussed at the Neos Sprint, we want to (re-)establish a weekly remote meetup to discuss aspects of the “Event Sourced Content Repository” (ESCR).
Our motivation and main goals are:
Keep up the pace (currently every progress stops very easily, but I hope that we can change that if we constantly keep the topics in our RAM aka brain)
Prevent “wood for the trees” symptom (i.e. get more “outside” opinions on important decisions)
Spread the knowledge (in order to attract more developers & prospects)
Enable more people to work on individual topics
The plan is to do a weekly meeting every Friday at 9.30am CET for about two hours and I would suggest to apply the following guidelines:
Everyone is invited to join (actively or silently)
The meetings will be in English, unless there are only German speaking participants (in which case it will be Russian of course *g)
I’ll create a discuss post (within this very thread) after each weekly to briefly summarize the discussed topics (important ideas/decisions should be tracked in Github issues)
In these posts we should also plan the agenda for the upcoming session
I’ll volunteer to organize the agenda and to timebox the topics during the discussion (I suggest to stick to the schedule as close as possible in order to allow others to join specific topics exclusively – and to prevent us from losing focus)
I would love to see that at the end of the meeting we are able to agree on a roadmap for a first release. So my topic proposal would be: What are the current blockers and how can we (support to) solve them?
Thanks for your feedback, it’s appreciated!
I’m afraid a final roadmap is a little ambitious for our first session because it depends on many factors, mainly: available development power.
Also: Sebastian won’t be there this time, and he definitely has a say in that *g
thanks for joining the first session and sorry for the somewhat chaotic execution. I hope that the next sessions will be a bit more structured and interesting for everyone!
=> I created a bug report and will work on a fix
=> IMO it is important that we keep the initial setup as easy and stable as possible. I wonder why the CI doesn’t fail currently. If it is not affected, we should maybe tweak it so that we catch setup regressions before a PR is merged already
Anatomy of the event stream
We took the chance to look at the event stream of the imported Neos.Demo site and stumbled upon two oddities:
There were 5 NodeAggregateWasDisabled events – fortunately this is not a bug of the ESCR but a curiousity of the Neos.Demo site export (=> https://github.com/neos/Neos.Demo/issues/118)
We got a little lost in discussions about “versioning”, "snapshots, “nested live event streams” and “Postgres support”.
I’ll try to summarize what I can remember (feel free to correct/amend):
Versioning seems to be a feature that many customers would love to see
With our current implementation especially the “live content stream” will grow substantialy over time. This will make it harder and more expensive to work with the ESCR (e.g. replays & new projections take longer, data consumption, …)
Snapshots in the original sense probably won’t help us (and is rather impractical for the graph projection)
Would it help to introduce some kind of “closing the books” process?
(How) can we implement a “compact event stream” mechanism that reduces the events to the minimum that is needed for the current state? (=> maybe even with an option to compact older events more than newer ones, => we’ll probably need that for a proper export format as well)
Postgres has some major advantages over MySQL/MariaDB, e.g. “hyper edges” (reduce the amount of edges/copy operations drastically, especially for large projects with many dimensions), better JSON support and fulltext search
Personally, I would suggest that we try to keep support for MySQL as much as possible, i.e. by avoiding too many features to be added into the Graph projection and rather provide clean PHP APIs so that the implementations can be replaced
Outlook
The time slot (9.30-11.30am) seems to be suboptimal since it collides with dailies and reduces the preparation time. I would suggest to move the slot to 11am-1pm instead. What do you think?
To be honest: I am not sure if it makes sense to keep this weekly, but I’m happy to give it another try. Feel free to suggest topics for the next agenda
The next “ESCR Talk” will take place on Friday, November 26th at 11.00am CET.
export format: line-delimited JSON, every line is an Event from the event store
Potential export formats (v2):
a) export events 1:1
b) export only selected streams
c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
d) (longer-term): “compact event stream” up to time X
=> Bastian will work on a v1
P3 - Rebasing fine tuning / conflicts (#144, #146)
Currently we rebase for every Neos login
idea: skip rebase if not required
in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)
ToDo: Asset Usage Projection maybe? (to export files as well)
Do we have to keep track of the full hierarchy?
Alternative: some kind of cleanup that removes entries from the projected read model (or emits new events)
Maybe we do need explicit NodeWasDeleted events for all descendant nodes of a removed nodes
To avoid huge amounts of events, we could detect those cases and force a “compraction”
Idea: maybe we can use the existing NodeSubtreeSnapshot for a simple compraction
Sebastian:
Updated README
Helped Bernhard with Postgres adapter (merged master into feature branch, adjusted behat tests, stumbled upon some quirks)
Questions:
How to deal with the distribution providing multiple adapter implementations (by default: leads to exception since no instance is explicitly configured)
Bernhard: Test suite will be better organized (=> configurable threshold to test multiple adapters in different maturity)
export format: line-delimited JSON, every line is an Event from the event store
Potential export formats (v2):
a) export events 1:1
b) export only selected streams
c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
d) (longer-term): “compact event stream” up to time X
=> Bastian has started on it, working on V1
P3 - Rebasing fine tuning / conflicts (#144, #146)
Currently we rebase for every Neos login
idea: skip rebase if not required
in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)
Decision & Kickstart: Try to run the docs.neos.io site on the ES CR! (#127)
ES CR Sync 11.1. (from memory)
(Sebastian, Bastian)
Sebastian was very busy and pushed 23 Commits to fix incompatibilities with the docs site
Remaining issues (currently):
Caches are not flushed properly
idea: callback mechanism to hook into before and after ContentGraph projector transactions so that we can trigger the cache flusher during projection time (via JobQueue!)
related: The GraphProjector should be a class in the ESCR package using a new interface for the actual read model interaction (this will allow us to implement (and potentially refactor) core features like GraphProjector::hasProcessed() and the hook mechanism mentioned before in a single place)
Problem: The NodeSearchService needs a custom implementation. Unfortunately it currently has some quirks:
the $startingPoint argument (which is not part of the NodeSearchServiceInterface!) is actively used and makes this implementation complex/slow with the ESCR
the return type is an array of NodeInterface instances but effectively it is indexed by the corresponding node path
Idea: For a v1 Bastian will try to implement the interface with a custom (DBAL-based) projection and couple of limitations:
The $startingPoint argument won’t be supported (if it’s specified an exception will be thrown(?))
The result will be limited to 15 nodes (reason: we’ll need to interact with the ContentGraph in order to get the node path and this is a rather slow operation)
For the final version we’ll probably need a new interface along the lines of SomeClass::someMethod(SearchTerm $term, NodeTypeConstraints $filter, ContentStreamIdentifier $contentStreamId, DimensionSpacePoint $dimensionSpacePoint): Nodes. (note: We might get around the startingPoint-requirement by using a CR per site, see below).
Publish button does not turn yellow for deleted/disabled nodes (#89)
Problem: The ContentGraph does not keep track of deleted nodes and a custom projection won’t know the affected document node of a deleted content node (unless it keeps track of the full node hierarchy)
Ideas:
We could implement some kind of “DeletionEdge” (similar to the VisibilityEdges)
For now we decided to add the parentNodeAggregateIdentifier to the payload of the NodeAggregateWasRemoved event – but that probably won’t be enough for detecting deletions in deeply nested content nodes…
We agree that the state of the Publish Button is closely related to the feature of diffing two content streams (e.g. like we gonna need it for a revamped Workspace Module)
We seem to come back to the issue of a “site context” and all the challenges that come with it
Ideas:
Allow multiple ContentRepository instances to be instantiated in parallel
In Neos a middleware could detect the correct “default” instance based on the current request
[IMO] This CR instance would be the major API for all core interactions, i.e. act as command bus factory for Content(Sub)Graph instances and other core read models and allow for registering custom event listeners (if we go that route we won’t have a global event store but one per CR instance, so we can’t have global event listeners either)
→ implement adapter for the neos/redirecthandler (#210)
(Bernhard) working on DimensionSpace to support PHP 8.1
→ adjustments to Neos CR
Flow Adjustments to support PHP 8.1
Other discussions
PHP 8.1 - Do we need/want a NodeInterface?
aspects:
ease of use for custom “entity node types”: ${product.specialPrice} vs ${My.Product(node).specialPrice} (or $product->someDomainLogic() vs $productService->getProduct(...)->someDomainLogic() in PHP)
(unplanned) extensibility
assumption 1 (Sebastian): it is easier to extend the behavior with an interface (e.g. lazy properties)
assumption 2 (Bastian): it is easier to change/extend the implementation if we have a final implementation
we are not sure yet
→ TODO: collect pain points from similar decisions in the past
add “affected dimension space points” to relevant events #163
The next “ESCR Talk” will take place on Friday, February 18th at 11.15am CETFriday, February 25th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel
if in doubt, “fail fast” and avoiding inconsistencies.
old and new CR should never run “in parallel”
avoid AOP and Objects.yaml “rewiring” as much as possible
mock implementations of removed classes in order to provide better errors?
We want people to start with new ES CR for new projects soon ™
Current Situation:
Neos.Neos and Neos.Neos.Ui works with Neos.ContentRepository
Neos.EventSourcedNeosAdjustments “patches” the above to work with the new stuff.
Neos.EventSourcedContentRepository depends (in small parts) on Neos.ContentRepository (Node Type Mgmt)
Target Situation
Neos.ESCR.ContentRepository (formerly Neos.EventSourcedContentRepository) does NOT depend on Neos.ContentRepository anymore
→ We simply copy over the NodeTypeManagment stuff and REBUILD with new type hints.
→ for migration, we directly read NodeData table (instead of using ORM)
→ Export NodeData table into events export (to be imported with new Export/Import functionality)
Try to avoid cases where two packages are installed that serve the same purpose (e.g. Neos.ESCR.ContentRepository && Neos.ContentRepository)
maybe even add conflicts to composer manifest
what about neos/neos-ui and neos/neos?
e.g. routing is deeply integrated in neos/neos
idea #1: extract routing into (Neos.Neos.Routing and Neos.ESCR.Routing)?
→ 10-20 packages in the old and new world
many “implicit knowledge” in the trenches.
idea #2: we maintain a separate branch?? for “new”?
Neos v8 vs neos v18
→ remove all @api annotations (except FlowQuery / Fusion) and communicate that this is an intermediate solution
idea #3: we keep some sort of “LegacyNeosAdjustments” package (so like Neos.EventSourcedNeosAdjustments today but the “other way around”)
→ we thought through this, and we will have a scenario where if you ONLY want to install the old world, the “new” upstream code will still depend on ESCR (which is not installed) → CRASH. → this idea will not work.
idea #4: we keep some sort of “EventSourcedNeosAdjustments” package like it exists right now
→ same as #3, just the other way round.
idea #5: use (composer) patches
dangerous because it partly defeats SemVer
idea #6
Neos.Neos stays Neos.Neos and will never be used “in the new world”. Instead we would extract “things” that we need for the ESCR to work into separate packages
idea #7
communicate “ES Neos” as new Product (“Neos Flow 9.0”, “Neos CMS 9.0”, “Neos {New Name} 1.0”)
→
idea #8 (our preference currently): simply release Neos 9.0 with EventSourcing
→ create new branch “escr” (or similar) and apply all changes from “EventSourcedNeosAdjustments” → that branch will become “main” in the future
→ we’ll have to take care to “upmerge” fixes and relevant changes (probably partly manually)
→ maybe we can automate that
Next steps
Bastian
finalize Neos.ESCR.Export package and create exporter from NodeData => events.jsonl
Sebastian
write discuss post with the suggestions above
Bernhard
experiment with Cypher queries
ran into issues with the PRs for type safety, will contact @albe for support
The next “ESCR Talk” will take place on Friday, March 11th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel
After a longer break (containing NeosCon and Sprint) we had another great session today with @sebastian and @Nezaniel:
Sebastian has been very active recently and managed to get most of Neos working again after the “big merge”.
One of the remaining tasks is the support for multiple sites and content dimensions (e.g. the ContentDimensionMenu).
The introduction of ContentRepository instances will affect all layers and the way content dimensions are configured.
It will also require some mapping between Sites and a CR instance as well as custom routing dimension resolution logic per site.
Some goals that we had while discussing possible solutions:
Keep configuration simple for the regular use cases
Avoid millions of options in favor of replaceable PHP implementations
Keep configuration & concepts explicit
Keep CR and Neos concerns separated
and some ideas we had to approach it:
Introduce content repository instance
Currently the CR configuration is global (e.g. underneath Neos.ContentRepository.contentDimensions).
In order to support multiple CR instances, a new level is required, for example:
some-cr-instance is an arbitrary name referring to a single Content Repository instance. It could be prefixed in order to avoid naming conflicts.
For the main Neos CR instance it could be named after the site package key, but since multiple sites can share the same instance, this might be confusing.
Maybe it makes sense to already define some default ContentRepository that is used by default (see below).
Site specific settings
We briefly discussed whether it makes sense to use ApplicationContext or another “context” to allow for site specific configuration on a global level.
For now we ditched that idea because it could be a Pandora’s box.
Instead we decided to require affected settings to explicitly refer to a CR instance by its name. This will be required for custom routing behavior for example (see below).
We did not come up with a concept to have node type definitions per site, yet.
Dimensions / Routing
Currently configuration for content dimensions and the resolution logic (get from URL to dimension vice versa) is done in a single place.
Example:
Neos:
ContentRepository:
contentDimensions:
language:
label: 'Some label'
icon: icon-language
default: en_US
defaultPreset: en_US
presets:
en_US:
label: 'English (US)'
values:
- en_US
uriSegment: en
en_UK:
label: 'English (UK)'
values:
- en_UK
- en_US
uriSegment: uk
de:
label: German
values:
- de
uriSegment: de
We decided that it makes sense to separate the (Neos specific) routing configuration.
Together with the suggestion above, the result would be something like:
Neos:
ContentRepository:
instances:
'default':
contentDimensions:
language:
label: 'Some label'
icon: icon-language
values:
en_US:
label: 'English (US)'
specializations:
en_UK:
label: 'English (UK)'
de:
label: German
specializations:
nl:
label: Dutch
fr:
label: French
da:
label: Danish
lv:
label: Latvian
The corresponding routing-related configuration would reside underneath Neos.Neos:
some-site-name would refer to a unique name for a site. It can be the package name but one package could contain multiple sites so it probably makes sense to keep those separated.
Note: We should consider using the SiteName as NodeAggregateIdentifier of the site’s root node
Default configuration
In order to avoid having to configure all of the above for all sites even though they might only use a single dimension resolution logic, we thought about introducing a special default site name, represented as * (Note: This slightly contradicts the goal of avoiding magic but in this case it’s probably worth the added complexity).
Neos could ship with a default configuration of (something like):
So that special configuration is only required when deviating from the default behavior.
Question: Does it make sense to take the first configured dimension value as default (or keep the default as part of the CR configuration), so that we don’t have to add any custom configuration for the most cases?
addendum: I briefly discussed this question with @sebastian and he thinks that it makes sense to make the “default dimensions” a concept of Neos (this way it will be possible to share the CR between multiple sites but have a different default dimension for example).
I agree but would prefer if we would not have to touch the Neos configuration to get a sane default behavior (and we need default dimension values at least for the homepage).
If we stick to configuring the default as part of the Neos site configuration, it should at least not be specific to routing IMO because we need it in other places, too (e.g. for the initial rendering in the Neos Backend).
Implementation / Extension points
Implementation of the dimension resolution will probably based on an older PR:
But that still worked with the idea of a “global content repository”.
With multiple CR instances we’ll have to detect the current site and CR instance beforehand.
We consider two middlewares:
One that
…detects the current site from the HTTP request (similar logic to how we currently do it in the FrontendNodeRoutePartHandler)
…detects the current CR instance (by reading configuration from `Neos.Neos.sites[<site-name-or-asterisk>].contentRepository
…stores site name and CR name into the Request attributes
And the (already existing) RoutingMiddleware that
…puts resolved site and CR instance into the RouteParameters
…starts the routing as before
Note: We decided to put Site/CR detection into a dedicated Middleware because we need the current CR in the rendering layer, too (e.g. for the fusion context).
Next up
Sebastian wants to work on the above
Bastian will continue to work on a solid foundation for Import/Export with support for importing from “legacy systems”
The next “ESCR Talk” will take place on Friday, June 24th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel
We looked at the current state of the “Legacy CR Migration”: I managed to export a Neos 7.3 Demo Site to events and to import that (< 1 Second on my MacBook + ~2 Seconds projection replay).
Next
Migrate assets
Improve UX (create site, better handling of “event store already contains events”, …)
We want to create a command that can create a plaintext file containing all nodes for “old” and “new world” in order to verify that the migration produces the right events in all cases.
The current structure of the NodeAggregateWasMoved is rather complex. Especially the NodeMoveMappings are hard to grasp.
Besides, the hash of the affected ContentDimensionSpacepoints is expected as keys for NodeVariantAssignments::createFromArray() which makes this error prone. Also it makes the internal implementation detail part of the event payload.
We collected some ideas on how to change that.
Dimensions / Routing
Sebastian worked on the Content Dimension resolution and already has most of it working.
We had a fruitful discussion about architecture and namings.
One idea we came up with is a central authority to get “details” of a specific site.
In practice it could look something like this (names and signatures to be finalized):
It might seem like there’s slow progress on the ESCR project currently, but the opposite is the case. @sebastian is incredibly active and we discuss multiple times per day. @Nezaniel also makes great postgress (pun intended).
I decided to no longer put the effort into a weekly update here because IMO that time is better spent on the implementation.
