Weekly ESCR Talk

Dear community,

TL;DR: we welcome you to join us every Friday at 9.30am 11am CET to discuss about the Event Sourced Content Repository (or just listen, if you prefer!)

As discussed at the Neos Sprint, we want to (re-)establish a weekly remote meetup to discuss aspects of the “Event Sourced Content Repository” (ESCR).

Our motivation and main goals are:

  • Keep up the pace (currently every progress stops very easily, but I hope that we can change that if we constantly keep the topics in our RAM aka brain)
  • Prevent “wood for the trees” symptom (i.e. get more “outside” opinions on important decisions)
  • Spread the knowledge (in order to attract more developers & prospects)
  • Enable more people to work on individual topics

The plan is to do a weekly meeting every Friday at 9.30am CET for about two hours and I would suggest to apply the following guidelines:

  • I created a recurring event in the Neos calendar starting next week (you should be able to subscribe to it via https://calendar.google.com/calendar/embed?src=bastian%40neos.io)
  • Meetings will be in form of a Slack call in the #project-cr-rewrite Channel
  • Everyone is invited to join (actively or silently)
  • The meetings will be in English, unless there are only German speaking participants (in which case it will be Russian of course *g)
  • I’ll create a discuss post (within this very thread) after each weekly to briefly summarize the discussed topics (important ideas/decisions should be tracked in Github issues)
  • In these posts we should also plan the agenda for the upcoming session
  • I’ll volunteer to organize the agenda and to timebox the topics during the discussion (I suggest to stick to the schedule as close as possible in order to allow others to join specific topics exclusively – and to prevent us from losing focus)

I hope to see many of you next Friday!

5 Likes

The next (and first official) “ESCR Talk” will take place on Friday, November 19th at 9.30am CET.

Agenda

  • 9.30 - 10.30 Initial (development) setup
  • 10.30 - 11.30 Debugging tools

Feel free to get in touch with me via Slack, E-Mail or by commenting this post if you want to suggest topics for the upcoming session.

I would love to see that at the end of the meeting we are able to agree on a roadmap for a first release. So my topic proposal would be: What are the current blockers and how can we (support to) solve them?

Thanks for your feedback, it’s appreciated!
I’m afraid a final roadmap is a little ambitious for our first session because it depends on many factors, mainly: available development power.
Also: Sebastian won’t be there this time, and he definitely has a say in that *g

Having said that: One of the aims of this weekly session is to attract more developers so we can hopefully come up with better estimates soon.
Furthermore we should keep https://github.com/neos/contentrepository-development-collection#feature-comparison up-to-date so that everyone can get a better overview of the current state.

Hi all,

thanks for joining the first session and sorry for the somewhat chaotic execution. I hope that the next sessions will be a bit more structured and interesting for everyone!

Recap

Setup

We followed the simple installation steps described at https://github.com/neos/neos-development-distribution/tree/event-sourced#quick-start but unfortunately the current setup is broken due to patches that don’t apply to latest Neos versions.

=> I created a bug report and will work on a fix
=> IMO it is important that we keep the initial setup as easy and stable as possible. I wonder why the CI doesn’t fail currently. If it is not affected, we should maybe tweak it so that we catch setup regressions before a PR is merged already

Anatomy of the event stream

We took the chance to look at the event stream of the imported Neos.Demo site and stumbled upon two oddities:

  1. There were 5 NodeAggregateWasDisabled events – fortunately this is not a bug of the ESCR but a curiousity of the Neos.Demo site export (=> https://github.com/neos/Neos.Demo/issues/118)
  2. There were 19 NodePropertiesWereSet events – also not a bug per se, but a possibility for improvements => https://github.com/neos/contentrepository-development-collection/issues/200

Tooling/Debugging

We got a little lost in discussions about “versioning”, "snapshots, “nested live event streams” and “Postgres support”.
I’ll try to summarize what I can remember (feel free to correct/amend):

  • Versioning seems to be a feature that many customers would love to see
  • With our current implementation especially the “live content stream” will grow substantialy over time. This will make it harder and more expensive to work with the ESCR (e.g. replays & new projections take longer, data consumption, …)
    • Snapshots in the original sense probably won’t help us (and is rather impractical for the graph projection)
    • Would it help to introduce some kind of “closing the books” process?
    • (How) can we implement a “compact event stream” mechanism that reduces the events to the minimum that is needed for the current state? (=> maybe even with an option to compact older events more than newer ones, => we’ll probably need that for a proper export format as well)
  • Postgres support is in the works
    • Postgres has some major advantages over MySQL/MariaDB, e.g. “hyper edges” (reduce the amount of edges/copy operations drastically, especially for large projects with many dimensions), better JSON support and fulltext search
    • Personally, I would suggest that we try to keep support for MySQL as much as possible, i.e. by avoiding too many features to be added into the Graph projection and rather provide clean PHP APIs so that the implementations can be replaced

Outlook

  • The time slot (9.30-11.30am) seems to be suboptimal since it collides with dailies and reduces the preparation time. I would suggest to move the slot to 11am-1pm instead. What do you think?
  • To be honest: I am not sure if it makes sense to keep this weekly, but I’m happy to give it another try. Feel free to suggest topics for the next agenda

The next “ESCR Talk” will take place on Friday, November 26th at 11.00am CET.

Agenda

  • 11.00 -

(feel free to suggest topics)

2 Likes

ES CR Sync 26.11.

(Christian, Sebastian, Bernhard, Bastian)

Current Status / Done

  • Bastian updated us on the last week’s discussions
  • Node Migrations merged and lots of other changes as part there

Next Priorities

Next Steps

  • Readme Update (Sebastian)

Fix Setup

P1 - Test Refactor / Postgres support

  • maybe try to merge “without Postgres being finished”, as long as the old tests run through

=> Bernhard will work on that

P1 - Workspace module

  • Decide what to do with the workspace module (currently broken)
    • Can we “outsource” this topic partly to the community (i.e. start with wireframes for a module v2)
    • Dream: $someone builts a new “diff” module that we can fill via a clean API (DTO, …)
  • Nested Workspaces
  • Introduce concept of “Changeset” (e.g. allow editors to switch between, annotate and explicitly mark for review → “Workflow API”)
    • Allow working in content streams that are not assigned to workspaces
  • can we release a first MVP without workspace module? (How to publish nested workspaces?)
    • PRIO 1 Sebastian: Idea: patch current workspace module and remove option to publish/discard individual nodes

=> Sebastian will work on a v1

P2 - Import/Export (#65)

  • PRIO 2 - v1: initial version: use events 1:1

    • export format: line-delimited JSON, every line is an Event from the event store
  • Potential export formats (v2):

    • a) export events 1:1
    • b) export only selected streams
    • c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
    • d) (longer-term): “compact event stream” up to time X

=> Bastian will work on a v1

P3 - Rebasing fine tuning / conflicts (#144, #146)

  • Currently we rebase for every Neos login
    • idea: skip rebase if not required
    • in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
      • Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)

=> Sebastian will work on a v1

Mid-term (but already started):

  • Content Stream Aggregate (#195) (Bastian)
  • Postgres support (#180) (Bernhard)

Mid-term (not yet started):

Versioning/Snapshots/…

Security

  • How to proceed with support for Node privileges?
    • release initial version without support for node privileges is OK
    • mid-term Idea: use command bus in order to extend command handling via middlewares

Versioning/Snapshots/…

Compacting of the stream

Overhaul Value Objects & DTOs

  • Readonly Properties (PHP 8.1+)?

The next “ESCR Talk” will take place on Monday, December 6th at 11.30am CET (!).

(feel free to suggest topics)

Oh missed the meeting again :see_no_evil:

ES CR Sync 06.12.

(Christian, Sebastian, Bernhard, Bastian)

  • Bastian:
    • Import/Export (#65)
      • based on EventSourcing package directly
      • (PHP 8.1)
      • ToDo: Asset Usage Projection maybe? (to export files as well)
        • Do we have to keep track of the full hierarchy?
        • Alternative: some kind of cleanup that removes entries from the projected read model (or emits new events)
        • Maybe we do need explicit NodeWasDeleted events for all descendant nodes of a removed nodes
          • To avoid huge amounts of events, we could detect those cases and force a “compraction”
          • Idea: maybe we can use the existing NodeSubtreeSnapshot for a simple compraction
  • Sebastian:
    • Updated README
    • Helped Bernhard with Postgres adapter (merged master into feature branch, adjusted behat tests, stumbled upon some quirks)
    • Questions:
      • How to deal with the distribution providing multiple adapter implementations (by default: leads to exception since no instance is explicitly configured)
        • Bernhard: Test suite will be better organized (=> configurable threshold to test multiple adapters in different maturity)
        • Bastian: Interactive CLI setup (dynamically detect graph projection implementations => generate global Objects.yaml)
        • Sebastian: Remove custom Objects.yaml from implementing packages and move it to global
      • Long term: allow multiple instances to run in parallel
        • Idea: Allow CR instance to be configurable per use case (e.g. site in a multi-site installation, taxonomies, …)
        • Challenge: Write and read models must stay consistent => implement command bus and some kind of “context”
        • Challenge: How to determine the right “context”, e.g. for routing
        • Maybe “realm” or “preset” is a better name

Next Priorities

Fix Setup (#199)

  • Idea: get rid of patches (use custom NodeConverter instead of NodeAddress, and NodeInterface instead of TraversableNodeInterface)
  • Alternative (Bastian): Open PR against dev collection with fixes and refer to PR patch in composer patches

P1 - Test Refactor / Postgres support #180)

  • maybe try to merge “without Postgres being finished”, as long as the old tests run through

=> Sebastian has started with that, will continue here

P1 - Workspace module #143)

  • Decide what to do with the workspace module (currently broken)
    • Can we “outsource” this topic partly to the community (i.e. start with wireframes for a module v2)
    • Dream: $someone builts a new “diff” module that we can fill via a clean API (DTO, …)
  • Nested Workspaces
  • Introduce concept of “Changeset” (e.g. allow editors to switch between, annotate and explicitly mark for review → “Workflow API”)
    • Allow working in content streams that are not assigned to workspaces
  • can we release a first MVP without workspace module? (How to publish nested workspaces?)
    • PRIO 1 Sebastian: Idea: patch current workspace module and remove option to publish/discard individual nodes

=> Sebastian will work on a v1

P2 - Import/Export #65)

  • PRIO 2 - v1: initial version: use events 1:1

    • export format: line-delimited JSON, every line is an Event from the event store
  • Potential export formats (v2):

    • a) export events 1:1
    • b) export only selected streams
    • c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
    • d) (longer-term): “compact event stream” up to time X

=> Bastian has started on it, working on V1

P3 - Rebasing fine tuning / conflicts (#144, #146)

  • Currently we rebase for every Neos login
    • idea: skip rebase if not required
    • in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
      • Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)

=> Sebastian will work on a v1

Mid-term (but already started):

  • Content Stream Aggregate (#195) (Bastian)
  • Postgres support (#180) (Bernhard)

Mid-term (not yet started):

Versioning/Snapshots/…

Security

  • How to proceed with support for Node privileges?
    • release initial version without support for node privileges is OK
    • mid-term Idea: use command bus in order to extend command handling via middlewares

Versioning/Snapshots/…

Compacting of the stream

Overhaul Value Objects & DTOs

  • Readonly Properties (PHP 8.1+)?

The next “ESCR Talk” will take place on Friday, December 17th at 11.15am CET (!).

(feel free to suggest topics)

1 Like

ES CR Sync 7.1. (from memory)

(Sebastian, Robert, Bastian)

  • Recap
  • Decision & Kickstart: Try to run the docs.neos.io site on the ES CR! (#127)

ES CR Sync 11.1. (from memory)

(Sebastian, Bastian)

  • Sebastian was very busy and pushed 23 Commits to fix incompatibilities with the docs site
  • Remaining issues (currently):
    • Caches are not flushed properly
      • idea: callback mechanism to hook into before and after ContentGraph projector transactions so that we can trigger the cache flusher during projection time (via JobQueue!)
      • related: The GraphProjector should be a class in the ESCR package using a new interface for the actual read model interaction (this will allow us to implement (and potentially refactor) core features like GraphProjector::hasProcessed() and the hook mechanism mentioned before in a single place)
    • FE search
    • BE search
    • Flowpack.NodeTemplates package needs manual work
    • probably some more :slight_smile:

ES CR Sync 14.1.

(Sebastian, Christian, Viktor, Bastian)

BE Search is not working yet (#204)

Problem: The NodeSearchService needs a custom implementation. Unfortunately it currently has some quirks:

  • the $startingPoint argument (which is not part of the NodeSearchServiceInterface!) is actively used and makes this implementation complex/slow with the ESCR
  • the return type is an array of NodeInterface instances but effectively it is indexed by the corresponding node path

Idea: For a v1 Bastian will try to implement the interface with a custom (DBAL-based) projection and couple of limitations:

  • The $startingPoint argument won’t be supported (if it’s specified an exception will be thrown(?))
  • The result will be limited to 15 nodes (reason: we’ll need to interact with the ContentGraph in order to get the node path and this is a rather slow operation)

For the final version we’ll probably need a new interface along the lines of SomeClass::someMethod(SearchTerm $term, NodeTypeConstraints $filter, ContentStreamIdentifier $contentStreamId, DimensionSpacePoint $dimensionSpacePoint): Nodes. (note: We might get around the startingPoint-requirement by using a CR per site, see below).

Publish button does not turn yellow for deleted/disabled nodes (#89)

Problem: The ContentGraph does not keep track of deleted nodes and a custom projection won’t know the affected document node of a deleted content node (unless it keeps track of the full node hierarchy)

Ideas:

  • We could implement some kind of “DeletionEdge” (similar to the VisibilityEdges)
  • For now we decided to add the parentNodeAggregateIdentifier to the payload of the NodeAggregateWasRemoved event – but that probably won’t be enough for detecting deletions in deeply nested content nodes…

We agree that the state of the Publish Button is closely related to the feature of diffing two content streams (e.g. like we gonna need it for a revamped Workspace Module)

Multi Sites (or: Multiple CR instances) (#100)

We seem to come back to the issue of a “site context” and all the challenges that come with it

Ideas:

  • Allow multiple ContentRepository instances to be instantiated in parallel
  • In Neos a middleware could detect the correct “default” instance based on the current request
  • [IMO] This CR instance would be the major API for all core interactions, i.e. act as command bus factory for Content(Sub)Graph instances and other core read models and allow for registering custom event listeners (if we go that route we won’t have a global event store but one per CR instance, so we can’t have global event listeners either)
// retrieve an instance by request (for Neos)
$neosCR = $this->neosContentRepositoryFactory->createFromRequest($serverRequest);
// dispatch commands
$neosCR->handle(new CreateNodeVariant(...));
// query nodes
$neosCr->contentGraph()->findNodeAggregateByIdentifier(...)

// instantiate custom instances
$tagsCR = $this->contentRepositoryFactory->get('My.Package:Tags');
$tagsCR->handle(new SetNodeProperties(...));

The next “ESCR Talk” will take place on Thursday, January 20th at 9.30am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 4.2.

(Bastian, Sebastian, Bernhard, Viktor)

Working on / planned

  • (Sebastian) Worked on compatibility issues (#127)

    • → continue with that
    • → Check if/where legacy nodes are still instantiated (#184)
  • (Bastian) Worked on import/export (#65)

    • → continue with that
    • → implement adapter for the neos/redirecthandler (#210)
  • (Bernhard) working on DimensionSpace to support PHP 8.1

    • → adjustments to Neos CR
    • Flow Adjustments to support PHP 8.1

Other discussions

PHP 8.1 - Do we need/want a NodeInterface?

  • aspects:
    • ease of use for custom “entity node types”: ${product.specialPrice} vs ${My.Product(node).specialPrice} (or $product->someDomainLogic() vs $productService->getProduct(...)->someDomainLogic() in PHP)
    • (unplanned) extensibility
      • assumption 1 (Sebastian): it is easier to extend the behavior with an interface (e.g. lazy properties)
      • assumption 2 (Bastian): it is easier to change/extend the implementation if we have a final implementation
  • we are not sure yet :wink:
  • → TODO: collect pain points from similar decisions in the past

central “ContentRepository” object ([#100]

(https://github.com/neos/contentrepository-development-collection/issues/100#issuecomment-1015339893))

  • EventListener API
  • ContentGraph
  • sendCommand(…)
  • → TODO: Diff?

(mid/longterm) Goal:

  • Content Repository Core as extra package WITHOUT Flow dependencies.
    • → does not contain YAML for Node Types Schema
  • for WRITE operations, you need to use commands (breaking!)
  • for READ operations in Fusion / FlowQuery, we additionally provide a best-effort “legacy shim” which should solve most real world Fusion issues
    • (LegacyApi package right now)
  • for READ operations in PHP based on Node, we do not provide upgrade support.
  • for other READ operations (ContentContext, …), we do not provide upgrade support.
  • Adapter Package to Neos.

Migration Path forward for Neos.ContentRepository (planning)

  • currently:
    • Neos.EventSourcedContentRepositoryNeos.ContentRepository
  • idea 1:
    • starting with Neos X, we do NOT support the old CR anymore. (dangerous)
  • idea 2:
    • Neos.EventSourcedContentRepository

The next “ESCR Talk” will take place on Friday, February 11th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 11.02.

Recap

Sebastian

  • Fixed further compatibility issues #127

Bastian

  • supported Redirect Handler with ES CR #210
  • Asset Usage implemented (WIP)
  • Export/Import implemented (WIP)

Bernhard

  • DimensionSpace package update
    • #212
      1. Core Changes need to go in #2695 and #3592
      • PHP 8.1 test runner needs to be configured
      1. Test Neos with rendering and backend with this change
      1. merge :slight_smile:

Next

  • frontend search in docs.neos.io needs to be tested/fixed (Sebastian)
  • create PR for asset usage & export/import packages in neos/contentrepository-development-collection (Bastian)
  • Finalize DimensionSpace package update (Bernhard)
  • prevent old node access #183
  • add “affected dimension space points” to relevant events #163

The next “ESCR Talk” will take place on Friday, February 18th at 11.15am CET Friday, February 25th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

1 Like

ES CR Sync 25.02.

Recap

Bernhard

  • Working on “Dimension Space 2 Update (PHP 8.1)” (#212)
    • still waiting for one more Core Patch (#3592)
  • Working on clean code basis (Flightweights, static code analysis, …)

Bastian

Decisions

  • We want to use PHPStan for our static code analysis (for now)
  • We want to rename ESCR-related packages with Neos.ESCR.* prefix (#214)
  • We want to use Composer Scripts to invoke common tasks (run behat tests, static analysis, …)

Next


The next “ESCR Talk” will take place on Friday, March 4th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 04.03.

Path to the “ESCR driven Neos”

Goals

  • “Smooth Transition”
  • if in doubt, “fail fast” and avoiding inconsistencies.
    • old and new CR should never run “in parallel”
    • avoid AOP and Objects.yaml “rewiring” as much as possible
    • mock implementations of removed classes in order to provide better errors?
  • We want people to start with new ES CR for new projects soon ™ :wink:

Current Situation:

  • Neos.Neos and Neos.Neos.Ui works with Neos.ContentRepository
  • Neos.EventSourcedNeosAdjustments “patches” the above to work with the new stuff.
  • Neos.EventSourcedContentRepository depends (in small parts) on Neos.ContentRepository (Node Type Mgmt)

Target Situation

  • Neos.ESCR.ContentRepository (formerly Neos.EventSourcedContentRepository) does NOT depend on Neos.ContentRepository anymore

  • → We simply copy over the NodeTypeManagment stuff and REBUILD with new type hints.

  • → for migration, we directly read NodeData table (instead of using ORM)

    • → Export NodeData table into events export (to be imported with new Export/Import functionality)
  • Try to avoid cases where two packages are installed that serve the same purpose (e.g. Neos.ESCR.ContentRepository && Neos.ContentRepository)

  • maybe even add conflicts to composer manifest

  • what about neos/neos-ui and neos/neos?

    • e.g. routing is deeply integrated in neos/neos
  • idea #1: extract routing into (Neos.Neos.Routing and Neos.ESCR.Routing)?

    • → 10-20 packages in the old and new world
      • many “implicit knowledge” in the trenches.
  • idea #2: we maintain a separate branch?? for “new”?

    • Neos v8 vs neos v18
    • → remove all @api annotations (except FlowQuery / Fusion) and communicate that this is an intermediate solution
  • idea #3: we keep some sort of “LegacyNeosAdjustments” package (so like Neos.EventSourcedNeosAdjustments today but the “other way around”)

    • → we thought through this, and we will have a scenario where if you ONLY want to install the old world, the “new” upstream code will still depend on ESCR (which is not installed) → CRASH. → this idea will not work.
  • idea #4: we keep some sort of “EventSourcedNeosAdjustments” package like it exists right now

    • → same as #3, just the other way round.
  • idea #5: use (composer) patches

    • dangerous because it partly defeats SemVer
  • idea #6

    • Neos.Neos stays Neos.Neos and will never be used “in the new world”. Instead we would extract “things” that we need for the ESCR to work into separate packages
  • idea #7

    • communicate “ES Neos” as new Product (“Neos Flow 9.0”, “Neos CMS 9.0”, “Neos {New Name} 1.0”)
  • idea #8 (our preference currently): simply release Neos 9.0 with EventSourcing :wink:

    • → create new branch “escr” (or similar) and apply all changes from “EventSourcedNeosAdjustments” → that branch will become “main” in the future
    • → we’ll have to take care to “upmerge” fixes and relevant changes (probably partly manually)
    • → maybe we can automate that

Next steps

Bastian

  • finalize Neos.ESCR.Export package and create exporter from NodeData => events.jsonl

Sebastian

  • write discuss post with the suggestions above

Bernhard

  • experiment with Cypher queries
  • ran into issues with the PRs for type safety, will contact @albe for support

The next “ESCR Talk” will take place on Friday, March 11th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 08.04.

Finally, another really nice session, this time with @sebastian, @Nezaniel and @christianm :

Road to 1.0

  • Can Neos 9.0 (scheduled August 2023) be based on ESCR?

    • Coordinate with team
    • Idea: (after NeosCon):
      • Remove “master” (i.e. turn readonly)
      • Create branches “8.1” and “9.0”
      • After “9.0” is out, a new “main” master can be introduced
      • Commit our adjustments into “9.0”
      • Keep branches in sync as much as possible (i.e. regular upmerges)
      • To decide: When to create a custom package?
        • e.g. Node migrations (currently work with NodeData)
      • What to do with Neos.Ui?
        • Extract (dirty) PHP code into separate package (in dev collection so that we can adjust it for “9.0”)
        • or: create the same branches and adjust PHP code in “9.0” branch
    • Communication challenges:
      • Coordinate fundamental changes to 8.x upfront
      • Coordinate fundamental changes to 9.x upfront
    • NodeData, ContentContext, … will be removed (i.e. extracted into some legacy compat package?)
  • Bigger ToDos before productive usage

    • Streamline and simplify PHP API (epic)
    • Introduce common package key prefix for ESCR related packages (#214)
    • 1x event updates: Remove Namespaces in events. (#220) (can also be done later via migration)
  • later:

    • “Aggregate” for Node/ContentStream (#195)

The next “ESCR Talk” will take place on Friday, April 22nd at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

We skipped the session today.

The next “ESCR Talk” will take place on Friday, June 17th at 10.30am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

1 Like

ES CR Sync 17.06.

After a longer break (containing NeosCon and Sprint) we had another great session today with @sebastian and @Nezaniel:

Sebastian has been very active recently and managed to get most of Neos working again after the “big merge”.
One of the remaining tasks is the support for multiple sites and content dimensions (e.g. the ContentDimensionMenu).
The introduction of ContentRepository instances will affect all layers and the way content dimensions are configured.
It will also require some mapping between Sites and a CR instance as well as custom routing dimension resolution logic per site.

Some goals that we had while discussing possible solutions:

  • Keep configuration simple for the regular use cases
  • Avoid millions of options in favor of replaceable PHP implementations
  • Keep configuration & concepts explicit
  • Keep CR and Neos concerns separated

and some ideas we had to approach it:

Introduce content repository instance

Currently the CR configuration is global (e.g. underneath Neos.ContentRepository.contentDimensions).

In order to support multiple CR instances, a new level is required, for example:

Neos:
  ContentRepository:
    instances:
      'some-cr-instance':
        contentDimensions:
          # ...

some-cr-instance is an arbitrary name referring to a single Content Repository instance. It could be prefixed in order to avoid naming conflicts.
For the main Neos CR instance it could be named after the site package key, but since multiple sites can share the same instance, this might be confusing.

Maybe it makes sense to already define some default ContentRepository that is used by default (see below).

Site specific settings

We briefly discussed whether it makes sense to use ApplicationContext or another “context” to allow for site specific configuration on a global level.

For now we ditched that idea because it could be a Pandora’s box.
Instead we decided to require affected settings to explicitly refer to a CR instance by its name. This will be required for custom routing behavior for example (see below).

We did not come up with a concept to have node type definitions per site, yet.

Dimensions / Routing

Currently configuration for content dimensions and the resolution logic (get from URL to dimension vice versa) is done in a single place.

Example:

Neos:
  ContentRepository:
    contentDimensions:
      language:
        label: 'Some label'
        icon: icon-language
        default: en_US
        defaultPreset: en_US
        presets:
          en_US:
            label: 'English (US)'
            values:
              - en_US
            uriSegment: en
          en_UK:
            label: 'English (UK)'
            values:
              - en_UK
              - en_US
            uriSegment: uk
          de:
            label: German
            values:
              - de
            uriSegment: de

We decided that it makes sense to separate the (Neos specific) routing configuration.
Together with the suggestion above, the result would be something like:

Neos:
  ContentRepository:
    instances:
      'default':
        contentDimensions:
          language:
            label: 'Some label'
            icon: icon-language
            values:
              en_US:
                label: 'English (US)'
                specializations:
                  en_UK:
                    label: 'English (UK)'
              de:
                label: German
                specializations:
                  nl:
                    label: Dutch
              fr:
                label: French
              da:
                label: Danish
              lv:
                label: Latvian

The corresponding routing-related configuration would reside underneath Neos.Neos:

Neos:
  Neos:
    sites:
      'some-site-name':
        contentRepository: 'default'
        routing:
          dimensionResolver:
            className: 'UriPathSegmentBased'
            options:
              segmentSeparator: '_'
              segments:
                'language':
                  defaultDimensionValue: 'en_US'
                  uriPathSegmentMapping:
                    'en_UK': 'uk'

(exact class name and options to be defined)

some-site-name would refer to a unique name for a site. It can be the package name but one package could contain multiple sites so it probably makes sense to keep those separated.

Note: We should consider using the SiteName as NodeAggregateIdentifier of the site’s root node

Default configuration

In order to avoid having to configure all of the above for all sites even though they might only use a single dimension resolution logic, we thought about introducing a special default site name, represented as * (Note: This slightly contradicts the goal of avoiding magic but in this case it’s probably worth the added complexity).

Neos could ship with a default configuration of (something like):

Neos:
  Neos:
    sites:
      '*':
        contentRepository: 'default'
        routing:
          dimensionResolver:
            className: 'UriPathSegmentBased'
            options:
              segmentSeparator: '_'

  ContentRepository:
    instances:
      'default':
        contentDimensions: {}

So that special configuration is only required when deviating from the default behavior.

Question: Does it make sense to take the first configured dimension value as default (or keep the default as part of the CR configuration), so that we don’t have to add any custom configuration for the most cases?

addendum: I briefly discussed this question with @sebastian and he thinks that it makes sense to make the “default dimensions” a concept of Neos (this way it will be possible to share the CR between multiple sites but have a different default dimension for example).
I agree but would prefer if we would not have to touch the Neos configuration to get a sane default behavior (and we need default dimension values at least for the homepage).
If we stick to configuring the default as part of the Neos site configuration, it should at least not be specific to routing IMO because we need it in other places, too (e.g. for the initial rendering in the Neos Backend).

Implementation / Extension points

Implementation of the dimension resolution will probably based on an older PR:

But that still worked with the idea of a “global content repository”.
With multiple CR instances we’ll have to detect the current site and CR instance beforehand.

We consider two middlewares:

One that

  • …detects the current site from the HTTP request (similar logic to how we currently do it in the FrontendNodeRoutePartHandler)
  • …detects the current CR instance (by reading configuration from `Neos.Neos.sites[<site-name-or-asterisk>].contentRepository
  • …stores site name and CR name into the Request attributes

And the (already existing) RoutingMiddleware that

  • …puts resolved site and CR instance into the RouteParameters
  • …starts the routing as before

Note: We decided to put Site/CR detection into a dedicated Middleware because we need the current CR in the rendering layer, too (e.g. for the fusion context).

Next up

  • Sebastian wants to work on the above
  • Bastian will continue to work on a solid foundation for Import/Export with support for importing from “legacy systems”

The next “ESCR Talk” will take place on Friday, June 24th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

2 Likes

Hey,

@bwaidelich Thanks for the detailed writeup :heart: :slight_smile: Much less confusing than what I have written in our WIP docs :wink:

All the best, and have a great weekend,
Sebastian

1 Like

ES CR Sync 24.06.

Another long session with @sebastian @christianm and @yvolution (great username btw *g)

Legacy CR Migration (#3795)

We looked at the current state of the “Legacy CR Migration”: I managed to export a Neos 7.3 Demo Site to events and to import that (< 1 Second on my MacBook + ~2 Seconds projection replay).

Next

  • Migrate assets
  • Improve UX (create site, better handling of “event store already contains events”, …)
  • Allow export part to be installed in Neos < 9

Verify migrations (#3736)

We want to create a command that can create a plaintext file containing all nodes for “old” and “new world” in order to verify that the migration produces the right events in all cases.

Possible command signature

./flow cr:dump <content-dimensions> <sites> <include-properties>

The resulting output should be one file per site & dimension combination.

Simplify NodeAggregateWasMoved event (#3825)

The current structure of the NodeAggregateWasMoved is rather complex. Especially the NodeMoveMappings are hard to grasp.

Besides, the hash of the affected ContentDimensionSpacepoints is expected as keys for NodeVariantAssignments::createFromArray() which makes this error prone. Also it makes the internal implementation detail part of the event payload.

We collected some ideas on how to change that.

Dimensions / Routing

Sebastian worked on the Content Dimension resolution and already has most of it working.
We had a fruitful discussion about architecture and namings.

One idea we came up with is a central authority to get “details” of a specific site.
In practice it could look something like this (names and signatures to be finalized):

$siteDetails = $this->siteDetailsRepository->finyByRequest($httpRequest);
// or: $this->siteDetailsRepository->findById($siteIdentifier);

$siteDetails->getContentRepositoryIdentifier(); // ContentRepositoryIdentifier
$siteDetails->getConfiguration(); // array
$siteDetails->getAssetCollection(); // ?AssetCollection
// ...

A date for the next “ESCR Talk” is not announced yet (I won’t be there next Friday). Feel free to join us in the #project-cr-rewrite Slack Channel

It might seem like there’s slow progress on the ESCR project currently, but the opposite is the case.
@sebastian is incredibly active and we discuss multiple times per day.
@Nezaniel also makes great postgress (pun intended).

I decided to no longer put the effort into a weekly update here because IMO that time is better spent on the implementation.

You can always follow the progress on ES CR - Neos 9.0 · GitHub
And, as usual, feel free to join the #project-cr-rewrite Slack Channel to get in touch if you have questions or want to participate in discussions!

1 Like