Weekly ESCR Talk

Dear community,

TL;DR: we welcome you to join us every Friday at 9.30am 11am CET to discuss about the Event Sourced Content Repository (or just listen, if you prefer!)

As discussed at the Neos Sprint, we want to (re-)establish a weekly remote meetup to discuss aspects of the “Event Sourced Content Repository” (ESCR).

Our motivation and main goals are:

  • Keep up the pace (currently every progress stops very easily, but I hope that we can change that if we constantly keep the topics in our RAM aka brain)
  • Prevent “wood for the trees” symptom (i.e. get more “outside” opinions on important decisions)
  • Spread the knowledge (in order to attract more developers & prospects)
  • Enable more people to work on individual topics

The plan is to do a weekly meeting every Friday at 9.30am CET for about two hours and I would suggest to apply the following guidelines:

  • I created a recurring event in the Neos calendar starting next week (you should be able to subscribe to it via https://calendar.google.com/calendar/embed?src=bastian%40neos.io)
  • Meetings will be in form of a Slack call in the #project-cr-rewrite Channel
  • Everyone is invited to join (actively or silently)
  • The meetings will be in English, unless there are only German speaking participants (in which case it will be Russian of course *g)
  • I’ll create a discuss post (within this very thread) after each weekly to briefly summarize the discussed topics (important ideas/decisions should be tracked in Github issues)
  • In these posts we should also plan the agenda for the upcoming session
  • I’ll volunteer to organize the agenda and to timebox the topics during the discussion (I suggest to stick to the schedule as close as possible in order to allow others to join specific topics exclusively – and to prevent us from losing focus)

I hope to see many of you next Friday!

5 Likes

The next (and first official) “ESCR Talk” will take place on Friday, November 19th at 9.30am CET.

Agenda

  • 9.30 - 10.30 Initial (development) setup
  • 10.30 - 11.30 Debugging tools

Feel free to get in touch with me via Slack, E-Mail or by commenting this post if you want to suggest topics for the upcoming session.

I would love to see that at the end of the meeting we are able to agree on a roadmap for a first release. So my topic proposal would be: What are the current blockers and how can we (support to) solve them?

Thanks for your feedback, it’s appreciated!
I’m afraid a final roadmap is a little ambitious for our first session because it depends on many factors, mainly: available development power.
Also: Sebastian won’t be there this time, and he definitely has a say in that *g

Having said that: One of the aims of this weekly session is to attract more developers so we can hopefully come up with better estimates soon.
Furthermore we should keep https://github.com/neos/contentrepository-development-collection#feature-comparison up-to-date so that everyone can get a better overview of the current state.

Hi all,

thanks for joining the first session and sorry for the somewhat chaotic execution. I hope that the next sessions will be a bit more structured and interesting for everyone!

Recap

Setup

We followed the simple installation steps described at https://github.com/neos/neos-development-distribution/tree/event-sourced#quick-start but unfortunately the current setup is broken due to patches that don’t apply to latest Neos versions.

=> I created a bug report and will work on a fix
=> IMO it is important that we keep the initial setup as easy and stable as possible. I wonder why the CI doesn’t fail currently. If it is not affected, we should maybe tweak it so that we catch setup regressions before a PR is merged already

Anatomy of the event stream

We took the chance to look at the event stream of the imported Neos.Demo site and stumbled upon two oddities:

  1. There were 5 NodeAggregateWasDisabled events – fortunately this is not a bug of the ESCR but a curiousity of the Neos.Demo site export (=> https://github.com/neos/Neos.Demo/issues/118)
  2. There were 19 NodePropertiesWereSet events – also not a bug per se, but a possibility for improvements => https://github.com/neos/contentrepository-development-collection/issues/200

Tooling/Debugging

We got a little lost in discussions about “versioning”, "snapshots, “nested live event streams” and “Postgres support”.
I’ll try to summarize what I can remember (feel free to correct/amend):

  • Versioning seems to be a feature that many customers would love to see
  • With our current implementation especially the “live content stream” will grow substantialy over time. This will make it harder and more expensive to work with the ESCR (e.g. replays & new projections take longer, data consumption, …)
    • Snapshots in the original sense probably won’t help us (and is rather impractical for the graph projection)
    • Would it help to introduce some kind of “closing the books” process?
    • (How) can we implement a “compact event stream” mechanism that reduces the events to the minimum that is needed for the current state? (=> maybe even with an option to compact older events more than newer ones, => we’ll probably need that for a proper export format as well)
  • Postgres support is in the works
    • Postgres has some major advantages over MySQL/MariaDB, e.g. “hyper edges” (reduce the amount of edges/copy operations drastically, especially for large projects with many dimensions), better JSON support and fulltext search
    • Personally, I would suggest that we try to keep support for MySQL as much as possible, i.e. by avoiding too many features to be added into the Graph projection and rather provide clean PHP APIs so that the implementations can be replaced

Outlook

  • The time slot (9.30-11.30am) seems to be suboptimal since it collides with dailies and reduces the preparation time. I would suggest to move the slot to 11am-1pm instead. What do you think?
  • To be honest: I am not sure if it makes sense to keep this weekly, but I’m happy to give it another try. Feel free to suggest topics for the next agenda

The next “ESCR Talk” will take place on Friday, November 26th at 11.00am CET.

Agenda

  • 11.00 -

(feel free to suggest topics)

2 Likes

ES CR Sync 26.11.

(Christian, Sebastian, Bernhard, Bastian)

Current Status / Done

  • Bastian updated us on the last week’s discussions
  • Node Migrations merged and lots of other changes as part there

Next Priorities

Next Steps

  • Readme Update (Sebastian)

Fix Setup

P1 - Test Refactor / Postgres support

  • maybe try to merge “without Postgres being finished”, as long as the old tests run through

=> Bernhard will work on that

P1 - Workspace module

  • Decide what to do with the workspace module (currently broken)
    • Can we “outsource” this topic partly to the community (i.e. start with wireframes for a module v2)
    • Dream: $someone builts a new “diff” module that we can fill via a clean API (DTO, …)
  • Nested Workspaces
  • Introduce concept of “Changeset” (e.g. allow editors to switch between, annotate and explicitly mark for review → “Workflow API”)
    • Allow working in content streams that are not assigned to workspaces
  • can we release a first MVP without workspace module? (How to publish nested workspaces?)
    • PRIO 1 Sebastian: Idea: patch current workspace module and remove option to publish/discard individual nodes

=> Sebastian will work on a v1

P2 - Import/Export (#65)

  • PRIO 2 - v1: initial version: use events 1:1

    • export format: line-delimited JSON, every line is an Event from the event store
  • Potential export formats (v2):

    • a) export events 1:1
    • b) export only selected streams
    • c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
    • d) (longer-term): “compact event stream” up to time X

=> Bastian will work on a v1

P3 - Rebasing fine tuning / conflicts (#144, #146)

  • Currently we rebase for every Neos login
    • idea: skip rebase if not required
    • in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
      • Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)

=> Sebastian will work on a v1

Mid-term (but already started):

  • Content Stream Aggregate (#195) (Bastian)
  • Postgres support (#180) (Bernhard)

Mid-term (not yet started):

Versioning/Snapshots/…

Security

  • How to proceed with support for Node privileges?
    • release initial version without support for node privileges is OK
    • mid-term Idea: use command bus in order to extend command handling via middlewares

Versioning/Snapshots/…

Compacting of the stream

Overhaul Value Objects & DTOs

  • Readonly Properties (PHP 8.1+)?

The next “ESCR Talk” will take place on Monday, December 6th at 11.30am CET (!).

(feel free to suggest topics)

Oh missed the meeting again :see_no_evil:

ES CR Sync 06.12.

(Christian, Sebastian, Bernhard, Bastian)

  • Bastian:
    • Import/Export (#65)
      • based on EventSourcing package directly
      • (PHP 8.1)
      • ToDo: Asset Usage Projection maybe? (to export files as well)
        • Do we have to keep track of the full hierarchy?
        • Alternative: some kind of cleanup that removes entries from the projected read model (or emits new events)
        • Maybe we do need explicit NodeWasDeleted events for all descendant nodes of a removed nodes
          • To avoid huge amounts of events, we could detect those cases and force a “compraction”
          • Idea: maybe we can use the existing NodeSubtreeSnapshot for a simple compraction
  • Sebastian:
    • Updated README
    • Helped Bernhard with Postgres adapter (merged master into feature branch, adjusted behat tests, stumbled upon some quirks)
    • Questions:
      • How to deal with the distribution providing multiple adapter implementations (by default: leads to exception since no instance is explicitly configured)
        • Bernhard: Test suite will be better organized (=> configurable threshold to test multiple adapters in different maturity)
        • Bastian: Interactive CLI setup (dynamically detect graph projection implementations => generate global Objects.yaml)
        • Sebastian: Remove custom Objects.yaml from implementing packages and move it to global
      • Long term: allow multiple instances to run in parallel
        • Idea: Allow CR instance to be configurable per use case (e.g. site in a multi-site installation, taxonomies, …)
        • Challenge: Write and read models must stay consistent => implement command bus and some kind of “context”
        • Challenge: How to determine the right “context”, e.g. for routing
        • Maybe “realm” or “preset” is a better name

Next Priorities

Fix Setup (#199)

  • Idea: get rid of patches (use custom NodeConverter instead of NodeAddress, and NodeInterface instead of TraversableNodeInterface)
  • Alternative (Bastian): Open PR against dev collection with fixes and refer to PR patch in composer patches

P1 - Test Refactor / Postgres support #180)

  • maybe try to merge “without Postgres being finished”, as long as the old tests run through

=> Sebastian has started with that, will continue here

P1 - Workspace module #143)

  • Decide what to do with the workspace module (currently broken)
    • Can we “outsource” this topic partly to the community (i.e. start with wireframes for a module v2)
    • Dream: $someone builts a new “diff” module that we can fill via a clean API (DTO, …)
  • Nested Workspaces
  • Introduce concept of “Changeset” (e.g. allow editors to switch between, annotate and explicitly mark for review → “Workflow API”)
    • Allow working in content streams that are not assigned to workspaces
  • can we release a first MVP without workspace module? (How to publish nested workspaces?)
    • PRIO 1 Sebastian: Idea: patch current workspace module and remove option to publish/discard individual nodes

=> Sebastian will work on a v1

P2 - Import/Export #65)

  • PRIO 2 - v1: initial version: use events 1:1

    • export format: line-delimited JSON, every line is an Event from the event store
  • Potential export formats (v2):

    • a) export events 1:1
    • b) export only selected streams
    • c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
    • d) (longer-term): “compact event stream” up to time X

=> Bastian has started on it, working on V1

P3 - Rebasing fine tuning / conflicts (#144, #146)

  • Currently we rebase for every Neos login
    • idea: skip rebase if not required
    • in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
      • Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)

=> Sebastian will work on a v1

Mid-term (but already started):

  • Content Stream Aggregate (#195) (Bastian)
  • Postgres support (#180) (Bernhard)

Mid-term (not yet started):

Versioning/Snapshots/…

Security

  • How to proceed with support for Node privileges?
    • release initial version without support for node privileges is OK
    • mid-term Idea: use command bus in order to extend command handling via middlewares

Versioning/Snapshots/…

Compacting of the stream

Overhaul Value Objects & DTOs

  • Readonly Properties (PHP 8.1+)?

The next “ESCR Talk” will take place on Friday, December 17th at 11.15am CET (!).

(feel free to suggest topics)

1 Like

ES CR Sync 7.1. (from memory)

(Sebastian, Robert, Bastian)

  • Recap
  • Decision & Kickstart: Try to run the docs.neos.io site on the ES CR! (#127)

ES CR Sync 11.1. (from memory)

(Sebastian, Bastian)

  • Sebastian was very busy and pushed 23 Commits to fix incompatibilities with the docs site
  • Remaining issues (currently):
    • Caches are not flushed properly
      • idea: callback mechanism to hook into before and after ContentGraph projector transactions so that we can trigger the cache flusher during projection time (via JobQueue!)
      • related: The GraphProjector should be a class in the ESCR package using a new interface for the actual read model interaction (this will allow us to implement (and potentially refactor) core features like GraphProjector::hasProcessed() and the hook mechanism mentioned before in a single place)
    • FE search
    • BE search
    • Flowpack.NodeTemplates package needs manual work
    • probably some more :slight_smile:

ES CR Sync 14.1.

(Sebastian, Christian, Viktor, Bastian)

BE Search is not working yet (#204)

Problem: The NodeSearchService needs a custom implementation. Unfortunately it currently has some quirks:

  • the $startingPoint argument (which is not part of the NodeSearchServiceInterface!) is actively used and makes this implementation complex/slow with the ESCR
  • the return type is an array of NodeInterface instances but effectively it is indexed by the corresponding node path

Idea: For a v1 Bastian will try to implement the interface with a custom (DBAL-based) projection and couple of limitations:

  • The $startingPoint argument won’t be supported (if it’s specified an exception will be thrown(?))
  • The result will be limited to 15 nodes (reason: we’ll need to interact with the ContentGraph in order to get the node path and this is a rather slow operation)

For the final version we’ll probably need a new interface along the lines of SomeClass::someMethod(SearchTerm $term, NodeTypeConstraints $filter, ContentStreamIdentifier $contentStreamId, DimensionSpacePoint $dimensionSpacePoint): Nodes. (note: We might get around the startingPoint-requirement by using a CR per site, see below).

Publish button does not turn yellow for deleted/disabled nodes (#89)

Problem: The ContentGraph does not keep track of deleted nodes and a custom projection won’t know the affected document node of a deleted content node (unless it keeps track of the full node hierarchy)

Ideas:

  • We could implement some kind of “DeletionEdge” (similar to the VisibilityEdges)
  • For now we decided to add the parentNodeAggregateIdentifier to the payload of the NodeAggregateWasRemoved event – but that probably won’t be enough for detecting deletions in deeply nested content nodes…

We agree that the state of the Publish Button is closely related to the feature of diffing two content streams (e.g. like we gonna need it for a revamped Workspace Module)

Multi Sites (or: Multiple CR instances) (#100)

We seem to come back to the issue of a “site context” and all the challenges that come with it

Ideas:

  • Allow multiple ContentRepository instances to be instantiated in parallel
  • In Neos a middleware could detect the correct “default” instance based on the current request
  • [IMO] This CR instance would be the major API for all core interactions, i.e. act as command bus factory for Content(Sub)Graph instances and other core read models and allow for registering custom event listeners (if we go that route we won’t have a global event store but one per CR instance, so we can’t have global event listeners either)
// retrieve an instance by request (for Neos)
$neosCR = $this->neosContentRepositoryFactory->createFromRequest($serverRequest);
// dispatch commands
$neosCR->handle(new CreateNodeVariant(...));
// query nodes
$neosCr->contentGraph()->findNodeAggregateByIdentifier(...)

// instantiate custom instances
$tagsCR = $this->contentRepositoryFactory->get('My.Package:Tags');
$tagsCR->handle(new SetNodeProperties(...));

The next “ESCR Talk” will take place on Thursday, January 20th at 9.30am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 4.2.

(Bastian, Sebastian, Bernhard, Viktor)

Working on / planned

  • (Sebastian) Worked on compatibility issues (#127)

    • → continue with that
    • → Check if/where legacy nodes are still instantiated (#184)
  • (Bastian) Worked on import/export (#65)

    • → continue with that
    • → implement adapter for the neos/redirecthandler (#210)
  • (Bernhard) working on DimensionSpace to support PHP 8.1

    • → adjustments to Neos CR
    • Flow Adjustments to support PHP 8.1

Other discussions

PHP 8.1 - Do we need/want a NodeInterface?

  • aspects:
    • ease of use for custom “entity node types”: ${product.specialPrice} vs ${My.Product(node).specialPrice} (or $product->someDomainLogic() vs $productService->getProduct(...)->someDomainLogic() in PHP)
    • (unplanned) extensibility
      • assumption 1 (Sebastian): it is easier to extend the behavior with an interface (e.g. lazy properties)
      • assumption 2 (Bastian): it is easier to change/extend the implementation if we have a final implementation
  • we are not sure yet :wink:
  • → TODO: collect pain points from similar decisions in the past

central “ContentRepository” object ([#100]

(https://github.com/neos/contentrepository-development-collection/issues/100#issuecomment-1015339893))

  • EventListener API
  • ContentGraph
  • sendCommand(…)
  • → TODO: Diff?

(mid/longterm) Goal:

  • Content Repository Core as extra package WITHOUT Flow dependencies.
    • → does not contain YAML for Node Types Schema
  • for WRITE operations, you need to use commands (breaking!)
  • for READ operations in Fusion / FlowQuery, we additionally provide a best-effort “legacy shim” which should solve most real world Fusion issues
    • (LegacyApi package right now)
  • for READ operations in PHP based on Node, we do not provide upgrade support.
  • for other READ operations (ContentContext, …), we do not provide upgrade support.
  • Adapter Package to Neos.

Migration Path forward for Neos.ContentRepository (planning)

  • currently:
    • Neos.EventSourcedContentRepositoryNeos.ContentRepository
  • idea 1:
    • starting with Neos X, we do NOT support the old CR anymore. (dangerous)
  • idea 2:
    • Neos.EventSourcedContentRepository

The next “ESCR Talk” will take place on Friday, February 11th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 11.02.

Recap

Sebastian

  • Fixed further compatibility issues #127

Bastian

  • supported Redirect Handler with ES CR #210
  • Asset Usage implemented (WIP)
  • Export/Import implemented (WIP)

Bernhard

  • DimensionSpace package update
    • #212
      1. Core Changes need to go in #2695 and #3592
      • PHP 8.1 test runner needs to be configured
      1. Test Neos with rendering and backend with this change
      1. merge :slight_smile:

Next

  • frontend search in docs.neos.io needs to be tested/fixed (Sebastian)
  • create PR for asset usage & export/import packages in neos/contentrepository-development-collection (Bastian)
  • Finalize DimensionSpace package update (Bernhard)
  • prevent old node access #183
  • add “affected dimension space points” to relevant events #163

The next “ESCR Talk” will take place on Friday, February 18th at 11.15am CET Friday, February 25th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

1 Like

ES CR Sync 25.02.

Recap

Bernhard

  • Working on “Dimension Space 2 Update (PHP 8.1)” (#212)
    • still waiting for one more Core Patch (#3592)
  • Working on clean code basis (Flightweights, static code analysis, …)

Bastian

Decisions

  • We want to use PHPStan for our static code analysis (for now)
  • We want to rename ESCR-related packages with Neos.ESCR.* prefix (#214)
  • We want to use Composer Scripts to invoke common tasks (run behat tests, static analysis, …)

Next


The next “ESCR Talk” will take place on Friday, March 4th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 04.03.

Path to the “ESCR driven Neos”

Goals

  • “Smooth Transition”
  • if in doubt, “fail fast” and avoiding inconsistencies.
    • old and new CR should never run “in parallel”
    • avoid AOP and Objects.yaml “rewiring” as much as possible
    • mock implementations of removed classes in order to provide better errors?
  • We want people to start with new ES CR for new projects soon ™ :wink:

Current Situation:

  • Neos.Neos and Neos.Neos.Ui works with Neos.ContentRepository
  • Neos.EventSourcedNeosAdjustments “patches” the above to work with the new stuff.
  • Neos.EventSourcedContentRepository depends (in small parts) on Neos.ContentRepository (Node Type Mgmt)

Target Situation

  • Neos.ESCR.ContentRepository (formerly Neos.EventSourcedContentRepository) does NOT depend on Neos.ContentRepository anymore

  • → We simply copy over the NodeTypeManagment stuff and REBUILD with new type hints.

  • → for migration, we directly read NodeData table (instead of using ORM)

    • → Export NodeData table into events export (to be imported with new Export/Import functionality)
  • Try to avoid cases where two packages are installed that serve the same purpose (e.g. Neos.ESCR.ContentRepository && Neos.ContentRepository)

  • maybe even add conflicts to composer manifest

  • what about neos/neos-ui and neos/neos?

    • e.g. routing is deeply integrated in neos/neos
  • idea #1: extract routing into (Neos.Neos.Routing and Neos.ESCR.Routing)?

    • → 10-20 packages in the old and new world
      • many “implicit knowledge” in the trenches.
  • idea #2: we maintain a separate branch?? for “new”?

    • Neos v8 vs neos v18
    • → remove all @api annotations (except FlowQuery / Fusion) and communicate that this is an intermediate solution
  • idea #3: we keep some sort of “LegacyNeosAdjustments” package (so like Neos.EventSourcedNeosAdjustments today but the “other way around”)

    • → we thought through this, and we will have a scenario where if you ONLY want to install the old world, the “new” upstream code will still depend on ESCR (which is not installed) → CRASH. → this idea will not work.
  • idea #4: we keep some sort of “EventSourcedNeosAdjustments” package like it exists right now

    • → same as #3, just the other way round.
  • idea #5: use (composer) patches

    • dangerous because it partly defeats SemVer
  • idea #6

    • Neos.Neos stays Neos.Neos and will never be used “in the new world”. Instead we would extract “things” that we need for the ESCR to work into separate packages
  • idea #7

    • communicate “ES Neos” as new Product (“Neos Flow 9.0”, “Neos CMS 9.0”, “Neos {New Name} 1.0”)
  • idea #8 (our preference currently): simply release Neos 9.0 with EventSourcing :wink:

    • → create new branch “escr” (or similar) and apply all changes from “EventSourcedNeosAdjustments” → that branch will become “main” in the future
    • → we’ll have to take care to “upmerge” fixes and relevant changes (probably partly manually)
    • → maybe we can automate that

Next steps

Bastian

  • finalize Neos.ESCR.Export package and create exporter from NodeData => events.jsonl

Sebastian

  • write discuss post with the suggestions above

Bernhard

  • experiment with Cypher queries
  • ran into issues with the PRs for type safety, will contact @albe for support

The next “ESCR Talk” will take place on Friday, March 11th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

ES CR Sync 08.04.

Finally, another really nice session, this time with @sebastian, @Nezaniel and @christianm :

Road to 1.0

  • Can Neos 9.0 (scheduled August 2023) be based on ESCR?

    • Coordinate with team
    • Idea: (after NeosCon):
      • Remove “master” (i.e. turn readonly)
      • Create branches “8.1” and “9.0”
      • After “9.0” is out, a new “main” master can be introduced
      • Commit our adjustments into “9.0”
      • Keep branches in sync as much as possible (i.e. regular upmerges)
      • To decide: When to create a custom package?
        • e.g. Node migrations (currently work with NodeData)
      • What to do with Neos.Ui?
        • Extract (dirty) PHP code into separate package (in dev collection so that we can adjust it for “9.0”)
        • or: create the same branches and adjust PHP code in “9.0” branch
    • Communication challenges:
      • Coordinate fundamental changes to 8.x upfront
      • Coordinate fundamental changes to 9.x upfront
    • NodeData, ContentContext, … will be removed (i.e. extracted into some legacy compat package?)
  • Bigger ToDos before productive usage

    • Streamline and simplify PHP API (epic)
    • Introduce common package key prefix for ESCR related packages (#214)
    • 1x event updates: Remove Namespaces in events. (#220) (can also be done later via migration)
  • later:

    • “Aggregate” for Node/ContentStream (#195)

The next “ESCR Talk” will take place on Friday, April 22nd at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

We skipped the session today.

The next “ESCR Talk” will take place on Friday, March 25th at 11.15am CET (!), feel free to join us in the #project-cr-rewrite Slack Channel

1 Like