Weekly ESCR Talk

The Neos Project
#1

Dear community,

TL;DR: we welcome you to join us every Friday at 9.30am 11am CET to discuss about the Event Sourced Content Repository (or just listen, if you prefer!)

As discussed at the Neos Sprint, we want to (re-)establish a weekly remote meetup to discuss aspects of the “Event Sourced Content Repository” (ESCR).

Our motivation and main goals are:

  • Keep up the pace (currently every progress stops very easily, but I hope that we can change that if we constantly keep the topics in our RAM aka brain)
  • Prevent “wood for the trees” symptom (i.e. get more “outside” opinions on important decisions)
  • Spread the knowledge (in order to attract more developers & prospects)
  • Enable more people to work on individual topics

The plan is to do a weekly meeting every Friday at 9.30am CET for about two hours and I would suggest to apply the following guidelines:

  • I created a recurring event in the Neos calendar starting next week (you should be able to subscribe to it via https://calendar.google.com/calendar/embed?src=bastian%40neos.io)
  • Meetings will be in form of a Slack call in the #project-cr-rewrite Channel
  • Everyone is invited to join (actively or silently)
  • The meetings will be in English, unless there are only German speaking participants (in which case it will be Russian of course *g)
  • I’ll create a discuss post (within this very thread) after each weekly to briefly summarize the discussed topics (important ideas/decisions should be tracked in Github issues)
  • In these posts we should also plan the agenda for the upcoming session
  • I’ll volunteer to organize the agenda and to timebox the topics during the discussion (I suggest to stick to the schedule as close as possible in order to allow others to join specific topics exclusively – and to prevent us from losing focus)

I hope to see many of you next Friday!

5 Likes
Event Sourced Content Repository: Roadmap
#2

The next (and first official) “ESCR Talk” will take place on Friday, November 19th at 9.30am CET.

Agenda

  • 9.30 - 10.30 Initial (development) setup
  • 10.30 - 11.30 Debugging tools

Feel free to get in touch with me via Slack, E-Mail or by commenting this post if you want to suggest topics for the upcoming session.

#3

I would love to see that at the end of the meeting we are able to agree on a roadmap for a first release. So my topic proposal would be: What are the current blockers and how can we (support to) solve them?

#4

Thanks for your feedback, it’s appreciated!
I’m afraid a final roadmap is a little ambitious for our first session because it depends on many factors, mainly: available development power.
Also: Sebastian won’t be there this time, and he definitely has a say in that *g

Having said that: One of the aims of this weekly session is to attract more developers so we can hopefully come up with better estimates soon.
Furthermore we should keep https://github.com/neos/contentrepository-development-collection#feature-comparison up-to-date so that everyone can get a better overview of the current state.

#5

Hi all,

thanks for joining the first session and sorry for the somewhat chaotic execution. I hope that the next sessions will be a bit more structured and interesting for everyone!

Recap

Setup

We followed the simple installation steps described at https://github.com/neos/neos-development-distribution/tree/event-sourced#quick-start but unfortunately the current setup is broken due to patches that don’t apply to latest Neos versions.

=> I created a bug report and will work on a fix
=> IMO it is important that we keep the initial setup as easy and stable as possible. I wonder why the CI doesn’t fail currently. If it is not affected, we should maybe tweak it so that we catch setup regressions before a PR is merged already

Anatomy of the event stream

We took the chance to look at the event stream of the imported Neos.Demo site and stumbled upon two oddities:

  1. There were 5 NodeAggregateWasDisabled events – fortunately this is not a bug of the ESCR but a curiousity of the Neos.Demo site export (=> https://github.com/neos/Neos.Demo/issues/118)
  2. There were 19 NodePropertiesWereSet events – also not a bug per se, but a possibility for improvements => https://github.com/neos/contentrepository-development-collection/issues/200

Tooling/Debugging

We got a little lost in discussions about “versioning”, "snapshots, “nested live event streams” and “Postgres support”.
I’ll try to summarize what I can remember (feel free to correct/amend):

  • Versioning seems to be a feature that many customers would love to see
  • With our current implementation especially the “live content stream” will grow substantialy over time. This will make it harder and more expensive to work with the ESCR (e.g. replays & new projections take longer, data consumption, …)
    • Snapshots in the original sense probably won’t help us (and is rather impractical for the graph projection)
    • Would it help to introduce some kind of “closing the books” process?
    • (How) can we implement a “compact event stream” mechanism that reduces the events to the minimum that is needed for the current state? (=> maybe even with an option to compact older events more than newer ones, => we’ll probably need that for a proper export format as well)
  • Postgres support is in the works
    • Postgres has some major advantages over MySQL/MariaDB, e.g. “hyper edges” (reduce the amount of edges/copy operations drastically, especially for large projects with many dimensions), better JSON support and fulltext search
    • Personally, I would suggest that we try to keep support for MySQL as much as possible, i.e. by avoiding too many features to be added into the Graph projection and rather provide clean PHP APIs so that the implementations can be replaced

Outlook

  • The time slot (9.30-11.30am) seems to be suboptimal since it collides with dailies and reduces the preparation time. I would suggest to move the slot to 11am-1pm instead. What do you think?
  • To be honest: I am not sure if it makes sense to keep this weekly, but I’m happy to give it another try. Feel free to suggest topics for the next agenda

The next “ESCR Talk” will take place on Friday, November 26th at 11.00am CET.

Agenda

  • 11.00 -

(feel free to suggest topics)

2 Likes
Team Tiga | Team Meeting | 2021-11-24
#6

ES CR Sync 26.11.

(Christian, Sebastian, Bernhard, Bastian)

Current Status / Done

  • Bastian updated us on the last week’s discussions
  • Node Migrations merged and lots of other changes as part there

Next Priorities

Next Steps

  • Readme Update (Sebastian)

Fix Setup

P1 - Test Refactor / Postgres support

  • maybe try to merge “without Postgres being finished”, as long as the old tests run through

=> Bernhard will work on that

P1 - Workspace module

  • Decide what to do with the workspace module (currently broken)
    • Can we “outsource” this topic partly to the community (i.e. start with wireframes for a module v2)
    • Dream: $someone builts a new “diff” module that we can fill via a clean API (DTO, …)
  • Nested Workspaces
  • Introduce concept of “Changeset” (e.g. allow editors to switch between, annotate and explicitly mark for review → “Workflow API”)
    • Allow working in content streams that are not assigned to workspaces
  • can we release a first MVP without workspace module? (How to publish nested workspaces?)
    • PRIO 1 Sebastian: Idea: patch current workspace module and remove option to publish/discard individual nodes

=> Sebastian will work on a v1

P2 - Import/Export

  • PRIO 2 - v1: initial version: use events 1:1

    • export format: line-delimited JSON, every line is an Event from the event store

  • Potential export formats (v2):

    • a) export events 1:1
    • b) export only selected streams
    • c) “compact event stream” (similar to current import from old CR) - via iterating over the projection (content graph) → Potentially use InMemory content graph at some point
    • d) (longer-term): “compact event stream” up to time X

=> Bastian will work on a v1

P3 - Rebasing fine tuning / conflicts

  • Currently we rebase for every Neos login
    • idea: skip rebase if not required
    • in case of conflicts: display modal (v1: provide CLI tools to solve conflicts, v2: with options to resolve)
      • Allow showing content streams that are not assigned to workspaces (adjust NodeAddress.php)

=> Sebastian will work on a v1

Mid-term (but already started):

  • Content Stream Aggregate (Bastian)
  • Postgres support (Bernhard)

Mid-term (not yet started):

Versioning/Snapshots/…

Security

  • How to proceed with support for Node privileges?
    • release initial version without support for node privileges is OK
    • mid-term Idea: use command bus in order to extend command handling via middlewares

Versioning/Snapshots/…

Compacting of the stream

Overhaul Value Objects & DTOs

  • Readonly Properties (PHP 8.1+)?

The next “ESCR Talk” will take place on Monday, December 6th at 11.30am CET (!).

(feel free to suggest topics)

Sorting feature workspace
#7

Oh missed the meeting again :see_no_evil: