RFC: Future of the Neos package architecture (`Neos.Neos` in particular)

Status Quo

We have three repositories for the bulk of Neos and Flow development:

Between these three, dependencies are (theoretically) unidirectional, so:

  • Neos.Neos.Ui depends on Neos.Neos, but not the other way around
  • A package in neos-development-collection may depend on a package in flow-development-collection, but not the other way around

The heart of Neos is the Neos.Neos package, which handles the following concerns:

  • Establish a User model and add user-related semantics on top of the Content Repository (e.g. Personal Workspace)
  • Establish web-related semantics on top of the Content Repository (e.g. Site, Domain)
  • Establish combined semantics relating the Content Repository and Neos.Media (e.g. AssetUsage)
  • Integrate the Content Repository and Neos.Media with Flow’s Routing- & Linking-mechanisms
  • Provide high-level integration APIs on top of the Content Repository, Neos.Media and its own domain models
    • FlowQuery
    • Eel
    • Fluid View Helpers
  • Provide main entry point for frontend requests (Frontend\NodeController)
  • Provide a Backend Module for Configuration inspection
  • Provide a Backend Module for (composer-/Flow-)Package inspection
  • Provide a Backend Module for Dimension configuration inspection
  • Provide a Backend Module for Sites & Domain management
  • Provide a Backend Module for User management
  • Provide a Backend Module for User settings
  • Provide a Backend Module for Workspaces management

Problems

1. Responsibility Overflow

The above (certainly incomplete) list shows, that Neos.Neos has a lot of responsibilities. Some of these are irreducibly necessary. Neos is composed of various parts (especially Content Repository & Media) that have useful applications on their own. Neos.Neos takes those parts and adds semantics aside and on top to turn the whole composition into a Content Management System.

Besides those domain-related responsibilities, Neos.Neos also provides APIs for integration and User Interfaces in the form of backend modules. It is these responsibilities that make the package appear quite monolithic.

2. Neos.NeosNeos.Neos.Ui

It is very often unclear whether specific functionality shall live in Neos.Neos or rather in Neos.Neos.Ui. We have for instance resorted to awkward workarounds to establish content element wrapping and out-of-band rendering in the UI (some of this has been already mitigated though).

In subtle ways, Neos.Neos sometimes has soft dependencies on Neos.Neos.Ui, or it anticipates UI-specifics. This mostly happens with translation labels, but there are also more pronounced cases, like the IconNameMappingPostprocessor.

Neos.Neos.Ui on the other hand establishes logic that should rather live in Neos.Neos. It has its own node property converter for example - that’s weird (and many have stumbled over this one, too).

3. Backend Modules

It has long been the plan to unify the content module (which runs on React and is developed separately in GitHub - neos/neos-ui: Neos CMS UI written in ReactJS with Immutable data structures.) with the other backend modules. We are currently developing redundant CSS and JavaScript split over two packages and repositories.

Some backend modules are about to grow significantly in size. In case of Workspaces (see: !!! TASK: Move workspace module to separate package by ahaeslich · Pull Request #4665 · neos/neos-development-collection · GitHub) and Media UI (see: GitHub - Flowpack/media-ui: The development repository for the new Neos media management interface), these modules have been moved to separate packages. If we continue in this direction (so: Move all backend modules into dedicated packages), we’ll need to provide an easy way to install all packages that are supposed to be part of the Neos core UX at once.

Proposal

During the post-con sprint 2024 in Karlsruhe, @ahaeslich, @christianm , @robert and I have briefly discussed the following two ideas:

1. Split domain repsonsibilities from Neos.Neos and turn it into a meta-package

The idea is to establish an new package Neos.Neos.Core (name tbd.), that takes on the domain responsibilities of Neos.Neos and provides high-level PHP APIs, so it can be used as a headless CMS.

Stripped of its responsibilities, Neos.Neos itself is then turned into a meta-package that requires all dependencies needed for the usual Neos user experience.

A simple composer require neos/neos shall suffice to bring Neos to life.

2. Create a new neos-ui-development-collection

To ease the unification of all UI-related code, a new development collection (following the pattern of flow-development-collection and neos-development-collection) shall be created.

This collection can be home to dedicated Backend Module packages, as well as UI tooling and framework code.

Neos.Neos shall require the packages from this collection as well.

Prior Discussions

What’s next?

We’d like to get your feedback on the above proposal. Please tell us what you think, add your ideas and criticism. We may add your feedback to the proposal and/or get an overview of counter-proposals on this matter.

As a result of this discussion, we’d like to reach a decision on how to move forward.

7 Likes

Thanks a lot, i really like this approach. Really nothing to add technically as i 100% agree.

Only question i have is wether you think this can be achieved incrementally or wether it needs a big change at once.

1 Like

I also agree to this plan, as you know as we discussed it :slight_smile: Thank you for the great overview post.

I wouldn’t mind calling the new meta package “Neos.Headless” as it describes the intend well IMHO. But we have the CR Core already, so maybe then having Neos.Core could be sensible.

I think we need to take some important steps right now if we agree @mficzel for example the high level workspace should reside in a separate package from Neos.Neos then as otherwise it wouldn’t be available to a headless installation. But that should still be relatively easy to do. Other than that it’s more a step by step transformation. We might need to check if Neos.Neos currently contains anything that a headless installation might need and move it out for 9.0.

@mficzel

Only question i have is wether you think this can be achieved incrementally or wether it needs a big change at once.

Devil’s in the details, I guess :slight_smile:

I believe there is an incremental way to achieve this architecture:

  1. Establish the Neos.Core package and move the high-level Workspace API there (Neos.Neos requires Neos.Core & Neos.Neos.Ui requires Neos.Core)
  2. Establish the neos-ui-development-collection
  3. Every time we touch one of the backend modules in Neos.Neos, we move it into a new package in neos-ui-development-collection, and let Neos.Neos require that package
  4. Whilst reforming the Backend API of Neos.Neos.Ui, we create new high-level APIs in Neos.Core until Neos.Neos.Ui no longer needs to have a dependency on Neos.Neos
  5. Once Neos.Neos is completely hollowed out, it can require Neos.Neos.Ui and we’re done.

The most disruptive step imho is going to be step 2. I have no experience with setting up those sub-splits and don’t have a strategy in mind how to make sure that commit history, issues, etc. stay intact for the UI repository. I’d love to hear some insight there from the people who facilitated the other development collections :slight_smile:

Additionally, step 3 may cause some disruption with upmerges, but I think this won’t be too bad. The first backend module that comes to mind is the Workspaces module. The plan with this one is to move it into a separate package anyway, and I suppose it will change so massively, that upmerges are no longer of any concern. Same with the media module (which doesn’t live in Neos.Neos to begin with iirc). The other backend modules aren’t under such stress and haven’t changed all that much in the past. We would move them only, when we start reforming them. The plan, as far as I understand, is to transform all of them from Fluid to Fusion. Doing that would disrupt upmerges anyway.

2 Likes

Unless we find a critical point why we need to do this as a big bang I would also rather do this incremental. Following the above roadmap from @wbehncke.

Thanks a lot for the thorough explanation!

I’m currently working on some of the Neos.Ui controller classes and things are… not optimal, to say the least :slight_smile:
So I’m all for splitting up the responsibilities and introducing proper dependencies between them!

One thing I am not fond of is the idea to introduce a third dev collection.
I forgot that I had suggested this before (four years ago :flushed:) but I would still very much prefer the Neos.Ui package(s) to be part of the neos-development-collection because currently it’s quite painful to keep those multiple mono-repos in sync.

I wonder about your reasoning against that?

Given that the UI is the biggest part of the new ui collection we could basically start that collection repo based on the history of ui and then just add everything else on top?

1 Like

@bwaidelich:

[…] I would still very much prefer the Neos.Ui package(s) to be part of the neos-development-collection […]
I wonder about your reasoning against that?

Thanks for chiming in! :slight_smile: I think it’s crucial we think this through thoroughly, and your objection marks an important point.

To explain my thought process: I’m thinking of the third dev-collection as an attempt at an Inverse Conway Manoeuvre. Starting with Conway’s Law:

Any organization that designs a system (defined broadly) will produce a design whose structure is a copy of the organization’s communication structure.

I would hypothesize that our repository structure determines our communication structure. Following Conway, this would mean: to support our system design, we must reflect its desired structure in our repository structure.

The key question then is: What is the desired structure of our system design?

My assumption is, that Neos.Ui is to Neos what Neos is to Flow. Flow is a general purpose application framework that supports (but not exclusively) Neos. Similary, Neos is a general purpose content application framework that supports (but not exclusively) the Neos UI.

To be clear: By this I don’t mean that we need to anticipate someone writing an entire UI of their own. I rather think that every Site built with Neos can be considered an Application supported by Neos, and the Neos UI simply lives in that same layer. In reverse conclusion this also means that every requirement the UI poses upon the Neos Core will result in an API that every other Application/Site will benefit from.

Admittedly, this is a very strong assumption and might very well be a case of YAGNI. I believe that as a team and a community, we need to make a decision here.

There’s some anecdotal evidence that I think supports my hypothesis:

For instance, the problems outlined in my original post bother us precisely because there’s a structural boundary and we’re palpably violating it. If the Neos.Neos.Ui package would be part of the neos-development-collection repository, we would probably notice some weirdness there, because the boundaries between Neos.Neos and Neos.Neos.Ui are unclear, but I’m not sure it would bother us enough, to consider it worthwhile to fix it. (And this would be okay, if we do not wish to follow a strategy of avoiding cohesion in this area)

The way it is now, the two repositories neos-ui and neos-development-collection are in a clear customer<->supplier relationship. Some desired functionality in the UI may require changes to the underlying domain model. I consider it a good thing that such changes need to go the extra mile, encouraging us to talk about broader concepts, and cannot be snuck in silently.

A good recent example for this is the “Publishing Bonanza” topic. There was one big PR in neos-development-collection which triggered a discussion with 5 participants (see: FEATURE: Highlevel Workspace API by nezaniel · Pull Request #4943 · neos/neos-development-collection · GitHub). The result was a design for a high-level Workspace API, that is useful to integrators and other backend consumers rather than just the UI. This is much more than we would have needed to solve the immediate problem in the UI, and I for one certainly didn’t see that coming :slight_smile:

This goes to show, that the boundary between neos-development-collection and neos-ui forces us to make a clear cut. Any functionality that spans both repositories must be split into two separate PRs that will be discussed under very different perspectives. It also means, that the overall change will (ideally) be reviewed by 4 rather than just 2 people, with 2 reviewers focusing on the domain and infrastructural impact, and 2 reviewers focusing on the UI impact.

This is all grey theory, but besides those strategic considerations, I also see more mundane things to consider. Just looking at some basic metrics about both repositories and what we’d end up with:

Repository Size Open Issues Open Pull Requests CI Duration
neos-ui ~300MB (Source) 234 53 15-20min
neos-development-collection ~170MB (Source) 471 93 35-40min
both, unified ~470MB 714 146 50-60min (possibly less thanks to parallelization)

I cannot actually judge how problematic such increase in size would be. I just have a strong urge to split things up, when their size grows beyond a (subjectively) comfortable level :D. Looking at issues and PRs though, I would definitely say that this would have a negative impact on the signal-to-noise ratio for both sides.

All that being said, I have to say that I’m not entirely opposed to the alternative approach of folding in neos-ui with neos-development-collection. There are some strong arguments for it:

Cross-Cutting changes would be much easier to implement. Especially if our mission is to disentangle the boundary issues between Neos.Neos and Neos.Neos.Ui, this would (paradoxically) be much easier to do, if they’d live in the same repository.

Running E2E tests on every change may be a good thing. Currently, we do not recognize if a change in neos-development-collection breaks the UI, unless a new PR in neos-ui is opened that triggers an E2E test run. We had multiple such incidents in the past, and of course the nature of this issue poses the risk of such defects falling through and being released to the public. However, there’s cost involved with this: The E2E tests are a delicate thing, often false-negative and quite difficult to maintain. So, the cost would be again a decrease of the signal-to-noise ratio for most PRs in neos-development-collection, and a lot of wasted compute power for false-negative runs and subsequent re-runs. The upside would be a better detection of UI-breaking bugs in the core.

There would be less confusion about where to open issues. From the outside it is hard to judge whether a bug needs to be reported in neos-development-collection or neos-ui. neos-ui is the less prominent repository of the two, but most end-user-facing bugs need to be reported there. Occasionally, we move issues around, because we find they’ve been reported in the wrong place. We must stay alert about this problem and make sure to check new issues in both places. A unified repository would solve this problem for good.

Our communication & review practice may actually improve. If we’d all be looking at the same repository all the time, chances are each individual maintainer will engage with issues & PRs they would otherwise have overlooked.

Long story, short: I do not have fully conclusive reasoning to offer. I believe both approaches have their advantages and disadvantages, and we need to make a decision. I don’t know if it’s too early for this, but maybe, or eventually we should put this up for a vote:

Q: How shall we approach step 2 in the above roadmap?

  1. Establish a neos-ui-development-collection
  2. Merge neos-ui with neos-development-collection
  3. Stay with status quo

(Provieded, of course, we decide collectively to proceed with the roadmap in general)

Wdyt?

2 Likes

Okay first of all thank you so much for discussing this and writing this down. When i looked at the image of the third ui-dev collection i was like: Yes this makes absolutely sense.

1.) Regarding the current Ui dependency dilemma

The bidirectional dependency from Neos.Neos to Neos.Neos.Ui its actually much worse, for example certain changes in the neos dev collection will break the ci because the coupling with the ui is too hard and their unit tests are executed as well. This makes reviewing needlessly complex.

2.) Regarding keeping Neos.Ui separate or move everything in the same dev collection

I really like what you said here because thats what i think as well.

My assumption is, that Neos.Ui is to Neos what Neos is to Flow .

I do fully agree with you and beyond i like to elaborate why i think that the neos-ui should keep its own repo:

  • independent release cycle, custom build tools, ci & e2e, minified package etc
  • dedicated neos core developers reviewing and merging there (looser/different rules)
  • 230 ui and accessibility issues that must be kept separate and not be mixed up with say the ESCR core
  • completely different tooling javascript vs php
  • the ui is an adapter and could technically be switched out
    • keeping it separate will enforce (hopefully at some point) correct dependency management

I also thought similar of you point:

It also means, that the overall change will (ideally) be reviewed by 4 rather than just 2 people

… but i have to admit that this is the ideal scenario, many refactorings require immediate small adjustments to be merged promptly and sometimes the adjustment prs will be even forgotten and are hard to maintain.

Further point for a big unified repo would be:

  • As you said, cross-cutting changes like disentangling the Neos Ui from Neos Neos fully actually never really happened and its unlikely going to happen soon without dedicated effort. One unified repo would make this easier.
  • Currently i usually run the Neos ui e2e test locally to verify that the Neos Ui still works and to approve a neos pr. We actually face a similar problem with Neos and Flow and need a solution for there as well as i guess the answer is not merging all three repos into one :wink: but having a daily trigger or something like discussed here:
    TASK: Add integration build on a neos distribution by albe · Pull Request #2431 · neos/flow-development-collection · GitHub

3.) Final thoughts

Every time we touch one of the backend modules in Neos.Neos, we move it into a new package in neos-ui-development-collection, and let Neos.Neos require that package

So at one point every fluid and fusion backend module will reside in the ui collection in a separate package?


Given that the UI is the biggest part of the new ui collection we could basically start that collection repo based on the history of ui and then just add everything else on top?

I would also like to rename the Neos.Neos.Ui repo to neos-ui-development-collection to keep all issues, prs, and the history.


People currently create issues for the flow core, neos core, or ui level and the latter is hard to correctly place as the neos-ui is independent but all backend modules look the same. This results in having to move a lot of issues to the correct repo. I think its a fair assumption that end users interacting with the ui should place the issues in the ui, neos intergrators in the neos core and hardcore flow users into flow :wink:


Regarding having all backend modules in one package to eventually use the same react instance and to unify the backend modules we already made some preparations: Turn Neos UI into a scaffold for Plugin/BackendModule integration · Issue #3119 · neos/neos-ui · GitHub and !!!FEATURE: Reform foundation and boot process of the UI by grebaldi · Pull Request #3682 · neos/neos-ui · GitHub

1 Like

@wbehncke Thanks for the detailed response!

I agree to everything you wrote. We urgently need to separate concerns of the different Neos parts.

I’m just a little uncertain, whether separating the repositories is the best move at this point in time.

Even though it might sound counter-intuitive, I would suggest to integrate the neos-ui package to the neos-development-collection as a first step.
That would allow us to clean up and move parts quickly and then extract a version that has less/no Neos dependencies to its own repository after the release of 9.0.

One thing that might be incorporated into the plan would be the separation of the FrontendController showAction (plus all relevant parts) and edit/previewAction into separate packages.

That way we could create projects with two tiers of hosts (production / no ui) vs. (editing / with ui + additional protection). Not sure how much value is in that but options are usually always nice.

I can see the value of working in a single repository to speed up the clean up process.

On the other hand I wonder if it would be more pain for maintaince in the long run (regarding upmerges, split jobs etc.) to go with this temporary integration.

1 Like

We already had a consensus to merge the ui into dev collection years ago and then it was somehow not done in the end. I think the reasoning was that the UI sometimes has additional releases because features get introduced there while staying backwards compatible. We would close that door if we do that.

2 Likes