Product Vision Flow / Neos

The Neos Project Neos Team
1

Hi folks,

I postponed writing this post several times, because I wanted to come up with a well-prepared, well-written proposal for a product vision which could be a basis for further discussion. But I realized that it is probably better to come up with some sketchy notes which could trigger a discussion, rather than not discussing this topic in the team.

So, without further ado, here are some of my thoughts regarding the future of Flow and Neos. I won’t get very specific here, but rather would like to point out a few thoughts regarding the “how”.

Flow

I regained passion and motivation for pushing Flow again. In the past I said that we probably need to position Flow as “the framework which powers Neos and is also useful for standalone applications” due to the fact that there are already many frameworks with big communities out there. However, I do hear from many people, and see it from my own experience, that there is still a lot of beauty in the approach Flow takes. And that there are a couple of things which make Flow unique today.

Considering what I hear from the community and am passionate about myself, I think we should strengthen Flow and position it as the Domain-Driven Design framework for PHP. We could take all the experience we made together and work on Flow 4.0 which provides excellent support for Domain-Driven Design, CQRS, Event Sourcing and so on. During the last couple of weeks I have experimented a lot with Flow + CQRS + ES and I really see an opportunity there.

While CQRS + ES don’t actually need a framework to work, I think that we can take Flow’s approach of being an opinionated framework which guides its developers. By that approach we can possibly open the world of DDD to a larger group of developers who would feel lost or overwhelmed without the guidance. And for experienced developers, Flow will provide many convenient features which simplify day-to-day programming tasks and testing / debugging of DDD applications.

Neos

I think that the picture of a Content Mix Deck is still valid and a good (part of a) vision for Neos. We need to make this picture more concrete and also extend the meaning of content. While I have been working with Commands and Events, I realized how much more meaning you produce when you develop using these techniques. We can take advantage of this for Neos as well and produce meaningful content and events generated by Neos or external systems. Rather than regularly importing XML from some CRM system, we try to deal with events (customer John is not interested in canoes anymore) and use these semantics in Neos.

A new Content Repository, based on CQRS / ES, would provide many more possibilites than the current one. The whole publishing process and also the authoring experience (undo …) could be improved and provide features which are not possible with traditional approaches.

There are lots of ideas around about rewriting / refactoring / improving the UI technically. And not only there - we continue to come up with new libraries, techniques and so on. And that is a very good thing, because we want to keep innovating, so we need to make use of the best tools and practices. On the other hand, we have a growing community who trusts us to provide realistic upgrade paths and a product which can be hosted and maintained in their typical customer projects.

There is no simple answer for this compromise between conservative and bleeding edge. But it is a technical challenge which experience, professional developers and architects can master. If we agree on the direction (using innovate tools and techniques), we can also find a path which leads there. But we can’t just jump and hope to land at the right spot. In programming, you start encapsulating and decoupling code parts in order to be more flexible regarding internal changes. In DDD you define Domains, Subdomains and Bounded Contexts. And we can do the same with Neos. If we want to develop a new UI, we need to strengthen the service API in order to decouple the UI more from the server side (like the Calypso project for Wordpress did).

And we need to stay shippable and be able to progress while we refactor: if we decide to switch the JS framework for our UI, we can’t develop a new UI for a year and stop implementing new features in the meantime. Maybe we can introduce the new library in a single screen for the first release, then two modules in the next one. It shouldn’t become an all or nothing decision.

I think that we can continue introducing new technologies, but that comes with a price: we have to be even more careful and quality concerned if we want to keep juggling with all this new technology. And we must not forget that developers working on Neos customer projects are our users. They deserve a great Developer Experience. And that includes that they don’t have to learn everything new on every second Neos release. For customers like for developers, staying with Neos and Flow must be worth the invest - I don’t like Neos to become famous for difficult upgrade paths.

Regarding target groups: I think it is important to have big corporations / projects as our major target group. Smaller target groups will benefit from that, but only these bigger projects will enable us to develop the features we want and in the quality we expect. Personally, I’m not in for Neos as a pure hobby project, because I’d like to work on Neos during my work time - and for that we need enough customers we can attend to.

Oookay, I warned you - this won’t be the polished pamphlet …

I’ll try to condense it to a few key points for now:

  • position Flow as a DDD framework, with its own website and major new features
  • further decouple, modernize and clean up Neos, but with the clear objectives to (in that order)
    1. provide more value to the users
    2. keep existing Neos devs motivated and attract new good developers
    3. be able to proceed with new features faster
  • continue to follow and make more concrete the vision of Neos as a content mix deck + authoring tool with the true meaning of content and the actual user in mind

Now shoot, I know you have a lot of ideas to contribute as well!

Oh and, I’m really looking forward to putting all that (and what you guys come up with) into practice. Because, FUN!

5 Likes
2

Hey Robert,

totally makes a lot of sense to me what you wrote and I support this direction :slight_smile: Pretty much what I had in my mind as well; although I wouldn’t have been able to express it that consisly.

For the Neos part, I 100% agree to what you wrote; regarding Flow I am fine with it but just see the limited resources we are having. So I’d really like if Flow could additionally be based on well-known framework components (e.g. like Symfony parts etc). I know this is some kind of “technical detail decision”; but still something I’d like to throw into a discussion: Positioning Flow as a separate product AND it (in the long term) technically being a symfony distribution would totally fit well I think :smile:

All the best,
Sebastian

3

Dear Robert!

Thanks for the comprehensive write up!

Flow

This is the part I’m least competent in, but I do totally agree to what Sebastian said about refactoring as much as possible on top of low-level Symfony components. As my professional experience grows, I try to withhold from large buy-in on large monolith frameworks, and on personal project see myself more and more in nodejs land, but it doesn’t matter so much here.

Neos

Breaking the thing apart

I totally agree with you here on need for decoupling our monolith baby into packages that have clear bounded contexts talking to each other via an API (content repository, universal/isomorphic(!) rendering engine, JS admin interface…), we talked it through many times already.

Neos JS admin interface

Structured editing + bulk records editing
In current editing experience the crucial thing for my company that we currently miss is the structured editing + bulk editing. We have tons of content, and we need great UI performance to edit all of it. For most of the stuff we need just plain raw editing mode, without inline editing. Also we don’t care for an RTE, just let it do bold/italic/links and we’d be happy with it.

Break down Neos UI into components and allow to use only the pieces we need in our own web apps

We discussed it multiple times as well, to have a library of structured content editing and UI components, rather than a monolith thing. This way we could craft custom editing experiences out of small components where it is required (e.g. editorial magazines and just about any large custom project).

I will pause for now and pause the topic of target audience later this evening.

1 Like
4

Thanks @robert for starting this discussion. You have my :+1: for more or less all your points. Bellow a short/unpolished resume from my POV

Flow

I’m fine with that as a goal, but it’s a bit too early to communicate on this point from my POV. DDD is hard (and we don’t do DDD in a really advanced way currently). If we decide to follow this path (and I think we should), we MUST include in the process guys like Mathias Verraes (http://verraes.net/), even if this cost us some money. As said before DDD is pretty hard, doing it well at application level is hard, but in an opinionated framework it MUST be done right from the ground.

I love Flow since the first day (maybe the second or the third), because of how useful it can be and improve the quality of the work in our team. Convention / Opinionated sounds good to me. If we can specialize Flow (like low level DDD support), i think we can have a market. But we currently lack a lots of ressource, so we need to work closely with Symfony, I remember a discussion with Fabien, that say that we can collaborate and maybe the next major version of Symfony can be adapted to support what we need. If we can have Flow 4 as a slim layer on top of Symfony that add the magic/useful layer that we like (AOP, Property Mapper, Fluid, …). For AOP we can collaborate with Alexander from GoAOP. If we move in the direction of a really open community, focused on building a framework a bit more opinionated and focused (DDD) that the others, we have a chance to gain more visibility and confidence from the market. Yes the first step will be very hard, but after this we can really focus on our vision, and stop maintaining the full stack alone.

For me, the framework is pretty cool now, but market wise to need to be more specialized on communicate more around key features. I don’t see a big opportunity to become the next hot framework, so we need the be aware resource wise.

Neos

Disclaimer, i spend 80% of my time between Flow and Neos project, if I can spend 110% of my time on Flow/Neos project before the end of the year, it’s pretty ok for me :wink:

  • Decouple & Modularize
  • More test / Less regression in the UI
  • Major features (Media Browser, HTTP Redirect, Asset Meta Data Handling, …) need to be in one or more dedicated Package, and not required in a basic setup
  • Less Page/Web centric as fast as possible
  • Become an opensource contentful, prismic, … (mix deck), maybe we can do it like Mattermost, fully compatible with one the competitor API (don’t reinvent the wheel, check Contentful API, API basics | Contentful), and extend to offer more in some area.
  • Target: Pro tools, focus on agency that now how to setup a server go away from the shared hosting hell as fast we can.
  • Open: Integrate more external service when possible (FilePreviews.io, Zapier, OData, …)
  • Roadmap: Need to public, even if no date are define, but need to communication the next major feature, for the next versions (roadmap is not written in stone, things/priority can change)
  • Communication: More, more, more … but that’s hard, and not my preferred hobby And from my POV the best things we can do for communication if working with other community and gaining visibility in the developper community (and it’s not communication but code, and we know how to code)
  • Resist to the NIH syndrome (sometimes).
4 Likes
5

Hey, back to where I left off, want to leave a few more comments to Robert’s thoughts.

Upgrade ~= Rewrite (in our case)

Looking at current state of UI of Neos, deep refactoring would be very close to full rewrite. We plan to drop about everything we had in our stack (CreateJS, VIE, Aloha, start using iframe, get rid of two-way binding etc etc) and shift major focus from inline editing to structured editing and so on.
Looking at current pains during Ember 2.x update, I really doubt the fresh React re-write would require much more energy, and no, I don’t think that would take a year.

Poly-client from the start

But I think this UI rewrite should happen at the same time with Neos decoupling. Provide new UI as a separate package, make it optional, and when it stabilizes make it the default choice. Of course this step requires having some sort of API and getting used to the thought that we now have multiple interface clients.

Compose frontend with npm/webpack

Also continuing the thought of new UI to be modular, I think we should do similar thing to what we currently do in Flow land: have a distribution package for frontend, that would not contain almost any code, but rather pull in dependencies from npm and compile them with webpack, just as we do with composer. We may provide the default Neos distribution with pre-built JS code (in the way we still provide zip downloads in addition to composer), so if the novice would want the default interface he would get just that. But say you want to add your own interface modules, remove some modules or build a completely custom UI for your app: there you are, just create custom frontend distribution, modify packages.json and you got it.
And the reason why I think it’s best to get with npm, is that it has become the de-facto package manager in frontend world and its not associated with any specific vendor or framework, even projects like ember-cli or meteor allow to use its modules. This way our UI modules would be available to any custom app developers.

Having a modern stack is also very important to draw more frontend developers to the team. I doubt a lot of people would want to navigate our Ember jungle.

6

Now for the target audience.

Target audience

I’m all for aiming at pro market and big projects. But there are very different corners on this market: compare some bureaucratized German corporate monster to some tech-driven startup. In my opinion we should not clinch so much to the same niche as TYPO3 did, or we risk becoming another TYPO3 quite soon. I suggest we stay who we are, and produce something we would ourselves use, and the market will find us (with some effort ofc).

There’s a great market of SaS and PaS solutions, and if we get more projects of that sort built on top of Neos, that would be a much more healthy and innovative ecosystem for the Neos project itself.

Practical outcome for this: use any technology that fits our purpose better and aim for professionals who know how to run modern web apps (on own infrastructure or on PaS), rather than on shared web hosting or corporations with legacy hosting systems.

7

Hey @dimaip,

I believe we could write the Neos backend with both Ember 2.x and React; both would certainly be doable – and I’d even expect the code to look somehow similar, as Ember 2 (same as e.g. Angular 2) has been heavily inspired by data-down-actions-up of React.

However, when we have migrated to Ember 2 and Ember CLI (which is some route to go still), I see most benefits because Ember 2 has strong conventions (the way Flow has strong conventions as well); and React e.g. does not tell you which file-structure you should use for writing your modules…

I feel that this discussion in the end it boils down to a “library” (=react) or “framework” (=ember) approach.

To me personally, Dries from Drupal has summed up the state of the discussion really well: http://buytaert.net/selecting-a-client-side-framework-for-drupal

However, I’d suggest to move this discussion (e.g. which FE framework to use) out of this thread… And rather focus on our goals and not so-much on how we want to archive them.

All the best,
Sebastian

3 Likes
8

Hey thanks for your feedback and for setting the boundaries to the discussion. I have replied here:

9

I decided to read some discussions here this morning and I must say I’m pretty much delighted by reading this thread :wink:

My 2 cents:

  • I pretty much agree to the visions above
  • Flow with CQRS / Event sourcing / DDD would be awesome.
  • I would really love to change Flow from being a full framework to a layer on top of Symfony. Even if that means a hell of a job. If we manage to put our vision / the current strong points of ‘the way Flow works’ in such a layer it would open a lot of possibilities… (and the codebase would probably be smaller too :stuck_out_tongue: )
  • I would like to discuss moving the UI from Neos to a separate package. I’m getting tired of 2 things: the lack of a solid API and the constant framework discussions for the ajvascript part. We use Neos now only as a content repository and find that the strong points of Neos are easy to sell that way. Selling the interface is a bit harder for us though…
2 Likes
10

Thanks for that writeup Robert. Since I’m not really involved with Neos, but only with Flow for the time being, I only respond to that part:

Considering what I hear from the community and am passionate about myself, I think we should strengthen Flow and position it as the Domain-Driven Design framework for PHP. We could take all the experience we made together and work on Flow 4.0 which provides excellent support for Domain-Driven Design, CQRS, Event Sourcing and so on. During the last couple of weeks I have experimented a lot with Flow + CQRS + ES and I really see an opportunity there.

I can not express enough how much I would love to go and push that direction. Flow (or rather extbase) was my introduction into the DDD world and I barely grasped what I was actually doing from the start, but it all just felt right. After some years now, I have a better understanding of what DDD is and what Flow already delivers in that regard (and also what not, like clean Value Object support or Domain Events). I really love the way of DDD and only last year entered the world of CQRS + ES, which just totally set my (dev-)world on fire, hence I would love to make Flow the (opinionated) DDD Framework in the PHP world.

Regarding the idea of others to just wrap a layer around Symphony I’m not sure about that, but most likely because I’m not deep enough into Symphony to really know if that would be feasible. If so, then I wouldn’t mind to have a solid base to work on, though I’m not sure how that would restrict us from changing that base to the needs. We would basically have a huge dependency on the Symphony Team to help us get our required changes into the core then.

2 Likes
11

Great, would love to do that! The question is if we can get that right with the little experience we have and how much of a rework that will mean which is not a lot of benefit for the average user right away.

I see a lot of risk in that. The CR is the major player in the backend for Neos and with the above mentioned lack of experience with CQRS / ES we are bound to make mistakes in our first implementations. Starting with something that big seems to me very risky with little gains for Neos users. That said I would love to have it. But I see it rather as a vision than a goal for short (or maybe even mid term).

I think we positioned ourselfs and the project as a future project, that means we need to move forward technologically so I see rewrites and changes to libraries as something we constantly need to do to stay at the front of the pack. We shouldn’t sacrifice a whole lot of stability but we need to move. Fear of change will drive us down a bad path and also code that is working fine should be touched from time to time to get it up to date and make it easier to handle when the time comes that it needs to be touched. IMHO we should strife for constant renewal with as less disruption for users as posible and I think that is possible. Something to make that easier is obviously more testing. If we increase our test coverage we can much more easily deal with changes to libraries and the code base as we can be sure that for users it’s still the same.

And we need to stay shippable and be able to progress while we refactor

Yes absolutely, we should never be blocked by a major refactoring on the way in master, but shippable master is still something I don’t agree with 100%, as mentioned in other places there must be a place for mistakes, hasty decisions and second thoughts. Obviously this is the code review, still we are humans and also 4 and 6 eyes might not see everything, so master it is and I think it’s fine that errors end up in master, as long as they are fixed right away.

Decouple & Modularize

Same for Flow, saying that for ages and yes definitely needed!

About the symfony base, I had a closer look at it and I don’t feel so strongly that we should and easily can do it TBH. I still think we should decouple flow and then start to replace parts of it with existing components, but the full stack symfony is a very different thing. Symfony components do not offer much we need right now…

So far my 2cc, I guess at large I agree with the direction.

As it’s not that clear so far I would like to add:

  • Gradually improving experience with your technology.
    What I mean is that you should be able to run a Neos on classical LAMP with some features (about what we have now) but be able to ramp up your stack (with redis, ES, Postgresql, whatever we decide is good) and gain benefits from it (speed and features)

I think with the right decoupling and thoughts going into the architecture that is feasible and allows us to keep the existing target groups while at the same time opening up new user groups in the more “high end” sector.

4 Likes
12

Hi there,

(as posted in Slack already) I worked on an idea for a future architecture for Neos. I see that as the underlying foundation of the product strategy / vision because it might ultimately change the way Neos as a product is defined.

I’ll write about the details in a separate topic, but the main idea for me is to define boundaries for individual services and have a real and more radical separation between them. This should pave the way to select the best fitting technology, release cycle, team structure for each service. Following the idea of microservices, the technical architecture and organisational structure influence each other here.

What does that practically mean? Neos itself should be a suite of individual components / services that can be composed for content management solutions. It should be possible to replace these components by a different implementation depending on requirements and the project. It should also be possible to build a headless Content API server using Neos, or a Content API solution with a structured editing UI. Or having an editing interface and using Neos as a static site generator to publish for static hosting. Building a two stack CMS solution should be directly supported by the architecture.

So the scenarios where Neos can be used could be magnitudes higher and especially more complex projects / SaaS solutions benefit from the added flexibility.

6 Likes
13

Hey everyone,

It’s nice to see such a passionate discussion regarding the project. Especially when the vision seems to be very similar among everybody participating. Really brings the a shared feeling of joy for the project.

This is a topic I’ve spent a lot of time thinking about as well. I think Sebastian is spot on that it’s better to focus on where we want to go than how we get there.

Half a year ago I tried to summarize:

Where is Neos today?

CMS with limited feature set that gives a simple & intuitive editing experience that editors and developers love to use.

Allows for building websites with application functionality seamlessly together using a powerful application framework.

What will Neos be in the future?

A modern and fully customizable content application platform with strong conventions, good architecture and great user experience for editors & developers that can be used for any web solution purpose that creates and/or manages content for any type of output and strong integration with other applications.

Thus I really agree with the vision of a flexible content application platform. That requires modularization/decoupling/boundaries, clear interaction points (APIs) and composition flexibility. The possibility of seamless integration between websites and applications is a strong factor, so we should ensure it remains that way, and even improve it further. Contrary to Rens, the UI is a key selling point in my experience, so I’d like to keep a strong focus on that. However it depends on the projects, so it surely makes sense making it easier to remove that part if it’s not needed. Another key factor is the developer satisfaction, which we need to maintain and hopefully improve along the way.

+1 for getting rid of web centric focus
+1 for a optional composable UI
+1 for focussing on structured content
+1 for focussing on non hierarchical data in combination with hierarchical data
+1 for keeping a smooth as possible (ASAP ;)) migration path
+1 for focussing on handling of content (CR, dimensions, workspaces, CQRS)
+1 for focussing on professional developers

All this would as an example enable Neos being able to use Neos as an open source Contentful/GatherContent.

Personally I’m a bit torn between keeping the LAMP stack compatibility versus requiring modern server setups as I see both sides being valid. However I do agree with Christian regarding being an innovator and we should strive to keep it that way, since most of us are motivated by that aspect. So there’s a fine line between the two we have to find. So we keep innovating and avoid getting far behind since we didn’t adopt new approaches and still keep stability/backwards compatibility.

Also agree with Dominique regarding there being quite some way to be able to call Flow the DDD framework. The lack of encouraging bounded contexts, read/write models etc. which is what makes DDD really beneficial shows that. However going in that direction would be really great.

I somewhat feel the same. However I do think we can improve this part by changing the way we work. If we start developing bigger/riskier changes in separate branches with PRs or direct pulls to those branches and have a strict merge policy where we need more than usual amount of reviews before merging to master, we should be able to improve this without making the ongoing review process a challenge.

6 Likes
14

Hey all.
So cool to have this discussion here!
A lot of things have been mentioned already and I fully agree on the general direction. So I’ll restrain myself to the Flow-part for now (with a strong focus on the needs of a more decoupled Neos that you all seem to agree to).

I agree that we need adjusted tutorials and communication in general. I don’t think that Flow lacks many features for that though as DDD is mostly a matter of application design.

IMO there are currently three major parts where Flow underperforms in those regards:

It should be really easy to serve and consume (Hypermedia) APIs

It’s a pain to create proper REST APIs in Flow and that gotta change. It’s not a simple task due to the lack of accepted standards and the inherent dependencies of domain model, -logic and routing. But it could be done in a completely separate package and I already have some rough ideas.

You don’t need to go CQRS/Event Sourcing to be “Domain-Driven”, but it should be an option

…at least for parts of your application.
It’s a tough one because of the risks of over-engineering like it happened with things like SOAP or almost all Java frameworks :wink:

Everything should be composable

I share Alexanders and Christians concerns regarding the feasibility and benefit of rebuilding Flow on top of Symfony – at least I can’t imagine how that should work in practice.
But I think that we really need to start thinking more decoupled when creating new behavior. Most features can be created in completely separate packages. And our task should be to adjust the core where that’s not the case (e.g. provide additional hooks/APIs).
In addition we should strive for decoupling Flow subpackages into more general-purpose packages (and Christian has already started with that!)

One last thing regarding:

The great guys from yeebase (you might now them as t3n) generously support us kickstarting a microsite for flow. The plan is to setup a (neos) website as quickly and friction-free as possible. Please get in touch if you have any concerns regarding that approach and/or follow the #guild-website channel on slack for updates.

15

I do too in some regard. Though I feel like I need to make bold statements regarding using Symfony as a foundation as in my opinion this was opted about 2 years ago. By then people felt it might be a good choice and worth some research. @christianm did some research and found that components would be easily usable but there are not that many feasible components. The Symfony team actually said they’d be willing to discuss things we would need for making this possible and see if they could support that in the new version. To make a long story short we all shift it to the low priority / too much work list and the end result is (and will probably be): we keep maintaining our own code in our own little community world.

While I didn’t really do an in depth check how such change could be done I do really fear we are not putting enough effort into it and as such throw it in. If (and keep in mind I’m not 100% sure if it would be technically possible at all) we could manage to put the stuff that creates Flow (AOP, conventions, Fluid, and so on) into Symfony components we would be forced to decouple. But hey, this is all highly hypothetical :wink:

1 Like
16

@radmiraal Fluid in Symfony isn’t as hypothetical anymore :wink: i’m toying around with it right now :slight_smile:

17

Just to prevent any misunderstandings: I’m all for decoupling and to adapt other libraries where possible!

I never understood this as rebuilding Flow “as a slim layer on top of Symfony”, but maybe the end result is the same: make it easier to use 3rd party libraries in Flow and at least as importantly allow our components to be reused outside of Flow.
I just doubt that it’s feasible to do this in one large project but IMO we should start by

  • adapting common (PSR) interfaces and moving subpackages into (optional) external packages, leaving only a slim adapter in Flow
  • go the extra mile to create external packages when adding new major features/behavior and put Flow magic like AOP etc. on top of it within Flow or a separate adapter package

Do you agree to that, @radmiraal or do you have other ideas on how to approach this?

2 Likes
18

I’m just quietly reading most of the posts here but one thing that came to my mind when reading about Flow as a “Symfony layer”: can this be done in a way that existing packages/projects can be updated?
I think there are a bunch of bigger Flow projects out there and it could cause some serious problems for these companies if there’s a point (event if it’s a few years away) where we need to tell them to rebuild the whole thing.

1 Like
19

Hey David,

I agree with you that upgrading strategies and ways in which backwards compatibility must be kept is extremely important – I think that basically all of us agree here :smile:

So, nothing to worry about!

All the best,
Sebastian

1 Like
20

To be honest I’m shouting out ideas without concrete solutions in mind :wink: I’m still in the problem phase so to say. I feel like we in general do agree that we should open up / make our code shareable and usable to others. But somehow it never leads to really concrete steps or so.

Adapting common interfaces would indeed have a +100, and far better separation of features and behavior in packages would be lovely.

Maybe this product vision thread would be an interesting place to collect what such a ‘layer on top of symfony’ would mean? As that could help to extend what was written above already :wink: