This is the part I’m least competent in, but I do totally agree to what Sebastian said about refactoring as much as possible on top of low-level Symfony components. As my professional experience grows, I try to withhold from large buy-in on large monolith frameworks, and on personal project see myself more and more in nodejs land, but it doesn’t matter so much here.
Neos
Breaking the thing apart
I totally agree with you here on need for decoupling our monolith baby into packages that have clear bounded contexts talking to each other via an API (content repository, universal/isomorphic(!) rendering engine, JS admin interface…), we talked it through many times already.
Neos JS admin interface
Structured editing + bulk records editing
In current editing experience the crucial thing for my company that we currently miss is the structured editing + bulk editing. We have tons of content, and we need great UI performance to edit all of it. For most of the stuff we need just plain raw editing mode, without inline editing. Also we don’t care for an RTE, just let it do bold/italic/links and we’d be happy with it.
Break down Neos UI into components and allow to use only the pieces we need in our own web apps
We discussed it multiple times as well, to have a library of structured content editing and UI components, rather than a monolith thing. This way we could craft custom editing experiences out of small components where it is required (e.g. editorial magazines and just about any large custom project).
I will pause for now and pause the topic of target audience later this evening.
Thanks @robert for starting this discussion. You have my for more or less all your points. Bellow a short/unpolished resume from my POV
Flow
I’m fine with that as a goal, but it’s a bit too early to communicate on this point from my POV. DDD is hard (and we don’t do DDD in a really advanced way currently). If we decide to follow this path (and I think we should), we MUST include in the process guys like Mathias Verraes (http://verraes.net/), even if this cost us some money. As said before DDD is pretty hard, doing it well at application level is hard, but in an opinionated framework it MUST be done right from the ground.
I love Flow since the first day (maybe the second or the third), because of how useful it can be and improve the quality of the work in our team. Convention / Opinionated sounds good to me. If we can specialize Flow (like low level DDD support), i think we can have a market. But we currently lack a lots of ressource, so we need to work closely with Symfony, I remember a discussion with Fabien, that say that we can collaborate and maybe the next major version of Symfony can be adapted to support what we need. If we can have Flow 4 as a slim layer on top of Symfony that add the magic/useful layer that we like (AOP, Property Mapper, Fluid, …). For AOP we can collaborate with Alexander from GoAOP. If we move in the direction of a really open community, focused on building a framework a bit more opinionated and focused (DDD) that the others, we have a chance to gain more visibility and confidence from the market. Yes the first step will be very hard, but after this we can really focus on our vision, and stop maintaining the full stack alone.
For me, the framework is pretty cool now, but market wise to need to be more specialized on communicate more around key features. I don’t see a big opportunity to become the next hot framework, so we need the be aware resource wise.
Neos
Disclaimer, i spend 80% of my time between Flow and Neos project, if I can spend 110% of my time on Flow/Neos project before the end of the year, it’s pretty ok for me
Decouple & Modularize
More test / Less regression in the UI
Major features (Media Browser, HTTP Redirect, Asset Meta Data Handling, …) need to be in one or more dedicated Package, and not required in a basic setup
Less Page/Web centric as fast as possible
Become an opensource contentful, prismic, … (mix deck), maybe we can do it like Mattermost, fully compatible with one the competitor API (don’t reinvent the wheel, check Contentful API, API basics | Contentful), and extend to offer more in some area.
Target: Pro tools, focus on agency that now how to setup a server go away from the shared hosting hell as fast we can.
Open: Integrate more external service when possible (FilePreviews.io, Zapier, OData, …)
Roadmap: Need to public, even if no date are define, but need to communication the next major feature, for the next versions (roadmap is not written in stone, things/priority can change)
Communication: More, more, more … but that’s hard, and not my preferred hobby And from my POV the best things we can do for communication if working with other community and gaining visibility in the developper community (and it’s not communication but code, and we know how to code)
Hey, back to where I left off, want to leave a few more comments to Robert’s thoughts.
Upgrade ~= Rewrite (in our case)
Looking at current state of UI of Neos, deep refactoring would be very close to full rewrite. We plan to drop about everything we had in our stack (CreateJS, VIE, Aloha, start using iframe, get rid of two-way binding etc etc) and shift major focus from inline editing to structured editing and so on.
Looking at current pains during Ember 2.x update, I really doubt the fresh React re-write would require much more energy, and no, I don’t think that would take a year.
Poly-client from the start
But I think this UI rewrite should happen at the same time with Neos decoupling. Provide new UI as a separate package, make it optional, and when it stabilizes make it the default choice. Of course this step requires having some sort of API and getting used to the thought that we now have multiple interface clients.
Compose frontend with npm/webpack
Also continuing the thought of new UI to be modular, I think we should do similar thing to what we currently do in Flow land: have a distribution package for frontend, that would not contain almost any code, but rather pull in dependencies from npm and compile them with webpack, just as we do with composer. We may provide the default Neos distribution with pre-built JS code (in the way we still provide zip downloads in addition to composer), so if the novice would want the default interface he would get just that. But say you want to add your own interface modules, remove some modules or build a completely custom UI for your app: there you are, just create custom frontend distribution, modify packages.json and you got it.
And the reason why I think it’s best to get with npm, is that it has become the de-facto package manager in frontend world and its not associated with any specific vendor or framework, even projects like ember-cli or meteor allow to use its modules. This way our UI modules would be available to any custom app developers.
Having a modern stack is also very important to draw more frontend developers to the team. I doubt a lot of people would want to navigate our Ember jungle.
I’m all for aiming at pro market and big projects. But there are very different corners on this market: compare some bureaucratized German corporate monster to some tech-driven startup. In my opinion we should not clinch so much to the same niche as TYPO3 did, or we risk becoming another TYPO3 quite soon. I suggest we stay who we are, and produce something we would ourselves use, and the market will find us (with some effort ofc).
There’s a great market of SaS and PaS solutions, and if we get more projects of that sort built on top of Neos, that would be a much more healthy and innovative ecosystem for the Neos project itself.
Practical outcome for this: use any technology that fits our purpose better and aim for professionals who know how to run modern web apps (on own infrastructure or on PaS), rather than on shared web hosting or corporations with legacy hosting systems.
I believe we could write the Neos backend with both Ember 2.x and React; both would certainly be doable – and I’d even expect the code to look somehow similar, as Ember 2 (same as e.g. Angular 2) has been heavily inspired by data-down-actions-up of React.
However, when we have migrated to Ember 2 and Ember CLI (which is some route to go still), I see most benefits because Ember 2 has strong conventions (the way Flow has strong conventions as well); and React e.g. does not tell you which file-structure you should use for writing your modules…
I feel that this discussion in the end it boils down to a “library” (=react) or “framework” (=ember) approach.
However, I’d suggest to move this discussion (e.g. which FE framework to use) out of this thread… And rather focus on our goals and not so-much on how we want to archive them.
I decided to read some discussions here this morning and I must say I’m pretty much delighted by reading this thread
My 2 cents:
I pretty much agree to the visions above
Flow with CQRS / Event sourcing / DDD would be awesome.
I would really love to change Flow from being a full framework to a layer on top of Symfony. Even if that means a hell of a job. If we manage to put our vision / the current strong points of ‘the way Flow works’ in such a layer it would open a lot of possibilities… (and the codebase would probably be smaller too )
I would like to discuss moving the UI from Neos to a separate package. I’m getting tired of 2 things: the lack of a solid API and the constant framework discussions for the ajvascript part. We use Neos now only as a content repository and find that the strong points of Neos are easy to sell that way. Selling the interface is a bit harder for us though…
Thanks for that writeup Robert. Since I’m not really involved with Neos, but only with Flow for the time being, I only respond to that part:
Considering what I hear from the community and am passionate about myself, I think we should strengthen Flow and position it as the Domain-Driven Design framework for PHP. We could take all the experience we made together and work on Flow 4.0 which provides excellent support for Domain-Driven Design, CQRS, Event Sourcing and so on. During the last couple of weeks I have experimented a lot with Flow + CQRS + ES and I really see an opportunity there.
I can not express enough how much I would love to go and push that direction. Flow (or rather extbase) was my introduction into the DDD world and I barely grasped what I was actually doing from the start, but it all just felt right. After some years now, I have a better understanding of what DDD is and what Flow already delivers in that regard (and also what not, like clean Value Object support or Domain Events). I really love the way of DDD and only last year entered the world of CQRS + ES, which just totally set my (dev-)world on fire, hence I would love to make Flow the (opinionated) DDD Framework in the PHP world.
Regarding the idea of others to just wrap a layer around Symphony I’m not sure about that, but most likely because I’m not deep enough into Symphony to really know if that would be feasible. If so, then I wouldn’t mind to have a solid base to work on, though I’m not sure how that would restrict us from changing that base to the needs. We would basically have a huge dependency on the Symphony Team to help us get our required changes into the core then.
Great, would love to do that! The question is if we can get that right with the little experience we have and how much of a rework that will mean which is not a lot of benefit for the average user right away.
I see a lot of risk in that. The CR is the major player in the backend for Neos and with the above mentioned lack of experience with CQRS / ES we are bound to make mistakes in our first implementations. Starting with something that big seems to me very risky with little gains for Neos users. That said I would love to have it. But I see it rather as a vision than a goal for short (or maybe even mid term).
I think we positioned ourselfs and the project as a future project, that means we need to move forward technologically so I see rewrites and changes to libraries as something we constantly need to do to stay at the front of the pack. We shouldn’t sacrifice a whole lot of stability but we need to move. Fear of change will drive us down a bad path and also code that is working fine should be touched from time to time to get it up to date and make it easier to handle when the time comes that it needs to be touched. IMHO we should strife for constant renewal with as less disruption for users as posible and I think that is possible. Something to make that easier is obviously more testing. If we increase our test coverage we can much more easily deal with changes to libraries and the code base as we can be sure that for users it’s still the same.
And we need to stay shippable and be able to progress while we refactor
Yes absolutely, we should never be blocked by a major refactoring on the way in master, but shippable master is still something I don’t agree with 100%, as mentioned in other places there must be a place for mistakes, hasty decisions and second thoughts. Obviously this is the code review, still we are humans and also 4 and 6 eyes might not see everything, so master it is and I think it’s fine that errors end up in master, as long as they are fixed right away.
Decouple & Modularize
Same for Flow, saying that for ages and yes definitely needed!
About the symfony base, I had a closer look at it and I don’t feel so strongly that we should and easily can do it TBH. I still think we should decouple flow and then start to replace parts of it with existing components, but the full stack symfony is a very different thing. Symfony components do not offer much we need right now…
So far my 2cc, I guess at large I agree with the direction.
As it’s not that clear so far I would like to add:
Gradually improving experience with your technology.
What I mean is that you should be able to run a Neos on classical LAMP with some features (about what we have now) but be able to ramp up your stack (with redis, ES, Postgresql, whatever we decide is good) and gain benefits from it (speed and features)
I think with the right decoupling and thoughts going into the architecture that is feasible and allows us to keep the existing target groups while at the same time opening up new user groups in the more “high end” sector.
(as posted in Slack already) I worked on an idea for a future architecture for Neos. I see that as the underlying foundation of the product strategy / vision because it might ultimately change the way Neos as a product is defined.
I’ll write about the details in a separate topic, but the main idea for me is to define boundaries for individual services and have a real and more radical separation between them. This should pave the way to select the best fitting technology, release cycle, team structure for each service. Following the idea of microservices, the technical architecture and organisational structure influence each other here.
What does that practically mean? Neos itself should be a suite of individual components / services that can be composed for content management solutions. It should be possible to replace these components by a different implementation depending on requirements and the project. It should also be possible to build a headless Content API server using Neos, or a Content API solution with a structured editing UI. Or having an editing interface and using Neos as a static site generator to publish for static hosting. Building a two stack CMS solution should be directly supported by the architecture.
So the scenarios where Neos can be used could be magnitudes higher and especially more complex projects / SaaS solutions benefit from the added flexibility.
It’s nice to see such a passionate discussion regarding the project. Especially when the vision seems to be very similar among everybody participating. Really brings the a shared feeling of joy for the project.
This is a topic I’ve spent a lot of time thinking about as well. I think Sebastian is spot on that it’s better to focus on where we want to go than how we get there.
Half a year ago I tried to summarize:
Where is Neos today?
CMS with limited feature set that gives a simple & intuitive editing experience that editors and developers love to use.
Allows for building websites with application functionality seamlessly together using a powerful application framework.
What will Neos be in the future?
A modern and fully customizable content application platform with strong conventions, good architecture and great user experience for editors & developers that can be used for any web solution purpose that creates and/or manages content for any type of output and strong integration with other applications.
Thus I really agree with the vision of a flexible content application platform. That requires modularization/decoupling/boundaries, clear interaction points (APIs) and composition flexibility. The possibility of seamless integration between websites and applications is a strong factor, so we should ensure it remains that way, and even improve it further. Contrary to Rens, the UI is a key selling point in my experience, so I’d like to keep a strong focus on that. However it depends on the projects, so it surely makes sense making it easier to remove that part if it’s not needed. Another key factor is the developer satisfaction, which we need to maintain and hopefully improve along the way.
+1 for getting rid of web centric focus
+1 for a optional composable UI
+1 for focussing on structured content
+1 for focussing on non hierarchical data in combination with hierarchical data
+1 for keeping a smooth as possible (ASAP ;)) migration path
+1 for focussing on handling of content (CR, dimensions, workspaces, CQRS)
+1 for focussing on professional developers
All this would as an example enable Neos being able to use Neos as an open source Contentful/GatherContent.
Personally I’m a bit torn between keeping the LAMP stack compatibility versus requiring modern server setups as I see both sides being valid. However I do agree with Christian regarding being an innovator and we should strive to keep it that way, since most of us are motivated by that aspect. So there’s a fine line between the two we have to find. So we keep innovating and avoid getting far behind since we didn’t adopt new approaches and still keep stability/backwards compatibility.
Also agree with Dominique regarding there being quite some way to be able to call Flow the DDD framework. The lack of encouraging bounded contexts, read/write models etc. which is what makes DDD really beneficial shows that. However going in that direction would be really great.
I somewhat feel the same. However I do think we can improve this part by changing the way we work. If we start developing bigger/riskier changes in separate branches with PRs or direct pulls to those branches and have a strict merge policy where we need more than usual amount of reviews before merging to master, we should be able to improve this without making the ongoing review process a challenge.
Hey all.
So cool to have this discussion here!
A lot of things have been mentioned already and I fully agree on the general direction. So I’ll restrain myself to the Flow-part for now (with a strong focus on the needs of a more decoupled Neos that you all seem to agree to).
I agree that we need adjusted tutorials and communication in general. I don’t think that Flow lacks many features for that though as DDD is mostly a matter of application design.
IMO there are currently three major parts where Flow underperforms in those regards:
It should be really easy to serve and consume (Hypermedia) APIs
It’s a pain to create proper REST APIs in Flow and that gotta change. It’s not a simple task due to the lack of accepted standards and the inherent dependencies of domain model, -logic and routing. But it could be done in a completely separate package and I already have some rough ideas.
You don’t need to go CQRS/Event Sourcing to be “Domain-Driven”, but it should be an option
…at least for parts of your application.
It’s a tough one because of the risks of over-engineering like it happened with things like SOAP or almost all Java frameworks
Everything should be composable
I share Alexanders and Christians concerns regarding the feasibility and benefit of rebuilding Flow on top of Symfony – at least I can’t imagine how that should work in practice.
But I think that we really need to start thinking more decoupled when creating new behavior. Most features can be created in completely separate packages. And our task should be to adjust the core where that’s not the case (e.g. provide additional hooks/APIs).
In addition we should strive for decoupling Flow subpackages into more general-purpose packages (and Christian has already started with that!)
One last thing regarding:
The great guys from yeebase (you might now them as t3n) generously support us kickstarting a microsite for flow. The plan is to setup a (neos) website as quickly and friction-free as possible. Please get in touch if you have any concerns regarding that approach and/or follow the #guild-website channel on slack for updates.
I do too in some regard. Though I feel like I need to make bold statements regarding using Symfony as a foundation as in my opinion this was opted about 2 years ago. By then people felt it might be a good choice and worth some research. @christianm did some research and found that components would be easily usable but there are not that many feasible components. The Symfony team actually said they’d be willing to discuss things we would need for making this possible and see if they could support that in the new version. To make a long story short we all shift it to the low priority / too much work list and the end result is (and will probably be): we keep maintaining our own code in our own little community world.
While I didn’t really do an in depth check how such change could be done I do really fear we are not putting enough effort into it and as such throw it in. If (and keep in mind I’m not 100% sure if it would be technically possible at all) we could manage to put the stuff that creates Flow (AOP, conventions, Fluid, and so on) into Symfony components we would be forced to decouple. But hey, this is all highly hypothetical
Just to prevent any misunderstandings: I’m all for decoupling and to adapt other libraries where possible!
I never understood this as rebuilding Flow “as a slim layer on top of Symfony”, but maybe the end result is the same: make it easier to use 3rd party libraries in Flow and at least as importantly allow our components to be reused outside of Flow.
I just doubt that it’s feasible to do this in one large project but IMO we should start by
adapting common (PSR) interfaces and moving subpackages into (optional) external packages, leaving only a slim adapter in Flow
go the extra mile to create external packages when adding new major features/behavior and put Flow magic like AOP etc. on top of it within Flow or a separate adapter package
Do you agree to that, @radmiraal or do you have other ideas on how to approach this?
I’m just quietly reading most of the posts here but one thing that came to my mind when reading about Flow as a “Symfony layer”: can this be done in a way that existing packages/projects can be updated?
I think there are a bunch of bigger Flow projects out there and it could cause some serious problems for these companies if there’s a point (event if it’s a few years away) where we need to tell them to rebuild the whole thing.
I agree with you that upgrading strategies and ways in which backwards compatibility must be kept is extremely important – I think that basically all of us agree here
To be honest I’m shouting out ideas without concrete solutions in mind I’m still in the problem phase so to say. I feel like we in general do agree that we should open up / make our code shareable and usable to others. But somehow it never leads to really concrete steps or so.
Adapting common interfaces would indeed have a +100, and far better separation of features and behavior in packages would be lovely.
Maybe this product vision thread would be an interesting place to collect what such a ‘layer on top of symfony’ would mean? As that could help to extend what was written above already
Regarding the “Flow on Symfony” part: I agree with @bwaidelich here - I don’t think that it’s feasible or would make sense for our existing community to start from scratch and implement the Flow feature set on top of Symonfy. Also I don’t think that we should - right now - look further into making Flow a Symfony application (that would mean, compatibility on bundle level).
That being said, we should definitely look into modularizing Flow more and also looking into replacing certain parts of Flow with third-party components. Typical examples would be logging, the HTTP stack and so on. We also should look into taking advantage of Symfony’s tooling, like the profiler.
What’s important to me in that regard is:
we should maintain and further improve the typical Flow developer experience, which means convenience features like DI (property injection …), automatic registration of certain components (command controllers and their documentation for example)
we should make it more natural for experienced Symfony developers to find their way around in Flow
we should replace existing code with third party components which is not part of the special Flow developer experience and is probably better maintained elsewhere (HTTP, our virtual browser, loggging, …)
we should strive for making Flow more lightweight (e.g. an option to use Flow without AOP, look for a solution without proxy class generation, make it possible to realistically create PHAR applications with Flow)
we must make sure to provide a realistic upgrade part for existing Flow projects
I think we should create technical story-tickets (possibly with sub-tasks) for those, explaining the desired long-term outcome in order to prevent situations like:
for more or less all your points. Bellow a short/unpolished resume from my POV
)
