Following is an RFC outlining the steps on how to organize the refactoring / rewrite; and we’ll update the RFC from time to time. I think we can still improve the RFC, but that’s here just a starting point. Feel free to directly edit the RFC (it is a “WIKI” post)
All the best,
RFC: JS Refactoring of the Backend – Structure
Build the Neos UI around loosely Components
Allow for planned and unplanned extensibility
do not pollute the global scope
allow to use components stand-alone, being in the long run able to very freely adjust the Neos UI.
Generally follow the “Data-Down-Actions-Up” paradigm popularized by React/Flux and also followed by Ember 2.0
We suggest to use ember 2.0.
Internally, the Node Data Model will (at first) still use the data-attributes; but in the mid-term, we might expriment with other ways of transmitting the node’s attributes.
We’d like to get rid of VIE / Create; which is mostly replaced by our custom Node Data Model
It is still open whether we want to use Ember Data or not. @christopher and @radmiraal have experiences with recent versions.
We want to use ES6 Modules and other ES6 features.
As consequence we need a cross compiler like BabelJS
As consequence, we suggest to get rid of Require.js
Grunt vs Gulp vs Ember CLI:
Now that we settled for Ember, the current suggestion is to use Ember CLI. @radmiraal has good experiences with it.
(previous argument): just use which is quicker in execution times (might be Gulp)
Note: Execution Time of the build is lot more critical than currently, because we’d rebuild on every code-change then.
Components should be “self-contained”, i.e. contain JS, Handlebars, SASS, Tests grouped into the component.
It is yet undecided whether we want to explore editing content in an iframe and go this direction or not. However, we agree that the bootstrap should be reversed: First, JS should be initialized, second, the content area is filled (which could be filled with either an iFrame or without an iFrame)
We will definitely move to CKEditor for editing.
(what is missing?)
Priority List Suggestion / Working Steps
We propose to have intermediate steps which can be merged directly in Master; trying to avoid a long-running fork.
Personally, I’m pretty happy with all the results except the “framework decision”. Don’t get me wrong I don’t want to transform this very refactoring aspect into a bottomless pit, but I guess sticking with Ember might be a problem in the long run.
Ember is a great framework when it comes to building single page applications with all it’s whistles and bells (routing, 2way binding, you name it …). From my point of view the Neos backend is not a classical SPA. I look at it more as a heterogeneous landscape of loosely coupled and therefore self-contained components. Something which can be achieved with a “framework” which follows the component-oriented paradigm in a more “aggressive” manner, like Flux + React, Angular 2 (not an option atm) or NodeProto + ComponentDomParser (nice because lightweight).
I’m pretty excited about the “new” Neos movement and happy to hear your thoughts.
None of the above libraries (afaik) handle the problem of computed properties in a good way – that’s the main abstraction we/I enjoy in Ember; as you do not have to handle state change implicitly only using events.
I’ve had a long discussion with @inkdpixels on the framework-vs-non-framework topic; so I’d also be curious what he writes
In my POV we need to avoid the NIH or reinvent the wheel too much.
One of my pain point for the current implementation is that we have a semantic backend, but if the frondend is semantic based (read use typeof attributes) we can have a lots of problem with the current implementation.
Not really related to the JS refactoring but did you have some discussion about Aloha (or other editor) during the sprint ? Maybe this can be continued in a related topic.
As @dfeyer said, I totally agree on not reinventing the wheel. But the question with frameworks is a bigger one with more facades in it. From my perspective, the rewrite needs a solid foundation, but the foundation and the neos code should also be as easy to learn as possible. I am not opinionated on Ember 2.0 at this moment, and I’ve also had further discussions about this with @akoenig and @wbehncke at work today.
At the bottom of it we all agreed on the following:
We should, as proposed in this post, write a prototype with Ember 2.0 and try to get the best out of it. At this stage we have more insights in how complex the Code looks, how easy it is for beginners to jump in, evaluate the performance and even criticize ourselves and our decision.
If we are happy with the prototype, we should head into the direction, but as well have an abstraction layer between Neos and Ember(thanks to @akoenig for this idea) - to reduce the friction on upgrading/switching to other frameworks in the future(this of course shouldn’t be a goal (hah!), but should be counted in since the JS landscape is really iterative and fast living.
If - and this of course may be possible - the arguments against Ember still count and hold after producing the prototype, we should reconsider this decision and have a look at the things we like and dislike of Ember.
a really easy to use and slim RTE, didn’t do advanced stuff with it and it would require a PR upstream to use p tags instead of divs, but I like how you can completely build your own toolbar and link the editor easily to that (or even better: by default you have to make your own toolbar)
I’d like to state out, that computed properties and data-binding for themselves are more or less just one paradigm for solving the problem of keeping your data model and the DOM in sync. But I think it’s a paradigm with huge drawbacks especially regarding readability and debugging. It sure is a way to cover complexity, but not to actually reduce it.
Virtual DOM technology is aimed at the same problem, but reduces complexity since rendering and syncing are disjunct processes here. For all who don’t know V-DOM already, here’s a pretty nice talk, that also puts it into perspective regarding embers computed properties and angulars dirty checking: https://www.youtube.com/watch?v=1h2G20A-AvY
Last but not least: I think “big framework yes/no” still needs discussion. I have difficulties to understand the Neos backend as a classical rich client application - which would be the common use case all those embers, backbones and angulars are designed for. The Neos backend tries to enrich an environment that for itself could be an entire application since it gets more and more difficult to distinguish between “web pages” and “web applications”. It throws itself on top of something that’s already there and imho needs to aim for as little interference as possible while still delivering the functionalities needed for in-place editing and such.
Having Neos as a platform is a big thing to me. We are (and have been) working in multiple Neos projects now where they needed some kind of frontend editing. And one thing has been proven: the Neos UI ain’t the solution for just some frontend editing as you get all the other stuff on top.
I’ve had usecases where we only wanted to show a subtree in the navigate component, or no navigate component at all… Or maybe just really advanced editors in the inspector, and no publish menu… or whatever. We will have numerous custom implementations of frontend editing soon if we don’t support that, and if we’re really serious about being a content editing platform we’ve no other choice than making it a platform on the JS side.
Saying all this I also realize that the chances are bigger that people will use their framework of choice for building stuff on top, and might see a choice for a big js framework as a bourdon. Vanilla js stuff is accepted far more easily because it doesn’t add dependencies. We all still know what we’ve seen as problems for VIE, hallo, aloha and so on: the dependencies on certain frameworks.
So I agree that we need arguments for picking a framework. And we don’t need arguments for ‘not picking them’ Just good solid arguments what is actually fixed by the framework, and if that’s just about the data binding for example, or just about the object model… than we might even find libraries around that offer just that.
+100 to what you generally wrote; I think we very much agree on the goals here.
I don’t know of libraries which solve the above two problems really solidly (except ember). For me it would be the above two parts, plus the View hierarchy (and virtual DOM with Ember 2.0) which I’d really want.
I’d suggest we build a custom Ember version just containing these parts. That should be fairly easy I think.
This can lead to very Java-like abstractions that make things harder to understand. I’d suggest to keep the code containing our logic / service interaction etc. as clean and modular as possible (in many cases we don’t need much Ember there, maybe some RSVP promises, but that’s not a big deal). But in many cases you’d want to write the implementation directly using the framework without any further abstractions (e.g. when creating components for the user interface). I never saw that these particular “we might want to switch the framework” abstractions are worth the effort if it’s not about having decoupled, modular code - but without too many abstraction layers.
Overall I don’t want to sound negative here or harm motivation in any way. When working on Neos for several years it’s certainly time to rethink very early decisions like the particular framework. My problem here is that we risk to do some very serious bike-shedding when we need to decide over everything again (and need to iterate, get it stable, find best practices, maintain it, etc.).
Regarding the discussion about Neos as a SPA. When we decide to go the COPE way (and forget about inline editing that is only one specific editing mode for web output) we need to tackle many more problems than “just” having a tree, inspector and RTEs (greatly simplified). There should be all kinds of components / modules that need to be composable and configurable by an integrator. And there will be the need for routing/state and composition of a larger application from many components. I’d not like to have a traditional server-side driven webapp for many of these parts because that means to split our UI into two separate layers and increases complexity again. My vision for Neos as a backend would be to provide solid services around content and producing output for various channels. The Neos UI should not be fixed (that means it needs to be pretty dynamic) and should be only one possible implementation over the backend services.
If we find out Ember or other “large” frameworks have serious drawbacks to achieve these goals, I’m totally fine to get back to the very essence because that will mean Neos as a product will have an advantage by choosing another kind of technology.
Here’s a short summary of what we’ve discussed on friday:
We agree on moving to ES2015
Refactoring and Rewrite can be handled as seperate tasks, so that the more urgent refactoring issues get a chance to be included in sooner releases
Since there are too many open questions, we didn’t reach a settlement regarding the framework decision
To get started on addressing the more urgent things and to get more technical insight on the future foundation for the backend in the medium term, we decided the following next steps:
Refactor the existing code base to get rid of require in favor of ES2015 modules and therefore integrating babel and browserify in the build process. This can be done immediately and should be targeted for the 2.1 release. (@inkdpixels, @akoenig and myself would volunteer for that)
Create a Proof of concept in Ember 2.0 for a complete rewrite/architecural redesign.
Create a Proof of concept in a Flux/React-like architecture.
Points 2 and 3 can be done in parallel and should help to reach clarity on the framework question.
Please feel free to amend anything that I might have missed.
Bookmark post, but hope it’s useful for other. For the refactoring we need a really solid “data storage” on the client side to handle data / sync / transform, and I just found this lib Orbit.js (https://github.com/orbitjs/orbit.js/tree/master) who support JSON-API and many other, also memory, local storage, … and complex configuration, like save in memory, and next async to the server rest API (with sync of the record ID when creating new record, …). Also support undo operation, …