Following is an RFC outlining the steps on how to organize the refactoring / rewrite; and we’ll update the RFC from time to time. I think we can still improve the RFC, but that’s here just a starting point. Feel free to directly edit the RFC (it is a “WIKI” post)
All the best,
Sebastian
RFC: JS Refactoring of the Backend – Structure
Goals
Build the Neos UI around loosely Components
Allow for planned and unplanned extensibility
do not pollute the global scope
allow to use components stand-alone, being in the long run able to very freely adjust the Neos UI.
Generally follow the “Data-Down-Actions-Up” paradigm popularized by React/Flux and also followed by Ember 2.0
Solution Parts
We suggest to use ember 2.0.
We need a Unified Node Data Model in JavaScript which is expressive; which should be used by all parts of the UI (editing, tree, inspector, …).
Internally, the Node Data Model will (at first) still use the data-attributes; but in the mid-term, we might expriment with other ways of transmitting the node’s attributes.
We’d like to get rid of VIE / Create; which is mostly replaced by our custom Node Data Model
It is still open whether we want to use Ember Data or not. @christopher and @radmiraal have experiences with recent versions.
We want to use ES6 Modules and other ES6 features.
As consequence we need a cross compiler like BabelJS
As consequence, we suggest to get rid of Require.js
We’ll very likely have a compile daemon running at all times when developing JavaScript.
Grunt vs Gulp vs Ember CLI:
Now that we settled for Ember, the current suggestion is to use Ember CLI. @radmiraal has good experiences with it.
(previous argument): just use which is quicker in execution times (might be Gulp)
Note: Execution Time of the build is lot more critical than currently, because we’d rebuild on every code-change then.
Components should be “self-contained”, i.e. contain JS, Handlebars, SASS, Tests grouped into the component.
We nee an extensible bootstrap process, so components can be activated / deactivated. How this works in detail, we do not know yet. Do we need something like Dependency Injection or will we do this in the build process?
→ Dependency Injection - Application Concerns - Ember Guides
It is yet undecided whether we want to explore editing content in an iframe and go this direction or not. However, we agree that the bootstrap should be reversed: First, JS should be initialized, second, the content area is filled (which could be filled with either an iFrame or without an iFrame)
We will definitely move to CKEditor for editing.
(what is missing?)
Priority List Suggestion / Working Steps
We propose to have intermediate steps which can be merged directly in Master; trying to avoid a long-running fork.
Personally, I’m pretty happy with all the results except the “framework decision”. Don’t get me wrong I don’t want to transform this very refactoring aspect into a bottomless pit, but I guess sticking with Ember might be a problem in the long run.
Ember is a great framework when it comes to building single page applications with all it’s whistles and bells (routing, 2way binding, you name it …). From my point of view the Neos backend is not a classical SPA. I look at it more as a heterogeneous landscape of loosely coupled and therefore self-contained components. Something which can be achieved with a “framework” which follows the component-oriented paradigm in a more “aggressive” manner, like Flux + React, Angular 2 (not an option atm) or NodeProto + ComponentDomParser (nice because lightweight).
My second argument concerns the community. I was under the assumption that Ember is one of the most used frontend frameworks. Yes, you’re right, the numbers which are flying through the web telling you that. I attended a pretty large JavaScript meetup in Hamburg yesterday. One of the speakers asked which framework the attendees use on a daily basis. Sure, AngularJS and Flux + React are the cool kids on the block and everybody uses them. I was shocked to see that nobody (!) uses Ember. You might say that this is only a tiny portion of the community and you’re right, but I think that community penetration should also be one criteria when it comes to building the foundation of the upcoming Neos releases. I don’t say to follow a hype just to attract more developers. What I mean is considering the technical aspects which lead to the trend of minimalistic component-oriented instead of monolithic “one to rule everything” frameworks.
I’m pretty excited about the “new” Neos movement and happy to hear your thoughts.
None of the above libraries (afaik) handle the problem of computed properties in a good way – that’s the main abstraction we/I enjoy in Ember; as you do not have to handle state change implicitly only using events.
I’ve had a long discussion with @inkdpixels on the framework-vs-non-framework topic; so I’d also be curious what he writes
In my POV we need to avoid the NIH or reinvent the wheel too much.
One of my pain point for the current implementation is that we have a semantic backend, but if the frondend is semantic based (read use typeof attributes) we can have a lots of problem with the current implementation.
Not really related to the JS refactoring but did you have some discussion about Aloha (or other editor) during the sprint ? Maybe this can be continued in a related topic.
As @dfeyer said, I totally agree on not reinventing the wheel. But the question with frameworks is a bigger one with more facades in it. From my perspective, the rewrite needs a solid foundation, but the foundation and the neos code should also be as easy to learn as possible. I am not opinionated on Ember 2.0 at this moment, and I’ve also had further discussions about this with @akoenig and @wbehncke at work today.
At the bottom of it we all agreed on the following:
We should, as proposed in this post, write a prototype with Ember 2.0 and try to get the best out of it. At this stage we have more insights in how complex the Code looks, how easy it is for beginners to jump in, evaluate the performance and even criticize ourselves and our decision.
If we are happy with the prototype, we should head into the direction, but as well have an abstraction layer between Neos and Ember(thanks to @akoenig for this idea) - to reduce the friction on upgrading/switching to other frameworks in the future(this of course shouldn’t be a goal (hah!), but should be counted in since the JS landscape is really iterative and fast living.
If - and this of course may be possible - the arguments against Ember still count and hold after producing the prototype, we should reconsider this decision and have a look at the things we like and dislike of Ember.
a really easy to use and slim RTE, didn’t do advanced stuff with it and it would require a PR upstream to use p tags instead of divs, but I like how you can completely build your own toolbar and link the editor easily to that (or even better: by default you have to make your own toolbar)
+1 for quill - I already used it in a couple of projects. Besides it’s neat integration, it is also pretty cool that it deals with deltas only. That enables it to be used collaboratively at realtime.
http://jakiestfu.github.io/Medium.js/docs/ might also be worth a look. I like how they explain the medium approach and anybody who ever worked with the medium editor knows that it pays off.
Regarding computed properties and data-binding:
I’d like to state out, that computed properties and data-binding for themselves are more or less just one paradigm for solving the problem of keeping your data model and the DOM in sync. But I think it’s a paradigm with huge drawbacks especially regarding readability and debugging. It sure is a way to cover complexity, but not to actually reduce it.
Virtual DOM technology is aimed at the same problem, but reduces complexity since rendering and syncing are disjunct processes here. For all who don’t know V-DOM already, here’s a pretty nice talk, that also puts it into perspective regarding embers computed properties and angulars dirty checking: https://www.youtube.com/watch?v=1h2G20A-AvY
Last but not least: I think “big framework yes/no” still needs discussion. I have difficulties to understand the Neos backend as a classical rich client application - which would be the common use case all those embers, backbones and angulars are designed for. The Neos backend tries to enrich an environment that for itself could be an entire application since it gets more and more difficult to distinguish between “web pages” and “web applications”. It throws itself on top of something that’s already there and imho needs to aim for as little interference as possible while still delivering the functionalities needed for in-place editing and such.
At the same time I would find it necessary to deliver the backend as something that is closer to a platform, others could build upon. It also needs to be easy for your frontend JavaScript to talk to it, without binding yourself to specific implementational boundaries. To achieve that, you would need an abstraction layer between the backend application framework and whatever API should be accessible in the frontend. And by having such an abstraction layer, the question pops into my mind if it wouldn’t be better to omit the framework alltogether…
Having Neos as a platform is a big thing to me. We are (and have been) working in multiple Neos projects now where they needed some kind of frontend editing. And one thing has been proven: the Neos UI ain’t the solution for just some frontend editing as you get all the other stuff on top.
I’ve had usecases where we only wanted to show a subtree in the navigate component, or no navigate component at all… Or maybe just really advanced editors in the inspector, and no publish menu… or whatever. We will have numerous custom implementations of frontend editing soon if we don’t support that, and if we’re really serious about being a content editing platform we’ve no other choice than making it a platform on the JS side.
Saying all this I also realize that the chances are bigger that people will use their framework of choice for building stuff on top, and might see a choice for a big js framework as a bourdon. Vanilla js stuff is accepted far more easily because it doesn’t add dependencies. We all still know what we’ve seen as problems for VIE, hallo, aloha and so on: the dependencies on certain frameworks.
So I agree that we need arguments for picking a framework. And we don’t need arguments for ‘not picking them’ Just good solid arguments what is actually fixed by the framework, and if that’s just about the data binding for example, or just about the object model… than we might even find libraries around that offer just that.
+100 to what you generally wrote; I think we very much agree on the goals here.
I don’t know of libraries which solve the above two problems really solidly (except ember). For me it would be the above two parts, plus the View hierarchy (and virtual DOM with Ember 2.0) which I’d really want.
I’d suggest we build a custom Ember version just containing these parts. That should be fairly easy I think.
This can lead to very Java-like abstractions that make things harder to understand. I’d suggest to keep the code containing our logic / service interaction etc. as clean and modular as possible (in many cases we don’t need much Ember there, maybe some RSVP promises, but that’s not a big deal). But in many cases you’d want to write the implementation directly using the framework without any further abstractions (e.g. when creating components for the user interface). I never saw that these particular “we might want to switch the framework” abstractions are worth the effort if it’s not about having decoupled, modular code - but without too many abstraction layers.
Overall I don’t want to sound negative here or harm motivation in any way. When working on Neos for several years it’s certainly time to rethink very early decisions like the particular framework. My problem here is that we risk to do some very serious bike-shedding when we need to decide over everything again (and need to iterate, get it stable, find best practices, maintain it, etc.).
Regarding the discussion about Neos as a SPA. When we decide to go the COPE way (and forget about inline editing that is only one specific editing mode for web output) we need to tackle many more problems than “just” having a tree, inspector and RTEs (greatly simplified). There should be all kinds of components / modules that need to be composable and configurable by an integrator. And there will be the need for routing/state and composition of a larger application from many components. I’d not like to have a traditional server-side driven webapp for many of these parts because that means to split our UI into two separate layers and increases complexity again. My vision for Neos as a backend would be to provide solid services around content and producing output for various channels. The Neos UI should not be fixed (that means it needs to be pretty dynamic) and should be only one possible implementation over the backend services.
If we find out Ember or other “large” frameworks have serious drawbacks to achieve these goals, I’m totally fine to get back to the very essence because that will mean Neos as a product will have an advantage by choosing another kind of technology.
thanks for voting We’ll do the meeting on Friday, 14:00. I am not yet sure whether we will meet using hangout or some telephone conferencing number – I’ll post details here shortly before.
Here’s a short summary of what we’ve discussed on friday:
We agree on moving to ES2015
Refactoring and Rewrite can be handled as seperate tasks, so that the more urgent refactoring issues get a chance to be included in sooner releases
Since there are too many open questions, we didn’t reach a settlement regarding the framework decision
To get started on addressing the more urgent things and to get more technical insight on the future foundation for the backend in the medium term, we decided the following next steps:
Refactor the existing code base to get rid of require in favor of ES2015 modules and therefore integrating babel and browserify in the build process. This can be done immediately and should be targeted for the 2.1 release. (@inkdpixels, @akoenig and myself would volunteer for that)
Create a Proof of concept in Ember 2.0 for a complete rewrite/architecural redesign.
Create a Proof of concept in a Flux/React-like architecture.
Points 2 and 3 can be done in parallel and should help to reach clarity on the framework question.
Please feel free to amend anything that I might have missed.
Bookmark post, but hope it’s useful for other. For the refactoring we need a really solid “data storage” on the client side to handle data / sync / transform, and I just found this lib Orbit.js (https://github.com/orbitjs/orbit.js/tree/master) who support JSON-API and many other, also memory, local storage, … and complex configuration, like save in memory, and next async to the server rest API (with sync of the record ID when creating new record, …). Also support undo operation, …
Just good solid arguments what is actually fixed by the framework, and if that’s just about the data binding for example, or just about the object model… than we might even find libraries around that offer just that.
