Project proposal: Rewriting the UI using React

[aertmann (Aske Ertmann)]

Like @sebastian I’m sure a similar prototype could have been achieved with Ember 2 in the same timeframe. Also took quite some time to read up/inspect on this, think and respond.

Unfortunately this whole discussion is being complicated with the whole React vs. Ember question. I don’t want to argue for or against here, just that I’m confident both will work out well.

What really matters is the question of doing a complete rewrite. Seeing a prototype like this doesn’t really tell much in my opinion. We had a similar prototype a couple of years ago, and it took a lot of effort to reach a state of stability and detail we have today. Version 1.2 was “usable”, 2.0 was “okay” and we’re still not near “great” in terms of being bug free and having a good user experience. Doing a completely rewrite will put us back to that state. It seems like I’m the only one who has the view, but I feel confident with it because I’ve built this before and know just about all the code. Changing foundation is not magically going to solve the problem of stability, it takes iteration, which takes effort/time. I’m certainly one of those who has spend the most time doing these fixes/improvements, the commit history proves so, and most of it has been on code others have started. So simply don’t see why this would be any different on this prototype compared to the one build by @sebastian and @hlubek. As much as I’d like to have a new code base instead of improving and old one, I can’t justify it as a business decision. To me it’s really about developer happiness, not about providing the most benefit to the end users.

Neos is currently at a point where people expect stability and thus the new UI cannot replace the existing before it’s on par or there are enough pros/cons to justify it. The idea of having two separate UIs simultaneously can certainly help, but at the same time it halts development and makes the process even longer.

Going step by step, improving one area at a time and releasing to the public is a much safer bet and can be done with far fewer stability concerns. It also allows us to keep all the already stabilized/polished parts and replace parts when there’s an actual end user benefit rather than just refactoring for the sake of refactoring. I’d much rather spend my time providing additional value than reimplementing things we already have for my own benefit.

Just to give you guys an idea of how much functionality there is to reimplement, I started a document describing the functionality of the UI while working on the upgrade. Even documenting it is a daunting task and only about 20% done in it’s current state.

I also see this as a very high risk project and one I fear will have one of two outcomes: 1) it takes way longer than expected or 2) the current state is replaced by a less stable/not on par state due to lack of resources/motivation to reach it. I’m not a big fan og high risk projects, especially when the biggest benefits are not towards the end users.

I do however support the whole concept of decoupling the UI into a separate package, although this can be done regardless.

One thing I’ve asked for before is an estimate on how much effort you think is needed to reach the current UI. The following question is the likelihood of there being people willing to put in that effort and when will they have the time for it, since that will affect the timeframe. This might be much to ask, but perfectly normal when making decisions like this. Otherwise it’s all on a whim, which is certainly not something I can base such a decision upon.

I hate to be the conservative negative grinch, but I simply cannot support a rewrite. Hope you can understand my reasoning why.

And keep in mind, we all want a really nicely build application, but how we get there is the question. And creating a prototype of our vision is certainly no waste of time, even if we don’t adopt it.

Btw. it’s mentioned we’d gain new contributors, we might also alienate some.

Counter arguments very welcome.

[sebastian (Sebastian Kurfuerst)]

Hey Aske,

We had a similar prototype a couple of years ago, and it took a lot of effort to reach a state of stability and detail we have today. Version 1.2 was “usable”, 2.0 was “okay” and we’re still not near “great” in terms of being bug free and having a good user experience. Doing a completely rewrite will put us back to that state.

that’s also the fear I am having generally: The first 80% are “easy”, fixing 20% of the bugs will take its time… So for me it is also a “higher-risk”-project. What Tyl writes looks exactly how Christopher and myself felt when we started the Ember prototype. “All Nice and Shiny”… and then reality kicked in and it became more complex and long than anticipated.

However, read on why right now I think it might still be worth a shot

Changing foundation is not magically going to solve the problem of stability, it takes iteration, which takes effort/time.

To me there is one major technical reason why I think we can make a more stable application in a shorter timeframe: Immutability and single-directional data flow. Let me explain:

• React is much more strict in what it expects its component to be; that is, non-idiomatic code is “less tolerated” than inside Ember. This means concepts like a “global event hub” or so simply do not exist. And you basically never listen to events. In contrast to Ember (where it’s easy to e.g. integrate a jQuery plugin into an Ember Component without noticing whether or not you break Ember’s assumptions), you feel that this is a dangerous route in React.
• Having an immutable data structure at the core of the state has many very nice properties: Makes debugging a lot easier, makes state changes extremely explicit, and makes it generally reason about “what is changed by which action”.
• Both combined, I feel, creates a very predictable programming and usage environment – and this in turn could help to reduce the bugs. I feel many bugs which have appeared were due to components needing to somehow “talk” to each other in some way, and lots of asynchrony in the user interface. With React, it feels having basically no asynchrony at all.
• The above points have no equivalent in Ember…

There are some areas where I think a “start-from-scratch-decoupled-package” is better than modifying the existing core into that: Especially when we talk about e.g. switching to CKEditor, I think starting with a completely separate package actually reduces risk instead of increases it… So I’m feeling a lot better integrating CKEditor in a new package than trying to replace it in the existing Neos code – simply because this way, CKEditor can mature independently of reducing stability for the main Neos package.

As of now, I don’t see another way of reducing risk than trying it out. And seeing how stable the experience will be.

I’d love if some more people chime into the discussion again, especially @christopher, @robert, @wbehncke, @inkdpixels …

All the best,
Sebastian

[radmiraal (Rens Admiraal)]

I was just thinking that we’ve a risk of staying in the discussion ‘rewrite’ or ‘refactor’ while we all seem to agree to at this point: we need to separate the UI.

We could lower the overall risk by spending a bit of time upfront by moving the current UI out of Neos into a separate UI package. That package would then be the current ember version which we could ship as long as the React version is not considered stable enough but still usable if one would prefer…

[dfeyer (Dominique Feyer)]

That package would then be the current ember version which we could ship as long as the React version is not considered stable enough but still usable if one would prefer…

I’m really in favor of that solution too. If we have a more or less ready Ember 2.0 upgrade and we have the human resource to make it decoupled from Neos, let’s do that. Build a rock solid API, and continue to work on the React stuff in parallel. I spend the last few days, to build a small customer project on top of the Redux/React/Immutable stack just to have a better understanding on what we can gain to move to this stack.

Immutability and single-directional data flow

And for me the gain is cleary what @sebastian write.

Being able to dump/restore the state of the UI state can be really useful to tackle bugs. You just need a copy of the website and restore the state to be exactly in sync with a client problem. The time traveler debug of redux also help a lots understanding what happen in the application.

I play a bit with Socket.io on to of the stack, and buidling a flow of Actions in sync between all editors is easy, so per example we can sync what element are currently in editing mode, without having a central server, by using some P2P Socket.io implementation. That’s really powerful to implement interaction between editors and stuff like give user warning to avoid conflict. All that can be done with a light Middleware on the store, really decoupled for all the rest of the code. Not sure how we can do that in really clean way with Ember.

About the framework vs libraries it’s a no go discussion. Using a Framework on composing your own … it’s just a matter of taste.

This project is really high risk, but lot’s of good thing can pop up. If we can follow to path, let’s make the Ember 2.0 decoupled, build a strong and solid API, continue to build the React stuff on top of that, and switch to the React stuff when it’s ready it’s fine for me.

[mgoldbeck (Markus Goldbeck)]

Hi all,

as it looks like that there is already a winning site and I don’t want to “argue” about frameworks But I share a lot of Aske’s points especially regarding in terms of

• being bug free,
• having a good user experience
• and also expected stability.

Nevertheless I had a quick look at react and the prototype on GitHub and also liked what I saw.

So I will just share with you some of my thoughts.

• It will take some time until the new ui will be finished. What will happen during this time with the old ui and its bugs? In general I like the idea of Rens

We could lower the overall risk by spending a bit of time upfront by moving the current UI out of Neos into a separate UI package.

• How can we play safe that we do not lose functionality which we already had?

Just to give you guys an idea of how much functionality there is to reimplement, I started a document describing the functionality of the UI while working on the upgrade. Even documenting it is a daunting task and only about 20% done in it’s current state.

• Won’t we have the same problems if we add jQuery plugins to React Components? Don’t see how this will change.

In contrast to Ember (where it’s easy to e.g. integrate a jQuery plugin into an Ember Component without noticing whether or not you break Ember’s assumptions), you feel that this is a dangerous route in React.

[mgoldbeck (Markus Goldbeck)]

This project is really high risk, but lot’s of good thing can pop up. If we can follow to path, let’s make the Ember 2.0 decoupled, build a strong and solid API, continue to build the React stuff on top of that, and switch to the React stuff when it’s ready it’s fine for me.

Same here. I think this way is the best agreement.

[inkdpixels (Tyll Weiß)]

Hey guys, sorry for not being that active here in the last days, had a lot of work to do.

Like mentioned in RFC: JS Refactoring Process of Backend, this is not exclusive to using React and can be achieved with Ember 2 as well.

Of course, the whole list on the top isn’t intended to be a Ember vs React comparison. It’s a list of things already implemented in the Guevara package.

In general, I think we don’t need that type of discussion here, as said by many others previously, everything can be done using Ember/React/Angular, and we all fully agree with that. What separates Ember from the stack we’ve chosen, is not only the technical points which @sebastian summed up perfectly, it’s also the community size/momentum which will bring us further in terms of developers who would like to contribute to Neos in the future.

Please note: I don’t say that by using Ember we won’t attract new developers at all, we just have a higher chance if we are using a “popular”/fancy stack, and again, from a technical perspective both solutions are as good as the other, so we should weigh in benefits like this one. Especially if we look at the topic of bugs/stability you’ve mentioned in the post afterwards @aertmann.

More developers contributing => faster fixes for bugs => stability increases => happy user.

And yes, we might loose some contributors, even though I don’t see anyone here who is strictly against using React, without having worked with it, and jump off the Neos train if we start using another framework than Ember. Also, the amount of contributors we could attract with such a stack will be way higher.

As much as I’d like to have a new code base instead of improving and old one, I can’t justify it as a business decision. To me it’s really about developer happiness, not about providing the most benefit to the end users.

The risks are clear and I think everyone involved has the same thoughts and fears, but in some cases starting from scratch is the better option.

Especially, if a refactoring to a way higher standard will take as much, maybe even more time than a rewrite. Starting from scratch also gives us more room to re-evaluate previously written code, new standards and re-think best practices. While of course we can refactor, we can only tackle one part at a time, and in between we also still need to tackle bugs of the code you are currently refactoring.

If we all work into the same direction, we will have a stable UI package which is on pair with the current UI in autumn/before winter. And yet, while we might introduce new bugs(we need to be realistic here), we will definitely eliminate other, and since our main goal should be to make the new UI as stable as possible, we will introduce tests from the beginning, ranging from end to end tests to behavior tests for the UI itself.

To be honest, while all of this stuff can be done in a refactoring, the risk of a refactoring taking longer than a rewrite is on pair, if not even greater.

Again, we all strive into the same direction, to make Neos a better product, and we all are aware that a rewrite might lead to other problems, but those will be solved if we announce a public beta phase of the new UI, and since the package is pretty much just a plug & play-package, we surely wont lose users or make them mad if done properly.

I was just thinking that we’ve a risk of staying in the discussion ‘rewrite’ or ‘refactor’ while we all seem to agree to at this point: we need to separate the UI.

We could lower the overall risk by spending a bit of time upfront by moving the current UI out of Neos into a separate UI package. That package would then be the current ember version which we could ship as long as the React version is not considered stable enough but still usable if one would prefer…

Could you elaborate which risk this would eliminate?

I think it will lead to a whole lot of work we could save ourselves/spend in maintaining the current code base, since it doesn’t have any benefits for the user or for developers(until we have a JS API) at all.

I don’t want to sound hasty, I might as well have overlooked something here

It will take some time until the new ui will be finished. What will happen during this time with the old ui and its bugs? In general I like the idea of Rens

The current UI still need to be maintained of course, and as long as the separate UI is not stable, the ember codebase will be the official supported route.

How can we play safe that we do not lose functionality which we already had?

This will be covered by having the “public beta” phase and the pluggable nature of the new UI package

Won’t we have the same problems if we add jQuery plugins to React Components? Don’t see how this will change.

In general I would try to avoid using jQuery and its plugin ecosystem at all cost. If however you need to do this, you can simply turn off the re-rendering process of React for this component with shouldComponentUpdate, and since React only handles the rendering, nothing will leak into your codebase.

Also, if anyone still has questions, reach out to me on slack or here, I will answer as soon as possible.

[wbehncke (Wilhelm Behncke)]

Hi you all,

I’d also like to apologize for not being active in this thread, but I’ve got knocked out by a flu and couldn’t do much at all the last couple of days

But I’m happy to see how much response has been there over the time and I will try to respond to as much of it as possible.

Unfortunately this whole discussion is being complicated with the whole React vs. Ember question. I don’t want to argue for or against here, just that I’m confident both will work out well.

I agree and would like to add, that actually the rivalry between Ember and React is pretty much pointless. We made the prototype merely to demonstrate the different architectural approach of Redux, componentization and the integration of an i-frame with an unoppinionated guest frame application regarding the methods of inline editing. React is just a natural choice when it comes to Redux and componentization. So I think, it’s actually the architecture, not the framework, worth discussing about.

What really matters is the question of doing a complete rewrite. Seeing a prototype like this doesn’t really tell much in my opinion. We had a similar prototype a couple of years ago, and it took a lot of effort to reach a state of stability and detail we have today.

I understand your reservation here @aertmann, but I think that one should take switched technologies and architecture into account, when it comes to evaluating this. I feel it is a bit of a generalization to predict, that all prototypes go down the same road. Of course our prototype is not safe to not share that fate, so it is absolutely correct to be aware of the downsides and pitfalls along the way and not let the prototype-inherent quick-win situations blind our judgement. But still, the fact that this is prototypical software should not diminish its purpose: demonstrating an alternative technological approach while being a foundation for further development in that direction. I am convinced that the experience of the Neos team with the ember prototype will in fact help figuring out a way to avoid shortages in quality and resources and thus get this done properly.

As much as I’d like to have a new code base instead of improving and old one, I can’t justify it as a business decision. To me it’s really about developer happiness, not about providing the most benefit to the end users.

This is somewhat a philosophical question to me Actually, developer happiness is imho not at all detached from end user benefits. These two are rather closely connected, since as developers, end user benefits is all we want to create at the end of the day. And from the business pov, I think it’s even more important, since developer happiness actually fixes what you stated out earlier:

I’m certainly one of those who has spend the most time doing these fixes/improvements, the commit history proves so, and most of it has been on code others have started.

I’m not arguing, that we should concentrate on having “cool code stuff” going on, because it goes without saying, that this is a false premise for running any kind of serious software project. I just want to say, that starting with a good development experience does not indicate, that things get worse for the end user. If done right, happy developers actually sound like a good foundation for the end user experience to improve as well.

Going step by step, improving one area at a time and releasing to the public is a much safer bet and can be done with far fewer stability concerns.

I have to disagree here. Touching the current code base would imho always endanger stability since there is no test coverage. Tests could still be done now, but this would already require some refactoring steps to be done sufficiently, and that alone could already affect stability in a way, that may be hard to reason about. I see a lot of concerns there, and imho both approaches at least bear the potential of being equally time-consuming.

One thing I’ve asked for before is an estimate on how much effort you think is needed to reach the current UI. The following question is the likelihood of there being people willing to put in that effort and when will they have the time for it, since that will affect the timeframe. This might be much to ask, but perfectly normal when making decisions like this. Otherwise it’s all on a whim, which is certainly not something I can base such a decision upon.

The amount of time, that we could spent on this cannot be estimated safely and partly depends on the outcome of this discussion (just like @dimaip’s situation, we can only know how much work time might be given to us, when there is an official statement regarding this). But @inkdpixels and I have spoken with @robert and @christopher about the desirable time-frame and came to the conclusion, that a stable release in december this year can and should be done. If this is agreed upon, we would concentrate our efforts to enable a public beta as early as possible before december.

We could lower the overall risk by spending a bit of time upfront by moving the current UI out of Neos into a separate UI package. That package would then be the current ember version which we could ship as long as the React version is not considered stable enough but still usable if one would prefer…

I absolutely agree with this exact plan

Now I have to admit, that @inkdpixels has already answered most of the other responses and I really have nothing to add there.

I really hope that we figure out a roadmap, everbody is comfortable with and I feel that we’re actually pretty close to it.

BTW, @aertmann: What is your opinion on @radmiraal’s suggestion?

As @inkdpixels, I’m available for further questions here (and in slack) as well. I’m looking forward to the further development of this discussion and wish you all a nice sunday evening!

[sebastian (Sebastian Kurfuerst)]

Hey everybody,

I’d like to reflect a few things on the discussions of the last days and share them with you; both from this thread and the #guild-js slack channel.

First off, I am extremely glad about the discussions taking place - both in terms of content and feelings; I feel that even if we disagree from time to time, we listen to each other and argue for the best solution.

However, there is a topic I’m increasingly seeing as danger to the successful rewrite, which I’d like to share with you: Basically I know that starting “fresh” creates a big motivation for many of us, also the opportunity to re-think many decisions from the past and re-decide them.
To me, this has both pros and cons: on one side, it will allow us to fix some long-term technical glitches (like by using an iframe, or using a mature RTE). On the other hand, there is the danger of worsening the experience by also re-creating what has worked well in the past in ways which do not take prior knowledge into account; basically I fear that if we do this, we might end up with a new solution which fixes glitches from the current implementation, but will have lots of glitches which do not exist in the current version.

So, I’d propose the following: Let’s identify the parts which we want to improve, and the parts which generally work well and we want to copy.

Things I would like to improve include:

• Make the whole behavior a lot more predictable and debuggable by using immutable state and avoiding side effects in the whole UI.
• Use an iframe for editing.
• Use CKeditor as editor to improve editing experience. (I am insisting on this one because we have done a very intense evaluation in June 2015 and the reasoning from back then still applies today)
• Generalize the concept of inspector editors, so they can be used e.g. in wizards or in the to-be-created “structured editing mode”, and allow editors to return asynchronously.
• Transfer mode properties as JSON instead of data attributes.

Well-working parts for me include:

• keep the general user experience the way we have it so far (+ stability and speed).
• when choosing a framework/library, stick to it and use it to its full extent. To me, that means that e.g. editors should be written in React; directly being called from inside other React components. I do not think that people need to be able to write editors in plain JS / ember / …; or if they do, they must take care of wrapping it in a React component themselves.
• The general concept of configuring editors and editing in NodeTypes.yaml
• Having the NodeTypes schema available client-side.
• The general approach of wiring together editors with their model. I feel e.g. Using reactive streams would be way too low level here I think.
• Allowing to create new editors in other packages, without re-compiling the User Interface. This is a kind of “simple extension of the UI”.
• Shipping Neos UI as composer package. For bigger adjustments, e.g. when you want to completely mix-and-match the Neos UI, of course, I think it is fine if you need to do a custom Webpack build.

As said, my current fears revolve around opening the box of rebuilding too-wide, and trying to tackle everything at once. I’d personally suggest that we focus on the following steps now:

• have a meeting focussed on structuring the client-side state of React.
• create a client-side Node model integrated into Redux.
• Re-create the Inspector.
• Re-integrate CKEditor (also with custom editing controls).

Thanks for reading!

All the best,
Sebastian

[christianm (Christian Müller)]

After much discussion a bit late the offical vote:

• EmberJs
• React

Show results

I think the question is clear, do we want to pursue the React prototype or stay with EmberJs in the long run.

TEAM MEMBERS make sure you add a short reply with your “official” vote below.
The poll is open until Sunday, 31st January 2016.

[christianm (Christian Müller)]

I vote for this because I believe the chances outweight the dangers.

[dfeyer (Dominique Feyer)]

check Project proposal: Rewriting the UI using React for my point of view on the transition.

[sebastian (Sebastian Kurfuerst)]

I vote because. I basically said it all in my previous posts

[dimaip (Dmitri Pisarev)]

I vote to continue moving in the technical direction laid by this prototype (not just React vs. Ember thing) to fulfil the shared Neos product vision.

[kdambekalns (Karsten Dambekalns)]

Voted in favour of pursuing the React idea, assuming it will be done in parallel until we can “switch”. And please, whoever works on it: do it well and fast.

Also, keep in mind we must maintain the current UI as long as we have supported versions using it out there. Alternative way out: Make the new UI a fully drop-in replacement from the user’s perspective and ship it with “old” releases. Good luck…

[robert (Robert Lemke)]

from me. But please let’s plan and proceed with this project cautiously, as Sebastian pointed out. Don’t just rewrite the UI with a new technology.

[radmiraal (Rens Admiraal)]

while keeping in mind that:

• IMHO the curent Ember UI should be separated from Neos first (to allow parallel development)
• IMHO the rewrite requires a clean server side API too (maybe also in a separate package to allow flavors)

[mgoldbeck (Markus Goldbeck)]

assuming:

• Ember Ui should be separated
• done parallel
• in general as Sebastian last post pointed out

[bwaidelich (Bastian Waidelich)]

in general. But IMO we should thoroughly evaluate a decoupled architecture (like initiated by @christopher) before we decide about too many details

[sebobo (Sebastian Helzle)]

assuming like most others:

• Ember UI separation (anyway a good idea)
• Parallel development
• Having a good business value we can communicate to the outside