I trust your technical decision here and decided to not look deeper into it myself, I already gave my for the soft facts so from my side good to go. But me being me I want to add a few points that should not be forgotten while doing this:
This will be a breaking chance no matter what as we allow custom editors and validators and the code needed for this will probably change
The beforementioned extension points have to be keept and new ones need to be deeply embedded into the rewrite IMHO
While at it we should think about reuse of components for (custom) backend modules, but I guess @dimaip will have an eye on this
Just wanted to correct my fanboyish tone here a bit. I didn’t post this comparison here as an Ember vs. React thing, I’m not that stupid and perfectly aware that we could get similar performance from starting a fresh with Ember.
It was just a shout of relief from long pains with current UI performance and was more in favor of rewrite with iframe than anything else. Yet internet communications are not my strong side so please forgive me for that
I think the best sort of validation and documentation here is writing more tests, and that could serve as such “schema” and documentation. And we should be truly test-driven here, i.e. first write tests then code. It’s very easy with functional style of code.
Yes, there’s always this discovery effort with npm ecosystem. But it’s more about ideas than about actual code libraries, usually a module encapsulates some new way of thinking, and it feels more like a refactoring we have to undertake anyways, just outsourced to 3rd party modules.
thanks for taking the time to have that detailed look on our prototype. I really appreciate your comprehensive feedback and will try to give some insight on what our thoughts on this are:
We agree that validation is a huge concern when dealing with this kind of data structure (which not only regards the store but also actions for example). @inkdpixels already had a look on GitHub - molnarg/js-schema: Simple and intuitive schema validator for validating actions on reducer level - it could be helpful for the store as well.
But I feel that the schema (with whatever tool it will be described) should not be our single source of documentation regarding the store. To completely understand the overall data flow, besides the static structure of the store, one must understand what smart components expect the store to deliver and what portion of the store reducer functions are going to mutate. Component-wise, I think, the @connect decorator does a great job for developers to immediately understand what section of the store is displayed by a component. Reducers on the other hand will profit from unit testing.
I also agree with @dimaip that testing will be worth a great deal to support the contractual character of the store, but runtime validation is definitely also important, since it helps to sort out corrupt server state.
I have been pretty overwhelmed by this as well when I started working with these tools. But after a while, I was under the impression that this speed of progress is actually healthy. Whenever a particularly better solution comes up for a problem, there’s no better time than right now to fix it. And the smaller your dependencies are (in amount and size), the more easy it will be to manage updates for them. Luckily, most of the npm-libraries innovate as fast as they ship, so change is not introduced in huge amounts. This might be just a feeling of mine, but I think that adopting progress doesn’t look like one, but is an overall time-saver.
I like the idea, but I would be afraid of this to die pretty quickly (just my experience with '.md’s other than README). But I do see the point of documenting third party libraries in such a way, so I would give it a shot.
I agree with prioritizing the recreation of the current UI, with keeping the benefits of some new paradigms in mind (like the I-Frame integration) - so that we will once in a while reflect on exisiting features, when and if they could be replaced with a better solution along the way (I don’t have anything particular in mind, but I feel like we should not be dogmatic about things, when there’s something better and more work-effective ahead).
The one I can not easily agree with is CK. I absolutely respect the level of sophistication behind this software, and certainly think that it is superior to aloha, but I do have great doubts if it is the right solution for Neos. It is just huge! Why should Neos integrate a Rich text editor, that looks like it aims to be a CMS itself? In my daily experience working with Neos, I always try to restrict the feature-set of Aloha (Which I can’t even do to the degree I would like to). With CK, I would have to reduce the feature set of a battleship down to that of a canoe - for me there would literally never be a scenario in which I could use a full blown CK in a project.
Of course I can only speak for myself, but I think this is a question of how much control you would like give up for a larger rich text editing experience. I always felt like restriction is the way to make the system more usable for editors.
I could live with CK as the editor shipped with Neos though (assuming it works as an npm depency) - but I absolutely disagree on the integrational part.
Our prototype is already agnostic about inline editors and I really would like to keep it that way. When Neos is actually able to provide a way to extend the inline editing feature with custom inline editors, this would be a huge win for the end users.
…
So, now I’ve written a lot myself I’m very happy about all the great and constructive feedback so far and really have the feeling, that we’re on the same page on most of the issues. And I’m really looking forward to figure out solutions on the remaining ones with you guys
Reading those posts I can only say “I’m happy to read this”. As a complete rewrite was (from the beginning) my most favorite solution although I heavily worry about the time it might take / breaking changes that also @christianm mentions. The main advantage is see from the full rewrite is that it gives us the chance to completely separate the UI from Neos / CR. In line with that I’d like to state that it would be a wise decision to
add a (REST / whatever) API to CR
add a similar interface to Neos for the domain specific stuff
make Neos just a ‘content management framework’ that is able to load a backend UI
make the now discussed rewrite the ‘official supported’ version but keep in mind that companies will write their own backend too
Regarding Ember vs React: as long as the stuff is clearly separated and has enough support from the team I’d be fine with either of them. Downside of my current view on that is the point I became tired of the framework discussion as a whole (also outside of the Neos project btw). Been there, done that… So let’s see what the future brings, maybe React might even make me more excited about the UI stuff in the end
For now: great to see this all moving, and great to see open and complete responses… keep doing that
Hey Wilhelm (and also others, thanks for your comments),
Full ack, and I have to admit I am still kind of lost at this point – so I am starting to feel that we really need a good documentation/test coverage/Architecture Diagram/… at this point. So that’s something we very much need to care about
I agree this might become out of date if we are not disciplined enough to do this; but I just think that we need to be very disciplined at this point, because it allows new people (which might be familiar with React) to become involved in the project at a later point in time. So for me that is crucial for improved “openness” of the project…
I think this part is more crucial in an Open Source project than in a company; because in a company usually there is a more well-defined and organized onboarding process; and a company usually grows slower than a well-developing Open Source project.
Of course, we are totally on the same side here. I just wanted to re-create the overal User Experience (in a faster & more stable way); of course the technical parts should and can be re-thought (like the iframe you mentioned; and there are ofc. lots of other parts where this might make sense!).
I’ve been learning React, Redux and related concepts over the weekend, and this all makes sense to me. However, there are two areas which I’d like to highlight as I think they are crucial to think about:
Documentation of the general architecture. I think we need to document the basic architecture, explaining how we understand the react ecosystem to be plugged together. As far as I understand right now, we have some “services/managers” (like ChangeManager) which trigger an action on the store, which then mutates the data structure. Unclear to me is yet why we f.e. need a separate factory for creating actions (like in actions.Transient.Changes.add), if we have a service layer in place which shall be called instead. To answer these kinds of question, I think it is crucial to document this for us; or at least post links to online resources which explain how we approach things and why.
Structuring State, especially Nodes: I am currently feeling that we need a unified Node model on the client side, which is represented in some normalized way; and then used e.g. by the tree. But how this model looks like and behaves is IMHO extremely important to get right So far I am thinking along the lines of creating a structure looking like what is the result of normalizr; along with using reselect to make e.g. provide a possibility to access children on a node.
Actually would like to correct you here. The difficult part (upgrading to Ember 2.2) is actually almost finished (90% ~ 1 day work left) and the reason it has been difficult is due to the fact that we haven’t used Ember in the way it’s supposed to be used and that we haven’t followed the upgrades gradually. Additionally upgrades are difficult for new comers and not very motivating. I can safely say that since I’ve personally done the primary work load on the 1.x and 2.x upgrades. So it’s simple due to the fact the little work has been put into the upgrade in those 5 months that it’s not finished yet. If more people had helped it would have been done long ago.
Next step is moving to ember-cli, which will certainly be a more interesting task and get us back on track with how Ember is supposed to be used and thus remove most of our frustration working with Ember.
Global CSS from Neos is actually not much of a problem. The problem is global CSS leaking into Neos, which using inline styles solves to some degree but doesn’t eliminate since all inherit styles still apply. The only way to getting rid of this problem is by using an iframe for the website. Also the real problem is updating CSS/JS on page reload and media queries, both solved by an iframe approach.
I don’t see great benefit from this to be honest. One of the reasons of doing this is to make use of native only API and performance. However neither are a big problem for Neos. Good performance can be achieved with just web and if you want a actual app, Cordova can do the trick. Keep in mind that even Facebook has to rewrite 30% of their react native code to build an app cross platform AFAIR. The great benefit of building web apps is that they don’t need to be downloaded and installed. You’d have to have very good arguments having to force that on your users and for Neos I simply don’t see them. Why haven’t anyone created any app icons for saving to homescreen? Same goes for packaging Neos in an Electron app e.g.
This is something I find a little odd being mentioned so often. Neos is a like a program and people would be okay waiting for it to load initially. Especially if we made the waiting experience better. The difference between Ember and React in this case is minimal and matters when you have a website that people don’t load often and don’t have clear intentions/needs of using. Thus this argument is completely void in my opinion.
Like @sebastian I’m sure a similar prototype could have been achieved with Ember 2 in the same timeframe. Also took quite some time to read up/inspect on this, think and respond.
Unfortunately this whole discussion is being complicated with the whole React vs. Ember question. I don’t want to argue for or against here, just that I’m confident both will work out well.
What really matters is the question of doing a complete rewrite. Seeing a prototype like this doesn’t really tell much in my opinion. We had a similar prototype a couple of years ago, and it took a lot of effort to reach a state of stability and detail we have today. Version 1.2 was “usable”, 2.0 was “okay” and we’re still not near “great” in terms of being bug free and having a good user experience. Doing a completely rewrite will put us back to that state. It seems like I’m the only one who has the view, but I feel confident with it because I’ve built this before and know just about all the code. Changing foundation is not magically going to solve the problem of stability, it takes iteration, which takes effort/time. I’m certainly one of those who has spend the most time doing these fixes/improvements, the commit history proves so, and most of it has been on code others have started. So simply don’t see why this would be any different on this prototype compared to the one build by @sebastian and @hlubek. As much as I’d like to have a new code base instead of improving and old one, I can’t justify it as a business decision. To me it’s really about developer happiness, not about providing the most benefit to the end users.
Neos is currently at a point where people expect stability and thus the new UI cannot replace the existing before it’s on par or there are enough pros/cons to justify it. The idea of having two separate UIs simultaneously can certainly help, but at the same time it halts development and makes the process even longer.
Going step by step, improving one area at a time and releasing to the public is a much safer bet and can be done with far fewer stability concerns. It also allows us to keep all the already stabilized/polished parts and replace parts when there’s an actual end user benefit rather than just refactoring for the sake of refactoring. I’d much rather spend my time providing additional value than reimplementing things we already have for my own benefit.
Just to give you guys an idea of how much functionality there is to reimplement, I started a document describing the functionality of the UI while working on the upgrade. Even documenting it is a daunting task and only about 20% done in it’s current state.
I also see this as a very high risk project and one I fear will have one of two outcomes: 1) it takes way longer than expected or 2) the current state is replaced by a less stable/not on par state due to lack of resources/motivation to reach it. I’m not a big fan og high risk projects, especially when the biggest benefits are not towards the end users.
I do however support the whole concept of decoupling the UI into a separate package, although this can be done regardless.
One thing I’ve asked for before is an estimate on how much effort you think is needed to reach the current UI. The following question is the likelihood of there being people willing to put in that effort and when will they have the time for it, since that will affect the timeframe. This might be much to ask, but perfectly normal when making decisions like this. Otherwise it’s all on a whim, which is certainly not something I can base such a decision upon.
I hate to be the conservative negative grinch, but I simply cannot support a rewrite. Hope you can understand my reasoning why.
And keep in mind, we all want a really nicely build application, but how we get there is the question. And creating a prototype of our vision is certainly no waste of time, even if we don’t adopt it.
Btw. it’s mentioned we’d gain new contributors, we might also alienate some.
that’s also the fear I am having generally: The first 80% are “easy”, fixing 20% of the bugs will take its time… So for me it is also a “higher-risk”-project. What Tyl writes looks exactly how Christopher and myself felt when we started the Ember prototype. “All Nice and Shiny”… and then reality kicked in and it became more complex and long than anticipated.
However, read on why right now I think it might still be worth a shot
To me there is one major technical reason why I think we can make a more stable application in a shorter timeframe: Immutability and single-directional data flow. Let me explain:
React is much more strict in what it expects its component to be; that is, non-idiomatic code is “less tolerated” than inside Ember. This means concepts like a “global event hub” or so simply do not exist. And you basically never listen to events. In contrast to Ember (where it’s easy to e.g. integrate a jQuery plugin into an Ember Component without noticing whether or not you break Ember’s assumptions), you feel that this is a dangerous route in React.
Having an immutable data structure at the core of the state has many very nice properties: Makes debugging a lot easier, makes state changes extremely explicit, and makes it generally reason about “what is changed by which action”.
Both combined, I feel, creates a very predictable programming and usage environment – and this in turn could help to reduce the bugs. I feel many bugs which have appeared were due to components needing to somehow “talk” to each other in some way, and lots of asynchrony in the user interface. With React, it feels having basically no asynchrony at all.
The above points have no equivalent in Ember…
There are some areas where I think a “start-from-scratch-decoupled-package” is better than modifying the existing core into that: Especially when we talk about e.g. switching to CKEditor, I think starting with a completely separate package actually reduces risk instead of increases it… So I’m feeling a lot better integrating CKEditor in a new package than trying to replace it in the existing Neos code – simply because this way, CKEditor can mature independently of reducing stability for the main Neos package.
As of now, I don’t see another way of reducing risk than trying it out. And seeing how stable the experience will be.
I was just thinking that we’ve a risk of staying in the discussion ‘rewrite’ or ‘refactor’ while we all seem to agree to at this point: we need to separate the UI.
We could lower the overall risk by spending a bit of time upfront by moving the current UI out of Neos into a separate UI package. That package would then be the current ember version which we could ship as long as the React version is not considered stable enough but still usable if one would prefer…
I’m really in favor of that solution too. If we have a more or less ready Ember 2.0 upgrade and we have the human resource to make it decoupled from Neos, let’s do that. Build a rock solid API, and continue to work on the React stuff in parallel. I spend the last few days, to build a small customer project on top of the Redux/React/Immutable stack just to have a better understanding on what we can gain to move to this stack.
And for me the gain is cleary what @sebastian write.
Being able to dump/restore the state of the UI state can be really useful to tackle bugs. You just need a copy of the website and restore the state to be exactly in sync with a client problem. The time traveler debug of redux also help a lots understanding what happen in the application.
I play a bit with Socket.io on to of the stack, and buidling a flow of Actions in sync between all editors is easy, so per example we can sync what element are currently in editing mode, without having a central server, by using some P2P Socket.io implementation. That’s really powerful to implement interaction between editors and stuff like give user warning to avoid conflict. All that can be done with a light Middleware on the store, really decoupled for all the rest of the code. Not sure how we can do that in really clean way with Ember.
About the framework vs libraries it’s a no go discussion. Using a Framework on composing your own … it’s just a matter of taste.
This project is really high risk, but lot’s of good thing can pop up. If we can follow to path, let’s make the Ember 2.0 decoupled, build a strong and solid API, continue to build the React stuff on top of that, and switch to the React stuff when it’s ready it’s fine for me.
as it looks like that there is already a winning site and I don’t want to “argue” about frameworks But I share a lot of Aske’s points especially regarding in terms of
being bug free,
having a good user experience
and also expected stability.
Nevertheless I had a quick look at react and the prototype on GitHub and also liked what I saw.
So I will just share with you some of my thoughts.
It will take some time until the new ui will be finished. What will happen during this time with the old ui and its bugs? In general I like the idea of Rens
How can we play safe that we do not lose functionality which we already had?
Won’t we have the same problems if we add jQuery plugins to React Components? Don’t see how this will change.
Hey guys, sorry for not being that active here in the last days, had a lot of work to do.
Of course, the whole list on the top isn’t intended to be a Ember vs React comparison. It’s a list of things already implemented in the Guevara package.
In general, I think we don’t need that type of discussion here, as said by many others previously, everything can be done using Ember/React/Angular, and we all fully agree with that. What separates Ember from the stack we’ve chosen, is not only the technical points which @sebastian summed up perfectly, it’s also the community size/momentum which will bring us further in terms of developers who would like to contribute to Neos in the future.
Please note: I don’t say that by using Ember we won’t attract new developers at all, we just have a higher chance if we are using a “popular”/fancy stack, and again, from a technical perspective both solutions are as good as the other, so we should weigh in benefits like this one. Especially if we look at the topic of bugs/stability you’ve mentioned in the post afterwards @aertmann.
More developers contributing => faster fixes for bugs => stability increases => happy user.
And yes, we might loose some contributors, even though I don’t see anyone here who is strictly against using React, without having worked with it, and jump off the Neos train if we start using another framework than Ember. Also, the amount of contributors we could attract with such a stack will be way higher.
The risks are clear and I think everyone involved has the same thoughts and fears, but in some cases starting from scratch is the better option.
Especially, if a refactoring to a way higher standard will take as much, maybe even more time than a rewrite. Starting from scratch also gives us more room to re-evaluate previously written code, new standards and re-think best practices. While of course we can refactor, we can only tackle one part at a time, and in between we also still need to tackle bugs of the code you are currently refactoring.
If we all work into the same direction, we will have a stable UI package which is on pair with the current UI in autumn/before winter. And yet, while we might introduce new bugs(we need to be realistic here), we will definitely eliminate other, and since our main goal should be to make the new UI as stable as possible, we will introduce tests from the beginning, ranging from end to end tests to behavior tests for the UI itself.
To be honest, while all of this stuff can be done in a refactoring, the risk of a refactoring taking longer than a rewrite is on pair, if not even greater.
Again, we all strive into the same direction, to make Neos a better product, and we all are aware that a rewrite might lead to other problems, but those will be solved if we announce a public beta phase of the new UI, and since the package is pretty much just a plug & play-package, we surely wont lose users or make them mad if done properly.
Could you elaborate which risk this would eliminate?
I think it will lead to a whole lot of work we could save ourselves/spend in maintaining the current code base, since it doesn’t have any benefits for the user or for developers(until we have a JS API) at all.
I don’t want to sound hasty, I might as well have overlooked something here
The current UI still need to be maintained of course, and as long as the separate UI is not stable, the ember codebase will be the official supported route.
This will be covered by having the “public beta” phase and the pluggable nature of the new UI package
In general I would try to avoid using jQuery and its plugin ecosystem at all cost. If however you need to do this, you can simply turn off the re-rendering process of React for this component with shouldComponentUpdate, and since React only handles the rendering, nothing will leak into your codebase.
Also, if anyone still has questions, reach out to me on slack or here, I will answer as soon as possible.
I’d also like to apologize for not being active in this thread, but I’ve got knocked out by a flu and couldn’t do much at all the last couple of days
But I’m happy to see how much response has been there over the time and I will try to respond to as much of it as possible.
I agree and would like to add, that actually the rivalry between Ember and React is pretty much pointless. We made the prototype merely to demonstrate the different architectural approach of Redux, componentization and the integration of an i-frame with an unoppinionated guest frame application regarding the methods of inline editing. React is just a natural choice when it comes to Redux and componentization. So I think, it’s actually the architecture, not the framework, worth discussing about.
I understand your reservation here @aertmann, but I think that one should take switched technologies and architecture into account, when it comes to evaluating this. I feel it is a bit of a generalization to predict, that all prototypes go down the same road. Of course our prototype is not safe to not share that fate, so it is absolutely correct to be aware of the downsides and pitfalls along the way and not let the prototype-inherent quick-win situations blind our judgement. But still, the fact that this is prototypical software should not diminish its purpose: demonstrating an alternative technological approach while being a foundation for further development in that direction. I am convinced that the experience of the Neos team with the ember prototype will in fact help figuring out a way to avoid shortages in quality and resources and thus get this done properly.
This is somewhat a philosophical question to me Actually, developer happiness is imho not at all detached from end user benefits. These two are rather closely connected, since as developers, end user benefits is all we want to create at the end of the day. And from the business pov, I think it’s even more important, since developer happiness actually fixes what you stated out earlier:
I’m not arguing, that we should concentrate on having “cool code stuff” going on, because it goes without saying, that this is a false premise for running any kind of serious software project. I just want to say, that starting with a good development experience does not indicate, that things get worse for the end user. If done right, happy developers actually sound like a good foundation for the end user experience to improve as well.
I have to disagree here. Touching the current code base would imho always endanger stability since there is no test coverage. Tests could still be done now, but this would already require some refactoring steps to be done sufficiently, and that alone could already affect stability in a way, that may be hard to reason about. I see a lot of concerns there, and imho both approaches at least bear the potential of being equally time-consuming.
The amount of time, that we could spent on this cannot be estimated safely and partly depends on the outcome of this discussion (just like @dimaip’s situation, we can only know how much work time might be given to us, when there is an official statement regarding this). But @inkdpixels and I have spoken with @robert and @christopher about the desirable time-frame and came to the conclusion, that a stable release in december this year can and should be done. If this is agreed upon, we would concentrate our efforts to enable a public beta as early as possible before december.
I absolutely agree with this exact plan
Now I have to admit, that @inkdpixels has already answered most of the other responses and I really have nothing to add there.
I really hope that we figure out a roadmap, everbody is comfortable with and I feel that we’re actually pretty close to it.
As @inkdpixels, I’m available for further questions here (and in slack) as well. I’m looking forward to the further development of this discussion and wish you all a nice sunday evening!
I’d like to reflect a few things on the discussions of the last days and share them with you; both from this thread and the #guild-js slack channel.
First off, I am extremely glad about the discussions taking place - both in terms of content and feelings; I feel that even if we disagree from time to time, we listen to each other and argue for the best solution.
However, there is a topic I’m increasingly seeing as danger to the successful rewrite, which I’d like to share with you: Basically I know that starting “fresh” creates a big motivation for many of us, also the opportunity to re-think many decisions from the past and re-decide them.
To me, this has both pros and cons: on one side, it will allow us to fix some long-term technical glitches (like by using an iframe, or using a mature RTE). On the other hand, there is the danger of worsening the experience by also re-creating what has worked well in the past in ways which do not take prior knowledge into account; basically I fear that if we do this, we might end up with a new solution which fixes glitches from the current implementation, but will have lots of glitches which do not exist in the current version.
So, I’d propose the following: Let’s identify the parts which we want to improve, and the parts which generally work well and we want to copy.
Things I would like to improve include:
Make the whole behavior a lot more predictable and debuggable by using immutable state and avoiding side effects in the whole UI.
Use an iframe for editing.
Use CKeditor as editor to improve editing experience. (I am insisting on this one because we have done a very intense evaluation in June 2015 and the reasoning from back then still applies today)
Generalize the concept of inspector editors, so they can be used e.g. in wizards or in the to-be-created “structured editing mode”, and allow editors to return asynchronously.
Transfer mode properties as JSON instead of data attributes.
Well-working parts for me include:
keep the general user experience the way we have it so far (+ stability and speed).
when choosing a framework/library, stick to it and use it to its full extent. To me, that means that e.g. editors should be written in React; directly being called from inside other React components. I do not think that people need to be able to write editors in plain JS / ember / …; or if they do, they must take care of wrapping it in a React component themselves.
The general concept of configuring editors and editing in NodeTypes.yaml
Having the NodeTypes schema available client-side.
The general approach of wiring together editors with their model. I feel e.g. Using reactive streams would be way too low level here I think.
Allowing to create new editors in other packages, without re-compiling the User Interface. This is a kind of “simple extension of the UI”.
Shipping Neos UI as composer package. For bigger adjustments, e.g. when you want to completely mix-and-match the Neos UI, of course, I think it is fine if you need to do a custom Webpack build.
As said, my current fears revolve around opening the box of rebuilding too-wide, and trying to tackle everything at once. I’d personally suggest that we focus on the following steps now:
have a meeting focussed on structuring the client-side state of React.
create a client-side Node model integrated into Redux.
Re-create the Inspector.
Re-integrate CKEditor (also with custom editing controls).
for the soft facts so from my side good to go. But me being me I want to add a few points that should not be forgotten while doing this:
But I share a lot of Aske’s points especially regarding in terms of