Project proposal: Rewriting the UI using React

The Neos Project Accepted Projects
41

:+1: in general. But IMO we should thoroughly evaluate a decoupled architecture (like initiated by @christopher) before we decide about too many details

1 Like
42

:+1: assuming like most others:

  • Ember UI separation (anyway a good idea)
  • Parallel development
  • Having a good business value we can communicate to the outside
1 Like
43

I can’t really vote based on a technical point of view but based on what I heard and read it seems to be quite some work to update Ember as well and is quite realistic to switch to React. And as some people are really interested in doing it with React and we could use some kind of hype around that I think it makes sense to switch to React. (Assuming that we will go the parallel path).

44

I abstain - don’t have enough technical insight :smile:

1 Like
45

:+1: from my side. My main point would be to make sure the server side API will be redone as well and we make sure the UI will be really separate by a clean (REST-)API. Will say, it has to be possible to replace the UI package by something else, if needed.

2 Likes
46

Everybody here,

as just mentioned in the retrospective, I wondered if you guys followed the post “selecting a client-side framework for Drupal” by Dries. It mentions licensing conflicts with GPL in case of Angular and a rather stupid patent clause by Facebook regarding React.

See also this issue in the Automattic issue tracker and this discussion in the React issues tracker.

Please do read that article if you haven’t and let’s discuss here if it makes a difference for us.

1 Like
47

My 2cts …

  1. I don’t like the patent clause, it’s not really in the Open Source spirit
  2. I don’t think that it makes any difference for us, also because the Apache license has a similar clause which wouldn’t affect any of us or using the software in practice. Let’s be grit our teeth and ignore it. Not nice, but hey.

Still I think it’s worth mentioning that Dries / Drupal seem to rather tend choosing Ember than React :wink:

2 Likes
48

Thanks for bringing this topic up @robert.

As you also stated, I think unless we also want to submit a patent for the Neos UI, this whole discussion does not affect us in any way.

Still I agree with you that this is not in the spirit of OSS in general, but I acknowledge the fact that the react core dev’s are also in the process with facebook-legal to remove this clause from the project, I think there is a fair chance that the clause will be removed. :+1:

2 Likes
49

I closed the voting above, I see eleven positive votes in the comments below. I guess that the result we expected and so we go ahead.

3 Likes
50

Hey guys,

Thanks for the responses. Since there was a clear preference towards starting from scratch, I focused on my time off.

Anyway I wanted to summarize a few consequences of this decision and share some thoughts, insights and advice.

  • During my involvement with this project I have really come to respect the extent of the solution we’re trying to build. This is where most of my reservation from doing a rewrite comes from. It has taken thousands of hours to achieve the current state, and there’s still so much to improve. Doing a rewrite means most of that work has to be redone with pretty limited resources.
  • There seem to be a very optimistic view of how much effort it will take to achieve the desired goal and estimates being made that have almost no backing. That’s very concerning and a sign of a disastrous project from my point of view. The fact is that we’ve more or less built this product already and we have a fairly good idea how long it takes to do. I’d be happy to help answering how long different aspects have taken. The first thing one should do is to get a clear idea of what one is trying to built, which no one seems to have done so far. Without it you will only have a vague idea how far you are and even if you’re building the right thing. I already started doing it during the Ember upgrade, and I suggest to finish it to have a clear scope of the project. See Neos UI overview - Google Docs
  • As I mentioned earlier I see two outcomes of this project, project extends beyond deadline or a less featurerich or potentially less stable solution replaces the existing one. Therefore I suggest to be realistic and accept to aim for the less featurerich outcome. Focus on the important aspects first and stabilize those. This is something we’ve been pretty poor at in the past, partly due to development being motivation based. We should have limited scope and focussed much more on delivering solid solutions on fewer things. My first suggestion would be to skip components like the edit/preview panel, inspector validation, data views, keyboard hotkeys, tab key navigation, code editor and similar for the first version.
  • During the development of the current UI, development stagnated. Around the time of releasing 1.0 we reached a point where it was mainly stability, handling edge cases and small improvements that was important. This wasn’t very motivating work to do and only few relied on the product. This was a pretty frustrating time lasting a whole year and took a lot of perseverance to get through. This maturing phase is likely to happen for the rewrite as well. Guess is how to prevent it from happening?
  • The success of this project is largely tied how much time people end up investing in it. It’s great to see Tyl and Williams drive, but the desired deadline is certainly more than they can achieve themselves. Looking at the past activity I must say I have my doubts if there will be the necessary support. I hope that it will motivate some who wouldn’t otherwise be, since there’s a popular stack and the chance of architecting things.
  • Most of the existing code base can’t be used for anything. Of course a lot of experience has been gained, which can be useful in a rewrite, but that’s very little benefit achieved from it. I hope that the existing code is taken into account seriously when rewriting to ensure all functionality is duplicated/tackled. Of course most of it will be rewritten in a new way causing different problems, but the functionality of the components remain very similar.
  • A lot of effort was saved previously by using jQuery plugins. A lot of complex functionality now potentially needs to be implemented ourselves unless React compatible components exists. This includes trees, drag & drop, sliders, date picker, cropping & upload. There’s a big chance of falling into the trap of not invented here syndrome. Hopefully there’s suitable components available.
  • Regardless of introducing tests, a new solution will have to go through it’s maturing phase. Tests help reduce regressions and approaching testing more methodically. However many of the stability issues Neos have had in the past, came mostly from not covering all aspects/use cases and the complexity of interacting components. Not from not being tested properly/not having tests or regressions.
  • The maturing phase will take time and effort, and requires a user base willing to accept it. I personally doubt many existing users will be interested in that (at least my customers won’t), since we’ve had a poor history in that regard. I do hope there will be new users or some that value the improvements the new solution has to offer will be available.
  • Attention to detail has been a sparse resource in the past. And something that’s required to deliver a user experience similar to the existing one. Therefore it’s important to ensure it’s a focus area covered by the people involved. Improving code quality doesn’t include this in any way.
  • It puts existing effort (WIPs) into limbo and it’s likely that most development on the current UI will cease. Meaning introducing new features are less likely, since implementing them in the rewrite is likely a big risk and difficult due to it’s progress phase. Not sure how we should tackle this.
  • It is certainly one the biggest and riskiest decision being made while I’ve been involved with the project and the first one I strongly disagree to. And I firmly believe it would be better achieved by refactoring our existing code base with much less effort and risk and a similar outcome.

While that has been said, I hope everyone involved takes ensuring the success of this project as a challenge and makes it happen. There’s is certainly a lot of good outcomes that can come from it, but it requires a lot of effort working together to achieve.

Technical alternations/suggestions:

  • When building the node state handling, take having the same node in different context into account. This is a current weakpoint and makes split screen difficult. Related to this is a bug where the same node exists twice on the same page through a reference, causing one of the two not to work.
  • There are some of the UX concepts that should be improved when rewriting instead of rebuild in the same way.
  • Streamline headers and collapsing of panels (documents, content, inspector) and collapsing of them – https://jira.neos.io/secure/attachment/12718/Neos-UI.png
  • Replace tabs in the inspector with dropdown – https://jira.neos.io/browse/NEOS-1790
  • Combine “Menu” and “Navigate” to tabs – https://jira.neos.io/browse/NEOS-1174
  • Implement a multi-functional tree – https://jira.neos.io/browse/NEOS-1789
  • Building the tree, take different sorting/filtering into account not just based on node hierarchy as well as lots of nodes on the same level
  • Property validation is currently based in the model, however only visible in the inspector editors. It should however also be applicable to inline editable properties.
  • URL handling should be decoupled from the website view. Instead of using the node URL, the identifier should be used to avoid making a website focused application. E.g. replace http://neos.dev/en/download@user-admin;language=en_US.html with neos ⚛ Universal Compiler and Bytecode JIT.
  • Give some thought how other modules should be bootstrapped and work in relation to the content module. There’s a need to reuse components across the two.
  • When implementing the select editor, consider making it the basis for the reference(s) and file/images editor. They all represent a list and will benefit from search, add/remove, reordering, search/select.
  • There are some areas with some pretty tricky CSS solutions which took a long time to achieve. These will be completely hidden and requires a lot of manual testing to get right. Therefore I recommend looking at the existing CSS to understand why it’s build like it is, regardless of the undesirable class structure. Some of these areas are the tree (overflow/scrolling/ellipsis), inspector fields (overflow/scrolling/ellipsis, modified/error borders, error messages), select2/chosen (overflow), edit/preview modes panel.

Lastly a have some questions / concerns about the new prototype:

  • The CSS approach might not be the best approach. Since we currently have both a JS and PHP rendering, re-using styles for components on PHP generated pages wouldn’t be possible. This comes back to the point I made regarding the how these two play together.

@sebastian: Completely agree with everything in your post, well put.

Cheers

2 Likes
51

Forgot one point. We’ve wanted to introduce theming for quite some time to better support poor screen, projector and bright light conditions. This is much easier to done when starting out than do later on. So I’d suggest to spent a little bit of theme thinking how to do that before it’s too late. Initially a light theme would be the goal, see https://dribbble.com/shots/2009015-Neos-white-theme

1 Like
52

Hey Aske, thanks a lot for your detailed feedback, it’s really valuable for the rewrite project at this point.
We will have a discussion tonight to review which points of UI we can change during the rewrite and which we can rewrite as is, and we’ll base it on the list that you brought up.
I think the focus for now would getting minimal working UI that could be installed alongside the current UI.
Will post the updates here soonish.

53

Hey everybody,

I just summed up the TODOs from the meeting @dimaip has been referring to:

@aertmann: I definitely understand your concerns; and as written previously, to me the main concern is that we cannot archieve the quality / bug-free-ness we need right from the start. I am willing to bet on this, as React/Redux state handling is a lot better understandable, simple and predictable (to what I see so far); hopefully reducing race-conditions etc.

Furthermore, I’d just like to point a very huge THANK YOU to you Aske for coming up with such a detailed posting even if you generally dislike the decision and feel it is very risky. Looking forward to hopefully see you soon again :slight_smile:

All the best,
Sebastian

54

hey Aske,

even though I can’t add much to discussion from a technical standpoint, I’d like to at least let you know that I really appreciate the way you handle this difficult decision. I’m glad that we can discuss this difficult topic in a constructive way, and even though I do agree on some on your points (especially the underestimation part), I also decided to fully stand behind this decision. Let’s give it a shot - sensing the spirit here at the sprint I think that there’s a good chance that this will really work out.

Cheers!
r

2 Likes
55

Hi Aske,

thanks for your detailed feedback. First off, let me say, that I really feel bad for seemingly having decreased your trust in the project, since this certainly was not my intention. Therefore I am just as thankful for you taking the time to bring in your perspective, experience and advice. And though I disagree with some of your points, I cannot emphasize enough the importance of your view on this entire topic and how much I value your advice. Now, I hope that I can say a few words, that hopefully help to give you a better feeling on what we are doing:

Estimation is without any doubt a problem, that we can’t solve for this project. The only thing we can do, is to time-box the effort and to frame hard criteria, after which to stop the project entirely. This is what we did at the sprint. The project is constrained now regarding the time it should take and if it bends its limits, the consequence will be its failure. I am strongly convinced that it will succeed wihtin these boundaries, since they are pretty reasonable, but of course I can’t foresee the future.

Feature-wise there is consensus not to re-implement everything, especially those features that are already planned to be improved. Dominique took the time to introduce Tyll and me to the UX improvements as suggested by Christoph Ono (https://jira.neos.io/browse/NEOS-1173), that are now basis of all reconsiderations on the way. We had a fruitful discussion at the sprint that resulted in four major improvements that should definitely be part of the rewrite (documented here: https://github.com/PackageFactory/PackageFactory.Guevara/wiki/Project-Planning). I agree to concentrate on the core experience of the UI first, though. We should aim to break the usual workflow as little as possible, except for explicitly improved features. Stability should be our primary goal.

Regardless of the resulting featureset, the project will - as you’ve said above - certainly go through a maturing phase. And while I do not believe that this is preventable at all, I had some thoughts on how this maturing phase will actually look like. Since we won’t just switch off the current UI but will allow a parallel installation at least during an alpha and beta phase, we can profit from the grown project and user base that Neos has by now. The UI has a chance to hit tons of edge cases before its final release.

Regarding the participation, the resonance during the code sprint was very positive and I hope the involvement of all those new contributors will sustain. Still I remain realistic about the progression of such projects. Participation tends to get more during the sprints and less between them, but I am certainly willing to keep working on this and I know, so many more are too. I also really hope that someday you will join the project as well, since I am certainly not the only one to consider not only your opinion and experience valuable, but who would also love to work together with you on this.

Finally, I totally agree with all of your technical suggestions. So let me just respond with a few headwords:

  • The handling of nodes is already designed with multiple, cross-dimension content views in mind
  • As said above, we will take the UX improvements you’ve mentioned into account
  • The tree is currently more a projection of the available nodes in the redux store, which can be freely sorted and filtered. Dominique even suggested to make the tree replaceable with different views, that would allow for a less web-centric editing experience - which wouldn’t be that hard given what we have implemented so far. I think the hardest part here will be to reason about server-side solutions e.g. for searching nodes.
  • Validation will take place at a different level. Currently our implementation makes no difference to the source of a change. Changes to a node (e.g. property change, creation of a node, deletion of a node) are handled as separate objects, that also could be validated separately. We haven’t spoken about this so far, but imho that would be the place to do validation.
  • Nice idea for the URL handling :slight_smile: I took the freedom to create an issue for that: https://github.com/PackageFactory/PackageFactory.Guevara/issues/88

CSS:

Using css modules is a controversial decision that without any doubts results to the drawbacks you’ve mentioned. We would have the option to switch to a modular BEM-driven CSS approach or to somehow find a way to let our server side rendering understand css modules. Luckily the component-driven approach gives us some time to think this through, though we shouldn’t postpone this for too long. As you’ve said, we also should take theming into account while re-visiting the css approach.

Now, to close this, I just want to express my hopes, that despite your reasonable doubt, there is a way to do this together.

Best wishes,
Wilhelm

3 Likes
56

That’s a good aim, just be aware that a minimal working UI probably doesn’t cut it for most users. There’s a lot that needs to come together to be satisfactory to work with. That’s what we saw in the beginning. Mainly there were three frustrating problems for the early adopters, a big one being actual bugs, second being things not being “wired” together or incomplete implementations and third missing functionality like a link editor, asset linking, asset editor, image cropping, placeholders, constraints etc.

It would have been possible to apply the exact same tactic to releasing a beta of a major refactoring as a rewrite.

It definitely is. Less magic than Ember. Forces a certain approach and structure, however it’s still a complex concept.

Making rough estimates and having a good overview of tasks will certainly help you guide the process and help you track progress more easily. Also I see this decision as a point of no return. There’s only one path and that is making this decision work. After all that’s what everyone who voted for it should take responsibility for making sure of. Otherwise we’ll just end up wasting a lot of resources at the expense of the users. However it would still be wise to evaluate along the way to ensure progress is on track.

There is a certain risk to rushing this project. After all being rushed is one of the main reasons the current code base is such a mess. Doing a rewrite should be done properly if the goal is to improve the quality of code. There are three reasons why there aren’t, running, tests in the current implementation: 1. poor test setup (cumbersome, slow), 2. lack of priority due to time pressure, 3. lack of know how/motivation.

Great, I’d be happy to help out with any questions or missing things in that regard. Most of the proposals came from me mentioning existing issues/ideas and discussion solutions to them. So I have more insight about the proposals and can help answer questions. Also some of it is not complete and needs more conceptualization, like the inspector behavior.

Since it wasn’t a team effort coming up with the prototype, a lot of insight and experience is being missed out on. I do hope that more of those who worked on the existing UI chime in, but also likely it won’t happen.

That’s great and it needs to be that way for this to succeed. The current refactoring certainly failed at that so far, although it never reached the point to actually refactor/reinvent things. Keeping interest between sprints has always been difficult and have proved to consist primarily of the most passionate contributors. Having sponsored work hours certainly help though, so aim for including as many as possible that are in such a situation.

Well thanks, even though I’ve being pushing for refactoring the code base for years I’m personally not motivated of doing a complete rewrite. I’ll leave that to you guys and those that join. However as stated, I’d be happy to help out with UX/UI questions. Another thing I could help out with is ensuring there isn’t a degrading experiences in some parts and pointing out things that might have been overlooked, when the time comes to make the new UI default.

That makes sense, as long as it’s the tree that populates what’s in the store. It certainly has limitations and I’d suggest to take a server-side approach for sorting/filtering. Unfortunately those things are missing at the moment. Regarding being replaceable, isn’t that what rolling a custom webpack config does? I think most usecases can be solved with the multi-purpose tree suggested in https://jira.neos.io/browse/NEOS-1789 / https://jira.neos.io/browse/NEOS-775

I wasn’t clear, that’s how it is today as well. However we have no concept of displaying errors for inline editable properties. So when doing validation, which should probably happen at a later stage, make sure to have a concept for displaying them that isn’t bound to the inspector only.

The key point here is modularization/componentization. As long as it’s well structured and side-effect free it can be reused. One idea would be to compile a stylesheet from those component stylesheets. That would be possible with a pre- or post-processor. But it makes more sense to think about how server side modules and the client side application will interact as I have suggested previously.

I have no doubt it’s achievable. My doubt is about the required resources needed to do it within desired timeframe. Where I believe those efforts would be better spent otherwise. A lot of resources will go into building something that already exists for the primary benefit of the developers instead of developing new/improving functionality for users. The joker is naturally the motivation part that amounts to resources available, which is difficult to argue/reason about since it’s hard to predict.

So far I haven’t commented or responded to the actual approach of the prototype. The React/Redux stack is definitely the best choice from what’s available at the moment (considering community as well) when starting from scratch IMO. Also many of the underlying decisions and how to use React/Redux are good. Redux hints towards event sourcing, which we have been wanting to implement for a long time. Additionally Redux is based on concepts from functional programming bringing predictability and determinism. I really like the server-side part of it and having those two work closely together. I hope it takes a step further into the content repository as well. So in that regard the rewrite is definitely going in a good direction.

While it’s awesome to be able to benefit from stuff like hot-swapping and time-travel because of redux, predictability/race conditions and such haven’t really been a big problem so far. The application is fairly simple in that regard of interaction. It’s more on the content repository side that event sourcing would be useful, since we’ve had a lot of issues there. However now it would be possible to reproduce some of those errors, since there’s now a way to record those actions.

3 Likes
57

Some more thoughts. It might make sense to optimize UI interaction by improving animations and transcations, since that’s much easier with React than CSS/jQuery.

Google Materiala Design guide contains a lot of good tips on that subject and a little can make a big difference.

First suggestion would be to adopt more authentic motion, where the current code base is all linear transitions.

Second suggestion would be to adopt more meaningful transitions, e.g. when opening modals, menus and such.

Additionally there are some ideas and concepts regarding a responsive UI that could be useful.

4 Likes
58

Meeting Protocol 23.03.2016

Participants: Philipp, Sebastian, Tyll, Dmitri, Wilhelm, Markus

Current State

Generally: The developer documentation at https://github.com/PackageFactory/PackageFactory.Guevara/wiki has improved a lot, so check it out!

  • Wilhelm: worked on React/Redux for Guest Frame; implemented inline toolbar for CKeditor; next steps: make plow immutable.js compatible
  • Tyll: (currently focussed on Neos website, will do reviews / help for questions in the next month)
  • Philipp: fixed initial work on acceptance testing (working again!); We can start writing acceptance tests; Currently working on select component based on https://github.com/JedWatson/react-select; Documentation for how to getting started contributing to React rewrite -> in Wiki
  • Dmitri: tried to fix publishing (refactored publishing service to sagas; refactored “changes” Saga). Next: Add tests for “add node” dialog (need help from Philipp on that); [also working on Rect rendering for Neos frontend just for fun]
  • Sebastian: Pretty occupied with work related things (sandstorm), reduced noise in the webpack build (awesome sauce), rebased the inspector feature branch, will continue working on the inspector(-editors), especially the image editor. Has some questions regarding the initial state generation via TS.
  • Markus: working on navigation component. Pretty busy with work related things also.

update: Old and New Acceptance Testing

worked on React/Redux for Guest Frame

  • originally, React was not planned for the Guest frame, because we had just little rendering to do.
  • With the inline toolbar, the state handling inside the guest frame has gotten a lot more complex. And it made sense to have Redux in there as well. This way, we can also centralize the synchronization between host frame and guest frame state.
  • Event handling in the guest frame in React binds to the document; so using preventDefault() in the client-side application is a problem. What if the website uses React itself? React core team is working on these issues.

Immutable data structures / Performance / Immutable.js with Plow

  • Plow generally should work with Immutable.js. Basically we’ll create a new version of all operations which works with Immutable.js. (Wilhelm is doing this)
  • Then we should use shallow comparison which is then possible.
  • We should decrease number of actions sent; focussing on actions which make sense semantically. We currently do not want to debounce actions as this would “hide” the problem but not solve it. Lateron, we might want to change this.
  • We should add performance measurement points to code (using react-addons-perf); lateron run performance measurements in CI system maybe helpful?

JSON Data communication from BE to FE (especially images etc)

  • Proposal/Decision: get rid of initial state in TypoScript; rather use YAML for that. I (sebastian) will work that over.

Next Meeting

  • Next virtual standup: 30. March (in Slack)
  • Next scheduled meeting on 6. April 13:30
1 Like
59

Virtual Meeting Protocol 30.03.2016

Participants: Philipp, Sebastian, Tyll, Wilhelm

Current State

  • Wilhelm
  • DONE: made Plow JS immutable compatible, integrated Immutable JS into the UI, created custom PropTypes for nodes and nodeTypes, refactored PageTree to a more performant solution .See https://github.com/PackageFactory/PackageFactory.Guevara/pull/102 for the current state.
  • NEXT STEPS: Fix the remaining issues with Immutable JS, continue working on the CK editor toolbar
  • Tyll:
    • DONE: nothing really, except some PM’ing and issue tracking
    • NEXT STEPS: stay sane, keep on working on the new website
  • Philipp:
    • DONE: nothing so far - unfortunately i had no time this week to work on Neos
    • NEXT STEPS: Select Component (Wrapper) - Documentation “Getting Started” - Will start/continue these tasks at the end of the week
    • OTHERS: i plan a meeting with my boss to take an agreement so i can work on neos during the week more often
  • Sebastian:
    • DONE: (Image Editor): improved JSON Data communication from BE to FE (especially images etc); Loading of Image metadata from the server; rebased based on immutable.js; optimized some performance; Displaying of Preview Image.
    • NEXT STEPS: Image Cropping; Image Upload
    • see https://github.com/PackageFactory/PackageFactory.Guevara/pull/78 for further details

worked on React/Redux for Guest Frame

  • Update: in the longer run, we will get rid of React in the guest frame again (at least that’s the plan as discussed by Wilhelm and Sebastian)

Next Meeting

  • Next scheduled meeting on 6. April 13:30