JS refactoring of the backend

This thread is there for showing the inital brainstorming phase; we’ve so far distilled the notes into an RFC which you’ll find at RFC: JS Refactoring Process of Backend
(comment by @sebastian)

After having built a user interface in a pragmatic way that has been proven to work, it’s time to refactoring this in a clean way to ensure a solid foundation for the future.

I created two documents to collect ideas so we make a good decision on the direction. Feel free to add stuff to the documents, and later we will decide based on those documents. It’s important to keep in mind that this will influence the next years of development.

One thing we discussed and decided during the last code sprint was to stick with Ember and upgrade towards the 2.0 version. We talked about if migrating to another framework like Angular or React would make sense, but due to the size of the current implementation it would require a lot of work to completely re-do and we also think that the direction Ember is heading is fitting for the project.

I would propose that we fork the repository in Github and everyone interested in the refactoring would be given access, this is due to the requirement for quicker iteration and since it will require many commits for migrating the backend before it can be merged back into the core.

We should set up a proper development setup with linting and tests, and possibly using travis to run the tests automatically as well.

JS: https://docs.google.com/document/d/1zLqy8HMalcyGK2bwz0vgMAHT9GDTZIqBWpL27Xc7Fog/edit#heading=h.an4us97vjq55

CSS: https://docs.google.com/document/d/1-JYm5PF-ql2mw9bWg1TuCMuZ15LcrtqO7R6GoGc2TE8/edit#heading=h.c0sqvrhsouw2

I began on writing my proposals down into the GoogleDoc, but found that this would lead to moving the discussion into GoogleDocs. So I am posting them here on discuss instead:

JavaScript (General / API):

Set up a proper Neos API which developers can access, I think of one window global which can be accessed via a getter method f.e. window.getNeosService() or window.getNeosAPI()

  • The global should hold a constant Object which can’t be modified(Non-Emumerable?)
  • The global should also include a PubSub/EventEmiter whom which you can fire events to the internals to Neos (Deleting/Creating Elements, hiding the inspector etc. etc.)
  • These EventNames should also be in the global object
  • I am thinking of an EVENT_NAMES object which the internal Neos code, as well as 3rd-Party Components from other developers can use.
  • Benefits:
  • Less repetition
  • 3rd-Party Components are bound to the same variable/string without copy/pasting things like event names from the documentation/wiki.
  • Another part of the global, should be to make all abstracted components from the Neos-Core accessible to 3rd-Party code.

JavaScript (General / Globals):
At this brief moment we have at least 10 globals from the Neos Code, two internals (window.T3 and window.Typo3Neos) as well as a bunch of third party globals.

These should be removed ASAP since this is a huge Problem if somebody is using the same window.global and is modifying it (Anti-Pattern, but it’s possible and we should take care of that).

JavaScript (DOM):
It seems like the neos code is accessing data from the DOM. The DOM is not an interactive Data-Exchange API and thus writing/reading data from the DOM should be avoided. It should be an overall goal to reduce the dependency to the DOM.

  • When possible, data should be in Collections/Models/youNameIt™
  • I found several Modules/JS Methods which sadly depend on fixed DOM classes/IDs
    • This is a general anti-pattern: Things like selectors should, if even needed, be pushed in from the outside, we should think about Properties/States for Components

Comment from @christopher:
The whole concept of semantic annotations driving the editing interface relies on this. There are certainly drawbacks of using the DOM, but we at least never had problems with consistency using this approach (which could easily happen when we provide the data through AJAX or embedded in one part of the document). (Christopher)

@christopher I think we should revisit each of the usages. For me, the DOM shouldn’t be used as a data-handling API. I could be totally wrong on this one, but it just seems like there is room for improvement here. I am mostly sure we can find a solution which is more performant, and reduces the friction which can be caused if anyone is modifying the DOM attributes of an element and accidently breaks the Neos backend trough this. (The same goes for window globals stated above.)

JavaScript (General / ES6 and ES7):
We should hook Babel into our Build-Process to allow ES6 and ES7 code. This way our Code is more future-friendly. We can also switch to native APIs/Methods as soon as these are in all major browsers.

So in the long run: Better maintainable code, less 3rd Party and more native APIs/Methods.

JavaScript (Frameworks):
I would also like to revive the discussion of using a big framework. We should consider removing libraries/frameworks from the Core whenever we can. Instead we should use smaller modules (From the NPM Eco-System) which perfectly fit our requirements.

If we miss something, we should write our own modules and publish them to NPM as well.

Benefits:

  • Reduced dependency on one big monolith
  • Faster changesets/releases
  • Decoupled code
  • in the long run, this will pay out (Less big releases / changesets / rewrites, Modules can be switched since the API surface for each Module should be as low as possible).
  • This should allow better attraction for more joining Frontend-Dev’s, since we aren’t bound to a single Framework, and instead are enforcing small an encapsulated modules.

Disadvantages:

  • The learning curve is steep
  • The refactoring process is more time consuming

Comment from @christopher:
We need to invent a lot of proven solutions (e.g. how to provide fast DOM updates / re-rendering / data binding) ourselves. When we would drop any existing framework we end up building our own. It needs a lot of continuously active people to support that amount of additional maintenance. (Christopher)

@christopher This is indeed not meant to replace existing logics, but more an enforcing of using smaller Modules, whom do one thing good, and not try to do 1000 things semi-great. There are a lot of existing DOM-Rendering Libraries, which solved the issues we all had. So we should use them of course! Ember also has a standalone version of their rendering logic (Will paste in the link ASAP).

JavaScript (Task-Runner):
Maybe we should switch from Grunt to Gulp. We could integrate a more modular/leaner build task, Gulp is known for being fast and I could provide a Gulp-Task-Application which creates standarized tasks on the fly from a passed in Config object. We should also consider switching to Node-Sass (Faster than RubySass and more or less on the same level regarding the language specification). This also comes in pair with Autoprefixer. It’s a tool that automatically writes the vendor prefixes we need, based on a standarized Browser-Array f.e.:

browserList: ['last 2 versions', 'ie9']
1 Like

I moved 4 posts to a new topic: CSS refactoring of the backend

Hey Tyll,

I agree to lots of what you wrote there :smile:

That’s basically the only thing where I disagree… Especially I think that Ember 2.0 really helps us in managing the application state in a good way, is well-maintained, has a sane object model and most importantly bindings and computed properties.

All the best,
Sebastian

Also agree to most of what you say, but this quote is what I disagree on :wink: Grunt pulls the job just fine imho, and it’s no requirement to switch the runner to use for example the suggested node-sass / libsass (witch seem to be the same?). Imho the biggest speed gain we can have is dropping compass / the current sass compilation stuff, and replace it by libsass. Not so sure if grunt vs gulp than still is a big difference, so I’d need a better argument for dropping things.

Hey @sebastian - thats great to hear! :smile:

I’he heard the decision was made on the last code sprint, but I haven’t found a document / topic on discuss which elaborates the statement on keeping Ember / A big framework. Thats pretty much why I would like to revive this discussion and make it a bit more open.

I’m not totally reserved upon frameworks, but as far as I can see it could be an option to re-develop it with smaller packages to increase the performance of the App(Neos) and of course shave some bytes of the build.

Best wishes,
Tyll.

Hey @radmiraal Thanks for the feedback! :smile:

Generally the aim of any build task/unit test should be to be as fast as possible, so it’s not a burden to run them after each change.

So Grunt of course pulls the job fine, but is pretty slow compared to task runners(like Gulp) which use streams instead of the R&W (Node-)FileSystem API. We could also take a look at other task runners like Brocoli or the like, but Gulp is the most accepted one out there which uses streams. Thats pretty much why I was considering Gulp, instead of Grunt - The speed benefit.

On Node-Sass/LibSass/Compass:
You are right, Compass slows down the build time. But not at the same level as RubySass does.
If we are just taking speed into account, the order of slow -> fast would be:
Compass with RubySass -> RubySass standalone -> LibSass/node-sass

Thats basically because of ruby itself, LibSass is written in C and compiles about 200-400% faster than RubySass(without Compass!). So why not take the opportunity and just switch to an even faster compiler(LibSass)? :smiley:

As stated, the language implementation between RubySass and LibSass is minimal/non existent, so switching shouldn’t be a problem after all.

Just pasting in the standard folder structure we use at sitegeist, maybe as an proposal/starting point for a future JS Folder-Structure:

| JavaScript
|   | Application
|       |-- ...
|   | Components -> Holds all abstracted Components like an accordion, menu, tab sytstem, modals etc.
|       | MyComponent -> Holds all files regarding a certain Component.
|           | View.js
|           | Config.js -> Holds the configuration for the Component.
|           | Template.html
|           | ...
|       | ...
|   | Utilities -> Utilities like a storage wrapper, logger, DomUtils, Events, i18n as well as the Global API etc.
|       | ...
| Config.js -> General Constants
| Index.js -> Kickoff for the application.

This is of course not finished, just a quickly done starting point and proposal.
The Application folder would hold stuff which isn’t possible to abstract and/or is too tightly coupled etc.

I’d like to mention that the above structure is hold in Resources/Private, unlike the current setup in Neos.

Private makes sense since the resources are compiled/processed and during that process made avaible in Resources/Public.

2 Likes

Indeed we should, there is already a public API and some events triggered. Additionally internally we use a event dispatcher, which can easily be replaced with something that’s used for internal and external events.
One thing to keep in mind is that when exposing these things, you also create an API you have to maintain and version. I’m not sure we want to do that for everything until we have a more polished interface.

We started doing that (https://forge.typo3.org/issues/48358), but never finished completely and didn’t bother anyone enough so far. Anyway would be good to keep that in mind with the refactoring.

I don’t see any way to get rid of relying on the DOM completely. Content can be rendered from various points and not just the page, so at minimum the collection ids and content ids would be a requirement. Additionally it avoids race conditions and problems with consistency. However the output is one argument against it and fetching the data could be done asynchronously, however that will also have drawbacks e.g. when using inline editing. So it’s a trade off, but I doubts there’s a perfect solution to this.

Fully agree.

I’m also not convinced it’s better to not use one. Doing that means we have to come up with our own solutions by combining modules, but I fear we’ll end up with something worse tbh. That will be harder to maintain, develop into conflicts because the libraries don’t depend on each other. Also it leaves more responsibility for the developer and requires us to document our homegrown “framework”. Doing our own framework will not necessarily attract more people, but making it easy to get started by componetizing, separating concerns and having a well defined/described strategy will regardless of framework.

So @dimaip, @wbehncke as well as me had a short discussion about how the order and possibilities of the refactoring should be.

Possible Roadmap:

  1. Move to Gulp / Setup a flexible Task-Runner
  2. CSS Refactoring
  3. Choosing the best methodologies for writing good CSS (Maybe BEM, OOCSS & Atomic Design)
  4. Rewrite
  5. JS Refactoring
  6. Write (Unit-) Tests/Specs for the current JS and it’s Modules
  7. Choose a more fitting & standardized folder-/application-structure
  8. Work out coding & quality guidelines for JS
  9. Eliminate Code smell (Globals, VIE, CreateJS)
  10. Create the global NeosAPI
  11. Complete Rewrite OR
  12. Step by step refactoring

Possible NeosAPI-Features

@ aertmann Exactly, this proposal isn’t meant to replace the current API, but more polish it up and justify it’s existence(Since at the moment there is only a reloadPage method).

  1. Make the implemented Ember Framework accessible (Instead of window.Ember) so Developers can use the constant Ember version as the Neos Backend does.
  2. Low-Level APIs
  3. For Events
  4. An NodeServiceAPI for interaction with nodes(Add, move, remove, hide etc.)
  5. StateAPI for interaction and the retrieve of the current status of the Neos Backend(isPublishing, isLoading, isEditing, getCurrentSelectedNode etc.)

Hey everybody,

after our meeting, I took the task to create a more specific proposal (which was the outcome of the meeting):

The RFC is at RFC: JS Refactoring Process of Backend

All the best,
Sebastian