Project proposal: Package Manager Refactoring

PROJECT PICKED BY TEAM UNICORN

Finalize Package Manager Refactoring decide on last open questions, polish, get merged.

Change is this: https://github.com/neos/flow-development-collection/pull/104

There are a few decisions to take regarding breakiness, and addition/removal of features.
The change was made to improve performance, code quality and future possibilites in the package manager.

ToDos:

  • Decide on removal of MetaData model
  • Decide on API changes
  • Decide on change to inactive packages
  • Reviews

Rather small project as @sebastian already reviewed, tests work, but still this needs some wrap up work.

3 Likes

In general: yes. But I think right now we experience more and more instability in Neos and Flow because of refactorings. It adds up, and every change is a wound if you don’t perform the surgery carefully.

Devil’s advocate: which value brings your change to Neos users?

I am not sure if the instabilities can be pinned to the refactorings. Certainly they uncover some broken concepts from time to time and of course any change has a change of breaking something, but lets not go there…

Generally the PackageManager needed osme refactoring but of course we should have solid reasons for this big change, so yes this change has some benefits:

  • improves performances noticably (you can easily see the improvement with just checking request times in browser) and that is for all contexts
  • package sorting more robust (you could have a wrong sorting order before)
  • removes the problem of casing of third party package Flow package keys, because it relies on the composer name which is stable.
  • Gets rid of complicated Metadata model (hopefully) in favor of directly using composer.json
  • Allows a (Flow) package to supply multiple namespaces
2 Likes

allright, thank you Christian. I think the performance improvement is clearly a gain for the user, but the others are more on the developer’s side.

So, +1 unless it is breaking something badly or removing some other important functionality I wasn’t aware of yet :wink:

:wink: I took care not to break things too much but that’s why I am putting this up as a project, to get it really nicely done, done.
In my book devs are users too. We didn’t build this rocket science system only for non-devs IMHO.

Hey everybody,

we (Team Unicorn) will work on this project!

All the best,
Sebastian

Hey all,

I remade this change and reduced the breakiness to be almost non existent. The change is still marked breaking due to the major changes, deprecations and rescan.

1 Like

I’m sorry, the package manager broke on me.

Because the tooltip mentioned improved performance, I selected all packages and clicked on freeze.

I now get an exception on all pages. Login is not possible.
I tried the unfreeze command I found in the docs, but nothing changed.
Can you suggest something?


./flow typo3.flow:package:unfreeze
Unfroze package “TYPO3.Eel”.
Unfroze package “doctrine.collections”.
Unfroze package “doctrine.inflector”.
Unfroze package “doctrine.cache”.
Unfroze package “doctrine.lexer”.
Unfroze package “doctrine.annotations”.
Unfroze package “Doctrine.Common”.
Unfroze package “Doctrine.DBAL”.
Unfroze package “symfony.console”.
Unfroze package “Doctrine.ORM”.
Unfroze package “symfony.yaml”.
Unfroze package “symfony.domcrawler”.
Unfroze package “neos.composerplugin”.
Unfroze package “TYPO3.Flow”.
Unfroze package “TYPO3.Fluid”.
Unfroze package “imagine.imagine”.
Unfroze package “TYPO3.Imagine”.
Unfroze package “TYPO3.Media”.
Unfroze package “TYPO3.Party”.
Unfroze package “TYPO3.Form”.
Unfroze package “TYPO3.Twitter.Bootstrap”.
Unfroze package “TYPO3.Setup”.
Unfroze package “Behat.Transliterator”.
Unfroze package “gedmo.doctrineextensions”.
Unfroze package “TYPO3.TYPO3CR”.
Unfroze package “TYPO3.TypoScript”.
Unfroze package “TYPO3.Neos”.
Unfroze package “TYPO3.Neos.NodeTypes”.
Unfroze package “TYPO3.Neos.Seo”.
Unfroze package “Flowpack.Neos.FrontendLogin”.
Unfroze package “TYPO3.NeosDemoTypo3Org”.
Unfroze package “phpdocumentor.reflectiondocblock”.
Unfroze package “sebastian.diff”.
Unfroze package “sebastian.recursioncontext”.
Unfroze package “sebastian.exporter”.
Unfroze package “sebastian.comparator”.
Unfroze package “doctrine.instantiator”.
Unfroze package “phpspec.prophecy”.
Unfroze package “sebastian.environment”.
Unfroze package “sebastian.globalstate”.
Unfroze package “sebastian.version”.
Unfroze package “mikey179.vfsStream”.
Unfroze package “phpunit.phpfileiterator”.
Unfroze package “phpunit.phptokenstream”.
Unfroze package “phpunit.phptexttemplate”.
Unfroze package “phpunit.phpcodecoverage”.
Unfroze package “phpunit.phptimer”.
Unfroze package “phpunit.phpunitmockobjects”.
Unfroze package “phpunit.phpunit”.
Unfroze package “TYPO3.Kickstart”.
Unfroze package “TYPO3.Neos.Kickstarter”.
Unfroze package “Flowpack.Behat”.


Uncaught Exception in Flow
#1295973082: No class schema found for “TYPO3\Neos\Domain\Model\Domain”.

TYPO3\Flow\Persistence\Doctrine\Mapping\Exception\ClassSchemaNotFoundException thrown in file
/html/neos/Data/Temporary/Development/Cache/Code/Flow_Object_Classes/TYPO3_Flow_Persistence_Doctrine_Mapping_Driver_FlowAnnotationDriver.php in line 123.
Reference code: 201511302337210d9fb3

On the login page it’s this:

#1186669086: The action “indexAction” in controller “TYPO3\Neos\Controller\Backend\BackendController” is not public!

More information
TYPO3\Flow\Mvc\Exception\InvalidActionVisibilityException thrown in file
/html/neos/Data/Temporary/Development/Cache/Code/Flow_Object_Classes/TYPO3_Flow_Mvc_Controller_ActionController.php in line 194.
Reference code: 2015113019553722f212

It was possible to work around the problem by issueing:
./flow flow:cache:flush --force

Thus, maybe this could just be done by the package manger to prevent such problems?

@cgat You should read up about freezing packages. You should only ever do that for packages that do not change. And if they change you need to flush the cache manually, that’s the use of freezing the package. We do no longer need to check for all changes. Generally freezing packages should work though. I will check that myself, but please use another thread for this problem as it is unrelated to the package manager refactoring.

Ok, thanks for your reply. Sorry, I did not make it clearer that I reported because the errors appeard without me (or sombody else) making any changes to packages, nor any other file (happened immediately after pressing “freeze”). I had looked at the bugtracker, but it seemed to require its own manual.

I could reproduce the errors after the recovering as reported by freezing all packages again (even though site breakage did not happen imediately the second time), and a ./flow flow:cache:flush --force actually made the site work again with all packages frozen. So breaking the site instead of just freezing (non effectiveness of changes) seems a instability you may be able to reproduce and fix. (Maybe already fixed or easier with your refactoring. Many thanks, and +1 for your code clean up!).