Could we build a version of Flow without doctrine orm / database requirement

Creating Neos & Flow
1

This is a description of a concept and is up for discussion. Please share you views, knowledge and comments on this topic

So, this other day I started looking at building a dashboard for a project. A local application that’s suppose to display on a monitor in a room showing widgets with data. I didn’t need any database stuff in my project as all would work with external apis. All I needed was

  • A http stack
  • Some caching (to save responses untill next refresh)

But when i install a clean version of Flow I get a lot of requirements installed that I would need and only slow down a process (imagine this in a larger scale app, that needed to download and deploy the same amount of packages). To show a example I did the following

 soren@kevin > ~/Projects/flow-no-database > composer why zendframework/zend-eventmanager
zendframework/zend-code  3.1.0  requires  zendframework/zend-eventmanager (^2.6 || ^3.0)
 soren@kevin  > ~/Projects/flow-no-database > composer why zendframework/zend-code
ocramius/proxy-manager  2.0.4  requires  zendframework/zend-code (3.0.0 - 3.0.2 || ^3.0.4)
 soren@kevin > ~/Projects/flow-no-database > composer why ocramius/proxy-manager
doctrine/migrations  v1.3.1  requires  ocramius/proxy-manager (^1.0|^2.0)
 soren@kevin  > ~/Projects/flow-no-database > composer why doctrine/migrations
neos/flow-development-collection  dev-master  requires  doctrine/migrations (~1.3.0)

What entities do we have in the core?

In the core we have the following enties, that requires the migrations and database related packages to be installed (please correct me if I’m wrong :thumbsup:)

  • Flow security account
  • Routing ObjectPathMapping
  • PersistentResource entity

Lately a lot of small utility packages have been created - could we do the same with these? My proposal is (And please comment and describe the blocking part of the different things)

Move the Security account to it’s own package

Create a package like Neos.SecurityAccount or even Neos.Security and move that part out there ?

Move the routing objectmapping to a cache

Could the object path mapping go to a cache instead of a entity? Perhaps introduce a Database cache backend if you are using databases anyway - otherwise use the file system or no cache at all?

Move the persistent resource to a own package

Yes, you guessed it. Move the whole ResourceManagement context of the Neos.Flow package to a own

What would we benefit

I did some testing using composer why and what I seem to have come up to was that a “Flow Light” could come down to the following packages

doctrine/annotations            v1.4.0            Docblock Annotations Parser
doctrine/cache                  v1.6.1            Caching library offering an object-oriented API for many cache backends
doctrine/collections            v1.4.0            Collections Abstraction library
doctrine/common                 2.7.x-dev 562b53a Common Library for Doctrine projects
neos/composer-plugin            2.0.1             Flow Composer Plugin
paragonie/random_compat         v2.0.10           PHP 5.x polyfill for random_bytes() and random_int() from PHP 7
psr/log                         1.0.2             Common interface for logging libraries
ramsey/uuid                     3.6.1             Formerly rhumsaa/uuid. A PHP 5.4+ library for generating RFC 4122 version 1, 3, 4, and 5 universally un...
symfony/console                 v2.8.19           Symfony Console Component
symfony/debug                   v3.0.9            Symfony Debug Component
symfony/dom-crawler             v2.8.19           Symfony DomCrawler Component
symfony/polyfill-mbstring       v1.3.0            Symfony polyfill for the Mbstring extension
symfony/yaml                    v2.8.19           Symfony Yaml Component
typo3fluid/fluid                2.1.3             The TYPO3 Fluid template rendering engine

And the moving the FlowAnnotationDriver and the whole Persistence context into a own package, would also remove a requirement of the doctrine/annotations package (as far as I can understand from the usage).

So, now it’s out. A few days of thinking… Please give your feedback. I’ll be happy if you took the time to response more than “+1”, “go go go” or “hey, are you mad?!”. Read it, suck it in and give it some thought and perhaps give a reponse next week.

I’m very well aware of this not being a “walk in the park” and “what about backward compatibility” - so this is a place for discussion, meaning sharing and your change to share your knowledge about Flow that could prevent this from happening or something that might make this “easier”.

I’m giving it a try in next week by moving the routing into a cache - wish me luck :slight_smile:

Have a great weekend!

4 Likes
2

Okay, I also needed a view part… Thank you Fluid! :slight_smile:

3

Can add that I know of potential users, who decided to find an alternative to Flow due to it’s requirement on having database.

So by all means, would be great to have!

And replace with interfaces for the account repository, account factory & account, right?

doctrine/annotations is used for parsing doc blocks for the reflection/AOP AFAIK, so that needs to stay, however it’s a light package with a single dependency on another light package

:+1:

4

I am all for it. Totally in line wit my evil masterplan :smiley:

5

Thanks for this, you really put some good research into this and that’s really awesome :slight_smile:

Could the object path mapping go to a cache instead of a entity?

The main issue there are the two queries inside the ObjectPathMappingRepository, findOneByObjectTypeUriPatternAndPathSegment and findOneByObjectTypeUriPatternAndIdentifier. Those would need to implemented through simple key-value lookups in order to be able to use e.g. the Cache framework for persistence.

Perhaps introduce a Database cache backend

There is already the PdoBackend Cache Backend, but that still only provides the normal Cache Backend interface of a key-value store.

Current state of affairs

Basically, the whole Persistence layer is already supposed to be pluggable with different implementations, and the original “generic” persistence implementation is still lying around, but pretty much unmaintained. So there is already some core principle to detach Flow from the massive Doctrine dependency in place. That of course would still leave you with the Persistence-Layer dependency.

So what I would see as a next step towards that goal of getting around the hard Doctrine dependency, would be to make the Doctrine implementation of the Persistence layer an own package, which could then be stripped from a custom (light) Distribution.

If the dependency to any Database is an issue, a Generic Persistence Layer backend could maybe be provided that works without a Database. BUT: That would mean we again have the huge effort to maintain a custom ORM layer and an underlying backend implementation.

Otherwise I guess the choice will always be: Want Security and/or ResourceManagent? Then you need the Persistence Layer (with a Database).

6

Yes, let’s slice up the elephant!

d6767b9d55877d2dffba12386863926bfb685748.jpg1619×971 213 KB

Just to split some hairs: Since a few versions, a database connection is no longer required by Flow.
But, yes, all the doctrine dependencies are.

Probably makes sense. Instead or in addition to that we should finally introduce some AccountInterface and code against that for the cases that need security (probably the large majority) but don’t want to use a db.

yes, could be done. But it got to be a persistent cache – you don’t want to lose those mappings if you flush the cache.
The lookup should be doable if we tag the cache entries.

I’d like to be able to use resources without a db. I mean in general it should be possible to store files just … in the filesysten (obviously you miss some of the metadata, but for the basics that should work IMO)

1 Like
7

Exactly, been seeing that comment inside the core. So yes, have a interface in the core and let developers implement that.

Can we create a project on github and try to collect issues without messing up the rest of the Flow issue tracker? Or should I just fork the flow collection/distributioni (?) repository and start collecting issues there ?

@aberl I was wondering if we could isolate the IdentityRoutePart to a own package aswell? Neos.Routing.IdentityRoute - but then I came across the DynamicRoutePart that requires the persistencemanager interface to get the identity of a object (used for ex. a “single view” route. So the splitting of the RoutePart classes might also be larger than expected, since everything excecpt static routes (controller/action based, no persisted object arguments) requires some sort of Persistence layer.

But of course we could use the Generic persistence implementation for that, as it already implements a generic way of getting that identifier. Or split the routing into different parts - you can still do routing with out having to pass around persisted object.

Which brings me to another topic: Are we touching other parts now aswell? TypeConverters, PropertyMapper ? Or are they good to leave and are not affected by these changes (more likely the other way around)

8

since everything excecpt static routes (controller/action based, no persisted object arguments) requires some sort of Persistence layer.

And Persistence(Manager)/entityManager is also part in a lot of other components, like MVC Views, PropertyMapping, Validation, Object Serialization, etc.

As much as I’d like to just separate a few components and make them optional to build a minimal Flow Distribution that works without Persistence, I think it’s a pretty huge effort and you either end up with many micro-components that need to be maintained, or a single big “Persistence component”, that contains component addons for pretty much every other component and feels like a full framework in itself.

Not trying to argue against it, I just want to show the amount of work that will be involved to reach the final goal. Of course starting to separate some components, like Security Account and Resource Management as you suggested would be a good thing to do in any case, even if it doesn’t fully resolve the dependency to a persistence layer.

As stated, getting rid of the Doctrine dependency is probably pretty doable. But for making Flow truly independent of a Database, I only see these two approaches:

  • factor out/rewrite all Parts that currently depend on the Persistence Layer, then make that Layer an optional part. A rewrite would surely be possible for Routing and Resource Management, but for example for Entity Privileges probably not so much.

  • write a Persistence Layer implementation that works without a database (ugh)

9

So, I took the time to collect a few weeks of thougt about this topic here, to create some sort of picture of the task

People with larger core knowledge, all comments are appreciated.

And if someone could give me a suggestion on how to organize a fork and a composer.json to read these new packages and similar (this is kind of the same thing when the FluidAdaptor was created - splitting up, and putting together), that would help me and avoid that I spend the first couple of days on that :slight_smile:

10

Sorry for not replying earlier. Hope you didn’t already spend the days to get the setup. If not, to setup your fork, it’s generally enough to add your own repository in the composer.json like this: https://getcomposer.org/doc/05-repositories.md#loading-a-package-from-a-vcs-repository

Basically just make flow-development-collection point to your own github fork. Afterwards any composer install/update will pull from there.

Anyway, thanks for the writeup! It’s highly appreciated! I’ll try to have a look and give more concrete feedback ASAP.

11

Thanks @aberl - I think I got it started :smiley:

And created a Neos.Security repository and added a bunch of issues

Two things I could really use some core developer insight on is the following (@aertmann @bwaidelich @christianm)

12

Perhaps I’m doing this wrong… Instead of creating a new giant package with the Security subcontext, would it be a better idea to just remove the entities and move that and database stuff to a new package.

that would also be easier for backward compatibilty

13

@sorenmalling Thank you for investing time and thought into this!

I see it as two different topics:

1. Extract the Security SubPackage

Following @christianm’s example we could create a new package neos/flow-security (or a similar name) and move the classes from Neos\Flow\Security there.
I’m always in favor of splitting up optional parts from the core. But in this case I’m not sure whether it’s feasible because security is so deeply integrated in many parts of Flow (but I might be wrong here!)

2. Make Authentication database independant

This is probably the low(er) hanging fruit here.
We should introduce a new AccountInterface and adjust all the places that currently refer to Account.
The existing Account entity could still stay in the core (for the start at least) – it doesn’t do any harm, does it?
At some point we could move all DB-related classes (PersistedUsernamePasswordProvider, Account, …) to a separate package, but that would require some B/C layer probably

14

BTW: I can offer to take care of replacing the Routing ObjectPathMapping implementation with a persistent cache

15

Thanks @bwaidelich

So, I might rethink stuff and create some smaller iteration.

First one being:

Introduce AccountInterface to Flow and make the Account entity implement that.

The interface should contain

  • accountIdentifier
  • authenticationProviderName (?)
  • credentialsSource (password, for sake simplicity and understanding)

Question: Supporting multiple AccountInterface implementations

Since now, the provider classes uses the generic AccountRepository. Could a second iteration make a “GenericUsernamePasswordProvider” class, that you can extend, and then simply set your own RepositoryClass as a property, and then “it just works”?

16

Just replying myself… Could a providerOption like “repository” or “implementaionClassName” or similar, be a great and simply solution. And have a default value to the Account class. that would also make it backward compatibile! <3

17

@sorenmalling If we introduce a new Interface I’d like to take the chance to use more ValueObjects from the start (only if it doesn’t produce a lot more work of course).
This is the interface I could imagine:

interface AccountInterface
{

  public function getIdentifier(): AccountIdentifier

  public function getProviderName(): AuthenticationProviderName

  public function getCredentialsSource(): CredentialsSource

  public function getRoles(): Roles

  public function hasRole(Role $role): bool

}

note: I renamed getAccountIdentifier() and getAuthenticationProviderName() to keep b/c with the current implementation (that doesn’t return ValueObjects)

And for the expiration-fields maybe a separate, more specific, interface along these lines:

interface ExpirableAccountInterface extends AccountInterface
{

    public function hasExpirationDate(): bool

    public function getExpirationDate(): ?\DateTimeInterface
}

note: At first I had the isActive() in that second interface, but IMO that logic should be moved out of the implementation

Not sure what to do with creationDate but all the mutating methods should be removed (set*()) and the whole failedAuthenticationCount / lastSuccessfulAuthenticationDate should never have been part of the entity in my opinion.

First we should think of reasons why you would want to have a different implementation. IMO we need to support some kind of “transient” Account implementation that is not persisted but will be returned from the provider. This will allow for db-less-authentication and for easier “augmented” authentication, i.E. using a 3rd party auth service.

I can’t think of a use case that would require a different DB-based account implementation

18

Could we define a CredentialsSourceInterface as well? I guess most projects would like to have individual control over validation here.

19

Not meant as a “different database” but meant as a different repository than AccountRepository - for example CustomerRepository if I treat my customer as someone with access to something.

That’s a great idea, especially in terms of validation and such.

20

meant as a different repository than AccountRepository - for example CustomerRepository if I treat my customer as someone with access to something.

But that promotes/opens the door for mixing up Account and User, which are two very different concepts. I always liked Flow making this explicit and hence forcing users to think about this (I myself had the two in a single mental concept until I learned better through Flow).

1 Like