RFC: Integrate a Job Queue into the Neos Distribution

Creating Neos & Flow Code
1

There are more and more situations, when I create features for Neos, where I think: What we really need here is a job queue!

For example, right now I am implementing a mechanism which allows for analyzing assets when they are uploaded, to detect features like faces and landmarks, and then adapt the cropping area accordingly. It would be possible, but not very clever, to run the analysis as part of the same request which is responsible for uploading / creating the asset. Not only because the response will take too long for the user, but also when something goes wrong. An error in the analysis would also cause the upload to fail. Therefore I’d like to put this operation into a queue and process it asynchronously.

I know that we have discussed this topic here and there, but I’m not sure about the current state and if there’s a consensus:

I propose that we add the Job Queue common package to the Neos distribution by default. As a Neos core dev, you should be able to rely on that there is some kind of job queue available. Certainly, you need to be able to use different backends and so on, but it shouldn’t be possible to remove the Job Queue common package.

What we’d need as well is a default backend which works without further configuration. That could be a PDO backend and then some kind of worker. This worker should be run in CLI mode, so it shouldn’t be backed by some kind of web-request-triggers-worker mechanism.

Before we go into details: What do you think about that? Has it been discussed or even decided elsewhere already?

3 Likes
2

I think that’s a good idea. And we’ll need it with the Event-Sourced CR anyways.

We already have that with the FakeQueue that can execute jobs synchronously or async by dispatching a sub request.

I would propose that we move the JobQueue package(s) from FlowPack to the Neos namespace for that

1 Like
3

I like the idea of having a jobqueue with at least the FakeQueue always available.
Already had several use cases where it would have been nice to have it to prepare some more time consuming tasks in the background.
For example prerendering a sitemap in the SEO package or prerendering optimized image variants for the frontend while the editor is still working in the backend.

4

I actually like the idea. However, i’m not sure if we would benefit that much from a namespace move? This would break kinda all existing 3rd Party implementations and force ppl to refactor their Packages.
If we would keep it in Flowpack we would also give the Flowpack namespace at all more relevance, right?

A always availabe FakeQueue would be a must have for me. Otherwise we would raise the hurdle to use Neos since you need something like a worker process to actually process the queue. This might be hard if you’d like to run your Neos on a shared hosting provider

5

We should of course keep the existing packages as is (with a pointer to the new repo for feature releases).

It would just be a new major version basically with a new name & repo.

It would be nice if we could overhaul the Flowpack packages. Right now it’s definitely not the “repository for high quality packages” that http://flowpack.org/ claims it to be and lots of code is out of date.

But I’m more concerned about the “reliability” – somehow it feels weird to me if the Neos Core depends on a Flowpack package…
It’s not a deal-breaker for me of course :slight_smile:

That’s part of the JobQueue.Common package.

We would have to decide whether we use it with the async flag set by default or not

6

I’m all for it. We use the package in most of our projects already for async / parallel processing and I think also the core could benefit at many places.

I would suggest to move the Common package as well as a at least the doctrine implementation.

1 Like