RFC: Overhauled JobQueue Packages

The Neos Project Neos Project Decisions
1

Hi all,

as you probably know there are these weird JobQueue packages floating around (previously TYPO3.Jobqueue.* now Flowpack.JobQueue.*).
But there is no documentation whatsoever and the code can use some cleanup/tweaking.

That’s what I started doing (with support by the great folks from t3n.de). And it turned out to be more than just a little tweaking so I’d like to get your opinion before pushing a lot of Pull-Requests.
So here’s what I did so far:

New Features

  • Documentation :wink:
  • Improved error handling: Failed Jobs will be re-released for a configurable amount of trials before they are “aborted”
  • Simple SerialQueue implementation that allows to execute jobs via sub request, so it won’t require any 3rd party tool nor server loop (i.e. as “poor mans solution” for async tasks in Neos)
  • New Flowpack.JobQueue.Doctrine package that allows db-based queues (SqlLite, MySql, … defaults to the current db)
  • Implementation specific options for the “Defer” annotation (e.g. @Job\Defer(queueName="some-queue", options="{delay: 123}"))
  • Signals for most events allowing 3rd party packages to hook into the processing of jobs/messages

(Breaking) Changes

The public API of JobManager, CLI and Defer annotation have been adjusted in a backwards compatible manner, but there are some changes that might break depending code:

The QueueInterface has been changed, including one important behavioral modification:

The signature of submit(Message $message, array $options = []) has been changed to submit($payload, array $options = []).
Previously a message with the same id of a previous message were discarded. This is no longer possible. The reason is that the behavior was not reliable (depending on the implementation the id was free again as soon as the message was reserved/processed) and is not supported by many queue implementations (for good reasons).

Now I wonder: Did anyone rely on that feature of a message id preventing the message from being published twice?

In any case I’d suggest to push the major changes to a new branch 1.x so that the existing implementations can still be maintained if required.

3 Likes
2

Hi there,

good to get this improved based on actual work. The first version was more or less constructed in the air ;).

There’s one thing though:

This would couple the PHP-Code (which cannot easily be swapped for different deployments) directly to a Queue implementation. This could hurt testing and different setups. What do you think about prefixing the options with some kind of implementation name so at least you could provide options for different implementations? E.g. options="{beanstalkd: {delay: 123}, doctrine: {transactions: false}, ...}"

I think there was a use case from my side for that, but I think it shouldn’t belong to the publishing / queue implementation, but rather be handled inside the job / worker. Also especially distributed message queue implementations provide different delivery guarantees and you’d end up with at least once delivery anyway ( You Cannot Have Exactly-Once Delivery – Brave New Geek ), so the jobs should behave idempotent to have a robust system.

3

Good point, but I think it works like this if we assume the following:

  • Unknown options must be ignored by the queue
  • We should only add options that make sense per submit/release

For example: A flag like transactions probably makes more sense on the queue (if at all). Remember, you can have multiple queues per backend.

In the Beanstalkd queue I have implemented the options delay, priority and TTR like supported by the Backend. The delay option I have already added to the Doctrine queue, too. And it will be straightforward to add it the Redis queue, too. Same for priority.

I can’t really think of any other options that would make sense… And TTR we can probably neglect as that’s probably rather a task for the JobManager to enforce timeouts.

Exactly!

Thanks a lot for your valuable feedback. I’m really looking forward to your code reviews once I get around finishing the PRs