Long running Neos Job worker vs Node caches

Hello,

we have a job worker (based on Flowpack\JobQueue\Common) that gets as input a “Node” from Neos CR (“afterNodePublishing”) and starts doing stuff on it. Works good - on first sight.

The problem is that there are several “node” caches involved and as a “long running” worker they get on the way of working with correct data. As soon as a node is once fetched with $this->context->getNodeByIdentifier($nodeId); the next time I fetch it (even if it changed meanwhile) I get the same old data back.

I already did $this->context->getFirstLevelNodeCache()->flush(); before fetching, which seems like a hack. But still there is probably the Doctrine\ORM query cache and maybe even other “caches” that are usually irrelevant as they are static to the classes and dropped as soon as a request is over, but in the long running case they are in the way.

Is there a generic “recipe” on how to deal with that in long running processes (like job workers)?

Thanks a lot for your help!

Kind regards,
Ernesto

I don’t really have a recipe, but indeed the identity map of doctrine could be a problem (which can be flushed as well).

Hi Ernesto,

Do you actually store the node instance in the queue?
I would suggest to only store the minimal amount of data in the job queue that you need to do the job.
In this case the node’s identifier could be enough. Or maybe the identifier and some context variables.

Besides, with the current version of the JobQueue packages you can use the executeIsolated flag to execute jobs in their own isolated request.

Yes, we have only the identifier in the queue.

Besides, with the current version of the JobQueue packages you can use the executeIsolated flag to execute jobs in their own isolated request.

That sounds great! To my shame I must confess that this project is still on Flow 3.0 and Neos 2.0. But an upgrade to at least 2.3 is “on the way”. Hope that your new updated JobQueue packages will work with that?

Do you know how to flush that?

I have now “solved” the problem by now running job:work with --limit 1 and running that in an endless loop. :slight_smile:

That should be compatible: jobqueue-common/composer.json at 1.0 · Flowpack/jobqueue-common · GitHub

1 Like