Improving the redis backend

Creating Neos & Flow Code
1

Hey there, I am currently digging more into redis and working on the improvement of redis cache backends as we have massive problems with flushes blocking our redis cluster.

As we are using a multi redis setup using redis sentinel I am currently improving the redis-sentinel package (https://github.com/flownative/flow-redis-sentinel/pulls). But most of these improvements as well as Ideas of @sebastian in https://github.com/sandstorm/OptimizedRedisCacheBackend should also fit into the default redis package.

In the OptimizedRedisCacheBackend, are some lines of code to also set a timeout for tags. In https://github.com/neos/flow-development-collection/pull/2052 @lars.lauger implemented a more sophisticated version which also handles write conflicts when recalculating the expiration times. But the PR was reverted as write conflicts happend even hough.

Now I have two questions:

  1. Why does this work in the OptimizedRedisCacheBackend but didn’t work in the standard package
  2. @lars.lauger did you thought about doing the recalculation entirely in a blocking lua script? Should that avoid the conflicts?
2

Hi @daniellienert,

I’m not quite sure but I don’t think that would help:

The problem with my PR was that other processes were modifying the keys with the same tags while the RedisBackend tried to calculate the TTL needed for the current keys that were being set.

When using a blocking lua script, no other processes will be able to write to redis while the script is running, so I assume that this would only change the error message to something like “BUSY Redis is busy running a script.”.

To be honest I’m not even sure if it’s really necessary to watch the keys for any modifications. Maybe the watch can be removed completely?

Here are some other problems I have with the current RedisBackend:

  • As mentioned in https://github.com/neos/flow-development-collection/pull/2052#issuecomment-813041802 I believe the entries should be removed completely. I think the same could be achieved by calling “KEYS [prefix]*” if really needed. But I’ve also never had the need to iterate over a cache so maybe the IterableBackendInterface can be removed as well?
  • The lua script currently used to flush a redis cache causes redis to freeze up for several minutes when flushing a db with a few million entries. I would prefer the RedisBackend to use one redis DB per Flow cache, as that would allow to use the FLUSHDB command which performs way better.
  • Since redis does not support tagging on it’s own I would like to have some sort of “SimpleRedisBackend” that does not implement the TaggableBackendInterface. When using redis as backend for routing caches, there will be a lot of tags that are never flushed. This seems to only be used by Neos currently. So having a variant that does not even write tags would be nice.