Imho the most expensive cache operation for the typoscript caches is the tag-based cache invalidation that happens during publish since the now invalid caches are thrown away immediately. The next reload takes a bit longer because some caches are rebuilt but not as long as the flushing if there are lots of documents.
You can easily validate this by running flow:cache:flush … if the system runs much better afterwards especially when publishing you have a peformance problem with cache invalidation.
I had that several times for sites with many nodes and usually solved it with redis.
# Use Redis Caching in Caches.yaml
TYPO3_TypoScript_Content:
backend: TYPO3\Flow\Cache\Backend\RedisBackend
Flow_Mvc_Routing_Resolve:
backend: TYPO3\Flow\Cache\Backend\RedisBackend
Flow_Mvc_Routing_Route:
backend: TYPO3\Flow\Cache\Backend\RedisBackend
If redis is not available you could try sqlite like here @christianm did in his experiment here Try this please, Cache configuration