Questions regarding resource management


(Christian Wellinghorst) #1

Hi there,

sorry if I missed this, but I couldn’t find answers to some questions I have regarding the resource management / temporary folder.

We developed an application using Flow (4.1 specifically) which allows users to upload images. Now we noticed that the temporary folder got pretty big and is not “cleaned up” when we flush the cache.
While the “Persistent” folder uses only about 4 GB of space, the Temporary folder currently contains 17 GB of data - 13.5 GB in the subfolder “ResourceFiles” and 3.5 GB of “ResourceUpload.tmp”-files in the main Production folder.

So here are my questions:

  • what is the “ResourceFiles” folder for and would it be useful to clean it up from time to time?
  • is it correct to have so many ResourceUpload files? We check the files after upload and sometimes do not want to keep them - do we need to clean them up or something to get rid of the tmp files?
  • to be on the safe side: the name “temporary” surely suggests so, but would it be safe to delete the whole “temporary” folder of a Flow installation without any harm?

Best,
Christian


(Christian Wellinghorst) #2

Hmm, apparently I’m not the only one who’s not sure how this works :wink:

Well, for the moment (to free up some space): does anyone know for sure whether it’s safe to delete the whole temporary folder?


(Sebastian Kurfuerst) #3

Hey Christian,

I just double-checked the files you mentioned:

  • ResourceFiles is created once a temporary local copy is created from a PersistentResource, i.e. when an image is scaled. source code - So this is safe to delete
  • ResourceUpload.tmp is created as a temporary file while uploading a new resource source code when open_basedir is enabled – So this is safe to delete
  • the Data/Temporary folder of a flow instance can be deleted with no harm; although if you do this, also the Flow caches are thrown away – thus your website is offline for maybe 30 seconds-1minute as the caches need to be rebuilt.

If you want to be absolutely safe, first move the Data/Temporary folder to a different place before deleting it (there are certain scenarios with mis-configured permissions on the server where this is helpful from time to time). In a “normal, well-configured Flow instance” it’s totally safe to delete the Data/Temporary folder.

All the best,
Sebastian


(Christian Wellinghorst) #4

Perfect, thanks for looking into this, I really appreciate it. I will try to figure out why the ResourceFiles folder got so big then.

By any chance, do you have an idea if it’s “normal” that ResourceUpload.tmp files are not cleaned up or is this an error within our app?

Best,
Christian


(Sebastian Kurfuerst) #5

Hey Christian,

I think this is missing in Flow; would be fine to clean up these files at the end of the request.

Often, I think this case does not surface, as e.g. we are having a deployment mechanism which removes Data/Temporary on every deployment (and I guess the same is true for many developers). How are you deploying the application?

All the best,
Sebastian


(Christian Wellinghorst) #6

Hi Sebastian!

Thanks again for your help. This is a bit embarrasing, but the project was started quite a while ago and we never really got around to defining a deployment mechanism (or even adding it to our GIT for that matter), so deployment is done manually and quite infrequently.
Lots of room for improvement :slight_smile:
But it’s good to know that those files are not cleaned up by Flow since we do have quite a lot of uploads and over time that takes up a lot of space (even though the project itself is rather old, it only went live about a month ago, so the 3.5 GB of temporary Uploads are probably the least to be expected each month :wink:

Anyway, thanks again for all your help!

Best,
Christian


(Sebastian Kurfuerst) #7

Hey Christian,

don’t worry about it, we all know this can happen :upside_down_face: I would really recommend to put the distribution to Git, so you at least have a cental changelog of what happened :slight_smile:

All the best,
Sebastian


(Christian Müller) #8

I am pretty sure those leftovers should only remain if something went wrong. IMHO we have cleanup for all of those in code. Eg. temporary upload files are removed after they are imported AFAIK and temporary local copies on shutdown. So it might be hint to unfinished requests (eg. due to errors and exceptions)