Neos S3 Migration Best Practices

Using Neos & Flow
1

Hello everyone,

i’ve recently migrated a project to use flownative/aws-s3 together with CloudFront, using the two-bucket setup. I’m building the architecture initially using Minikube, and later pushing the project to a real Kubernetes cluster.

Now I’m considering the “migration from environment A → environment B” workflow, and would love to hear feedback about best practices.

Here’s my current (partially automated) workflow:

Environment A (source):

  1. flow site:export --site-node vendor-neos --package-key vendor.Neos
  2. Backup of the Persistent/Resources folder
  3. Dump the PostgreSQL 17 database (using --data-only)

Environment B (target):

  1. Spin up a fresh DB
  2. Deploy Neos and run doctrine:migrate
  3. Run flow site:import --package-key vendor.Neos using the export from A
  4. Copy the Resources backup (from A) into Data/Persistent/ of B
  5. Truncate all media-related tables in the new DB, then import the DB dump from A
    Reason: The import would otherwise point to assets with different hashes that do not match the imported dump
  6. Flush caches, publish resources, etc.

My questions:

Now that B uses an S3 bucket, are there changes I should consider in my A → B workflow?

For example, I currently keep references like my site-package in repositories such as:

“repositories”: {    
    “distributionPackages”: {        
        “type”: “path”,        
        “url”: “./DistributionPackages/*”    
    }
},

Afterwards, I always run resource:clean to delete unused local assets.
I also clear the contents of /Data/Persistent/Resources/* and /Web/_Resources/Persistent/*.

This approach works acceptably, but it feels somewhat hacky.
Is there a cleaner, more robust way to migrate between A and B in the presence of the flownative/aws-s3 two-bucket setup?

Are there known pitfalls, best practices, or commands (e.g. resource:copy, configuration flags) I should use in this migration scenario to avoid inconsistencies, missing assets, cache issues, or hash mismatches?

Any insights, references to docs or real-world experience would be greatly appreciated.

Thanks in advance for your help! :slight_smile:

2

Hi there!

I’m not sure, if I get your use-case right. Did I get it right, that you are NOT using S3 in the source-environment, but in the target you will? Or am I getting something wrong?

When we did migrations from non-s3 to s3 setups, we installed flownative/flow-aws-s3 into the old environment and used the referenced ./flow resource:copy command, to copy all Resources into S3.
Since you can configure a new resource-storage beside neos’ default one, this can be done on the running environment. You only need external access to the S3-Storage.

Afterwards we just dumped the DB and moved it to the new instance and configured the S3-Storage target as default storage. Since the S3-Package stores the Resources based on the SHA1, the new instance will also find it there. :slight_smile:
(we are using single-bucket setup, but should be similar there).

I hope this is was the point, you wanted to know. If not, just let me know.

1 Like
3

Hey @ndroste

Thanks for your answer, it helped a lot :slight_smile:

You got it perfectly right, basically it is a non-S3 → S3 setup. I think i’ve understood the most, but I’ve got one more practical question regarding the site:export and the binary files it generates.

When I run:

./flow site:export --site-node vendor-neos --package-key vendor.Neos

Neos writes a bunch of files into
/DistributionPackages/mb.Manufacturing/Resources/Private/Content/Resources/
plus the updated Sites.xml.

Now, when i later import on B with:

./flow site:import --package-key vendor.Neos

are those binaries actually used in the import process, or are they redundant once S3 is configured and the database already knows all resource hashes?

From what I’ve understood:

When migrating from a local setup (A) to an S3 setup (B), these binaries are needed once, because Neos reimports them and uploads them into S3.

But when migrating from one S3 setup to another (B → C), they’re not needed anymore, since all assets already exist in the bucket and are matched by SHA1 in the DB.

So basically:

A → B: Export (A) → Import (B) → Publish (B) → Clean (B)

B → C: Export (B) → Import (C) → Publish (C) → Clean (C). But: The Import command is looking for the SHA1 references within the Sites.xml, right? So if the DB is not yet having the dump from B, it looks for the references in the Resources folder?

4

Perfect, then we are talking about the same thing :slight_smile:

In the scenario I described above, we did not even do a site:export. We just copied the Resources directly from the A-Environment to S3, using the resource:copy command. Then we directly dumped the DB, transferred the dump to B and imported the DB on B-Environment. When you configure S3 on B the exact same way, it will find all resources in S3, without an import.

That’s basically the same process, of how you could move a non-S3 from A to B. (Dump DB, import DB on target and copy Resources-Folder).

You could also do it like:

  1. Dump DB on A and import it on B
  2. Copy Resources-Folder from A to B
  3. ./flow flow:cache:flush & ./flow resource:publish on B. (Now Neos should run exactly as on A)
  4. Configure S3-Package on B as secondary resource-storage.
  5. Flush Cache and do the resource:copy command on B to copy files to that newly configured resource-storage (S3).
  6. Switch configuration to the new Storage → cache:flush & resource:publish.

It’s basically the same procedure. Also without need to use site:export.

I’m not sure, who told me back then, but somebody said that site:export could cause a lot of trouble with mid- or large-sized setups. And I also had a bad experience with that.

Would that maybe be an easy alternative for you? Maybe just give it a try without activating S3 first and see how easy it could be :slightly_smiling_face:

1 Like