Upgrade Neos from 2.1.2 to 8.3

Daniel_Oliver · March 11, 2024, 12:08pm

Due tue the ending basic support for projects below PHP version 8.1 on Mittwald, I’m looking for a way to upgrade Neos to the latest version. I have created this project around 2015 on a voluntary basis for my sports club. Neos 2.1.2 has been running smoothly for all these years and has never been upgraded.

Is there a recommended way to perform this upgrade?
I have checked the upgrade instructions from the documentation. Do I need to upgrade to every minor release one at a time, or is there a faster way?
Or would it be easier to start over with a fresh installation of Neos 8.3 and try to rebuild the project there?

Thank you very much for your help
Daniel

lorenzulrich · March 11, 2024, 4:51pm

Hi @Daniel_Oliver

Welcome to the forum. I’ve done this before and I suggest to take these steps:

2.3 to 3.0, in version 3.0, the resource management had breaking changes, also, the namespace was changed to neos/. The upgrade guide will help you through all migrations.
3.0 to 4.0 will bring you the formerly new Neos UI.

As far as I remember, you can then go straight up to Neos 8.3 as long as you apply all core, node and Doctrine migrations and migrate breaking changes not covered by the migrations. So you don’t need to upgrade to each version, but take into account the breaking changes in all upgrade guides inbetween. I’d like to point out the removal of the prototype generators in 5.0 which will most likely require some manual work.

Without seeing the project, it’s hard to tell if starting over is easier for your project. If you start over, you will most likely use new best practises, e.g. Fusion AFX instead of Fluid templates. This will be better going into the future, but take more time. Templating with Fluid is still possible in current Neos versions, so if you just want to get the job done as quick as possible, you better upgrade.

Daniel_Oliver · March 11, 2024, 6:32pm

Hi @lorenzulrich,

thank you very much for your reply, I will try to do the upgrade like you described.

One question. When I first created the project I didn’t know better and defined some of my Node Types within the Neos core packages, instead of my own project. Of course the NodeTypes.yaml is overwritten on every update. Is there a simple way to migrate these Node Types to my project, meaning I just move them to the other yaml, or do I need to do some kind of database migration when doing this?

Thank you very much again
Daniel

christianm · March 12, 2024, 8:22am

Just move them to a yaml inside your own site, the contents of all NodeTypes.yamls are merged anyways.

Daniel_Oliver · March 12, 2024, 11:11pm

I have started with the upgrade from 2.1 to 2.2, as shown in the upgrade instructions, but I’m already failing.

Highlited notices during composer upate:

The “neos/composer-plugin” plugin requires composer-plugin-api 1.0.0, this WILL break in the future and it should be fixed ASAP (require ^1.0 for example).

Warning from https://packagist.org: Support for Composer 1 is deprecated and some packages will not be available. You should upgrade to Composer 2. See https://blog.packagist.com/deprecating-composer-1-su pport/

After the update I’m getting “HTTP ERROR 500” loading the frontend.

The only Exception within System.log:
24-03-12 23:13:24 535045 CRITICAL Flow Exception in line 109 of /html/neos/Packages/Framework/TYPO3.Flow/Classes/TYPO3/Flow/Core/Bootstrap.php: Call to undefined method TYPO3\Flow\Http\Component\ComponentChain::getResponse() - See also: 2024031223132455bfe8.txt

PHP version: 5.6

composer.json I have used for the update:

Any hint would be greatly appreciated
Daniel

lorenzulrich · March 13, 2024, 12:02pm

I suggest to use the package versions of the Neos packages as per the Neos Base Distribution of the respective version. This helps you building a working installation.

Personally, I’ve only made sure that the Flow CLI works on the intermediate steps, so you can execute Core and Doctrine migrations. I did not try to have the frontend running because I didn’t want to waste time.

You will also need to delete Web/index.php when going from 2 to 3 as far as I remember. It will be regenerated with a composer update.

Starting with Neos 4, I would switch to Composer version 2.

Daniel_Oliver · March 13, 2024, 4:28pm

Thank you Lorenz, I will try it again with the original composer.json from version 2.2.

Unfortunately the frontend needs to keep running without much downtime, because we don’t have a development system and the website needs to be available. Maybe it would be better to create a clone of the environment on my local machine, where I can work without time pressure and without the need to restore backups. The docker installation does not really work for me, so I will try to install a local webserver + database and copy all of the files from the SFTP. Or do you know a better way of cloning the system?

Thanks again
Daniel

lorenzulrich · March 13, 2024, 5:11pm

@Daniel_Oliver Oh, I wasn’t aware you wanted to update the live system. I’d strongly suggest not to do that. We are using DDEV which is based on Docker. No need to setup PHP und co. locally and having multiple versions in parallel, you just change the version in the config file and restart DDEV. It may not be the fastest alternative, but for such a project, I think it’s a great thing.

christianm · March 13, 2024, 6:00pm

I you NOT want to do this upgrade live. Get a development setup, it’s waaaay too old a version that this will go fast and smooth.

Daniel_Oliver · March 18, 2024, 8:06am

Hi @lorenzulrich and @christianm,

thank you for the hints. It took me a few days, but I was abe to ge ddev up and running. I also managed to update Neos from version 2.1 up to version 2.3.

I’m currently struggeling with the update to version 3.0. There seems to be an issue with the package “neos/behat”. Error message:
Source directory Packages/Application/Flowpack.Behat/ has unpushed changes on the current branch:

M composer.json*

I have tried both the composer.json config from version 3.0, as well as the patched composer.json changed by “./DistributionUpgrader.phar”. Tried versions: 3.0.0, 4.0.x-dev, 4.0.1.

Is there a way I can get this package to work?

Thanks again
Daniel

christianm · March 18, 2024, 8:37am

Sounds like someone changed said file Packages/Application/Flowpack.Behat/composer.json this file shouldn’t be changed at all, if nothing else helps you could just delete the whole Flowpack.Behat folder and let composer install the new version.

lorenzulrich · March 18, 2024, 1:27pm

What @christianm says is always a good idea: If any external dependency makes you problems, just delete the code and run Composer again. Saves you time.

If it still doesn’t work with Behat, just remove the package. You don’t need it for running a website, it is just used to run the test suite.

Marc · March 19, 2024, 12:15pm

Btw its now Neos.Behat and no longer in FlowPack i would suggest to look at the composer.json of an up to date neos distribution neos-base-distribution/composer.json at eb709befbd85dab93af7a9edcf0b4e6dbdb8e0dc · neos/neos-base-distribution · GitHub

and use this as a template and add your additional dependencies back into it.

Daniel_Oliver · March 24, 2024, 10:17am

Hi all,

thank you for your input so far. After removing the Behat package I was able to run the composer update and update to 3.0, following the upgrade guide. Unfortunateley this upgrade completely broke flow and I’m not able to run it. Error message for calling “./flow”:

As a hobby web developer, I feel like this upgrade is way to complex for me. Fixing a problem will result in other problems. And I’m not even close to the latest version yet. Maybe it would be easier for me to start from scratch with a fresh installation of 8.3. Also looking at all the changes that happened within the last 8 years.

From your experience: Do you see a way to reuse parts of the database, so we don’t have to manually enter all of the data again? Is there also a way to migrate Node Types within the database, should they change in the new project?

Daniel_Oliver · May 5, 2024, 6:58pm

In the meantime I have recreated my project with fusion and afx and managed to migrate the database as well to version 8.3. Everything is looking good so far, except for a few minor issues. I would be happy if you had some tips for me.

1. Removing the language menu in the backend
In Version 2.1 I didn’t configure languages. Looking in the database, the default language was “en_US”. There also was no language menu. Is there a way to hide the language menu from the backend? Alternatively I would be happy to remove all the other languages and maybe migrate all content to German “de”, if possible. I tried the followings settings in my Settings.yaml, but without any effect:
Neos:
Neos:
Backend:
localization:
defaultLanguage: en_US
dimensions:
- en_US

2. Nodes have no dimension
Probably this is realted to issue 1.
When I run “./flow node:repair”, I’m getting the following information for all of my nodes:

Run checks for basic node integrity in the content repository
Checking for nodes with abstract or undefined node types …
Checking for orphan nodes …
Checking for disallowed child nodes …
Checking for undefined properties …
Checking for broken entity references …
Checking for nodes with invalid dimensions …
Checking for nodes with invalid workspace …
Checking for nodes with inconsistent identifier …
Checking for missing child nodes …
Checking for child nodes that need reordering …
Checking for missing default values …
Skipping node /sites because it has no dimension values set
Skipping node / because it has no dimension values set
Skipping node /sites/schnait/node-56db5e9b41b89/node-571510a308168/main/node-56ddee0771ece/column0 because it has no dimension values set
Skipping node /sites/schnait/node-56db5e9b41b89/node-571518db24120/main/node-56ddee0771ece/column0/node-57838685d0d0a/column0 because it has no dimension values set
Skipping node /sites/schnait/node-56db5e9b41b89/node-5713f58b62c82/main/node-56ddee0771ece/column0/node-5712887e374fc/column1 because it has no dimension values set

3. Issue with Images
I have copied all of my images from “Web/_Resources/Persistent” from neos 2.1 to my new project with neos 8.3. In the old version all images were stored flat in folders (havaing an ID name) within the same directory. The new version expects to find the images in sub directories, where the structure is the first 4 characters of the folder name.
For example the asset in folder “10283dfd6c0844d2b2538cb0a97a0bcc9ee697dd” is expected to be found with the following path: “1/0/2/8"10283dfd6c0844d2b2538cb0a97a0bcc9ee697dd”.
So I wrote a script that created this structure for alle folders and moved them there. This worked and all images are displayed correctly within the backend and frontend.

The issue:
Thumbnails are not shown within the Media area. If I run “./flow resource:publish”, I’m getting the following error message for all of my images:

Publishing resources of collection “static”
Publishing resources of collection “persistent”

The resources were published, but a few inconsistencies were detected. You can check and probably fix the integrity of the resource registry by using the resource:clean command.
Error: Could not publish resource X.jpg with SHA1 hash 3eed38ddb62534d0d1735727b0552973ebcedb85 of collection persistent because there seems to be no corresponding data in the storage.

Meaning that Neos thinks the images don’t exist, even though they are rendered correctly within back and frontend.

Thank you for your help
Daniel

lubitz · May 8, 2024, 11:47am

Hi @Daniel_Oliver ,

you can migrate node dimensions with NodeMigrations:

https://docs.neos.io/guide/content-repository/node-migrations
https://neos.readthedocs.io/en/stable/References/NodeMigrations.html

No sure if you need dimensions at all. Do you have just german content, but with the dimensionvalue en_US? Maybe it’s better to remove all dimension configuration and run the website without them?

The errors in #2 happen, because your code dimensions configration does not match the configuration of the nodes in your database. You need to get this aligned by a node migration (change in db) or your configuration.

Daniel_Oliver · May 8, 2024, 12:12pm

Hi @lubitz ,

yes, the website only containts german content with the dimensionvalue “en_US”. We don’t need any dimensions. So if I could remove them, this would be great. Is there a way to also remove the language menu in the backend, so the users can’t maintain multiple languages?

Thank you
Daniel

lubitz · May 8, 2024, 3:45pm

Yes, if you remove the dimension configuration, the dimension switcher on the backend will also get disabled.

lubitz · May 9, 2024, 7:03am

For better understanding please have a look here:

https://docs.neos.io/guide/content-repository/content-dimensions

At the end you should have no contentDimensions configured in your Settings:

Neos:
  ContentRepository:
    contentDimensions: {}

For the migration you need to reset all dimensions with a migration like this:

up:
  comments: 'Remove dimension values'
  migration:
    - filters:
        - type: DimensionValues
          settings:
            dimensionValues:
              language:
                - "en_US"
      transformations:
        - type: SetDimensions
          settings:
            dimensionValues: []
            addDefaultDimensionValues: false

down:
  comments: 'No down path'
  migration: []

(!) Please make a backup of your data before, in case something went wrong.

Daniel_Oliver · May 9, 2024, 5:03pm

Hi @lubitz,

thank you for the configuration and the migration. I was able to remove all dimensions from the database. However, the settings in Settings.yaml don’t seem to have any effect and the languages “English (US)”, “Engisch (UK)” and “German” still appear in the Neos Backend.

Also if I’m running ./flow node:repair, I’m still getting the info that all of my nodes have no dimension values set, which is correct of course. But it looks like this is still a problem.

Having German as the default language and migrating all content to “de” would be ok as well. I tried to use the following settings:

Neos:
ContentRepository:
contentDimensions:

  'language':
    default: 'de'
    defaultPreset: 'de'
    label: 'Sprache'
    icon: 'icon-language'
    presets:
      'de':
        label: 'Deutsch'
        values: ['de']
        uriSegment: ''

Then I ran the migration " ./flow node:migrate 20150716212459", which is mentioned in the documentation. It should migrate all dimensions to the default dimension, in my case “de”. This didn’t work. For all nodes the dimension “en_US” was added instead and I had to roll back the database. So it seems like my configuration above does not have any effect at all.

Do you have another idea on either how to remove all dimensions without Flow/Neos complaining about empty dimensions and removing the language menu in the Backend, or to migrate all content to “de” as an alternative?

Thank you very much
Daniel