Neos 9 Beta-15 Release

Announcements
1

Today we released another Neos 9 Beta.

An astonishing amount of work was done at the Neos sprint in Dresden as well as around the sprint and on the train :wink:

Thanks to all who contributed <3.

Upgrade instructions from Beta 14

previous beta → Neos 9 Beta-14 Release

In your composer.json you should require 9.0.0-beta15 for all Neos packages from the development distribution and for the NeosUi as well as Flow:

"neos/flow": "9.0.0-beta15",
"neos/neos": "9.0.0-beta15",
"neos/neos-ui": "9.0.0-beta15"

Maybe its necessary to flush the code cache due to changes in Flow itself:

./flow flow:cache:flush --force

The database can be migrated the following way:

A setup to apply the latest schema adjustments:

./flow doctrine:migrate
./flow cr:setup

Additionally, PRs marked with :boom:⛁ need a dedicated migration

  • :boom:⛁¹ Migration to adjust serialization to multiple references per event / command
./flow migrateevents:migratesetreferencestomultinameformat
  • :boom:⛁² Migration to prune “lying” events from temporary content streams and further cleanup for a replay

Note that pruning from the event stream is to be used with care because of its implications see: Note regarding pruning from the event stream below.

# 0.) backup the events (cr_default_events) 
./flow migrateevents:backup
# 1.) mark dangling content streams as removed (temporary ones and ones that are no longer in use)
./flow contentStream:removeDangling
# 2.) prune removed content streams that are not required for a full replay to work
./flow contentStream:pruneRemovedFromEventStream
# 3.) replay all projections
./flow cr:projectionReplayAll

Features

1.) :sparkles: Highlight :sparkles: Refactored, stabilised and faster publishing

The last beta uncovered a fundamental problem with the new workspace based publishing.
A partial publish / rebase could break the change and uri path projection.

You might have been faced with this error: “Changes in document test could not be published”:

Exception while catching up to sequence number drölf

or

Rebase failed

Bugfixes in publishing

  • publish workspace will now behave exactly like partially publish and won’t throw an exception if the workspace is outdated but will attempt a rebase
  • preserve always the initiating timestamp of nodes
  • we repoint the workspace via WorkspaceWasPartiallyPublished before applying the applying remaining events
  • no dangling content streams that are not used but bloat up the projections

Neos Ui conflict resolution after failed publish
The new conflict resolution of the Neos Ui from Beta 10 was further polished so that a possible conflict after publishing can be solved directly without syncing explicitly.

Performance optimisations

  • we do not rebase explicitly before the publishing but the publishing itself is a rebase (reducing the need of one fork)
  • we run the partially publishing in simulation (reducing the need of one fork)
  • a partial publish will be interpreted as full publish if all nodes are to be published

Excerpt of the new internal workings :wink:

With the new conceptualised publishing we only emit events to the other projections and catchup hooks if they are indeed to be published.
The constraint checks during the rebase / publish are done in simulation now:
We start a database transaction with later rollback for the content graph projection and then handle the rebased commands to see if they would pass today again.
The events to publish are first written into an in memory event store and later applied to the real content stream and fully catch up’d.

That means no projection or catchup hook has to deal with the complexity of temporary events that were partially a lie.

2.) Overhaul content stream pruner

As migration strategy for the in 1.) mentioned publishing problem we advise to use the content stream pruner.
It was reworked to operate fully on the event store instead of relying on projections to be up-to-date.
This also allowed us to remove lots of logic from the content stream projection like soft removal and also trying to track the in use status of a content stream.

In result the following commands can also be used in case the projection was not well, either to clean up or fully prune your system:

  • ./flow cr:prune
  • ./flow contentStream:removeDangling (previously ./flow contentStream:prune)
  • ./flow contentStream:pruneRemovedFromEventStream

The need of removing dangling content streams should be an issue of the past after the new publishing.
But to check for integrity ./flow contentstream:status can be used.

It detects if dangling content streams exists and which content streams could be pruned from the event stream.

:bulb: A note regarding pruning from the event stream

Content streams that were removed ContentStreamWasRemoved e.g. after publishing, and are not required for a full replay to reconstruct the current projections state.
That’s why we detect those and offer to prune them.
The ability to reconstitute a previous user workspace or shared workspace state will be lost.
Only the event stream for the root workspace (live) and for pending changes is untouched.

Dangling content streams: Content streams that are not removed via the event ContentStreamWasRemoved and are not in use by a workspace.

!!! 3.) One main stable content graph core projection

The workspace and content stream projections (which were just one table each) are now moved into the content graph projection.
This stabilizes the behaviour as we don’t have to deal with one projection not being up-to-date
Also this change was long planned so the getContentGraph call doesn’t have to look up foreign tables to fetch the workspace content stream id.

With this change the deprecated WorkspaceFinder was removed and also the internal ContentStreamFinder.
Instead, the ContentRepository offers the API:

/**
 * Returns the workspace with the given name, or NULL if it does not exist in this content repository
 */
public function findWorkspaceByName(WorkspaceName $workspaceName): ?Workspace;

/**
 * Returns all workspaces of this content repository. To limit the set, {@see Workspaces::find()} and {@see Workspaces::filter()} can be used
 * as well as {@see Workspaces::getBaseWorkspaces()} and {@see Workspaces::getDependantWorkspaces()}.
 */
public function findWorkspaces(): Workspaces;

/** @internal */
public function findContentStreamById(ContentStreamId $contentStreamId): ?ContentStream;

For full upgrade instructions regarding the removal of the deprecated WorkspaceFinder please look into the last beta.

Also, as we decided with Beta10 to catch up the projections synchronously (without ->block()),
the code base was finally cleaned up of these old async artefacts and a lot of thought in general has been put in wiring everything together.

4.) Reliable and tested change node type behaviour

We introduced a more comprehensive test suite for ChangeNodeType and added the following features:

  • default values are now added if the property is yet missing
  • !!! obsolete properties are now removed
  • missing tethered children are now added
  • disallowed children are now properly removed (in DELETE strategy)
  • already present tethered children will be adjusted in type if necessary
  • all the above is now applied recursively for nested tethered declarations
  • the change projection is now able to deal with these aggregate scoped changes (NodeAggregateTypeWasChanged, NodeAggregateNameWasChanged)

Further a bug in NodeRemoval was fixed where only one child node was removed per removed parent

5.) Fixes regarding workspace status and base workspace changing

The workspace status (Workspace::$status) is now calculated more reliable on querying directly.
Previously changes to live would not render dependant workspaces outdated.

Also, a minimal concept of changes inside a workspace was introduced as boolean: Workspace::hasPublishableChanges().

And changing the base workspace now works completely again and shows the content of the new base.

6.) New node serialization format for all places

With Beta 11 the node uri building was overhauled.
Now the Neos Ui was also adapted to use the new node serialisation format, getting rid of Neos\Neos\FrontendRouting\NodeAddress.
And custom ui plugins and endpoints will only need to deal with this one format:

localhost:8081/neos/content?node={"contentRepositoryId":"default","workspaceName":"marchenry","dimensionSpacePoint":{"language":"en_US"},"aggregateId":"000b42ff-c9d3-4b5e-b6de-56d35832dc0e"}

!!! 7.) Overhaul node references writing

Reworks references so that multiple reference properties can be set via a single command and also references can be attached to CreateNodeAggregateWithNode.
Most importantly, references are now also respected when copying nodes.

The API to set references changed a little and code must be adjusted accordingly.
Note that SetNodeReferences does not require the $referenceName field anymore, which is now encapsulated inside NodeReferencesForName.

This is a simple example which covers all cases:

SetNodeReferences::create(
    $node->workspaceName,
    $node->aggregateId,
    $node->originDimensionSpacePoint,
    NodeReferencesToWrite::create(
        // sets the 3 nodes as references
        NodeReferencesForName::fromTargets(
            ReferenceName::fromString('first-reference'),
            NodeAggregateIds::fromArray(['node-aggregate-id-1', 'node-aggregate-id-2', 'node-aggregate-id-3'])
        ),
        // unset previously set references (as we do NOT merge the values)
        NodeReferencesForName::createEmpty(
            ReferenceName::fromString('second-reference')
        ),
        // create a simple node reference and additionally a reference with an additional property
        NodeReferencesForName::fromReferences(
            ReferenceName::fromString('third-reference'),
            [
                NodeReferenceToWrite::fromTarget(NodeAggregateId::fromString('node-aggregate-id-6')),
                NodeReferenceToWrite::fromTargetAndProperties(NodeAggregateId::fromString('node-aggregate-id-5'), PropertyValuesToWrite::fromArray([
                    'propertyOnReference' => 'Hi im living on the edge ;)'
                ])),
            ])
        )
    )
)

!!! 8.) :bulb: Remainder :bulb: Removal of deprecated node access from beta 14

The FlowQuery accessors id() nodeTypeName() and label() were removed.
And the helper Neos.Node.getNodeType() was renamed and behaviour changed to be nullable.

Neos 8 (Removed) Neos 9 Beta 13 (Removed) With Beta 14 and above
node.identifier q(node).id() node.aggregateId
node.nodeType.name q(node).nodeTypeName() node.nodeTypeName
node.label q(node).label() Neos.Node.label(node)
node.nodeType Neos.Node.getNodeType(node) Neos.Node.nodeType(node)

For full upgrade instructions please look into the last beta.

Further a tracer was introduced to log if the Neos 8 Node field access was used.
To enable full strict mode set Neos.Fusion.deprecationTracer to "EXCEPTION" so it throws during rendering.

24-09-24    DEBUG   The Node field "label" is deprecated in "${q(site).property('titleOverride') || site.label}"
24-09-24    DEBUG   The Node field "label" is deprecated in "${item.label}"
24-09-24    DEBUG   The Node field "identifier" is deprecated in "${node.identifier}"

Bugfixes

Neos Content Repository

Neos Neos

Flow

Neos Ui

Dev only relevant

List of things

Neos

Neos Ui

6 Likes