One possible guide to replay projection locally and deploy remotely

Using Neos & Flow
1

Opinionated guide to replay projection locally

:warning: Please make sure you have a full backup available. The steps do NOT make an attempt to an atomic change of a projection in a deployment. Instead these steps are to be executed on a STAGING system to avoid any downtimes. Once the staging instance is working you can change the instance to production.

Reasons for a local replay can be:

  • speed up the replay via local processing power and less latency to the database
  • speed up the replay via experimental changes that are not release (PR to speed up subtree tagging for replay)
  • ensure a local consistent state and better local debugging

A content freeze is necessary as we intend to switch stage and production after the migration is finished. Any changes in the meantime one production would be lost otherwise.

These steps where used for the neos.io update for Neos 9.2

Step by step

  1. Remote: Validate the current sequence number in the events
./flow cr:status

Should report that the events are at position say 100.000 and the projection intended to replay is also at that position and ACTIVE

  1. Remote: Dump the events insert-SQL into cr_default_events.sql
mysqldump -u [user] [database] --no-create-info --compact cr_default_events > cr_default_events.sql
  1. Remote: Compact the dump into cr_default_events.sql.tar.gz
tar czfP cr_default_events.sql.tar.gz cr_default_events.sql
  1. Local: Download and extract the events

Check that your absolute path matches. E.g. /application/cr_default_events.sql.tar.gz

scp user@my.domain.net:/application/cr_default_events.sql.tar.gz cr_default_events.sql.tar.gz

With jump host for beach:

scp -J beach@xxx.flownative.cloud beach@instance-xxx:/application/cr_default_events.sql.tar.gz cr_default_events.sql.tar.gz

And extract

tar -xzvf cr_default_events.sql.tar.gz
  1. Local: Insert the events into your cr_default_events table

With an empty events table and empty content repository (drop all cr tables and run ./flow cr:setup )

mysql -u [user] -p[password] [database] < cr_default_events.sql
  1. Local: Replay the projection

Replay the content graph (drops previous tables and setup)

./flow crupgrade:resetupandreplaycontentgraph --force

Replay another projection - to speedup and avoid interference, please disable all catchup hooks configured by setting to NULL:

./flow subscription:replay [subscription]
  1. Local: Dump the projection insert SQL

Get all table names from the projection. In the content graph all prefixed with cr_default_p_graph_

mysqldump -u [user] -p[password] [database] --no-create-info --compact \
  cr_default_p_graph_contentstream \
  cr_default_p_graph_contentstreamlayer \
  cr_default_p_graph_dimensionspacepoints \
  cr_default_p_graph_hierarchyrelation \
  cr_default_p_graph_node \
  cr_default_p_graph_referencerelation \
  cr_default_p_graph_workspace \
  > content-graph-replayed.sql
  1. Local: Compact the dump into content-graph-replayed.sql.tar.gz
tar czfP content-graph-replayed.sql.tar.gz content-graph-replayed.sql
  1. Local: Upload the replayed projection
scp content-graph-replayed.sql.tar.gz user@my.domain.net:/application/content-graph-replayed.sql.tar.gz

With jump host for beach:

scp -J beach@xxx.flownative.cloud content-graph-replayed.sql.tar.gz beach@instance-xxx:/application/content-graph-replayed.sql.tar.gz
  1. Remote: Extract the replayed projection
tar -xzvf content-graph-replayed.sql.tar.gz content-graph-replayed.sql
  1. Remote: Drop projection tables and import dump

Drop all cr_default_p_graph_ tables and run ./flow cr:setup

In the empty tables import the dump

mysql -u [user] -p[password] [database] < content-graph-replayed.sql
  1. Remote: Validate the events and state

The highest sequence number for the events MUST still be the one acquired in Step 0. Otherwise a concurrent change happened which is not accounted for here. The replayed projection would be outdated and the system would not know.

./flow cr:status