Blackfire profiling in CI pipeline

Sometimes we discover regressions in performance in Flow or Neos and most of the time this is by chance. We seldomly actively look for performance issues and we don’t measure performance continuously while Flow and Neos are being developed.

Therefore, I wondered if we could integrate performance measurements into our CI pipeline / Github Actions? I don’t have a lot of first-hand experience with automating this with Github, but to me it sounds like its worth the effort.

In order to make this more attractive, I asked Blackfire / platform.sh if they could sponsor a license to the Neos project. And I just got the confirmation that they will! I’ve got all the instructions here to set up the account accordingly.

Now, here are some questions for you:

  • do you think that integrating performance tests will benefit the development of Neos and Flow?
  • are you interested in leading or contributing to this effort, setting up Blackfire for the Neos CI pipeline?

Right now (June) I don’t have any time left which I could spend on this topic, but maybe someone in the team finds this interesting enough to take the first steps.

3 Likes

While I totally like the idea my personal experience with performance metrics is a bit mixed.

Most times I tried I got quite big fluctuations in the results. While beeing interesting to watch over time I always struggled to define criteria’s for passing / failing tests.

So the question to me is how reproducible the numbers of blackfire really are.

1 Like