RFC: Flow / Neos Metric Storage & Reporting

I start a pet project to generate metrics in a Flow / Neos application, the main idea is to have a generic solution solution for any kind of metrics. This can be useful for debugging performance, but also for content performance monitoring and other analytics tools.

As this topic is highly complex and can generate tons of data the solution need to be as modular as possible. For a big project it’s not an option to use MySQL for this kind of storage and ELK stack can be a nice choice.

So first thing first, I need to build a team, this pet project is highly too big for me and too big for my free time. If you feel intersted by the topic or know a client/agency that need this kind of tooling feel free to comment this topic and let’s start how we can build a nice solution.

Currrently i work a prototype ( https://github.com/ttreeagency/Flow.DevHelper ). The name is stupid, I understand the more generic usage of this only recently :wink: especially during the reading of this article http://www.cmswire.com/web-cms/7-web-cms-trends-for-2016-and-beyond/

Currently the package focus on a central point to register metrics, at the end of the request all metrics a converted to logfmt format (plain text) and stored in a local log file. The storage emit a signal when flushing the metrics, so an other package or a suite of package can forward this log stream to a queue system or store them in an other format in MySQL for further analytics.

Each Metric is a small class that gather information on creation, it’s really a value object once it’s created it can not be modified. You can check a simple example for a MemoryUsageMetric ( https://github.com/ttreeagency/Flow.DevHelper/blob/master/Classes/Domain/Model/MemoryUsageMetric.php )

The next step will be to generate some kind of RequestAggregate to bind all metric per request. The RequestAggregate will store information regarding user session, remote ip address, … Further analytics (geoip, aggregate more metadata) should be handle later in the log stream post processing. The current challenge is the include in the RequestAggregate all subrequest (compile time or other) to have a clear overview of the experience of the user during this request. The RequestAggregate must not be “web centric” and need to be usable in the context of an API call from an external system too.

The final workflow pipeline should be something like this:

  1. Trackers (JS or Aspect to instrument the Flow/Neos internals)
  2. Collectors (basically my current prototype can be called LogFmtCollector, we can imagine some kind of collector that send data in small file to an S3 bucket for further analysis)
  3. Extenders Enrich metric generated by the Collectors (geoip, attach user meta data, …)
  4. Stores Store the data some where (SQL DB, S3, NO SQL, ELK)
  5. Data modeling / Analytics Transform the extended events in something useful and readable by the enduser (understand nice graph, dash board, …) Can be done in external tools / or internally … in CQRS terms the goal is the generate read model for fast presentation of content.

Every block of the pipeline should be fully decoupled. We can have different strategy (scheduled job take collected events every X hours, push then to extenders and store) or something based on NATSIO to handle everything in async way but limiting the delay between collection an analysis.

Look like a cool project to work on, and I think we need to put more dedication in the direction of content performance and this kind of tools can help a lots here.

The goal is not to rebuild Google Analytics we can even imagine to use GA as a store by connecting to their API. But think that many client can not use GA (data outside their own country, privacy, …).

Information about logfmt: https://brandur.org/logfmt
Tooling to work with logfmt: https://github.com/brandur/hutils