Neos API Proposal - Current State and Next Steps

Dear Neos Community and Neos Team,

tl;dr: our way towards a standardized Neos API; plus using neos/jsonschema for exposing validation rules of core DTOs

After this year’s Neos Conference, Bastian and myself revived our efforts about building an official Neos API (and later a CLI tool). So far we improved the tooling, and are working right now on the API.

We started exploring multiple ways, but finally settled on an OpenAPI compatible API now, as this is the most common denominator of current APIs right now. We then joined efforts with Daniel Kestler, because he also built a Neos API as part of Neos Studio - this gives us a real world API we can explore and improve, so this way we will be quite certain that all relevant use cases will be covered.

What we built so far

As we did not find any type-safe, code-first OpenAPI Generator in PHP, we built our own:

This allows us to build APIs which are fully type-safe and complete, i.e. where we can guarantee that the published spec matches the implementation, including error conditions.

(Personal side-note: This fixes IMHO one of the most annoying pain points of the usual OpenAPI server generators - and it’s all simple Library packages; so this can be easily used outside Neos.)

These packages also solve composability; so it is possible to extend the existing API in a planned and well-defined manner.

Next Steps (short term)

  • Move above packages to the Neos organization on Github (they already use the neos namespace but were created under Bastians account for the PoC phase)
  • We’d like to adjust the Neos Core Value Objects to depend on neos-jsonschema, so that means that constraints like a NodeAggregateId has a certain format or a DimensionValue has a given limitation will be exposed using JSON Schema.
  • This means that the core will have a (small) dependency to neos/jsonschema. We do not want to introduce this lightly, but Bastian and myself think this would be worth doing for API consistency; and generally as best practice for type-safe code.

All other features do not need any Neos Core adjustments, but will happen in separate packages:

  • We would like to have proper oAuth server support in the Neos Core, including support for fine-grained oAuth scopes - currently prototyping this via neos/oauth
  • We’ll introduce neos/api (name TBD) which implements the core Neos API
  • We’ll ask for a proper review once we are happy with the code quality of these packages :slight_smile:

We’re also planning a budget proposal to push this topic forward more swiftly.

Next Steps (medium term)

After the core API is ready in v1, there are two branches forward:

  • starting an official Neos CLI tool which interacts with this API.
  • seeing how we can expose this functionality via MCP as well.

Please add feedback by answering here and/or adding some reaction emoji :slight_smile:

Thanks and all the best,
Sebastian

6 Likes

I like the approach. We decided to go code first + OpenApi ourselves when we did very similar things for project apis & dtos with GitHub - sitegeist/Sitegeist.SchemeOnYou: A OpenApi integration for Neos.Flow. See also https://github.com/api-platform/json-schema/ · GitHub.

Instead of a dependency we used a set of rules + some Attributes for dto php objects but having json-schema as small dependency should allow to specify things that go beyond php types. Important to us was that those objects have to enforce all rules themselves so it is impossibe to instantiate an item that does not satisfy its schema.

If you need any feedback @Nezaniel and myself are very interested.

1 Like

Hi there,

first of all thank you a lot for your effort, I really like the general direction this is taking. It contains an approach for tackling limitations of the PHP type system, which is important.

What I’d like to discuss is some implementation details and I’d like to hear your reasoning on this because you probably already talked about this.

Having worked intensively with the aforementioned SchemeOnYou package, which relies completely on the PHP type system and Reflection and thus neglects finer grained constraints like string format or number ranges, I’d like to compare the approaches to get a clearer view on this.

I’m used to write my DTOs as follows (Starship content for example):

final readonly class SetAlertStatusAndRedirect
{
    public function __construct(
        public StarshipRegistryNumber $registryNumber,
        public ScopedAlertStatus $status,
        public ?NodeAggregateId $targetConsoleId = null,
    ) {
    }
}

with ScopedAlertStatus again being a complex nested DTO.

These objects are then directly passed to an endpoint:

...
    public function postAction(
        #[RequestBody(RequestBodyContentType::CONTENT_TYPE_JSON)]
        SetAlertStatusAndRedirect $command
    ): AlertStatusWasSet|AlertStatusSettingFailed|ResponseInterface {
...

If I understood correctly, to make this work with the schema based approach, I’d have to additionally implement something similar to

...
    public static function schema(): ObjectSchema {
        return AutoDiscoveringSchema::analyze(self::class);
    }
...

This gives me the opportunity to add additional validation rules in the members (e.g. StarshipRegistryNumber) and potentially for primitive members if the result AutoDiscoveringSchema::analyze is or gets processable.

I think this is very doable in general and I have a few followup thoughts:
It should be possible to just skip implementing ::schema, whoever de/serializes the object could automatically fall back to AutoDiscoveringSchema.

Would it be possible to expose the schema classes as attributes? Then I could do something similar to

final readonly class SomeDto
{
    public function __construct(
        #[StringSchema(pattern: 'someregex')]
        public string $value,
    ) {
        AutoDiscoveringSchema::validate(self::class, func_get_args());
    }
}

This would allow colocation of schemas and properties, reducing the risk of mismatches and easier implementation of phpstan rules to check whether I accidentally declared an IntegerSchema on a string property. It would also express the character of an addition to the PHP type system, not a replacement. And I would be rid of the interface :innocent:

It would also instantly work with future additions to the PHP type system; given we eventually got something like string<someregex>, I just needed to drop the attribute and schema call:

final readonly class SomeDto
{
    public function __construct(
        public string<someregex> $value,
    ) {
    }
}

and be done.

WDYT?

1 Like

@mficzel & @Nezaniel thanks a lot for your valuable feedback, greatly appreciated!

We certainly looked at SchemeOnYou and my types package and took a lot of inspiration from it (we shall attribute both as prior art!).

We decided to slightly deviate from the other approaches in a couple of specific cases with the following heuristics:

  • As little coupling as possible
    • i.e. not one Swiss Army Knife package, but a slim one that only provides simple ways to define schemas and which we would be fine adding a dependency to in core code. And then higher level abstractions for type conversion/coercion/API definition for the outer layers
    • not limited to (HTTP) APIs but a “general extension of the PHP type system” (e.g. it should be possible to do SomeStringVO::schema()->maxLength to determine the required size of a corresponding db schema, etc.)
  • As explicit as possible
    • it should be easy to understand the behavior even if one does not know the packages in detail, i.e. as little magic as possible
  • Make invalid states impossible
    • basically what you, @mficzel, wrote above: “it is impossibe to instantiate an item that does not satisfy its schema”
    • not entirely possible with PHP (e.g. we can’t prevent someone from writing a VO that does not enforce the schema, but we could add some detections – I did so with a dedicated PHPStan extension for my package
    • In my package, I introduced a global instantiate() function that one would have to use to safely enforce schemas, but with the new version this is much more explicit

If I understood correctly, to make this work with the schema based approach

I don’t think that we should expose commands/entities to the API layer directly in the Neos API case. But, yes, technically your example would work!

It should be possible to just skip implementing ::schema

Good call! I also didn’t require any attribute on “Shape” objects (aka records) in my types package since all their members validate themselves.

Here’s a first attempt at this: FEATURE: Fall back to AutoDiscoveringSchema for classes without ProvidesSchema by bwaidelich · Pull Request #2 · bwaidelich/neos-schematic · GitHub

Would it be possible to expose the schema classes as attributes?

That’s something I would advice against with the following reasoning: If a type gets any special rule beyond the scalar types (regex pattern, length restriction, …) it deserves to be an explicit type IMO. Otherwise it can’t really be reused or referenced via schema components.

In your example, if the SomeDto represents a single value, you’d have to do

final readonly class SomeDto implements ProvidesSchema
{
    private function __construct(
        public string $value,
    ) {
    }

    public static function schema(): StringSchema
    {
        return StringSchema::create(pattern: 'someregex');
    }
}

at least to make it work.

Or, if it was in fact a record, something like

final readonly class SomeDto
{
    public function __construct(
        public TheValueVo $value,
        public string $unconstrainedString,
    ) {
    }
}

Having said that, I just proposed to introduce an optional attribute that allows to give classes and instances (e.g. in constructor params) a dedicated description.

I think I probably misunderstood your example.
For single-value Value Objects, the attribute syntax would work (the schema name would be SomeDto).

However, I would still advice against this route since it would make auto discovery and instantiation quite a bit more “expensive” since it would require multi-step reflection into the constructor params.
Even if moving the attribute to the class (like I did in the types package) turned out to be that notch too much magic (for my personal taste at least).
I was tempted to introduce a caching layer just for the reflection part.. and this felt like a path into darkness :slight_smile:

Having implemented multi-step reflection at least thrice now (see SchemeOnYou, the component cache in Repositories (i.e., file:// or a. and the OPGM itself: Between these matches is considered to match. ) I’d say its very well manageable in terms of complexity and performance. I shared your concern back then and tried to measure how long it took to unserialze a component or map a node in horror movie-grade deeply nested edge cases. I failed miserably; what my operating system did in the background had orders of magnitude more impact than my nesting level. Reflection in PHP8 is just so fast that you can basically neglect the runtime performance aspect, no cache or whatsoever required.

What I love about the attribute-based approach is the following:

final readonly class SomeDto {
    public function __construct(
        #[StringSchema(pattern: 'someRegex')] /* here is the pattern */
        public /* here is the string|null union */ ?string $something = null; /* here is the declaration as optional*/
    ) {
    }
}

This is the bare minimum of coupling to the Schema: Everything derivable from the type system is derived from it, additions like the pattern are opt-in.

The same level of ease of use - and safety - can hardly, if at all, be achieved without reflection, can it? As ::schema() ist always userland, it needs (more) knowledge about the schema implementation. The best I could do would be something similar to AutoDiscoveringSchema::analyze(self::class)->merge(StringSchema(property: 'something', pattern: 'someRegex')), which is already both more verbose and error-prone.
Moving that effort from userland to the package imho is very much worth the complexity; It is completely deterministic and thus testable.

I wasn’t so much concerned about runtime performance, more about complexity in the code paths and memory usage.

Regarding the latter we suggest to write

public static function schema(): StringSchema
{
  static $schema = null;
  return $schema ??= StringSchema::create(minLength: 1, pattern: '^[a-z]+$');
}

as to prevent an object instance explosion for types that occur multiple times in one request/response.

I don’t know how we could prevent this with “attribute magic” without having to introduce another cache layer.

I get you’re point regarding “ease of use”. But having only a single way of describing a schema can also be an argument in favor of DX.

Anyways, I have no hard feelings against adding those attributes.
But if we do, I’d vote to add them on the class not its properties:

#[StringSchema(pattern: 'someRegex')] 
final readonly class SomeDto {
    public function __construct(
        public string $value,
    ) {
        // validate
    }
}

to clarify that this is not to be used for Shapes/Records (for the reasons in my original response)

Ah, yes! Restricting the attributes to classes enforces good design, we should absolutely do this. So the - for sake of the argument - simplified application level command (the domain level one is not exposed but derived, just as a side note) would be

final readonly class TriggerRedAlertAndRedirect
{
    public function __construct(
        public StarshipRegistryNumber $registryNumber,
        public ?NodeAggregateId $targetConsoleId = null,
    ) {
    }
}

with

#[StringSchema(pattern: 'someRegex')]
final readonly class StarshipRegistryNumber
{
    public function __construct(
        public string $value,
    ) {
    }
}

and

#[StringSchema(pattern: 'someRegex')]
final readonly class NodeAggregateId
{
    public function __construct(
        public string $value,
    ) {
    }
}

As for caching: using the language level static cache is super convenient, agreed; but actually using it is again in userland code :frowning: .
I don’t think it’s super important as requests/responses should™ not be gigantic beasts with millions of objects.
That said, we could still have a runtime cache like

final class SchemaRegistry {
    /**
     * @var array<class-string<object>,?Schema>
     */
    private static array $instances;

    public static function tryFromClassReflector(\ReflectionClass $reflector): ?Schema
    {
        self::$instances[$reflector->getName()] ??= self::extractFromClassReflector($reflector);

        return self::$instances[$reflector->getName()];
    }
}

Yes, this has a runtime cache and caches are bad™ in general; to its defense I can say it is rather trivial and userland cannot forget to use it. So it’s potentially even better than the native one.

Well… I’d have no objections on scrapping the interface and go all in on attributes and reflection :innocent:

You’re right about the runtime caches and the attributes idea is growing on me :slight_smile:

Regarding your examples, those are not entirely correct IMO because they would not prevent invalid instances from being created and this a must I think:

#[StringSchema(pattern: 'someRegex')]
final readonly class StarshipRegistryNumber
{
    public function __construct(
        public string $value,
    ) {
        AutoDiscoveringSchema::analyze(self::class)->validate($value)->throwIfInvalid();
    }
}

should work though (or a shortcut of it, like you suggested above)

@sebastian what do you think?

Yes, you’re right, I forgot the validation :smiley: This obviously has to be phpstanned, which should not be too hard as it only affects the constructor.

btw: Sorry for getting so deep into technical discussions – we should probably move these to GitHub or Slack => Assessment: Introduce attributes to represent schemata · Issue #4 · bwaidelich/neos-schematic · GitHub

1 Like

Since the main point of discussion seems to be additional Validation for String / Int / Float ValueObjects on top of php types we could provide 3 very simple base classes for that.

abstract class StringValueObject {
   protected const REGEX = null;
   protected const MIN_LENGTH = null;
   protected const MAX_LENGTH = null;

   final public function __construct(public readonly string $value) 
   {
       if (static::REGEX && !preg_match(static::REGEX, $value)) {
            throw new \Exception('regex did not match');
       }
       // validate constraints
   }  

   final public function schema(): Schema
   {
      // create schema for constraints
   }
}

So one can create a string based value object by only specifying the rules

final class StarshipRegistryNumber extends StringValueObject
{
    protected const REGEX = "/NCC-.*/"  
}

So far we mainly looked into the schema definition and i am pretty sure we will find a good way for that.

The custom-middleware approach in neos/openapi is very bold and looks really nice. However i think we should really discuss wether we want to establish that as a pattern as it completely circumvents flow routing.

Maybe an configuration based approach with pathPrefixes could be an option that scales better when a project has a dozen different APIs. In the namespace neos/openapi i would expect a generic solution that also supports project APIs.

I am fine with eventually kicking flow routing but it should be discussed wether we want to go the “there is no routing write middlewares” way.

I agree that we should consider Flow Routing and GitHub - bwaidelich/Neos.OpenApi.FlowAdapter: Serve neos/openapi APIs through Flow · GitHub shows a potential approach (PoC!) on top of the lowlevel openapi package (that does not depend on Flow/Neos)

1 Like

Totally like the approach in the flow adapter package. Really elegant that the routes and docs, specs are handled in one place.

1 Like

Are there any reusable best practice insights from @daniellienert’s gRPC API?
e.g. useful endpoints and sets of data …

Good question!
I haven’t seen the details of Daniels API yet (is it publicly available?) but there’s a general point to be made regarding gRPC etc:

gRPC might be another sensible API protocol (as well as tRCP, JSON-RPC, GraphQL, Websockets/SSE, …).

We decided to go for REST + OpenAPI for now since it provides the greatest level of support on Server and Client, i.e. we should not require Neos to provide a gRPC server (which is not possible without 3rd parties like Roadrunner AFAIK).

Having said that, the goal of these libraries is to make it very easy to provide custom endpoints/protocols once the core APIs and types are explicit and safe!

Right, gRPC is not what I’d consider, either for API discoverability.
I did have a quick look at TypeAPI/Typeschema by Chris Kappenstein https://chrisk.app/

It would be interesting to see if any of Daniel’s experience with practical requirements and challenges of interprocess communication can be transfered.

The system Daniel showed was not based on Neos/Flow, it was based on another PHP framework (spiral/roadrunner) so this is not really applicable in this case. And yes, roadrunner providing the necessary tooling for gRPC was essential in making this work

My question is rather about the implementation-agnostic learnings.