Although I’d much prefer GraphQL instead of REST, I agree that it might be too big a task. Pretty cool that Drupal has it. Did you use a library like https://github.com/webonyx/graphql-php to implement it or?
1 Like
Hey there.
Here’s just a view thoughts about implementing “kind of jsonapi.org”.
To me there are couple different things on the agenda.
Feels like a first prototype of this can be implemented as a package without any changes to the core.
Dispatcher request looping over both, the requested data as well as all included relations.
Let’s talk about it as a root request ending up in a controller performing sub requests.
I’m not completely sure if using a controller for this is the best idea or getting the Http Component involved is better. But for the sake of not getting too complex in the first shot, I’m talking of the root request as ending up in a controller.
The idea
The RootRequest gets resolved to an JsonrgApiController where an empty array is created.
The actual “thing to do” of the RootRequest is transformed to a WorkerRequest.
Every WorkerRequest gets executed separately, “while (count($workerRequests))”.
Whenever a WorkerRequest holds relations to other objects, a new WorkerRequest is created and pushed to the $workerRequests array.
The benefit
As soon as all relations are handled as subrequests, every subrequest can be cached individually.
Even the actual implementation of how those subrequests are handled can be adjusted per project, depending on the available software environment.
The very basic idea is just using Flow subrequests, but this could be enhanced to e.g. doing real HTTP web requests targeting a varnish proxy, or whatever.
The thing is: All relations are handled not nested but iterative and independent, providing a distinct spot where caching is supposed to go later.
DTOs wrap domain objects that provide configuration by PHP code
Not targeting jsonapi.org but to be used with the $resource mechanism of AnuglarJS, I did such a thing for Extbase and ported it to Flow some time later.
This shure needs some improvement in general and some adjustment in order to match the jsonapi.org requirement, but I’d like to think of it as an idea that has proven to be not too bad.
https://github.com/netlogix/Netlogix.Crud
Think about something like this:
https://github.com/netlogix/Netlogix.Crud/blob/master/Classes/Netlogix/Crud/Domain/Model/DataTransfer/AbstractDataTransferObject.php
All property names meant to be available externally are named separately and provided by a getter of the Dto.
This can be enhanced by privilege mechanism, context checks and so forth.
I’d leave it to a plain getter method in the AbstractDto, so it can be easily implemented per distinct Dto class. This leaves room for complexity when its necessary but no complex configuration required.
They are both at the same time, input and output converters. Just as they provide every information that should be exposed to the public in getter methods, they can cover all the code needed to transform the external data to internal data in setter methods.
The easiest way, of course, is to just pass getters and setters right through the payload object. And again: Room for complex mapping if necessary, but no mapping at all required by default.
The object can be used to alias internal attribute names to external attribute names, simply by providing getter methods and setter methods respectively. They can even be used to travers through nested objects. Think about “person.identifier” which could be a getter method calling “return $this->getPayload()->getAccount()->getIdentifier()”.
In contrast to my implementation available on Github, those objects should not be used to expose nested structures.
Lots of magic of my Github package is related to providing a deeply nested array that can be passed to json_encode. This, of course, is obsolet when it comes to jsonapi.org and should be replaced by “relationships”.
How to map a DTO class
I suggest introducing DtoConverters.
Just like TypeConverters, they should be have a priority and asked for “canConvert” by providing a distinct model object.
Mapping Model objects to Dto objects
That’s the easy part. In general, one can simply create a new Dto($payload). Of course that’s the obligation of the DtoConverter, but it’s no big deal.
Build URIs
The more complex task of DtoConverts should be to provide enough information to create absolute resource URIs pointing to the API endpoint for an individual Domain object or Dto.
Those are meant to be used for both, creating the exposed public URI for “links.self” as well as filling the HTTP sub request properly that acts as the WorkerRequest for related objects.
Creating a view
A JsonView dedicated to Dtos can easily iterate through all properties named by Dto.propertyNamesToBeApiExposed.
Every scalar value goes into an associative array.
Every object goes back to the list of WorkerRequests.
Every other object throws an exception.
Sparse Fieldset
There’s that nice feature of a Sparse Fieldset in the API documentation.
I’d like to handle that as a “post processing of a complete result”. Instead of passing all of this information to the view and let the view limit the “properties to be api exposed”, I’d rather let the view create a complete response and have the RootRequest apply those restrictions, in favor of advanced caching on one hand and a single point of implementing it on the other.
Open glitches
The jsonapi.org requires the “MM relation data” to be exposed.
http://jsonapi.org/format/#fetching-relationships
This means there need to be “relationship” routes that expose the inverse side of the association as well as some information about what owning side it belongs to.
I don’t really know how to tackle this. Maybe we could treat that “MUST” as a “MAY” and postpone this to an improved version.
Response
… well, not talking about the API response. If some of you have doubts or ideals, I’d really love to hear back. I might start creating a little prototype as soon as I have some minutes to spare :).
Regards,
Stephan.
2 Likes
We work on a generic package to have entity based JSON API:
https://github.com/ttreeagency/JsonApi
Adding CR support will be doable.
Hi Stephan,
sorry I hadn’t got the time to get back to you about this earlier. Did you already start with a prototype for this approach?
I totally like the idea of using DTOs around Flow entities (and pure objects) to specify the JSON API mapping. We should definitely use a whitelist approach (when exposing properties) for security here.
But I think we need further discussion around how to process / create the actual response. I see that this is very performance critical and must be done in a pretty efficient way. Especially with nodes / entities forming a graph it can happen that a single request results in a lot of SQL queries on the server.
Caching responses in Flow is another story that might be an option, but my first thought is, that the request processing in Flow might be too slow if the response has lots of relations.
One of the advantages of specific @include@ and @fields@ requests is, that the server doesn’t need to process additional relations / fields because traversing relations (and deep serialization) is the major performance bottleneck here.
I’d suggest to have the least abstraction that we need here and start with pretty specific custom views to collect the needed data (and with a very slim controller). We had a good experience in another project with this lightweight approach. I’ll post something about that in another topic later. We could think about going to the HttpComponent level here, that might give us a performance benefit (especially with read-only operations).
Anyway, thanks for your input and ideas here, it’s perfectly fine if you just implement a prototype following your initial idea and we see how it works out.
@christopher Check this library https://github.com/neomerx/json-api/ my package is based on this one. Framework agnostic, and nicely done.
Indeed, looks good as the base to generate the JSON API response and pretty similar to what we developed here.
2 Likes
Regarding a futur public API for the CR (not only for Neos Services), i think it should be really nice to use the same API as Contentful (like Mattermost did, use the Slack API and improve where possible, but basically have the future CR API compatible with Contentful can be nice. Not sure if it’s doable.
2 Likes
I like this idea to not re-invent the API wheel where possible.
I’m not deep into the topic, so Just wondering, why do we choose to follow Contenful and not Prismic or Garhtercontent?
We don’t choose anything currently, it’s just an example. We have other options, like this really pragmatic article Your CMS REST API should be WordPress Compatible … we can even support multiple API.
I will write an RFC next week about this topic.
The idea to adapt / improve an existing API has come up yesterday too when discussing the JS concepts / Neos architecture. What we’ll have to balance out is an API that works efficiently for the UI (JS client) and is generic enough for all the other use-cases. I’d not like to support multiple APIs from the start to keep things simpler (formats are okay, but I mean fundamentally different).
1 Like
Hey there.
I had quite some time now to think about it and give it a little try.
That’s basically what I came up with:
The example data package only contains a bunch of classes:
- The entity “Node” is handled by regular ORM.
- A corresponding “NodeResource” class knows about attributes and relationships of the public API.
- A “NodeResourceInformation” targeting Package, Controller and Action for the responsible endpoint.
I think the best way to get my idea is to look at the example data package first and just keep the jsonapi.org document structure in mind.
Basically every document structure element is implemented as JsonSerializable, so exposing json_encode($resource) or json_encode($topLevel) is just enough for the read api.
What I described as DTO in my post from dec. 22 just became the Resource class described in jsonapi.org document structure. A Resource, according to jsonapi.org, is the element having an identity, attributes and relations. So using that exaclt class as DTO seemd to be a good idea to me.
What I described as DtoConverter in my post from december got a ResourceInformation since it doesn’t only convert from Entity to Resource but it also provides endpoint information builds endpoint URIs.
Opposed to my former idea of using sub requests, and since I implemented a TypeConverter to convert the jsonapi.org resource array to a PHP Resource object by its type and id property anyway, I used that converter for creating the expected result of sub requests, too.
I didn’t completely drop the idea of having sub requests. But since my first attempt of using real curl based HTTP request for that was literally ten times slower than mapping the default setup uses mapping.
Regards,
Stephan.
Hey again.
I don’t want to amend my former post another time but keep it as a description what I did differently compared to my post from december.
This one is, in contrast, to explain what I did in general, since this came up currently. I think I need to go deeper into
- how the jsonapi.org document structure works and
- how its schema objects interact in my implemetation.
I basically implemented every object defined in the document structure providing the actual schema structure and required/optional attributes as an individual class. No schema object is dynamically, implicitly constructed or by just knowing array keys . Of course when it comes to e.g. the Meta object or the Links this seems a bit overengineered, but dropping existing classes, using arrays instead and reducing the footprint can be done at any time I guess.
Every schema object in my implementation can json_encode() itself.
The actual question asked was: How does an exposed relationship know about its self link and thus can create that URI in the “links” structure? More to that later.
How jsonapi.org connects on the server side
The general response element of the jsonapi.org document structure is the TopLevel object.
The most important result parts of the TopLevel object is data which represents the business content of a particular request and optional field links, that can contain the currently request URI as well as e.g. next and last to allow pagination. So the TopLevel more or less represents meta structure of a request. It does not directly contain any payload data.
The pretty awesome third important but unfortunately optional (in terms of "may not even be implemented on client side or server side) feature of the TopLevel object is the included array.
The TopLevel is the desired response type for listAction, showAction and showRelatedAction, so basically every action that is meant to result not meta data or structures but business data.
Resources
The data field of a TopLevel can either be a single Resource in case a single result is expected (e.g. from the showAction. But no matter if the TopLevel object returns an array or a single one, a Response object is always constructed the same way.
A resource object brings its id and type to be identified uniquely, as well as attributes, links and relationships.
This means: The resource needs to know about a single business data object, which makes it the level I choose as DTO.
Resource-Attributes
The attributes property of a Resource object is an Attributes object.
According to the jsonapi.org specification, this one is only a key => value store.
BUT: I implemented it as an Attributes class knowing its Resource object. So every getOffset() and setOffset() call (in terms of \ArrayAccess) is internally passed to the Resource, which makes the Attributes object not contain any payload data itself but only a mapper object for creating the jsonapi.org hierarchy.
Resource-Relationships
The relationships property of a Resource object is an Relationships object.
Just like the Attributes object, the Relationships object does not contain any payload data but just acts as a wrapper to access its parent Resource object.
When being json_encoded, its result is not plain key => value but a rather simple object exposing a self link (links to the metadata of the relationship), a related link (links to the business data being meant to be exposed by that relationship) and optional data which embeds the id and typeof one or more linked data.
The Relationships response is used to provided the output data for the showRelationshipsAction. Not the whole thing is exposed but $resource->relationships[$relationshipName] falling into the \ArrayAccess getter.
Which thingy builds URIs?
I added a class called ResourceInformation.
It’s one and only purpose is: Know a single implementation of the abstract class ResourceInformation to handle and which combination of Package, Subpackage, Controller, Action and Arguments have to be used to create a public URI for that resource type (the self link of that Resource class) or the public related URI for relationships of that resource type (the related link of a relationship of that Resource class).
This means:
- All links are created by the
ResourceInformationby using anUriBuilderwith proper arguments.
Theselflinks ofResourcesis initiated right inside of theResourceclass, considering the payload of theResource. - The
selflinks andrelatedlinks ofRelationshipsare initiated right inside of theRelationships, considering both, the payload of the parentResourceclass as well as the relationship name.
There is obviously room for improvement.
The NodeResourceInformation I added to my ExampleData package shows the fact that the ResourceInformation object is the one thing to know controller argument names. If a controller does not use custom argument names but sticks to $resource just like the abstract ApiEndpoint starts with, I could think about a GenericResourceInformation object getting its mapping from a Settings.yaml instead of object attributes.
What’s the controller made for?
My JsonApiOrg package brings an abstract ApiController that only reroutes indexAction() calls to other actions, depending on available request arguments and the given request method.
I intended to use a distinct controller class for every Resource Type handled through this API.
First reason: This allows for security constraints through Policy.yaml based on resource types.
Second reason: Meaningful routes are easily set up.
Every Resource of the jsonapi.org schema always comes with its type attribute. If a single controller is implemented to take every abstract Resource instead of a distinct extension of it, there is no mechanism in place that prevents e.g. Account objects to be added where Node objects are meant to arrive. The first situation such a misuse of an API endpoint catches and hopefully fails would be the NodeRepository, hence the persistence. But making such a thing only based on the framework internals throw proper type mismatch exceptions doesn’t seem right to me.
So I strongly suggest to create distinct, routable, policy-protectable and “one concern only” endpoints.
Things to implement in a custom package
I tried to keep my ExampleData as short as possible.
- The
Domain\Model\Nodewould be the business model, so that’s not a requirement for the jsonapi.org structure but the business data one would expose. - The
Controller\NodeControlleronly takes JsonApiOrg schema data and implements every GET, POST, PUT and DELETE request the object type should support. - The
Domain\Model\Dto\NodeResourceInformationpoints to thatNodeController - The
Domain\Model\Dto\NodeResourceprovides attribute names and relationship names to be exposed by the API. - In the
Objects.yaml, the internal classNetlogix\JsonApiOrg\ExampleData\Domain\Model\Nodeis always mapped to the stringnodeswhen exposing thetypeproperty of theResourceaccording to the the jsonapi.org schema. - In the
Routes.yaml, all routes target the indexAction.
I’d be glad to receive your feedback, especially since this is kind of a completely different approach to that jsonapi.org idea than @dfeyer implemented.
Regards,
Stephan.
Hello folks,
i’ll try to put my own two cents as understandable as possible.
By reading the posts here didn’t get “aggregate root” keyword.
So i’m thinking on Flow api highly dependent on jsonapi.org with Flow magic for aggregate roots.
So as perfect solution for Flow developers i see defining action controllers for agregate roots only. The JsonApiAggregateRootControllers generate JSON-Api compatible JSON.
So non-aggragate Entities MUST be contained in the JSON data (NOT in “relationships”) in associative way. Using PATCH to manipulate properties in the aggregate tree by URL
PATCH /blog/flow-is-avesome/post/45 HTTP/1.1
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "post",
"id": "45",
"attributes": {
"title": "DDD in JSON API",
"content": "………………………………………………………………………………………"
}
}
}
or as PATH in the JSON
PATCH /blog/flow-is-avesome HTTP/1.1
Content-Type: application/vnd.api+json
Accept: application/vnd.api+json
{
"data": {
"type": "blog",
"id": "flow-is-avesome",
"attributes": {
"posts": {
45: {
"title": "DDD in JSON API",
"content": "………………………………………………………………………………………"
}
}
}
}
}
Relationships, which are aggragates MUST be contained in “relationships” and CRUD manipulations follow Json-API standard.
Routing: Defining routing only for Aggregates.
Decision about Read/Write accessing could be configurable but using setter/getter/add/remove/etc. from Domain Model classes should be default.
So i think on AbstractAggregateRootApiController with some service classes, which provides JSON-Api functionalities with possibility to override magic methods like in repository.
I hope you got my idea.
If we can spend sometimes during the next code sprint to discuss all those API topics, could be nice to write a first RFC Draft, and continue the discussion based on that.
2 Likes
Hi folks and sorry for digging out this old thread.
As a little disclaimer: The reason to use HTML as Hypermedia format originated in an idea of Jon Moore (one of the Hypermedia gurus out there).
I still find the idea strikingly persuasive but I also see some drawbacks at least with todays constraints.
In the meantime I played a lot around with different APIs and had the chance to test different media types in complex scenarios.
Recently I had a close look at JSON API. I had high expectations that this would finally end the “API format wars” given it’s written by some very clever REST experts.
I have to admit: I didn’t get along with it very well (and I really tried!).
To cut a long story short: My favorite is currently still HAL as it seems to hit a good balance between pragmatic and over-sophisticated.
Regarding hypermedia controls: Yes, HAL is missing a notion of HTTP method for links. But so is JSON API. And I don’t think it’s important – at least not until clients get so smart that they really only need the base URL of the API and explore all of their logic from there.
And there are extensions allowing for specifying forms including method and payload definition if that’s needed.
Having written all this I think that our main issue & challenge is to find a good abstraction level for our APIs.
I’d especially suggest not to fall into the CRUD-trap that will lead to high coupling and very complex code (as we already experience today).
I liked the idea of combining REST APIs with a CQRS-kind-of-style, described in these slides: http://de.slideshare.net/fatmuemoo/cqrs-api-v2
A (very simplified) API for the CR could look a bit like this:
{
"_links": {
"self": {
"href": "/api/nodes"
},
"curies": [
{
"href": "https://neos.io/rels/{rel}",
"name": "neos",
"templated": true
}
],
"neos:create": {
"href": "/api/nodes",
"rel": "command",
"title": "Create new node"
}
},
"_embedded": {
"neos:node": [
{
"id": "36a63168-2042-8ef7-3787-4a215f4dff75",
"type": "Some:Node.Type",
"properties": {
"text": "Some Text"
},
"_links": {
"neos:hide": {
"href": "/api/nodes/36a63168-2042-8ef7-3787-4a215f4dff75/hide",
"rel": "command",
"title": "Hide this node"
},
"neos:move": {
"href": "/api/nodes/36a63168-2042-8ef7-3787-4a215f4dff75/move",
"rel": "command",
"title": "Move this node"
}
}
}
]
}
}
The nice thing with this is that we could just omit the “command links” on the server if the current user hasn’t got the required privileges. The client / UI wouldn’t have to know about those, it would only show the “Hide”-Button if the neos:hide relation is found…
@bwaidelich hey, what do you think of the API written by @wbehncke for the new Neos UI? I think it’s similar to CQRSish example from your last example. And it goes well together with our Redux-driven client.
And yes, this kind of command api is cool for data manipulation. For data querying we use client-side implementation of FlowQuery. It’s easy to grasp, flexible and familiar to Neos developers.
https://github.com/PackageFactory/PackageFactory.Guevara/tree/master/Resources/Private/JavaScript/API
Is there a wrap-up somewhere that shows how this would be used by clients? The link only leads to some command classes as far as I can tell
For querying data REST APIs used to work quite well already. Using some DSL (like FlowQuery) makes sense for complex things IMO and as long as that’s encapsulated in some query parameters I don’t see a problem with this, e.g. /api/nodes?filter=<some-flow-query>
1 Like
Ah, sorry, I don’t think there are any docs for this yet, but here’s an example of how to create a node: https://github.com/PackageFactory/PackageFactory.Guevara/blob/master/Resources/Private/JavaScript/Host/Containers/AddNodeModal/index.js#L57
Basically, you have a stream of change objects, each change has a type, subject (contextPath of a node it’s acting upon) and a payload.
You POST arrays of change objects to API endpoint and for such request:
{
"changes": [
{
"type": "PackageFactory.Guevara:Create",
"subject": "/sites/neosdemotypo3org@user-dimaip;language=en_US",
"payload": {
"nodeType": "TYPO3.Neos.NodeTypes:Page",
"initialProperties": {
"title": "test"
}
}
}
]
}
you get this response:
{
"timestamp": "2016-03-16T14:51:37+0300",
"feedbacks": [
{
"type": "PackageFactory.Guevara:Info",
"description": "Information available",
"payload": {
"message": "1 change(s) successfully applied.",
"severity": "INFO"
}
}
]
}
Don’t know if what I write to you makes sense. If not, perhaps @wbehncke may provide a better explanation.
Hi all
@dimaip Thanks for the example, it’s much clearer for me now - And sorry for the lack of feedback.
In the meantime I deep-dived into this topic (including GraphQL and graph databases) and started to write a (very long) response. But I realized this would be better discussed in person (or at least in sync).
There are quite a few people currently working on related topics and we think it’s time to stick our heads together.
@andi started a doodle poll for such a meeting and it would be great to have you guys join it!
Everyone is welcome to join the meeting, but please fill out the poll so we can find other channels in case there are too many for a google hangout
3 Likes
Hi all,
for everyone who could not attend the hangout. Basti worte a short summary and we decided to collect our API and format ideas, maybe even small prototypes to eventually come to a general architecture/standard/format decision: RFC: Neos API - next steps
Would be great to add your proposals/findings there.
Greets Andi
1 Like