We just had an interesting discussion regarding next steps for a Neos API.
It’s obvious that the fully fledged API needs to cover all sorts of resources like User settings, resources and accounts. But for now we’d concentrate on the ContentRepository because that’s the most important (and probably most complex) domain.
In general we still think that the CR should be Event-sourced in the long run, as this would give us some very very neat opportunities. There are currently a couple of us experimenting different approaches regarding CQRS/ES and Flow. But it’s apparent that a CR rewrite won’t happen very soon.
However the UI rework is in full swing and we should not block that of course. So the idea is to come up with an API that is more flexible & consistent than what we have today. So we need to agree on a format and on the structure of such API.
tl;dr
Let’s collect some exemplary request/response pairs for some common CR interactions. Feel free to suggest something in whatever hypermedia format and don’t be afraid to be “wrong” or incomplete. This is just to get a better feeling and basis for further discussions.
What about remove, copy (to some context), listing nodetypes. Maybe also a render action using out of band rendering.
Did you already think about how the api should be split over packages? I would prefer it a lot if we could separate this cleanly from the beginning, meaning we have a CR API which is either in the CR package or a dedicated CR API package, and the Neos specific in Neos.
Remove is probably pretty trivial and copy very similar to move but feel free to add any scenario to the examples of course! Context is a good point. That’s what makes the whole “node” API so hard to model.
NodeTypes, Workspaces, Dimensions… all to come. This is mainly an attempt to get the discussion started and get a feeling for pros and cons of different approaches.
Not yet, but I think it should definitely go into a separate package
I just checked the api of an ember app we built, and I don’t see many extra examples. Security should of course be added later
One thing we struggled with was updating nodes that had linked attachments, this was mostly ember-data specific but maybe asset / file handling would be good to add to the examples to think about?
Hey @bwaidelich, sorry for missing the hangout and thanks for putting it up here.
I have a few points to add:
Following the CQRSish approach, it makes sense to separate quering API from command API.
I really like the format for command API that Wilhelm developed, don’t have anything to add to it.
For quering content repository, FlowQuery is the perfect candidate. It does all you have written above, except for handling nested data response. So far we don’t use it anywhere, but could be handy. Also I would add abillity to request partial data, smth like q(node).get('title, description').
I see a trend in web app development, where client gets more freedom in shaping the data it wants from the server (see GraphQL, Falcor). GraphQL has amazing ways of fetching nested data in one request, but it’s very app-centric, so if we want that for CR we really need to think out something different. For now FlowQuery seems like the best fit to me.
I spent quite some time to prepare some requests and responses. My goal was providing material for the jsonapi.org format.
I ended up with not only some HTTP headers and body strings but created a package working with an actual Neos (Demo site).
So all my data is not made up and not only my mind trying to interpret the jsonapi.org format but those are actually taken from the “Test RESTful Web Service” of my PhpStorm.
Please excuse that every now and then I talk about how to improve my current jsonapi.org implementation. But to me it’s really important to make clear lots of code in its current state was required to be written because my library in its current state is not as driven by conventions and smart guessing as I’d love it to be.
This resulted in me creating an enormous wall of text in addition to my code.
If you want only the plain reqest/response data, skip every introduction and footer and step through the numbered list.
I used five different files, one for each request/response task because pretty-print JSON tends to be spread over a huge number of lines.
The ResourceInformation provides information about how arguments are named and which controller/action is to be targeted for creating links. In a later step this could be an optional class. Instead, a default implementation could “guess” by FQCN just like currently a Repository guesses the Entity it is resonsible for.
The NodeResource is a read model wrapping an actual node. For nodes this is a pretty custom thing because nodes themself are created by configuration.
In a later step I can think of making both classes optional and have default implementations working on annotations. But for now there’s only this pretty explicit configuration. Especially since the CommandResource class uses a hole lot of configuration that just duplicates the object structure.
Of course there are Command objects. They don’t do anything except getting mapped. And executed by a controller. The rest is up to the cqrs package.
Once again: This controller is only related to nodes when it comes to argument naming and type hints required by the property mapper. Providing a generic ApiController responsible for all ApiInterface entities should be no probem.
I try to answer you point by point by my jsonapi.org point of view.
That’s nothing we discussed in our hangout, but to me that’s the way to go when jsonapi.org would be the format to use. Just take those as “my personal idea of what a natural implementation would look like”.
And after having spent several hours deeply into jsonapi.org im kind of a fanboy now.
That’s exaclty what I tried to do. The jsonapi.org uses a pretty RESTy approach. But that’s not a problem at all when it comes commands. More later.
For Nodes I dropped every writing part and only implemented the reading parts. So reading goes in a RESTy way of the known domain model. If you don’t want to write but only read (e.g. if you skip server side fluid rendering and make a client-only application), the read model looks pretty RESTy and follows the current node tree by 100%.
I tried to think of Commands as “Domain objects that only represent input arguments”. So I actually created a domain model class CreateNodeCommand that has a property “parentNode” and a property “type”. From the client perspective, the command execution looks like a RESTy “create new CreateNodeCommand object”. So it’s done by posting a command JSON to the CommandController::createAction().
This makes both channels, the read channel as well as the write channel, use the same format (jsonapi.org) although the domain content being transported is different.
Same goes for events. An event is, after all, just a domain model that can be exposed in a RESTy way to the client.
Of couse this concept is not bound to jsonapi.org. It’s not even bound to JSON. Instead, I’d suggest to go this way with whatever format we decide on. Treating commands as “command entities” kind of boils lot of things down to the a well known level.
There’s no “changes” and no “feedback” in jsonapi.org since that is not only something that changed or is a response. Instead, jsonapi.org just always calls that level “data”
The “type” exists in jsonapi.org in exactly the same way. It is just for being able to distinguish different object types, the actual value doesn’t matter to jsonapi.org. But as you can see by comparing your request and response, the “type” property differs, so there’s no need to make the top level property (changes and feedback) different as well.
The “subject” is called “id” in jsonapi.org. The content doesn’t matter to jsonapi.org. Uuids are prefered, but using node paths go very well, too.
What @wbehncke called “payload” is called “attributes” in jsonapi.org in case its trivial information or “relationships” in case those are aggregate roots.
But there is one important difference: Request and response by definition of jsonapi.org “target the same resource”. You’re not allowed to “post a Create object and receive an Info object in return”. If you’re posting a Create object, you always return either that very same Create object back or “204 No Content”.
In case CQRS with a distinct events channel is not possible, a way to circumvent the “input equals output” rule would be to make the info response a property of the command object.
Just give the Command object a property $info. After a command is executed, its $info array is to be filled with the response information.
Now you can POST the Command object not to /api/commands but to /api/commands?include=info. That would internally reach the very same CommandController::createAction() and in addition, the jsonapi.org view is automatically configured to include the info relation of the Command object.
And here you are: You get the exact same command as a response as you pushed in, enhanced by every information the creation process collected.
But to be honet, I wouldn’t go that way.
I strongly suggest to separate read channel, write channel and event channel.
The read channel is our RESTy way of GETing nodes from /api/nodes.
The write channel is our RESTy way of POSTing commands to /api/commands.
The event channel should be a RESTy way of GETing events from /api/events by polling.
The polling interval can be managed smart. I’m thinking of once every 30 seconds in general, 5 times with a reduced delay to 5 seconds right after a write command received its 200, 201, 204 or 400 status code. Call it"polling schedule influenced by request responses".
An advanced implementation could drop the polling and use a web socket for that. Since every tiny part of information of a jsonapi.org object is completely covered by the JSON payload and no URL is required to e.g. guess the type or the ID of a thing, using web sockets as a response channel is pretty easy in terms of the protocol.
The /api/events list can be either stateful or stateless. Stateful would leave it up to the user session which events should be returned to the client and which are already transported. Stateless would leave it up to the client asking only for events starting by a distinct sequence number the client provides.
I thought about that, too. But I’m not completely sure if that’s the power we want an API consumer to have.
Have a look at this one:
Sine it’s an Eel query string being evaluated anyway, making it a full input argument is an easy thing. But that would require really rock solid content security. By no means a consumer should be able to break out of its privileges just by traversing nodes.
The “nested data response” is up to the client if jsonapi.org is the selected format.
Just issue this query:
GET /api/nodes?filter[eel]=%24%7Bq(node).parent().find(%22%5Binstanceof%20TYPO3.Neos%3AShortcut%5D%22).get()%7D&include=childNodes,parents,workspace.user.party.electronicAddress
Accept: application/vnd.api+json
This request would
return every shortcut in a raw list as requested by the eel query,
have every childNode of those shortcuts included,
the rootline (“parents”) for the shortcut,
the workspace for the shortcut,
the corresponding user to the workspace,
the corresponding party to the user and
the corresponding electronicAddress to the party.
Once again this is a huge thing for content security, but it’s the actual feature set of basic jsonapi.org include arguments in conjunction with the Eel based search.
I find this an amazingly powerfull feature. Only we need a client that can put this to good use.
Hey @goli, thanks for the amazingly detailed reply!
I’ll try to let this info sink into my poor brain and reply by Monday, though I’m not clever enough to give any definitive feedback, @wbehncke would be much better suited for that
Just wanted to give a quick feedback on this:
That’s exactly my point: with advent of GraphQL and the like, the consumer has become more demanding in terms of shaping data format that it wants to receive. GraphQL is almost like giving the client an SQL access: do with it what you want, query for data in any way you like. And yes, as you said, we really must believe in our security layer (which I hope we do!) to do that.
One of the use-cases I have in mind, is completely ditching TypoScript2 for rendering (optionally ofc), and doing the rendering fully in (say) React. If we provide a full FlowQuery access to JavaScript world, it immediately becomes as suited for rendering as TypoScript2 (in which we get all data for rendering via FlowQuery as well).
And here’s an example of current format of FQ API:
JS call q('/sites/neosdemotypo3org/metamenu@user-admin;language=en_US').children('[instanceof TYPO3.Neos:Document]').get()
is transformed into GET request to /service/flow-query with payload:
@goli Thanks again for putting all the effort into this and thanks for talking me through some of the remarks/questions I had the other day!
I started to write a respond a couple of times but there are so many specifics to be argued that I would prefer a “synchronous discussion”… I’ll prepare something for the sprint next week
And one more thing, here’s a rough sketch how GraphQL schema could look like for our CR, took me a while to get that:
interface Node {
name: String!
path: String! //or contextPath?
nodeType: String! // optional, could also be accessed from GraphQL __type introspection
properties: [Scalar]
// All relevant FQ operations here, extendable
parent: Node
prev: Node
next: Node
children(path: String): [Node]
find(filter: String): [Node]
parents(filter: String): [Node]
siblings(filter: String): [Node]
}
// Autogenerated from NodeTypes.yaml for each nodetype:
type TYPO3.Neos:Node: Node {
properties: {
title: String
}
}
type TYPO3.Neos:DocumentNode: Node {
properties: {
_uriPathSegment: String!
title: String
}
}
// Root query object. Only allow to get node by contextPath, the rest would be possible via operations (fields in GraphQL term) on each node (e.g. find, children etc, see above)
type Query {
node(contextPath: string): Node
}
This would allow to query CR like this:
query node('someContextPathOfSomeNode') {
properties {
title
teaser
}
coverImage: children(path: 'coverImage') {
properties {
image
altText
}
}
mainContent: children(path: 'main') {
...${ContentCase.getFragment('type')} // figure out how to get fragment by type of current node, but should be possible
// Find all youtube videos inside main content
youtubeVideos: find(filter: '[instanceof MyType:YouTube]') {
properties {
videoId
title
}
}
}
}
When I looked into GraphQL to evaluate it I ended up implementing the whole TYPO3CR API with it. But instead of just sharing the code here I wanted to discuss it with you guys next week because there are some caveats/challenges to think about.
I just wanted to ask you about your current state? We are currently implementing a mini API based on jsonapi.org and our CQRS-Package for Neos content. It will most likely not be the final generic API we strive for in Neos, but it should be a nice use case how all the things @goli already posted above could be applied to Neos content (nodes).
I just wanted to keep you updated and ask about your current plans regarding the content API? We definitely should keep in sync and don’t implement the same things multiple times…
Thanks for bringing this up again. I was certain I posted an update to this thread after Inspiring Con, but apparently nope…
As you know I experimented with GraphQL to see how this could work out and was positively surprised by it. I still do have my concerns regarding general usage but I think it’s a perfect fit for the “Content Repository Backend” with its current architecture.
I created a general GraphQL package and one on top of it for Neos.
You can see it in action here (login editor:editor)
So, after Inspiring Con the idea was to play with the different approaches and see how they fit in the (reworked) Neos backend. And I’m afraid it’s still the current state.
But @wbehncke wanted to have a look into the GraphQL implementation at some point to see how easily it could be integrated with the react-based UI.
However, I still think it’s a good idea to come up with a more general package that allows for providing (and maybe consuming?) REST-APIs, especially in conjunction with CQRS. And I think that’s where a “regular” Hypermedia format like json-api can score (and maybe for the CR once it is event-sourced).
But that’s more or less my personal opinion and in any case it would be very interesting to see how a finalized json-api implementation of the CR would look like and how it would integrate with the Backend
I really like the “by default” error handling of the GraphQL, and the extensibility / flexibility (ex. call the CR, but also an external REST endpoint to get additional data, …).
