Documenting Fusion

I miss the definition of a fusionDoc standard - documentation is important for reusability, and as many developers don’t read or write separate documentations :see_no_evil:, there should be a standard for Fusion like phpDoc for PHP.

there is Atomic Fusion Proptypes and I learned recently, this only works for Components but for everything else it could at least be used as documentation…

and when looking at the core Fusion prototypes, not every prototype has comments and when there are, only some properties are documented with comments in the fusion code but others are just documented in the docBlock of the PHP class - I think all the API properties/paths should be documented in the Fusion code

/**
* Render each item in items using itemRenderer.
*/
prototype(Neos.Fusion:Loop) {
 
  # The array or iterable to iterate over
  items = null
  items.@type = ${PropTypes.Iterable}

  # Context variable name for each item
  itemName = 'item'
  itemName.@type = ${PropTypes.string}

  # Context variable name for each item key, when working with array
  itemKey = 'itemKey'
  itemKey.@type = ${PropTypes.string}
}

if the above would not work or not be performant enough, it could also be written like that using the existing propTypes logic

/**
* Render each item in items using itemRenderer.
*/
prototype(Neos.Fusion:Loop) {

  # The array or iterable to iterate over
  items = null
  @propTypes.items = ${PropTypes.Iterable}

  # Context variable name for each item
  itemName = 'item'
  @propTypes.itemName = ${PropTypes.string}

  # Context variable name for each item key, when working with array
  itemKey = 'itemKey'
  @propTypes.itemKey = ${PropTypes.string}
}

eventually a Fusion reference could be compiled from the fusionDoc. :slight_smile:

3 Likes

about that proptypes dont work with anything else than components: i kickstarted the idea with runtime @type checks GitHub - mhsdesign/MhsDesign.FusionTypeHints

2 Likes

I just came up with that @type syntax myself, but it seems it is kind of intuitive. :grinning:

I really like the concept of propTypes to use Flow-Validators, because it is flexible and extensible. But I’ll have a deeper look at PropTypes and also the package you created.

But the formal type verification is only one part of my issue, the descriptions of prototypes and properties/paths are as important.

1 Like

I would really like to have an approach that allows us to generate the whole fusion prototype documentation automatically like for the eel helper docs. That would need descriptions for each property plus examples for the whole prototype in addition to the suggested @type. Problem is that this cannot be extracted from the parsed fusion anymore as all comments get lost during parsing.

I could imagine something like this:

/**
* 
*/
prototype(Neos.Fusion:Loop) {
  @description = "Render each item in items using itemRenderer." 
  
  items = null
  items.@type = "array|iterable"
  items.@description = "The array or iterable to iterate over"

With the @description beeing dropped in normal parsing and not beeing cached.

1 Like

I’ve just discussed with @sjsone (author of the VSCode Editor Extension) how we can improve the documentation in the code. This suggestion covers how to provide documentation using fusion features and some ideas how this would improve the everyday life of a fusion developer. This must not affect rendering at all.

Proposal

We suggest to use @doc:

  • @doc = "This prototype does xy".
  • @propTypes.[field].@doc = "This field does xy"

This would look something like this:

prototype(Vendor.Website:Atom.Test) < prototype(Neos.Fusion:Component) {
    @doc = "Description for Vendor.Website:Atom.Test"

    @propTypes {
        field = ${PropTypes.string}
        field.@doc = "This is a field"
    }
}

Thoughts to why we chose this route

Previously this was the suggested syntax for the docs:

prototype(Vendor.Website:Atom.Test) < prototype(Neos.Fusion:Component) {
  itemKey = 'itemKey'
  itemKey.@type = ${PropTypes.string}
  itemKey.@description = ${PropTypes.string}
}

If we rely on the @propTypes meta property / decorator “inside” the actual field, then the description would be lost if we process, overwrite or delete the prop. With our suggestion this won’t be a problem, as the description is not directly attached to the prop itself. Also this uses @propTypes which is already established in the Neos community.

Impact

With this we could document Fusion more easily and closer to the code.
This will solve at least 2 problems:

  • Documentation Reference stays in sync with the code and is always accessible in the project
  • Machine readable reference
    • For the IDE plugins this would also remove the need to keep the plugin in sync as the sources are available inside the project
    • For docs.neos.io this provides versioned Fusion object definitions

Requirements

As this suggestion is partly based on propTypes, we would need propTypes to move into the core. (Currently the reference is far away from the actual code: Inside Neos.Neos instead of Neos.Fusion)

Example of Neos.Fusion:Loop
prototype(Neos.Fusion:Loop) {
    @class = 'Neos\\Fusion\\FusionObjects\\LoopImplementation'
    
    @doc.text = "Render each item in items using itemRenderer."
    @doc.description = "
        Example using an object itemRenderer:

        ```neosfusion
        myLoop = Neos.Fusion:Loop {
                items = ${[1, 2, 3]}
                itemName = \"element\"
                itemRenderer = Neos.Fusion:Template {
                        templatePath = 'resource://...'
                        element = ${element}
                }
        }
        ```
    "

    @propTypes {
        items = ${PropTypes.arrayOf(PropTypes.any).isRequired}
        items.@doc = "The array or iterable to iterate over (to calculate iterator.isLast items have to be countable)"
        
        itemName = ${PropTypes.string}
        itemName.@doc = "Context variable name for each item"
        
        itemKey = ${PropTypes.string}
        itemKey.@doc = "Context variable name for each item key, when working with array"
        
        iterationName = ${PropTypes.string}
        iterationName.@doc = "A context variable with iteration information will be available under the given name: `index` (zero-based), `cycle` (1-based), `isFirst`, `isLast`"
        
        itemRenderer = ${PropTypes.any.isRequired}
        itemRenderer.@doc = "The renderer definition (simple value, expression or object) will be called once for every collection element, and its results will be concatenated (if `itemRenderer` cannot be rendered the path `content` is used as fallback for convenience in afx)"
        
        @meta.glue = ${PropTypes.string}
        @meta.glue.@doc = "The glue used to join the items together"
    }

    items = null
    itemName = 'item'
    itemKey = 'itemKey'
    iterationName = 'iterator'
    @glue = ''
}

We think it would be also beneficial to be able to differentiate between a short description (just @doc='' or @doc.text='') and more content (like examples, links, …) .

Next steps

  • Discuss solution with the core team
  • Move propTypes to the core
  • Move Fusion reference to the .fusion files and update the ReadTheDocs generation (with small to no changes for the reference users)
  • Create package for docs.neos.io to be able to use the reference as content
  • Implement IDE improvements to suggest props/docs based on defined @propTypes/@doc

Outlook

Eventually this could be implemented by the IDE plugin (like @sjsone’s VSCode plugin as well as the docs.neos.io)

Links, bugs & other ideas

  • Neos.Fusion:Join reference has property [key].@ignoreProperties and should probably be .@ignoreProperties (without [key])

  • Bug in PropTypes

    PropType definitions can't be deleted with >

    We think this should delete the proptype definition:

    prototype(Vendor.Website:Atom.Test) < prototype(Neos.Fusion:Component) {
        @propTypes {
            @strict = true
            field = ${PropTypes.string}
        }
        
        field = 'test'
    }
    
    prototype(Vendor.Website:Atom.Test2) < prototype(Vendor.Website:Atom.Test) {
        @propTypes {
            field >
        }
        
        field >
    }
    

    Currently this is not possible. We encounter the following error: propType for prop field must implement the ValidatorInterface array found instead

  • @deprecated Idea
    Create a meta property @deprecated. This could show a message on hover in the IDE, write a.
    warning in the log, …

  • Make the fusion api references more newcomer friendly

  • Redesign and restructuring

4 Likes

Lovely. Looking at the example I thought, maybe @doc.summary is better than @doc.text, because text seems rather… generic.

1 Like

I also like the @doc proposal. We could also make sure to exclude this from the fusion ast in Production context to save memory.

Technical idea: We could configure a list of meta keywords that are to be ignored by the fusion parser.

1 Like

How should/can we continue with this effort?

The autogenerated docs are created with this package here. GitHub - neos/doctools: Tools for generating and rendering Flow and Neos documentation .

A prototype or pr to showcase how we could integrate fusion here would be great.

Some time has passed so here is a small Update/POC using the neos/doctools: GitHub - sjsone/doctools: Tools for generating and rendering Flow and Neos documentation which implements most of the proposed ideas.

Implemented Features

The features and especially the naming of these features are happily open for discussion and feedback is more than welcome.

Prototype related

@doc.summary or the short version @doc for a few sentences. These get rendered at the beginning.

@doc.description is a long description which gets rendered after the PropTypes.

@deprecated is just text which gets rendered as a Deprecation-Warning in the end.

Property related

Every normal propType definition will be rendered into the reference. For that the property name, defined type (using the PropTypes-EEL-Helper) and, if existing, the defined default value are used.

If the optional @doc meta tag is provided it will be used as the summary text describing the property. (@doc.description does work but is currently just not used inside the reference template)

Meta tags like @glue are currently supported using the @meta meta tag…

If there is the need to document a non existing property, @doc.additionalProperty can be used. This is needed for the Neos.Fusion:Join reference.

@propTypes {
    itemName = ${PropTypes.string}
    itemName.@doc = "itemName summary"
    
    @meta.glue = ${PropTypes.string.isRequired}
    @meta.glue.@doc = "@glue summary"

    @doc.additionalProperty."[key]" = ${PropTypes.string.isRequired}
    @doc.additionalProperty."[key]".@doc = "every key inside the prototype"
}

itemName = 'item'
@glue = ''

Examples

Neos.Fusion:Loop:

Fusion
prototype(Neos.Fusion:Loop) {
  @class = 'Neos\\Fusion\\FusionObjects\\LoopImplementation'

  @doc.summary = "Render each item in items using itemRenderer."
  @doc.description = " Example using an object itemRenderer:..."

  @propTypes {
    items = ${PropTypes.arrayOf(PropTypes.any).isRequired}
    items.@doc = "The array or iterable to iterate over (to calculate iterator.isLast items have to be countable)"
    
    itemName = ${PropTypes.string}
    itemName.@doc = "Context variable name for each item"
    
    itemKey = ${PropTypes.string}
    itemKey.@doc = "Context variable name for each item key, when working with array"
    
    iterationName = ${PropTypes.string}
    iterationName.@doc = "A context variable with iteration information will be available under the given name: `index` (zero-based), `cycle` (1-based), `isFirst`, `isLast`"
    
    itemRenderer = ${PropTypes.any.isRequired}
    itemRenderer.@doc = "The renderer definition (simple value, expression or object) will be called once for every collection element, and its results will be concatenated (if `itemRenderer` cannot be rendered the path `content` is used as fallback for convenience in afx)"    
    
    @meta.glue = ${PropTypes.string.isRequired}
    @meta.glue.@doc = "The glue used to join the items together (default = ‘’)"
  }

  itemName = 'item'
  itemKey = 'itemKey'
  iterationName = 'iterator'
  @glue = ''
}
reStructuredText
.. _`Neos Fusion Reference: Neos.Fusion:Loop`:

Neos.Fusion:Loop
----------------
Render each item in items using itemRenderer.


* ``items`` (Array&lt;mixed&gt;): The array or iterable to iterate over (to calculate iterator.isLast items have to be countable)

* ``itemName`` (string, *optional*, defaults to `'item'`): Context variable name for each item

* ``itemKey`` (string, *optional*, defaults to `'itemKey'`): Context variable name for each item key, when working with array

* ``iterationName`` (string, *optional*, defaults to `'iterator'`): A context variable with iteration information will be available under the given name: `index` (zero-based), `cycle` (1-based), `isFirst`, `isLast`

* ``itemRenderer`` (mixed): The renderer definition (simple value, expression or object) will be called once for every collection element, and its results will be concatenated (if `itemRenderer` cannot be rendered the path `content` is used as fallback for convenience in afx)

* ``@glue`` (string): The glue used to join the items together (default = ‘’)



      Example using an object itemRenderer:

      ```neosfusion
      myLoop = Neos.Fusion:Loop {
              items = ${[1, 2, 3]}
              itemName = "element"
              itemRenderer = Neos.Fusion:Template {
                      templatePath = 'resource://...'
                      element = ${element}
              }
      }
      ```
  

Neos.Fusion:Array

Fusion
prototype(Neos.Fusion:Array) {
  @class = 'Neos\\Fusion\\FusionObjects\\ArrayImplementation'

  @doc = ""
  @propTypes {
    @doc.additionalProperty."[key]" = ${PropTypes.string.isRequired}
    @doc.additionalProperty."[key]".@doc = "A nested definition (simple value, expression or object) that evaluates to a string"
    
    @doc.additionalProperty."[key].@ignoreProperties" = ${PropTypes.arrayOf(PropTypes.string).isRequired}
    @doc.additionalProperty."[key].@ignoreProperties".@doc = "A list of properties to ignore from being “rendered” during evaluation"
    
    @doc.additionalProperty."[key].@position" = ${PropTypes.anyOf( PropTypes.string, PropTypes.integer ).isRequired}
    @doc.additionalProperty."[key].@position".@doc = "Define the ordering of the nested definition"
  }
  @deprecated = "The Neos.Fusion:Array object has been renamed to Neos.Fusion:Join the old name is DEPRECATED;"
  
  @sortProperties = true
}
reStructuredText
.. _`Neos Fusion Reference: Neos.Fusion:Array`:

Neos.Fusion:Array (deprecated)
------------------------------



* ``[key]`` (string): A nested definition (simple value, expression or object) that evaluates to a string

* ``[key].@ignoreProperties`` (Array&lt;string&gt;): A list of properties to ignore from being “rendered” during evaluation

* ``[key].@position`` (string|integer): Define the ordering of the nested definition





**DEPRECATED** The Neos.Fusion:Array object has been renamed to Neos.Fusion:Join the old name is DEPRECATED;

4 Likes

This seems pretty neat already! Super cool that you took care.

1 Like

I like this approach but i would prefer an to not introduce a dependency to Packagefactory.PropTypes since this is not a core package.

We also would have to settle upon the keys like @type, @deprecated, @description and ensure we can strip those in Production Context before the fusion is cached.

1 Like

Thank you for your feedback!

The idea was to incorporate Packagefactory.PropTypes into the core. It is however, not strictly necessary, as long as Packagefactory.PropTypes and the Fusion Doc share the exact same Syntax/Fusion/EEL-Helper/Stuff.

So if we introduce some new meta properties and EEL-Helper to define property types, a new version of Packagefactory.PropTypes is needed. Once the naming is done I will happily create a Pull-Request against it if necessary.

I personally do not see any real need to strip the properties for now. Yes, it would be cleaner without them, as there is no real need in Production, but I do not think they add any significant amount of parse-time or cache-size. So it should not be a requirement for this feature in my opinion. (Are @propTypes removed in Production?)

I agree that stripping of certain parts of the fusion AST for production is optional.

I am not against bringing PropTypes to the core but that would require a team decision and probably a bigger overhaul as PropTypes in the current form is merely a prototype that was to useful to not use. I personally would like to do some major changes nowadays and i am the original author. That is why personally would advice to do this without if possible. Also you would not need that dirty override eel helper pattern :smirk: that will make the review more interesting,

@mficzel why would you do this without? Because our proposal depends on Packagefactory.PropTypes or as very similar fusion structure, otherwise we would probably need a totally different solution.

What changes would you want to make to the PropTypes?

I nowadays would want to implement prop types in a cleaner way embracing php8.

Also the neos-team would have to decide wether we want to adopt it to the core and we should also decide wether we like title = ${PropTypes.string} or title = PropTypes:String better.

That is why i would like a syntax like title.@type = 'string' a bit better for docs as it is independent and does not assume propTypes in the core.

We (someone ) could then (sometime in the future) quite “easily” create a package like proptypes that validates those at least for components in dev mode. But that would not have to happen in the step that updates the docs.

Of course. :+1: That’s why we published this proposal and discussed it already with several team members. Should @sjsone and @manuelmeister join sometime in a team meeting, to present our proposal, to let the neos team decide if it wants to proceed with this proposal?

Also I don’t care if it is inside Eel or not (this is only parser relevant)


This has a major flaw. If clear the title

title.@type = 'string'
// either delete
title > 
// or just process
title.@process.map = ${String.toUpperCase(value)}

then the docs and type definitions would be gone, whether this is intended or not. The current mechanism with @propTypes is disconnected from the actual property as described in my post:

While I would like such a notation visually and closer to the property as you described it, it would clash with the way fusion (from my puny understanding) works.


Yes, exactly. It is fully independent from our effort.

What is my goal with these docs?

From my point of view I see the PropTypes and the PropDocs as an interface definition for Fusion objects / prototypes. Therefore I want a solution that is independent of its inner workings. That’s why I liked the PropTypes solution, that is separate from the actual properties.

Besides that you could write it closer to the actual prop:

@propTypes.title = ${PropTypes.string}
title = 'This is the headline'

or with the Object

@propTypes.title = PropTypes:String {
    @doc = 'Headline of the article'
}
title = 'This is the headline'
1 Like

The downside of PropTypes is that those are only for Components and only dev-mode. I always hoped we would find a better solution eventually for all properties. Not against it in any way but if you want to build the docs on top of it that would be the first building block to add to the core.

One thing i would like to achieve wold be a standard way to define validators in fusion. Currently PropTypes and RuntimeForm Schema use similar but different approaches for that.

Maybe a unified @doc annotation can do that:

title = "foo"
@doc {
   title.validator = "eel or processor to create a string validator"
   title.description = ' ... helpful stuff'
}
1 Like

(Sorry for my late response)

Interface definition & Naming

The existing @propTypes structure fits our need quite nicely so it should be used. Albeit with another name. I propose to use @types. It is short and still lets us use @type if needed, while implying that there are a bunch of types inside.

EEL or Prototypes or Strings or …

EEL

By keeping the trend of just removing prop from @propTypes the Type-EEL-Helper would then be called like ${Type.string.required}.

For now these helpers just return null but represent an interface which can be used by just overwriting the default EEL context. Either for building the documentation or for evaluating types at runtime.

Future

In the (distant) future some syntax sugar like:

title: string = ''

would be parsed into the same tree shape as:

@types {
    title = ${Type.string}
}

title = ''

Even some intermediate representation (using a typing-DSL) like:

@types {
    title = t`string`
}

would be possible

Strings

I am not a big fan of just using a string to define types:

title.@type = 'string'

Because then we loose any consistency and people could write stuff like "Strings" instead of "string[]" or "Array<string>" which creates frustration when “real” types are introduced.

@doc anywhere

By having the types as the “fixture” and adding @doc onto that, instead of the other way around, allows us to attach the @doc meta property anywhere and creating some kind of documentation for that path. For starters we would only look for @doc directly in the prototype and inside the @types properties (as already proposed).
But in the future @docs could be allowed everywhere and provide additional documentation or hints. This could be useful for cases inside the root = Neos.Fusion:Case for example.

@doc

That means the @doc meta property comes in two flavours:

Inside prototype(...)

Essentially used to add documentation text to the Prototype. Allows for the summary-shorthand @doc = "short text":

@doc {
    summary = "short text"
    description = "long text"
}

Inside @types.@doc

Used to add summary and description to either meta properties (like @glue) or non existing properties:

@doc {
    meta { }
    additionalProperty { }
}

Inside @types.bla.@doc

Essentially used to add documentation text to the property. Conceptually only summary should be used (TBD). Allows for the summary-shorthand @doc = "short text":

@doc {
    summary = "short text"
    description = "long text"
}
1 Like

So far the following looks best to me overall:

prototype(Vendor.Site:STuff) {
   @description = 'this makes things'
   @type = ${Types.string}

   title = null
   title.@type = ${Types.string}
   title.@description = 'The title for the stuff'
}

Downside is that by doing title > you can get rid of the annotation probably by accident. Not sure how much of a problem that is especially as we are currently talking about documentation.

Maybe we should simple get rid of the remove path operator eventually as it is weird anyways.