Currently Fusion can be used to express many data transformation needs in a nice way while still providing unplanned extensibility that often enables us to solve the last 5% of weird customer requests in projects. At some places this even leads to use cases where fusion is used without templates at all. The main argument against such a use-case is the sometimes bloaty syntax of fusion compared to a html-template.
This RFC aims to provide a possible solution by providing an extension point that can be used to integrate custom dsls into fusion.
Background
In a discussion about the protopye afx-language extension for fusion (https://github.com/PackageFactory/atomic-fusion-afx) we came to the conclusion that it might be valuable to provide extension points in fusion to integrate domain specific languages. That way the creation of things like AFX would be possible without modifying or aspecting the fusion core.
Proposal
I suggest that we define a dsl-syntax-block in fusion that will be passed to a somehow registered service at parse-time. The external service returns fusion code wich is afterwards processed by the fusion parser. That way the caching and runtime performance are not affected. It would also be possible to provide a generic cli eject-command that expands the DSL-Code directly in the fusion-files to enable the later removal of the fusion-dsl if needed.
In a later step this might be extended to allow dsl-services to directly return a fusion-ast instead of fusion but i suggest to postpone that decision because this would make the structure of the fusion ast part of the dsl-interface.
Questions to discuss
Do we agree that adding a dsl-extension point to the fusion-parser is valuable?
How should dsl-services be registered?
The first options that come to mind are Settings.yaml or Package.php. A possible drawback of Settings would be that this adds visibility for that feature and that people might use this too often for minor use-cases.
What would be a good generic-syntax for DSLs in fusion?
Here are some syntax options i tried so far for an AFX-DSL:
#
# First experiment ... matches all expressions that start with 'AFX::' and end with an empty line
# looks ok but matching an empty line as end probably is bad idea
#
renderer = AFX::
<div>
<h1 @key="headline" class="headline">${props.title}</h1>
<h2 @key="subheadline" class="subheadline" @if.hasSubtitle="${props.subtitle ? true : false}">${props.subtitle}</h2>
<Vendor.Site:Image @key="image" uri="${props.imageUri}" />
</div>
#
# Heredoc-ish syntax
#
# The main problem i see here is that i fear this will be recognized as a template string and
# not as a sublanguage. This might lead to misconceptions. Nevertheless this is my favorite syntax so far.
#
renderer = <<<AFX
<div>
<h1 @key="headline" class="headline">${props.title}</h1>
<h2 @key="subheadline" class="subheadline" @if.hasSubtitle="${props.subtitle ? true : false}">${props.subtitle}</h2>
<Vendor.Site:Image @key="image" uri="${props.imageUri}" />
</div>
AFX;
#
# ES6 tagged template string syntax
#
# Same as with heredoc i fear this will be recognized as a template string and
# not as a sublanguage. This might lead to misconceptions.
#
renderer = afx`
<div>
<h1 @key="headline" class="headline">${props.title}</h1>
<h2 @key="subheadline" class="subheadline" @if.hasSubtitle="${props.subtitle ? true : false}">${props.subtitle}</h2>
<Vendor.Site:Image @key="image" uri="${props.imageUri}" />
</div>
`
#
# Inline Assembler-ish syntax
# well i tried ... looking not really handy
#
renderer = __afx__ (
<div>
<h1 @key="headline" class="headline">${props.title}</h1>
<h2 @key="subheadline" class="subheadline" @if.hasSubtitle="${props.subtitle ? true : false}">${props.subtitle}</h2>
<Vendor.Site:Image @key="image" uri="${props.imageUri}" />
</div>
)
Wouldn’t it then be an idea to have the $ for eel and have the same structure but different prefix before {foo.bar} ?
Multiline strings would be good anyway.
That would be a huge refactoring of whole runtime and the fusion-ast since currently eel gets special treatment everywhere. Offcourse the EEL-Evaluator could be implemented as a fusion prototype.
I would not even try to do this directly but rather find a syntax that will support a refactoring like that in the future.
On the other hand … refactoring eel to a real dsl inside fusion definitely is a good idea. That would help to separate the concerns of the fusion an the eel packages.
The definition of the connection between both packages could be done inside an eel-fusion-connector-package so neither eel nor fusion would have to depend on each other.
If you develop a feature that need to be hidden, don’t do it
The documentation should be clear about the type of use case for this. And that it’s an advanced extension points.
Go to Settings.yaml, try to avoid Package.php has much as possible … impossible to overload from an other package, … And convention is configuration go to Settings.yaml.
An other option, can be an Interface + Annotations + StaticCompile.
As said in Slack, I love the ES6 tagged template, but nothing blocking from my side to use something else (love the nerd assembly based proposal).
also, dont expose too much internals.
When this new feature is rock solid, fine to move EEL handling on top of it, but first thing first
true … but it is still imortant to make clear that the easy extension points are eel-helpers, flowQuery operations and fusion objects.
OK valid point … Settings it is for now. I played with a possible configuration a bit.
Neos:
Fusion:
parser:
dsl:
# this would come from the eel fusion connector package
eel:
pattern: %Neos\Eel\Package::EelExpressionRecognizer%
multilineStartPattern: '${'
multilineEndPattern: '}'
className: Neos\FusionEelConnector\Service\DslService
# this would come from the afx package
afx:
pattern: %PackageFactory\AtomicFusion\Afx\Package::AfxExpressionRecognizer%
multilineStartPattern: '<<<AFX'
multilineEndPattern: 'AFX;'
className: PackageFactory\AtomicFusion\Afx\Service\DslService
```
The multiline configuration stuff surely does not look pretty yet but is necessary because of line by line way the fusion parser works currently. If it detects an eel-statement that is not valid for the current line it tries to add more and more lines and checks wether the expression is now valid. (see https://github.com/neos/neos-development-collection/blob/master/Neos.Fusion/Classes/Core/Parser.php#L736-L760)
Another possible problem with such a generic pattern approach would be third party tool support, If we would define a wrapper syntax for all dsls tools could at least exclude this part from syntax highlighting. Offcourse we also could suggest a best-practice wrapper syntax that most fusion code-highlighting tools will support.
@mficzel Convention over configuration is at the root of Flow, so I think the start/end + pattern should not be configurable.
If we decide to go to ES6 tagged template or anything else all implementation should use this. The only setting should be className and a “tagName” (if the array key can not be used, like for EEL (the tag should be $ and not EEL
Using a generic syntax definitely has advantages especially for third party tools but gives us the problem to find the end of an expression. If EEL has to fit into the same pattern the problem gets even harder since most dsls will have to include eel in one way or the other.
In general … if we include EEL as a dsl we have be find a pattern that allows us to safely detect the outer dsl boundaries while it includes expressions of other dsls inside. If eel is handled differently we could state that dsls cannot contain other dsls. This gets juicy especially for multiline expressions.
Minor side note: the strong pro of what we have done so far is trying to borrow a syntax/ideas from other popular technology that we all already know (e.g. jQuery -> FlowQuery). Thus I really like the idea to use the syntax of ES6 tagged templates.
That would make the detection of the expression-end easy since we do’nt use backticks for other stuff. On the other hand i cannot easily get eel into that picture. And we still need to figure out wether we want dsl’s to be able to include each other.
sorry for not responding for so long, definitely really interesting topic
My take on it:
I really like the ES6 Tagged String syntax, so ```afx`foo```` it is for me.
I would NOT integrate Eel into the picture just yet; as there is quite some logic in the system ensuring that Fusion objects override Eel; Eel overrides Fusion; etc etc. We can do this at a later point in time I think.
DSLs to include each other: I don’t think this really is useful or could be implemented in a general case. Basically a DSL could implement the “inclusion” itself if it makes sense for it; but otherwise I would try to refrain from it.
I’m actually preferring configuration via Package.php; but @dfeyer has had good arguments in favor of Settings.yaml… So I’d be OKish with it as well; provided we put a very very big notice there saying this part is UNSTABLE and subject to change.
We should introduce officially something like “preview technology” for this kind of interface that can change, but it’s usable. Like this we don’t reinvent the wheel each time we introduce something bleeding edge So we also have the responsability to make preview tech stable in the next few release. A bit like a inversed “deprecated” tag. So a preview tech can have an announcement in a release and release + N should make this tech a stable one.
props for the ASM style tagging, I doubt a lot would actually get the reference though and as is it just feels awkwardly arbitrary and verbose. I currently prefer the ES6 style tags too. The idea to use the curly brackets as general tags and have an (aliasable) prefix is nice at first look, but IMO the curly brackets are overused. I think it works for EEL, because the combination with $ is so rare that it makes it pretty much conflict-free.
And definitely don’t make the tags configurable, it just creates a hazard when trying to share DSL snippets.
Thanks for the feedback so far. I think the following points are common ground:
The es6 tagged template string syntax has some friends.
EEL should be taken out of this picture at least for now. Maybe the implementation can share the internal mechanics eventually but no need to habe a common syntax.
If DSLs include EEL or other DSLs it is their own business and responsibility.
Wether this is configured via Settings.yaml or Package.php is still subject of discussion.
Since the registration can be figured out later i can start implementing as soon as i find time. I think this will be my topic for the hamburg-sprint before neos-con.
IMHO Package.php is so low level I would avoid introducing anything that needs it. It’s basically pre boot stuff and we really don’t need to know about Fusion DSLs at that point in time.
I guess I should also state that I am by now okay-ish with the inlined template thingy. It will be abused and I will hate that point I get a project with 1000 lines of inlined templates, but so be it. I understand the case, I think it can be a nice extension of our ideas.
I might have a bit more problems with some implementation details that are indicated in the code sample, but let’s wait and see
true … but it is still imortant to make clear that the easy extension points are eel-helpers, flowQuery operations and fusion objects.