[SOLVED] Robots.txt editing best practices/suggestions

Using Neos & Flow
1

Hi guys,
just a quick and hopefully easy question.

Is there a way to programmatically edit the robots.txt file (something like sitemap.xml or via Settings.yaml or similar) or should I just provide a static one from my project repo?

Thanks,
Nicola

2

Hi,

one example: you could add some property to your homepage nodetype with a textarea.
And via fusion it should be renderable like a sitemap.

Regards,
Sebastian

3

Hi @sebobo,
thanks for the advice even if I understood what you suggest I think I’m missing something here.
How does it update or override the physical file Web/robots.txt ?

Thanks,
Nicola

4

It doesn’t, you don’t need an actual file robots.txt anymore.

Like the sitemap you just tell neos to serve the robots.txt.
The vhost configuration should ask neos to provide it if there is no actual file.

1 Like
5

Exactly what I supposed but I was doubting about getting it generated again as it happens when you launch composer install for the first time (I presume as I don’t bring it around in our repos and the only operation with provide is via composer).

Out of curiosity how/when that file gets generated on the first instance? Deleting it does not get it recreated, fortunately :grin:

Thanks,
Nicola

1 Like