Robots.txt is not in root but in /de directory

Using Neos & Flow
1

Hi there

Does anybody know why the robots.txt file is in the language directory and not in the root directory? Is it possible to move it to the root?

Thanks for your help
Ursula

2

Hi and welcome to the Neos community!

Neos doesn’t create a robots.txt file anymore since some versions and the Neos.SEO package has a dynamic robots.txt file defined by a route.

I assume that you have a language dimension and if you don’t have a preset with an empty uriSegment the url will contain the segment of the current language.

Hope that clarifies a bit.

3

Hi Sebastian, thanks for the welcome :slight_smile:

Yes, I do have a language dimension. Here is my language setting:

Neos:
  Flow:
    i18n:
      defaultLocale: de
      fallbackRule:
        order: ['en']
  ContentRepository:
    contentDimensions:
      language:
        label: DUMMY.Country:Main:settings.languages
        icon: fas fa-language
        default: de
        defaultPreset: de
        presets:
          de:
            label: DE
            values:
              - de
            uriSegment: 'de'
      country:
        label: dUMMY.Country:Main:settings.countries
        icon: fas fa-globe
        default: at
        defaultPreset: at
        presets:
          at:
            label: 'Österreich'
            values:
              - at
            uriSegment: ''
  Neos:
    userInterface:
      defaultLanguage: de
      navigateComponent:
        nodeTree:
          loadingDepth: 1

Can you tell me where the empty preset should be?

4

You can try to change the following line to an empty string instead of “de” (I think that should work):

Keep in mind that if you add more languages later, they of course require a segment and will allow urls like “/en/robots.txt” to resolve in addition to the default one under “/robots.txt”.
I don’t think we can prevent this currently.

5

Unfortunatly we do also have a Swiss site (own Neos instance) with three languages. When I empty the uriSegment for ‘de’ I get 404 on the pages because they are not below /de anymore… So I rather won’t touch this.

Do you know if it is possible to place a physical robots.txt in the Web directory and deploy it to the server?

6

Yes sure. If the web server finds an actual file it should skip the one Neos generates.

7

We found this in the docs Neos SEO dev-master Documentation — Neos SEO Addon dev-master documentation

You also have to edit the .htaccess:: Change the line RewriteRule ^(_Resources/Packages/|robots.txt|favicon.ico) - [L] to RewriteRule ^(_Resources/Packages/|favicon.ico) - [L].

And it seams to work. Thanks for your support.

1 Like