Flowpack.SearchPlugin: unknown key [analysis] for create index

itoop · October 16, 2020, 5:24am

Good morning,

I use the Flowpack.SearchPlugin in combination with the Flowpack.ElasticSearch.ContentRepositoryAdaptor and autocompletion on the search input with the js libary autocomplete.
As described here therefore I set the following config:

Flowpack:
  ElasticSearch:
    indexes:
      default:      # client name used to connect (see Flowpack.ElasticSearch.clients)
        zweicom:    # your (custom) index name
          analysis:
            filter:
              autocompleteFilter:
                max_shingle_size: 5
                min_shingle_size: 2
                type: 'shingle'
            analyzer:
              autocomplete:
                filter: [ 'lowercase', 'autocompleteFilter' ]
                char_filter: [ 'html_strip' ]
                type: 'custom'
                tokenizer: 'standard'

When I try to use the autocomplete-source https://develop.itoop.de/flowpack/searchplugin?contextNodeIdentifier=6803d200-7c87-4ccf-9ec5-bbcf228a662a&dimensionCombination={"language"%3A["de"]} by setting the term parameter manually like this…

https://develop.itoop.de/flowpack/searchplugin?term=kontakt&contextNodeIdentifier=6803d200-7c87-4ccf-9ec5-bbcf228a662a&dimensionCombination=%7B%22language%22%3A%5B%22de%22%5D%7D

…I get an Elasticsearch error:

{"completions":[],"suggestions":[],"errors":["Could not execute query: Elasticsearch request failed:\n\n[POST http:\/\/myuser:mypassword@94.130.131.131:8080\/zweicom-fb11fdde869d0a8fcfe00a2fd35c031d\/_search]: Array\n(\n [root_cause] => Array\n (\n [0] => Array\n (\n [type] => illegal_argument_exception\n [reason] => no mapping found for field [neos_suggestion]\n )\n\n )\n\n [type] => search_phase_execution_exception\n [reason] => all shards failed\n [phase] => query\n [grouped] => 1\n [failed_shards] => Array\n (\n [0] => Array\n (\n [shard] => 0\n [index] => zweicom-fb11fdde869d0a8fcfe00a2fd35c031d-1602280348\n [node] => oTxLkVlUQuGZlAU4gJrKlw\n [reason] => Array\n (\n [type] => illegal_argument_exception\n [reason] => no mapping found for field [neos_suggestion]\n )\n\n )\n\n )\n\n [caused_by] => Array\n (\n [type] => illegal_argument_exception\n [reason] => no mapping found for field [neos_suggestion]\n [caused_by] => Array\n (\n [type] => illegal_argument_exception\n [reason] => no mapping found for field [neos_suggestion]\n )\n\n )\n\n)\n\n\n\n\nRequest data: \n\n{\"query\":{\"bool\":{\"must\":[{\"match_all\":{\"boost\":1}}],\"filter\":{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"term\":{\"neos_parent_path\":\"\\\/sites\\\/itoop-zweicomde\"}},{\"term\":{\"neos_path\":\"\\\/sites\\\/itoop-zweicomde\"}}]}},{\"terms\":{\"neos_workspace\":[\"live\"]}},{\"prefix\":{\"neos_completion\":\"kontakt\"}}],\"should\":[],\"must_not\":[{\"term\":{\"neos_hidden\":true}},{\"range\":{\"neos_hidden_before_datetime\":{\"gt\":\"now\"}}},{\"range\":{\"neos_hidden_after_datetime\":{\"lt\":\"now\"}}}]}}}},\"_source\":[\"neos_path\",\"title\"],\"size\":0,\"aggregations\":{\"autocomplete\":{\"terms\":{\"field\":\"neos_completion\",\"order\":{\"_count\":\"desc\"},\"include\":\"kontakt.*\",\"size\":10}}},\"suggest\":{\"suggestions\":{\"prefix\":\"kontakt\",\"completion\":{\"field\":\"neos_suggestion\",\"fuzzy\":true,\"size\":10,\"contexts\":{\"parent_path\":\"\\\/sites\\\/itoop-zweicomde\",\"workspace\":\"live\"}}}}}\n\n\n\nResponse body:\n\n{\"error\":{\"root_cause\":[{\"type\":\"illegal_argument_exception\",\"reason\":\"no mapping found for field [neos_suggestion]\"}],\"type\":\"search_phase_execution_exception\",\"reason\":\"all shards failed\",\"phase\":\"query\",\"grouped\":true,\"failed_shards\":[{\"shard\":0,\"index\":\"zweicom-fb11fdde869d0a8fcfe00a2fd35c031d-1602280348\",\"node\":\"oTxLkVlUQuGZlAU4gJrKlw\",\"reason\":{\"type\":\"illegal_argument_exception\",\"reason\":\"no mapping found for field [neos_suggestion]\"}}],\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"no mapping found for field [neos_suggestion]\",\"caused_by\":{\"type\":\"illegal_argument_exception\",\"reason\":\"no mapping found for field [neos_suggestion]\"}}},\"status\":400}"]}

“Normal” search queries (e.g. http://94.130.131.131:8080/_search?q=kontakt) are working:

When I use ./flow nodeindex:build I get

Creating indices and apply mapping... Elasticsearch request failed:
[PUT
http://myuser:mypassword@94.130.131.131:8080/zweicom-fb11fdde869d0a8fcfe00a2fd35c031d-1602824440/]:
Array
(
    [root_cause] => Array
        (
            [0] => Array
                (
                    [type] => parse_exception
                    [reason] => unknown key [analysis] for create index
                )

        )

    [type] => parse_exception
    [reason] => unknown key [analysis] for create index
)


Request data:

{"analysis":{"filter":{"autocompleteFilter":{"max_shingle_size":5,"min_shingle_size":2,"type":"shingle"}},"analyzer":{"autocomplete":{"filter":["lowercase","autocompleteFilter"],"char_filter":["html_strip"],"type":"custom","tokenizer":"standard"}}}}

Response body:

{"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key
[analysis] for create index"}],"type":"parse_exception","reason":"unknown
key [analysis] for create index"},"status":400}

  Type: Flowpack\ElasticSearch\Transfer\Exception\ApiException
  Code: 1338977435
  File:
Data/Temporary/Development/Cache/Code/Flow_Object_Classes/Flowpack_ElasticS
        earch_Transfer_Response.php
  Line: 54

Open Data/Logs/Exceptions/2020101605004008b81f.txt for a full stack
trace.

  Type: Neos\Flow\Core\Booting\Exception\SubProcessException
  Code: 1355480641
  File: Packages/Framework/Neos.Flow/Classes/Core/Booting/Scripts.php
  Line: 699

Open Data/Logs/Exceptions/202010160500342f5752.txt for a full stack trace.

Any idea what I’m doing wrong?

daniellienert · October 28, 2020, 6:13am

Hey Andre,

which versions of the SearchPlugin, the CR-Adapter and Elasticsearch are you using?

itoop · October 28, 2020, 6:19am

Hi Daniel,

SeachPlugin: 5.0.0
CR-Adapter: 7.0.4
Elasticsearch: 7.9.3

itoop · November 27, 2020, 6:16am

I think I got it
I had to add “settings” between the custom index name and “analysis:”:

Flowpack:
  ElasticSearch:
    indexes:
      default:      
        zweicom:    
          settings:  # at least in my case this is necessary
            analysis:
              filter:
                autocompleteFilter:
                  max_shingle_size: 5
                  min_shingle_size: 2
                  type: 'shingle'
              analyzer:
                autocomplete:
                  filter: [ 'lowercase', 'autocompleteFilter' ]
                  char_filter: [ 'html_strip' ]
                  type: 'custom'
                  tokenizer: 'standard'

After that I tried ./flow nodeindex:build and got this:

./flow nodeindex:build
Creating indices and apply mapping... Elasticsearch request failed:
[PUT
http://myuser:mypass@myip:8080/zweicom-fb11fdde869d0a8fcfe00a2fd35c031d-1606456985/]:
Array
(
    [root_cause] => Array
        (
            [0] => Array
                (
                    [type] => illegal_argument_exception
                    [reason] => In Shingle TokenFilter the difference
between max_shingle_size and min_shingle_size (and +1 if outputting
unigrams) must be less than or equal to: [3] but was [4]. This limit can
be set by changing the [index.max_shingle_diff] index level setting.
                )

        )

    [type] => illegal_argument_exception
    [reason] => In Shingle TokenFilter the difference between
max_shingle_size and min_shingle_size (and +1 if outputting unigrams)
must be less than or equal to: [3] but was [4]. This limit can be set by
changing the [index.max_shingle_diff] index level setting.
)




Request data:

{"settings":{"analysis":{"filter":{"autocompleteFilter":{"max_shingle_size":5,"min_shingle_size":2,"type":"shingle"}},"analyzer":{"autocomplete":{"filter":["lowercase","autocompleteFilter"],"char_filter":["html_strip"],"type":"custom","tokenizer":"standard"}}}}}



Response body:

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"In
Shingle TokenFilter the difference between max_shingle_size and
min_shingle_size (and +1 if outputting unigrams) must be less than or
equal to: [3] but was [4]. This limit can be set by changing the
[index.max_shingle_diff] index level
setting."}],"type":"illegal_argument_exception","reason":"In Shingle
TokenFilter the difference between max_shingle_size and min_shingle_size
(and +1 if outputting unigrams) must be less than or equal to: [3] but
was [4]. This limit can be set by changing the [index.max_shingle_diff]
index level setting."},"status":400}

  Type: Flowpack\ElasticSearch\Transfer\Exception\ApiException
  Code: 1338977435
  File:
Data/Temporary/Development/Cache/Code/Flow_Object_Classes/Flowpack_ElasticS
        earch_Transfer_Response.php
  Line: 54

Open Data/Logs/Exceptions/20201127060305b3a96b.txt for a full stack
trace.

  Type: Neos\Flow\Core\Booting\Exception\SubProcessException
  Code: 1355480641
  File: Packages/Framework/Neos.Flow/Classes/Core/Booting/Scripts.php
  Line: 699

Open Data/Logs/Exceptions/202011270603052556f4.txt for a full stack trace.

So I set
max_shingle_size: 5
to
max_shingle_size: 4

and after that it works

I’m wondering how it is with other versions and if I should pull a request to change the docs