Highlight search word in ES fulltext search result

Using Neos & Flow
1

Hi everyone,

I have a question.
I’m using elasticsearch with Flowpack.Searchplugin and don’t get a em-Tag around my search word.

In the ES doc they write I have to add to my query:

    "highlight" : {
        "fields" : {
            "content" : {}
        }
    }

So I guess I have to add this to my Settings.yaml, something like this:

Flowpack:
  ElasticSearch:
    ContentRepositoryAdaptor:
      driver:
        version: 2.x
        mapping:
          2.x:
            query:
              arguments:
                request:
                  highlight:
                    fields:
                      - _all

But this doesn’t work.
Has anyone a hint for me?

2

Hey Alexander,

For the common use cases you don’t need to tweek the settings. The.fulltext() in the search query which the Search plugin uses already activates the highlighting. So it should work out-of-the-box. The highlighting can be adjusted using .highlight() .

Have you adjusted the search query / output somehow?

3

Hi Daniel,

thank you for your help.

I didn’t changed the search query. In the mean time I found out that the issue has to be elasticsearch.
I have activated logging for the search query via .log() and could see that the highlight:{} part is in the query. The result directly from ES via curl has the search word is in the __fulltext* fragment, but ES didn’t highlighted it.

Any clue what could cause this behavior? I’m using ES 2.4.6 via docker and I have added the following, as mentioned in the doc, to my elasticsearch.yml:

network.host: 0.0.0.0
script.inline: true
cluster.name: es2xdev

# the following settings are well-suited for smaller Elasticsearch instances (e.g. as long as you can stay on one host)
index.number_of_shards: 1
index.number_of_replicas: 0

Example search query:

curl 127.0.0.1:9200/typo3cr-dev-1503909818/_search -d 
'{"query":{"filtered":{"query":{"bool":{"must":[{"match_all":[]},{"query_string":{"query":"\"Exporte\""}}]}},"filter":{"bool":{"must":[{"bool":{"should":[{"term":{"__parentPath":"\/sites\/templates"}},{"term":{"__path":"\/sites\/templates"}}]}},{"terms":{"__workspace":["live"]}},{"term":{"__dimensionCombinationHash":"fb11fdde869d0a8fcfe00a2fd35c031d"}},{"term":{"__typeAndSupertypes":"Neos.Neos:Document"}}],"should":[],"must_not":[{"term":{"_hidden":true}},{"range":{"_hiddenBeforeDateTime":{"gt":"now"}}},{"range":{"_hiddenAfterDateTime":{"lt":"now"}}}]}}}},"fields":["__path"],"highlight":{"fields":{"__fulltext*":{"fragment_size":150,"no_match_size":150,"number_of_fragments":2}}},"size":10}'

Example Result:

[...]
{  
   "_index":"typo3cr-agaportal-1503909818",
   "_type":"Neos-NodeTypes:Page",
   "_id":"9aebf7e28d04b49b48b3ee800b12a360386ff381",
   "_score":11.017394,
   "fields":{  
      "__path":[  
         "/sites/templates/node-582db5867c067/node-5800ce5b81d57"
      ]
   },
   "highlight":{  
      "__fulltext.h3":[  
         "Fragen und Antworten zum Thema Allgemeines (Exporte) Fragen"
      ],
      "__fulltext.h1":[  
         "FAQ FAQ - Fragen und Antworten"
      ]
   }
},
[...]

Any idea?

4

It seems to me that somehow the require_field_match doesn’t match.

If I change the highlight part of my query to add "require_field_match":false the default <em></em> Tags are added.

But know clue why this is the case, yet.

5

@daniellienert I have also tested it with the Neos.Demo package, added "flowpack/searchplugin": "3.0.*" and had the same issue. If I’m using the example from elastic.io docu where “match” is used the higlighting works like expected. I tested it with the curl and the contenten of Neos.Demo Page:

curl http://127.0.0.1:9200/typo3cr-neos31demo-1503939206/_search?pretty=true -d '{
   "query":{
     "match":{
        "__fulltext.text":{
           "query":"Neos"
        }
     }
   },
   "highlight":{
      "fields":{
         "__fulltext*":{"fragment_size":30, "number_of_fragments":2}
      }
   },
   "size":5
}'

Would you mind if you can reproduce my issue or do you see on the following steps I did my mistake?

I’m using ES 2.4.5 from Docker, Neos 3.1 with Neos.Demo Package and PHP 7.1.8.

My steps:

  1. Pulled docker image, started the image and added the settings to the elasticsearch.yml

     network.host: 0.0.0.0
 script.inline: true
 cluster.name: es2xdev

 # the following settings are well-suited for smaller Elasticsearch instances (e.g. as long as you can stay on one host)
 index.number_of_shards: 1
 index.number_of_replicas: 0

  2. Changed my \Configuration\Settings.yaml to :

     Neos:
   ContentRepository:
     Search:
       elasticSearch:
         indexName: typo3cr-neos31demo

 Flowpack:
   ElasticSearch:
     ContentRepositoryAdaptor:
       driver:
         version: 2.x
     clients:
       default:
         - host: '127.0.0.1'
           port: 9200

  3. Build my index ./flow nodeindex:build

  4. Added the search to a Page in the backend

  5. Started a search e.g. for “available” and get two hits but no highlighting <em>-tags around the search word

Would be great to know if I forgot something or if I did something horrible wrong…

6

So it seems I found a solution for my problem.

I have created a PR to fix it.

A quick workaround is to change the search query in your site package like this:

prototype(Flowpack.SearchPlugin:Search) < prototype(Neos.Neos:Content) {
    queryStringFields = Neos.Fusion:RawArray {
        fields = Neos.Fusion:RawArray {
            0 = "__fulltext*"
        }
    }
    searchQuery = ${this.searchTerm ? Search.query(site).fulltext(this.searchTerm).request('query.filtered.query.bool.must.1.query_string', this.queryStringFields).nodeType('Neos.Neos:Document') : null}
}

Info: Make sure your site package is loaded after the Flowpack.SearchPlugin