Full text search with Elasticsearch of both Neos nodes and custom Elasticsearch documents

Hi guys,

I am currently using the Flowpack.ElasticSearch.ContentRepositoryAdaptor and skurfuerst/Flowpack.SearchPlugin packages for integration between Elasticsearch and Neos - for indexing Neos nodes and performing a full text search.

But in my site I have a couple of custom plugins, which fetch and display data from external data sources - Contentful and Podio. And I need the site full text search results to include not only Neos pages, but also the pages with external data, which my custom plugins create.

I guess that I need to create custom documents in Elasticsearch and index the data from the external sources in it. But after that I am not sure how should I proceed and display the combined full text search results of both Neos pages and custom documents, paginated and ordered by relevance.

Any ideas of how this should be resolved are greatly appreciated!

Thanks in advance!

Hey Pavlina,

I’ve been trying to think this case through a little. I fully agree that you “somehow” need to get the data into ElasticSearch first (through custom plugins etc). I’d probably create a custom index for this.

To include the index during search, I’d probably add it to the index alias here; so we’d need some custom additions or a custom hook in the CR adaptor here. Feel free to open a Pull Request here :smile:

For indexing, I’d suggest that you use the _all field; so ensure that this is properly filled when indexing your “proprietary” documents.

After the index is included during search, I think there need to be some adjustments during rendering – but you’ll probably see where errors occur once the “other” documents are included in the search results.

Don’t hesitate to ask further if you have more questions :slight_smile: I’d be happy to help and get this feature into the core of the CR adaptor.

All the best,
Sebastian

Hi Sebastian,

thanks a lot for your answer!

After some consideration I decided to make a separate application to take care of indexing the external data in ES, using a separate index or two indexes and adding them to the same alias, which the typo3cr index uses.

I also had a look at the TYPO3.TYPO3CR.Search and the Flowpack.ElasticSearch.ContentRepositoryAdaptor packages and noticed that the ElasticSearchQueryBuilder class can be replaced with other implementation. And I am thinking to reuse this class and adjust the ES query, so it searches inside typo3cr index and my indexes and returns the combined search results. After that rendering of the search results must be adjusted too. I need also to make some aggregation filters of all data, which this class supports.

What do you think, does this make sense and seems as a working solution?

KR,
Pavlina