Implementing "search-as-you-type" autocomplete with Elasticsearch and Symfony


♥ You can tap this article to like, comment, and share.


Attention developers! We have an open vacancy for a Senior Full-Stack Web Developer.


Good search results are very important to the end-user, which means helping them find the best possible search terms. For a commerical site, helping users get to the correct product page quickly is a value proposition that would positively impact the revenue stream of the business. With this in mind, could we do better by suggesting possible search results before the user has even finished typing? It's possible with Elasticsearch's completion suggester.

Elasticsearch is a highly scalable full-text search and analytics engine that helps developers build applications that bring out the best in data, whether it's a sophisticated search for products or custom dashboards that can perform complex business intelligence queries. It's open-source, well-supported, and used by industry-leading companies such as eBay, Netflix, Microsoft, and Facebook.

Let's say we're developing a website where users can find restaurants to view menus, booking information, leave reviews etc. The client wants a search box for users to type in a restaurant name to search on and get instant feedback. However, a restaurant could go by different names, or the user may type "Birmingham" or "Italian", wishing to search by properties like location or cuisine style. The client expects to scrape restaurant data from around the world, potentially storing millions of records. This article explores an approach on how to satisfy this requirement using Elasticsearch.

As always, there's a companion Symfony project that offers a working code example.

Integrating Elasticsearch into your Symfony project

Important: The following article was written with Elasticsearch 2.4 in mind. We've since upgraded the code example to be compatible with 5.x and included a Docker environment to run it in. The original is tagged as v1.0; if you're upgrading your own project feel free to reference this commit!

To begin with, let's assume we've yet to install Elasticsearch in our development environment. For Linux/Windows visit the Elasticsearch download page and follow their instructions. For Mac OS X, I'd recommend using Homebrew.

brew tap caskroom/cask && brew update
brew install elasticsearch
brew services start elasticsearch

I'd also highly recommend installing Kibana & Sense, which gives us a nice web-interface to RESTfully query our Elasticsearch instance.

brew install kibana
kibana plugin --install elastic/sense
brew services restart kibana

Next, install and configure the FOSElasticaBundle, making sure that we enable the bundle in our AppKernel. At the time of writing there is no stable 4.x release of FOSElasticaBundle, and so I've had to work around the Elastica library constraint to use a release that supports Elasticsearch 2.x. If there is a stable release, feel free to only require FOSElasticaBundle without a version constraint specified.

composer require friendsofsymfony/elastica-bundle:"^3.2" ruflin/elastica:"3.1.1 as 2.99.0"

Here is a minimal FOSElasticaBundle configuration required for us to begin indexing our Restaurant entity. Add it to config.yml, making sure to also define and set elasticsearch_host and elasticsearch_port in parameters.yml (the host default should be "localhost" and the port default should be 9200).

fos_elastica:
  clients:
      default: { host: "%elasticsearch_host%", port: "%elasticsearch_port%" }
  indexes:
      app:
          finder: ~
          settings: ~
          types:
              restaurant:
                  mappings: ~
                  persistence:
                      driver: orm
                      model: AppBundle\Entity\Restaurant
                      provider: ~
                      listener: ~
                      finder: ~
View on GitHub

There are a couple of things going on here. In Elasticsearch, one index can contain many different types of documents, each with their own fields. This is not too dissimilar from Doctrine's own entity manager, and so for our use-case it makes sense to have a single "app" index where our entities will reside. Our restaurant configuration enables the bundle to hydrate Elasticsearch results from a restaurant type query back into Restaurant objects. It will also listen for any changes to them and update their Elasticsearch document counterparts automatically, since it synchronises the entity and document by primary ID.

Moving on, we still need to define some field mappings so that the bundle knows which Restaurant fields to persist in an Elasticsearch document for us to query on.

Completion suggester mapping

Our goal is to start typing a restaurant name and get suggestions. However, a user may struggle to search for "L'Escargot Blanc" unless they type exactly how the name begins. This is where analyzers come into play. By default, Elasticsearch will apply simple analysis and lowercase all letters when the document is stored at index time, and when the document is queried at search time (note: this doesn't affect how the field's data is actually stored). For our French restaurant this won't be good enough, as savvy users may start typing "es" (omitting "the") and so it won't be suggested. By applying a number of token filters and wrapping it up in a custom analyzer, we can still suggest restaurants with elisions or accents in the name, even if the user doesn't include them in their search term.

settings:
    index:
        analysis:
            analyzer:
                name_analyzer:
                    type: custom
                    tokenizer: standard
                    filter: [standard, lowercase, asciifolding, elision]
View on GitHub

Next, we need to store the restaurant name so it can be queried by the user's search term. Rather than simply map the field, we need to provide a data structure for the completion suggester that describes possible input and what the output is. We'll start by creating a function in our Restaurant class to do this.

public function getNameSuggest()
{
  return array(
    'input' => $this->getName(),
    'output' => $this->getName(),
    'payload' => array(
      'id' => $this->getId(),
    ),
  );
}
View on GitHub

For now the input and output parameters will simply be the restaurant's name, so when a user starts typing the name will be returned as a suggestion, keeping things nice and consistent. We've also included a payload parameter which allows us include whatever we wish (as long as it's serializable to JSON!) and returned as part of the suggestion result. Most of the time you'll want to include the object's ID, as this will enable you to load the suggested entity without performing another search. This is important since our completion query won't be performed using FOSElasticaBundle's finder service, meaning we won't be relying on the bundle to hydrate the results (because suggestions aren't returned as documents).

Let's add our mapping using the new analyzer we configured earlier.

mappings:
    name_suggest:
        type: completion
        analyzer: name_analyzer
        search_analyzer: name_analyzer
        payloads: true
View on GitHub

With the mapping in place we can create and populate the index with a console command. Note: you'll want to do this each time you adjust your index and type configuration during development, or if you manually change a value in the database.

php app/console fos:elastica:populate

Using Sense and executing GET app/restaurant/_search, we can see our restaurants are now stored in Elasticsearch.

{
  "_index": "app",
  "_type": "restaurant",
  "_id": "1",
  "_score": 1,
  "_source": {
    "name_suggest": {
      "input": "Pizza Express",
      "output": "Pizza Express",
      "payload": {
        "id": 1
      }
    }
  }
}

Implementing search-as-you-type

Now that our restaurants are indexed correctly we'll need a UI element that takes a user's search term (while they're typing), performs a suggestion query, and returns any results. For brevity I've omitted the Symfony controller actions, as these can be viewed in the companion project.

First we need a suggestions endpoint. The query itself is quite simple: a new completion suggester is created that uses the name_suggest mapping that we defined earlier ("suggest" is just a name that's used to identify different groups of results if you use more than one suggester). We then use the service automatically exposed by the fos_elastica configuration to search restaurant types by wrapping the suggester in a query.

$completion = new Suggest\Completion('suggest', 'name_suggest');
$completion->setText($text);
$resultSet = $this->get('fos_elastica.index.app.restaurant')->search(Query::create($completion));
View on GitHub

Next I'll be using the Select2 library to supercharge a select box, which will call our suggestions endpoint whenever a user types in it, displaying any restaurants returned from the backend. Let's make it easier for ourselves by mapping the results set to a data structure more suitable for Select2.

foreach ($resultSet->getSuggests() as $suggests) {
  foreach ($suggests as $suggest) {
    foreach ($suggest['options'] as $option) {
      $suggestions[] = array(
        'id' => $option['payload']['id'],
        'text' => $option['text']
      );
    }
  }
}
View on GitHub

To be able to cleanly specify the endpoint URL we'll use the excellent FOSJsRoutingBundle to expose application routing in the frontend. There are multiple ways to expose routes; in this example I'm using annotations so I'll add the expose option to the endpoint.

@Route("/restaurants", name="suggest_restaurants", options={"expose"=true})
View on GitHub

Now it's quite straightforward configuring Select2 to use the suggestions endpoint, since we don't have to process the results. When dealing with a large dataset we'd want to extend this implementation to include pagination (as demonstrated in the Select2 ajax example), and increase the minimum input length (3 is recommended) so the potential search space is smaller.

$search.select2({
  placeholder: 'Type to search',
  ajax: {
    url: Routing.generate('suggest_restaurants'),
    dataType: 'json',
    delay: 250,
    data: function (params) {
      return {
        q: params.term
      }
    },
    processResults: function (data, params) {
      return {
        results: data.suggestions
      }
    }
  },
  cache: true,
  minimumInputLength: 1
});
View on GitHub

There is a lot of scope for what can be done next, such as custom results formatting (the suggestions payload could include an image URL or description to help with this). I've opted for the simple approach of sending the user off to the restaurant page once they select a result.

$search.on('select2:select', function (e) {
  window.location.href = Routing.generate('show_restaurant_redirect', {id: e.params.data.id});
});
View on GitHub

In reality, if the user has no suggestions to select from, they should be able to transition to performing a full-text search. Although it's outside the scope of this article, it's worth keeping in mind that we're building a feature that will improve the user's search experience, not something that will replace search entirely.

Improving results

We're off to a good start: our users can start typing a restaurant name and get meaningful suggestions back. But what if they don't quite know the name of the restaurant they're searching for? Perhaps they start typing "grill", why isn't "PJ's Bar & Grill" returned?

This is because Elasticsearch isn't searching for words in a phrase using a regular match query, rather it's using something called a finite state transducer (FST). An FST is a big graph that the completion suggester builds once it's fed all possible completions (i.e. our inputs) where each input is broken into indivdual paths by character. Starting from left-to-right, Elasticsearch traverses the graph until it runs out of input before returning all possible endings of the current path, allowing for extremely fast low-memory access to valid suggestions. Additionally, common suggestions are treated with higher relevance, as word frequency isn't taken into account like it would do with a full-text search.

So, our solution to the problem of the user starting mid-sentence or using related phrases is providing the completion suggester multiple inputs to assist in covering different search phrases. This means we can add our restaurant's city and cuisine styles as inputs so that users can search by these terms and get suggestions.

array_merge(
  array(
    $this->getName(),
    $this->getCity(),
  ),
  $this->getStyles()
);
View on GitHub

We can also control the order in which results are returned. For example, perhaps we'd like to bubble up restaurants with good ratings, or those who have paid to be promoted. Here's a very simple demonstration of how that can be achieved.

public function calculateWeight()
{
  $weight = 0;

  if ($this->isPromoted()) {
    $weight += 5;
  }

  return $weight;
}
View on GitHub

And the final result of our updated completion suggester data structure.

array(
  'input' => array_merge(
    array(
      $this->getName(),
      $this->getCity(),
    ),
    $this->getStyles()
  ),
  'output' => $this->getName(),
  'weight' => $this->calculateWeight(),
  'payload' => array(
    'id' => $this->getId(),
  ),
);
View on GitHub

We can also modify the completion suggester query to introduce a factor of fuzziness. This means that if a user starts typing "pza" instead of "pizza", they'll still get back suggestions such as "Pizza Express". However, fuzziness can be a double-edged sword, as you may find less relevant results are included.

$completion->setFuzzy(array('fuzziness' => 2));
View on GitHub

On the subject of relevance, Elastic (the creators of Elasticsearch) recommend the following:

  • Continuously log searches. Find the most important/common ones, and add good suggestions first.
  • Additionally, log the suggestion(s) selected by the user.
  • Refine your inputs based on what users are searching and what they're selecting. See if there are any obvious gaps.
  • Use weights with reasonable logic behind them, in accordance with whatever your definition of a best result is.

Rounding up

In the end it was fairly straightforward implementing search-as-you-type functionality thanks to existing open-source libraries, with the query itself only being a handful of lines long. In-fact, most of the work effort was upfront in understanding how Elasticsearch and the completion suggester works. Today we only scratched the surface regarding what is possible, and while (thankfully) the online documentation is great, I can't stress enough the need to fully understand how you wish to index and query your data to get the most out of Elasticsearch.

As for the completion suggester: it is extremely fast, but it does require getting ahead of your user's search terms and fine-tuning inputs to provide the most relevant results (so they don't have to search at all!) For searches that are less predictable in terms of word order, another technique called edge n-grams may be better suited. If your problem space involves dealing with the human language (e.g. stemming), deep-searching of documents or where relevance is king, then the completion suggester is not a viable path and you'd be better off simply performing a slower but more relevant full-text search.


What did you think? Rate this article below.

(Rated 4 by 4 ratings)

Comments

Leave a comment

Simple markdown syntax supported: *italic*, **bold**, > quote, `code`, etc.

Subscribe to our newsletter

We don’t send many emails. You’ll just be notified of new blog articles (for now)

Let’s talk about how our software can work for you.

Get in touch