December 10, 20207 min read

Development Environment Set-up for a Custom Elasticsearch Plugin

Originally published on Medium

Image

In the last few months, I have been spending most of my time creating a search engine using Elasticsearch. The team decided to go with this technology based on its simple REST APIs, distributed nature, speed, and scalability.

I started prototyping in Python. It was easy to port most of the implementation’s components to Elasticsearch; however, there is a custom ranking model that is not available in the similarity module. The complexity of the algorithm is high. It would take forever to apply it to all the retrieved documents in a search.

The rescore functionality was the solution to the problem. The rescore model is only applied to the top N documents that result from an initial query. The latter allows the user to define a window size that meets their requirements of response time, recall, and precision. The final decision was to create a custom plugin on top of the Elasticsearch rescore module.

The Challenges

The implementation of a custom plugin requires knowledge of how Elasticsearch works and its components. The initial tasks that helped to finish on time an initial proof of concept and followed by the deployment in production were:

  1. Define a set of tools to navigate through the code, highlight errors, autocorrect, code formatting, and debugging.
  2. Understand the classes and methods that form the plugin and the importance of each of them.

Considering the previous topics, I am splitting the information into two articles. In this article, I explain the process of selecting an IDE, and the configuration of the project to effectively work on it. In an additional blog post, I will explain the most important classes and methods of a custom plugin and how to modify them to create a simple rescore plugin.

The Tools

The main ingredients to efficiently build the plugin are an integrated development environment (IDE) and Docker.

IDE

The IDE that I used to develop the plugin is Visual Studio Code (VS Code). I liked the fact that it is an open-source editor, and there is a big community maintaining it. Another option is Intellij IDEA. There is a free version that has a subset of features but you have to pay for the complete version.

VS Code is technically only an editor, but you can extend its functionality by installing plugins. The plugins that I used to develop the plugin are:

In case you are also new to developing in Java, the Java in Visual Studio Code documentation is useful to understand the IDE capabilities and some extra information.

The Project Structure

An Elasticsearch plugin can be stand-alone in a repository. The Gradle plugin build-tools handles the dependencies of Elasticsearch. The project structure looks like this:

├── build.gradle
└── src
    ├── main
    │   └── java
    ├── test
    │   └── java
    └── yamlRestTest
        ├── java
        └── resources
            └── rest-api-spec

The build.gradle contains the specifications of the project. The src/main and src/test are the implementation and unit tests. The yamlRestTest folder has the definition of the integration tests.

Dependencies:

  • Open JDK 15
  • Gradle 6.7.1

The focus of this post is to set up the project. We start from the example rescore plugin in the Elasticsearch repository. There is no need to keep all the files. We just need to copy the rescore plugin to its own folder. Make sure to checkout to version 7.10.0 before copying the folder.

$ git clone https://github.com/elastic/elasticsearch.git ~/elasticsearch
$ cd ~/elasticsearch
$ git checkout v7.10.0
$ cp -r ~/plugins/examples/rescore ~/rescore
$ cd ~/rescore

At the end of this article, you can find a GitHub repository that contains the project. You can review the commits to analyze the step by step to get to the final project.

Setting up the project

After opening the project with VS code, a new file .vscode/settings.json is automatically generated. If not, you can create it yourself. Inside this file, you have to specify the path to the OpenJDK and Gradle folders.

{
    "java.home": "/path/to/jdk-15.0.1",
    "java.import.gradle.home": "/path/to/gradle-6.7.1",
}

If you try to build the java project, it won’t work. Before, we need to install the dependencies of Elasticsearch. So, at the beginning of the build.gradle add the following buildscript object:

buildscript {
  ext {
    version_es = "7.10.0"
  }

repositories {
    mavenCentral()
    jcenter()
  }

dependencies {
    classpath "org.elasticsearch.gradle:build-tools:${version_es}" 
  }
}

The last step is to create an empty file NOTICE.txt, as it is required to compile the plugin. If you are interested to know the ideal content of this file you can read Assembling LICENSE and NOTICE files — Apache Infrastructure. However, it can be empty.

To verify that the configuration process is correct, run the unit and integration tests:

$ gradle test yamlRestTest

=======================================
Elasticsearch Build Hamster says Hello!
  Gradle Version        : 6.7.1
  OS Info               : Linux 4.19.128-microsoft-standard (amd64)
  JDK Version           : 15 (OpenJDK)
  JAVA_HOME             : /opt/java/jdk-15.0.1
  Random Testing Seed   : 6B35F021BB7E61CB
  In FIPS 140 mode      : false
=======================================

BUILD SUCCESSFUL in 10s
12 actionable tasks: 3 executed, 9 up-to-date

Install the Plugin

All the tests passed, which is a good sign that the plugin is compiled and runs as expected. To execute some queries, we need to install it. The process is simple, only assemble it:

$ gradle assemble

The assembled package is stored in ./build/distributions/example-rescore.zip. The following is the command to install it, if you want more details you can click the link to the official documentation.

$ ./bin/elasticsearch-plugin install file:///path/to/plugin.zip

The process of creating the zip file, installing the plugin, and starting Elasticsearch can be automated. I have created a Docker file with each of the stages to prepare and run Elasticsearch. The automation reduces everything to run (Make sure to clone this project before running the command):

$ docker-compose up --build cache

To confirm the correct plugin installation, just call the endpoint:

$ curl -X GET http://localhost:9201/_cat/plugins

It should print the information about the plugin.

The Bulk API performs multiple indexing or delete operations in a single call. Here is an example to insert some records:

$ curl -X POST http://localhost:9201/_bulk -H 'Content-Type: application/json' -d '
{ "index" : { "_index" : "test", "_id" : "1" } }
{ "field1" : "value1", "field2": 1.2 }
{ "index" : { "_index" : "test", "_id" : "2" } }
{ "field1" : "value2", "field2": 1.1 }
{ "index" : { "_index" : "test", "_id" : "3" } }
{ "field1" : "value3", "field2": 5 }
'

Finally, we can use the plugin. An example of a query that applies the rescore model to all the documents looks like this:

$ curl -X GET http://localhost:9201/test/_search?pretty=true -H 'Content-Type: application/json' -d '
{
   "rescore": {
      "example": {
         "factor": 2,
         "factor_field":  "field2"
      }
   }
}'

The rescore calculation is the value of the field2 attribute times the factor. As stated previously, we can define a first query that applies a fast ranking model to sort the documents and then rescore them. Just add a query at the beginning of the body.

$ curl -X GET http://localhost:9201/test/_search?pretty=true -H 'Content-Type: application/json' -d '
{
   "query": {
      "match": {
         "field1": "value1"
      }
   },
   "rescore": {
      "example": {
         "factor": 2,
         "factor_field": "field2"
      }
   }
}'

How to Debug

Debugging is one of the most critical processes in the software development. It is important to understand the codebase and to find incorrect behavior.

Create the configuration in the file .vscode/launch.json. This configuration indicates the IDE to connect to a remote JVM that is mapped to the port 5005.

{
    "version": "0.2.0",
    "configurations": [
        {
            "type": "java",
            "name": "Debug",
            "request": "attach",
            "hostName": "localhost",
            "port": 5005
        }
    ]
}

To debug Elasticsearch, you need to start it in debug mode:

$ ./bin/elasticsearch -Xdebug -Xrunjdwp:server=y,transport=dt_socket,address=*:5005,suspend=y

The server waits until the IDE is connected. For some reason, it does not start at the first attempt, so click the start debugging button two or three times.

There is also another configuration that automates starting the Elasticsearch debugging server. In the same project, the file docker-compose.debug.yml has the definition. The way to run it is:

$ docker-compose -f docker-compose.yml -f docker-compose.debug.yml up --build cache

Some extra resources about debugging in VS Code:

Conclusion

In this article, it is described step by step the steps you need to follow to set up a project for an Elasticsearch plugin. Also, it is explained how to insert some documents in the database and to search using the plugin.

Repository

ocampor/plugin-example

← Back to the blog