Getting started with Full-Text Search (part 1 of 5)

You will not find any quick, easy-to-apply advice on deploying Solr or Elasticsearch here, nor how to trigger your first search queries using one of these. Instead we will touch on some other topics: the fundamental principles behind full-text search and the kinds of problems you may encounter in creating your own search engine.

But you ask: “Why would I need to know the inner-workings of full-text search?”

After all, it isn’t likely that you will ever have to build your own search engine from scratch. There are plenty of time-tested, out-of-the-box search engines on the market for you to use.

So why can’t you simply trigger a search request on Solr and get great results? Of course, you can!

But if your search is a bit more complex than a default request with some filtering added, then—sooner or later—you will discover that the search system is not magic. And to get the best results, you will need to understand how it behaves and how to make it perform to meet your requirements.

In other words, at some point, there’s a big chance that you will need to understand exactly how your system gathers results, and the purpose of this or that portion of the code. You will need to learn what tokenization is; how ranking works; why stemming is required; and, finally, when normalization (what’s that, by the way?) should be applied.

So, in this article, we will give you a brief introduction to each of these principles, explaining the basic components involved in nearly every full-text search process.

How will we keep all this technical stuff interesting along the way?

Simple! We’ll code something real, creating our own small search engine as we go. By the time we finish, our search engine will be capable of performing full-text searches and of returning an adequate result set.

What will it look like?

By the end of this article we will have created a shell application capable of performing a full-text search through a list of specified text files. The search will return a result-set displaying the names of those files with contents logically related to the query supplied.

If you want to fully understand the concepts being presented and see the whole process of the evolution of the search engine application, we strongly recommend that you read this article from start to finish. If you do, you will clearly grasp of concepts and methods used as you see them introduced step-by-step into the application.

If, on the other hand, you’re a “too long; didn’t read” person, then simply scroll to the end of the article. There, in the “Tutorial Conclusion,” you can get a very brief synopsis of the entire article.

So, let’s get started.

Before we really get underway, we need to lay some groundwork. In the end, we will have a CLI (Command-Line Interface) application that will accept user queries and return the relevant result-set to the user. However, we don’t really want to spend too much time implementing the shell itself. We would rather use something readily available and easily adaptable to our requirements. After all, this is an article about search technologies, and not about how to create an effective CLI utility.

Speaking of technologies: for building the CLI we will use Spring Boot and Spring Shell. We have chosen to use Kotlin as our programming language as we believe it will be a great fit for our use-case. Even more importantly, we find it is a pleasure to do something in a Kotlin way. (However, feel free to use your favorite development tool and adapt the examples according to your wishes.)

So, how should we get started?

First, we will create an empty Maven project and define dependencies on Spring Boot, Spring Shell, Kotlin and a couple of other libraries we will need.

Next, and before we start working on the search engine itself, we will check that our initial setup works properly and assure ourselves that we have a working CLI utility.

Since we have not yet added any commands, only built-in commands are currently available. To get a list of them, just type help and then press Enter.

All seems to be working fine. To exit the shell, type the exit command and press Enter again.

Okay! Now we are ready to get started working on our new search engine.

Let’s begin by formulating the concept on which we will be working.

We have put together (in the repository linked above) a set of fairytales (this is the content of two books available through Project Gutenberg—“The Yellow Fairy Book” and “Stories by English Authors”) with each story in a separate text file. In total, you should have 56 files when retrieved from the repository.

By the end of this project, we expect to have an application that we can use to get a list of stories matching the text search query we supplied. Or, speaking in more technical language, we will have full-text search capabilities across the 56 files and our result-set will be the names of the files where the contents of the files match our search request. For this case, we assume that the file name is a unique identifier of a story (which is technically correct since every filename in the folder is unique and defines a specific story).

Although it doesn’t really matter exactly where the files are to be located, for simplicity, let’s place all available content into the resources/stories folder. Later we will get the location of the files as part of our service’s configuration.

As we get started, let’s externalize the application’s configuration. This will allow us to gather all of the settings for the application from a config file, or even from the command-line. By doing this, we can easily change the parameters for our application without have to rebuild the whole program to do so.

We already have some data we want to store in our configuration. Specifically, we want the configuration to remember where the raw content files (that is, the stories in which we are going to search) are located.

Since we have selected Spring Framework as our main building tool, the default configuration should be stored in the application.yaml file.

How do we go about getting these configuration data out of application.yaml?

Well, Spring’s standard way would be to use the @Value annotation on the component’s property where you want to initialize the value from the application.yaml file data.

However, since our expectations are that we will have multiple configuration parameters in the future, we will put them all together into a separate component. It is more convenient to have a single object responsible for all the settings than it is to spread different parameters across different parts of the application.

In the configuration component we will create two methods: one to get the path to the contents to be searched, and another that simply resupplies all of the predefined application parameters.

Also, for making new features available to the end-user, we would need to define new shell commands. In this case, we need the one that displays all current system settings.

Now, let’s check how our newly added command works. To do that, we need to run our application and then ask for help.

As we can see, there is the new command config listed in the shell. Let’s try it and see what it does.

Are we ready to start working on the search?

Hmm, probably not!

We’re still missing two things: 1) we need a list of all the files, and 2) we need to have the ability to retrieve the contents of each individual story, along with the story’s unique identifier. (Remember: the file name is the story’s unique identifier.)

In order to meet these requirements, we will create a new component responsible for providing each document’s data. We will call this component DocumentService.

Let’s make sure that everything works as we expect. But in order to do that, we have to first add another command.

Okay. Let’s see how it works.

Continue reading - Getting started with Full-Text Search (part 2 of 5).