ElasticSearch & Kibana – Must for All Search Engine Development

How often there is a need to create a search engine within an organisation or even on a customer portal where search is fast, secure, accurate and learns over a period of time , the article shows how you can use Elasticsearch along with Kibana to build a world class search products.

Interms of ElasticSearch , you can consider each index as a database table and each index can have multiple line items , example a bookshop index can have many books which needs to be searched.

Kibana is a front end UI that interacts with Elastic Search where most of these content management activities can be done

No alt text provided for this image

Most Important part about search is not only being able to return results faster but also be able to manage synonyms example – user searching for ‘Bakery item‘ can have results that has ‘Cooking cakes ‘ , user looking for book on ‘Paneer‘ might actually be looking for a book on ‘Cottage cheese

let us start by creating an index , Use Dev Tools in Kibana Web UI

Request 
PUT bookshop
{
"settings" : {
"index" : {
"number_of_shards" : 1,
"number_of_replicas" : 1
}
}
}
Response
{
"acknowledged" : true,
"shards_acknowledged" : true,
"index" : "bookshop"
}
No alt text provided for this image

Let us add few books to bookshop

 Request
PUT bookshop/books/1
{
"title": "Bake with Shivesh",
"description" : "Food styling has become a skill many want to master, but don't know how. Popular food blogger and maverick baker Shivesh Bhatia is here to help. Twenty-two-year-old Shivesh enjoys a massive following on his blog and Instagram. Brands love him and so do people. In Bake with Shivesh, the ace baker reveals foolproof tips on food styling that can be easily followed at home, in your kitchen, with tools you already own. He also talks about his favourite styling techniques, and what works or doesn't on different social media platforms. This is a book for everyone looking to elevate the way they present food, to help boost their blogs and businesses, and to make food look as good as it tastes.",
"author": "Shivesh Bhatia"
}
Response
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "1",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 0,
"_primary_term" : 1
}

Add few more books

 Request
PUT bookshop/books/2
{
"title": "Can't Believe It's Eggless!",
"description" : "If you’re someone that does not eat eggs, baking is the final frontier. And this exact problem is what master baker and acclaimed entrepreneur, Pooja Dhingra, resolves with her third book, I Can’t Believe It’s Eggless. Having baked without eggs herself, and catered to customers at Le15, her beloved chain of patisseries and cafes, she knew that a book of eggless recipes written for the Indian kitchen was a long time coming. And so Pooja combined her skills as a classically-trained patissier with her love for taking on a challenge to create over a hundred eggless recipes for everything from cookies and brownies to decadent cakes, doughnuts and mousses. I Can’t Believe It’s Eggless lives up to its name with delicious recipes that are second to none. Plus, Pooja offers invaluable tips on how to stock your pantry, choose the right substitute for eggs, and perfect the art of baking. ",
"author": "Pooja Dhingra"
}
PUT bookshop/books/3
{
"title": "How to Cake It: A Cakebook",
"description" : "YOLANDA GAMPP is a self-taught master cake artist who started out baking novelty cakes in her mother’s kitchen and is now the host of the successful YouTube channel “How to Cake It.” “How to Cake It” has built a community of more than six million cake enthusiasts from around the world. Yolanda and her work have been featured on the Today show, The Social, Cityline, Breakfast Television, the Food Network show SugarStars, and in BuzzFeed. A graduate of George Brown College’s culinary arts program in Toronto, Yolanda quickly discovered that her true passion was working with sweets, and she was further inspired by her father, who worked as a baker. She lives in Toronto. ",
"author": "Yolanda Gampp"
}
Response
PUT bookshop/books/2
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "2",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 1,
"_primary_term" : 1
}
PUT bookshop/books/3
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "3",
"_version" : 1,
"result" : "created",
"_shards" : {
"total" : 2,
"successful" : 1,
"failed" : 0
},
"_seq_no" : 2,
"_primary_term" : 1
}

Search Books

GET bookshop/_search
{
"query": { "match": { "title": "Cake" } }
}
Response
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.8474839,
"hits" : [
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "3",
"_score" : 0.8474839,
"_source" : {
"title" : "How to Cake It: A Cakebook",
"description" : "YOLANDA GAMPP is a self-taught master cake artist who started out baking novelty cakes in her mother’s kitchen and is now the host of the successful YouTube channel “How to Cake It.” “How to Cake It” has built a community of more than six million cake enthusiasts from around the world. Yolanda and her work have been featured on the Today show, The Social, Cityline, Breakfast Television, the Food Network show SugarStars, and in BuzzFeed. A graduate of George Brown College’s culinary arts program in Toronto, Yolanda quickly discovered that her true passion was working with sweets, and she was further inspired by her father, who worked as a baker. She lives in Toronto. ",
"author" : "Yolanda Gampp"
}
}
]
}
}

PROBLEM comes when user is searching for a book on baking and we instead have a book on making cakes, We get 0 results as shown below

GET bookshop/_search
{
"query": { "match": { "title": "Baking" } }
}
Response
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 0,
"max_score" : null,
"hits" : [ ]
}
}

ElasticSearch Synonyms come our to rescue.. – Would it not be great if we say that a book on baking is same as a book on making cakes, or a book on tandoori chicken is same as Indian non veg cooking ?

So let us start building a Synonym where we say Cake, Bakery, Baking are all related words and see how it works

PUT /bookshop/_settings
{
"settings": {
"index" : {
"analysis" : {
"filter" : {
"synonym_filter" : {
"type" : "synonym",
"synonyms" : [
"Cake, Bakery, Baking"
]
}
},
"analyzer" : {
"synonym_analyzer" : {
"tokenizer" : "standard",
"filter" : ["lowercase", "synonym_filter"]
}
}
}
}
},
"mappings": {
"books": {
"properties": {
"title": {
"type": "text",
"analyzer": "synonym_analyzer"
}
}
}
}
}
Response
{
"error": {
"root_cause": [
{
"type": "illegal_argument_exception",
"reason": "Can't update non dynamic settings [[index.analysis.filter.synonym_filter.type, index.analysis.filter.synonym_filter.synonyms, index.analysis.analyzer.synonym_analyzer.filter, index.analysis.analyzer.synonym_analyzer.tokenizer]] for open indices [[bookshop/2QUYHij-TfKLOLcGVindog]]"
}
],
"type": "illegal_argument_exception",
"reason": "Can't update non dynamic settings [[index.analysis.filter.synonym_filter.type, index.analysis.filter.synonym_filter.synonyms, index.analysis.analyzer.synonym_analyzer.filter, index.analysis.analyzer.synonym_analyzer.tokenizer]] for open indices [[bookshop/2QUYHij-TfKLOLcGVindog]]"
},
"status": 400
}

Since the was index is already made it is not allowing us to add a Synonym , its important to close the index , create synonym and then reopen

You might want to delete the index created before adding Synonyms or just close and reopen index

DELETE  /bookshop/?pretty
POST /bookshop/_close
PUT /bookshop/_settings
{
"settings": {
"index" : {
"analysis" : {
"filter" : {
"synonym_filter" : {
"type" : "synonym",
"synonyms" : [
"Cake, Bakery, Baking"
]
}
},
"analyzer" : {
"synonym_analyzer" : {
"tokenizer" : "standard",
"filter" : ["lowercase", "synonym_filter"]
}
}
}
}
},
"mappings": {
"books": {
"properties": {
"title": {
"type": "text",
"analyzer": "synonym_analyzer"
}
}
}
}
}
POST /bookshop/_open
Response
POST /bookshop/_close
{
"acknowledged" : true
}
PUT /bookshop/_settings
{
"acknowledged" : true
}
POST /bookshop/_open
{
"acknowledged" : true,
"shards_acknowledged" : true
}
No alt text provided for this image
GET bookshop/_search
{
"query": { "match": { "title": "How to Bake It: A Cakebook" } }
}
Observe the Results Ranking below -
Response
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 4.237419,
"hits" : [
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "3",
"_score" : 4.237419,
"_source" : {
"title" : "How to Cake It: A Cakebook",
"description" : "YOLANDA GAMPP is a self-taught master cake artist who started out baking novelty cakes in her mother’s kitchen and is now the host of the successful YouTube channel “How to Cake It.” “How to Cake It” has built a community of more than six million cake enthusiasts from around the world. Yolanda and her work have been featured on the Today show, The Social, Cityline, Breakfast Television, the Food Network show SugarStars, and in BuzzFeed. A graduate of George Brown College’s culinary arts program in Toronto, Yolanda quickly discovered that her true passion was working with sweets, and she was further inspired by her father, who worked as a baker. She lives in Toronto. ",
"author" : "Yolanda Gampp"
}
},
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "1",
"_score" : 1.1220686,
"_source" : {
"title" : "Bake with Shivesh",
"description" : "Food styling has become a skill many want to master, but don't know how. Popular food blogger and maverick baker Shivesh Bhatia is here to help. Twenty-two-year-old Shivesh enjoys a massive following on his blog and Instagram. Brands love him and so do people. In Bake with Shivesh, the ace baker reveals foolproof tips on food styling that can be easily followed at home, in your kitchen, with tools you already own. He also talks about his favourite styling techniques, and what works or doesn't on different social media platforms. This is a book for everyone looking to elevate the way they present food, to help boost their blogs and businesses, and to make food look as good as it tastes.",
"author" : "Shivesh Bhatia"
}
}
]
}
}

If We want to fine tune our search with Title containing Bake and Author as Shivesh, we need to use a bool and must , match together

 GET bookshop/_search
{
"query": {
"bool": {
"must": [
{ "match": { "title": "Bake" } },
{ "match": { "author": "Shivesh" } }
]
}
}
}
Response
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 2.102898,
"hits" : [
{
"_index" : "bookshop",
"_type" : "books",
"_id" : "1",
"_score" : 2.102898,
"_source" : {
"title" : "Bake with Shivesh",
"description" : "Food styling has become a skill many want to master, but don't know how. Popular food blogger and maverick baker Shivesh Bhatia is here to help. Twenty-two-year-old Shivesh enjoys a massive following on his blog and Instagram. Brands love him and so do people. In Bake with Shivesh, the ace baker reveals foolproof tips on food styling that can be easily followed at home, in your kitchen, with tools you already own. He also talks about his favourite styling techniques, and what works or doesn't on different social media platforms. This is a book for everyone looking to elevate the way they present food, to help boost their blogs and businesses, and to make food look as good as it tastes.",
"author" : "Shivesh Bhatia"
}
}
]
}
}

So there 2 magic’s that ElasticSearch has done

  1. Searching for Bake will bring in a book by title Cake
  2. It Ranks 2 books containing the word Bake but the one with the closest search word match is ranked higher in search results

Enjoying Searching 🙂 and hit like if you really liked reading about it

https://www.linkedin.com/pulse/elasticsearch-kibana-must-all-search-engine-development-rao/