The Blast is a full text search and indexing server written in Go built on top of the Bleve, Bolt and Raft.
Blast server provides functions through gRPC (HTTP/2 + Protocol Buffers) or traditional RESTful API (HTTP/1.1 + JSON), and uses Raft to achieve consensus across all the instances of the nodes, ensuring that every change made to the system is made to a quorum of nodes, or none at all.
Blast makes it easy for programmers to develop search applications with advanced features.
- Full-text search and indexing
- Faceting
- Result highlighting
- Easy deployment
- Bringing up cluster
- Index replication
- An easy-to-use HTTP API
- CLI is also available
Building Blast requires Go 1.9 or later. To build Blast on Linux like so:
$ git clone git@github.com:mosuka/blast.git
$ cd blast
$ make buildIf you want to build Blast other platform, please set GOOS、GOARCH like following:
$ make GOOS=darwin buildYou can see the binary file when build successful like so:
$ ls ./bin
blastRunning a Blast node is easy. Start Blast node like so:
$ ./bin/blast start --bind-addr=localhost:10000 --grpc-addr=localhost:10001 --http-addr=localhost:10002 --node-id=node0 --raft-dir=/tmp/blast/noade0/raft --store-dir=/tmp/blast/node0/store --index-dir=/tmp/blast/node0/indexYou can now put, get, search and delete the document(s) via CLI.
Putting a document is as following:
$ cat ./example/doc1.json | xargs -0 ./bin/blast put --grpc-addr=localhost:10001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"success": true
}Getting a document is as following:
$ ./bin/blast get --grpc-addr=localhost:10001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"id": "1",
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"success": true
}Deleting a document is as following:
$ ./bin/blast delete --grpc-addr=localhost:10001 --pretty-print 1
You can see the result in JSON format. The result of the above command is:
{
"success": true
}Bulk update is as following:
$ cat ./example/bulk_put_request.json | xargs -0 ./bin/blast bulk --pretty-print
You can see the result in JSON format. The result of the above command is:
{
"put_count": 7,
"success": true
}Searching documents is as like following:
$ cat ./example/search_request.json | xargs -0 ./bin/blast search --grpc-addr=localhost:10001 --pretty-printYou can see the result in JSON format. The result of the above command is:
{
"search_result": {
"facets": {
"Category count": {
"field": "category",
"missing": 0,
"other": 0,
"terms": [
{
"count": 4,
"term": "library"
},
{
"count": 3,
"term": "server"
}
],
"total": 7
},
"Popularity range": {
"field": "popularity",
"missing": 0,
"numeric_ranges": [
{
"count": 2,
"max": 4,
"min": 3,
"name": "more than or equal to 3 and less than 4"
},
{
"count": 2,
"min": 5,
"name": "more than or equal to 5"
},
{
"count": 1,
"max": 2,
"min": 1,
"name": "more than or equal to 1 and less than 2"
},
{
"count": 1,
"max": 3,
"min": 2,
"name": "more than or equal to 2 and less than 3"
},
{
"count": 1,
"max": 5,
"min": 4,
"name": "more than or equal to 4 and less than 5"
}
],
"other": 0,
"total": 7
},
"Release date range": {
"date_ranges": [
{
"count": 4,
"end": "2010-12-31T23:59:59Z",
"name": "2001 - 2010",
"start": "2001-01-01T00:00:00Z"
},
{
"count": 2,
"end": "2020-12-31T23:59:59Z",
"name": "2011 - 2020",
"start": "2011-01-01T00:00:00Z"
}
],
"field": "release",
"missing": 0,
"other": 0,
"total": 6
}
},
"hits": [
{
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"fragments": {
"description": [
"Bleve is a full-text search and indexing library for Go."
],
"name": [
"\u003cmark\u003eBleve\u003c/mark\u003e"
]
},
"id": "1",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"bleve": [
{
"array_positions": null,
"end": 5,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java.",
"name": "Lucene",
"popularity": 4,
"release": "2000-03-30T00:00:00Z"
},
"fragments": {
"description": [
"Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java."
],
"name": [
"\u003cmark\u003eLucene\u003c/mark\u003e"
]
},
"id": "2",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"lucene": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python.",
"name": "Whoosh",
"popularity": 3,
"release": "2008-02-20T00:00:00Z"
},
"fragments": {
"description": [
"Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python."
],
"name": [
"\u003cmark\u003eWhoosh\u003c/mark\u003e"
]
},
"id": "3",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"whoosh": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Ferret is a super fast, highly configurable search library written in Ruby.",
"name": "Ferret",
"popularity": 2,
"release": "2005-10-01T00:00:00Z"
},
"fragments": {
"description": [
"Ferret is a super fast, highly configurable search library written in Ruby."
],
"name": [
"\u003cmark\u003eFerret\u003c/mark\u003e"
]
},
"id": "4",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"ferret": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project.",
"name": "Solr",
"popularity": 5,
"release": "2006-12-22T00:00:00Z"
},
"fragments": {
"description": [
"Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project."
],
"name": [
"\u003cmark\u003eSolr\u003c/mark\u003e"
]
},
"id": "5",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"solr": [
{
"array_positions": null,
"end": 4,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Elasticsearch is a search engine based on Lucene, written in Java.",
"name": "Elasticsearch",
"popularity": 5,
"release": "2010-02-08T00:00:00Z"
},
"fragments": {
"description": [
"Elasticsearch is a search engine based on Lucene, written in Java."
],
"name": [
"\u003cmark\u003eElasticsearch\u003c/mark\u003e"
]
},
"id": "6",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"elasticsearch": [
{
"array_positions": null,
"end": 13,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Blast is a full-text search and indexing server written in Go, built on top of Bleve.",
"name": "Blast",
"popularity": 1,
"release": "2017-01-13T00:00:00Z"
},
"fragments": {
"description": [
"Blast is a full-text search and indexing server written in Go, built on top of Bleve."
],
"name": [
"\u003cmark\u003eBlast\u003c/mark\u003e"
]
},
"id": "7",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"blast": [
{
"array_positions": null,
"end": 5,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
}
],
"max_score": 0.12163776688600772,
"request": {
"explain": false,
"facets": {
"Category count": {
"field": "category",
"size": 10
},
"Popularity range": {
"field": "popularity",
"numeric_ranges": [
{
"max": 1,
"name": "less than 1"
},
{
"max": 2,
"min": 1,
"name": "more than or equal to 1 and less than 2"
},
{
"max": 3,
"min": 2,
"name": "more than or equal to 2 and less than 3"
},
{
"max": 4,
"min": 3,
"name": "more than or equal to 3 and less than 4"
},
{
"max": 5,
"min": 4,
"name": "more than or equal to 4 and less than 5"
},
{
"min": 5,
"name": "more than or equal to 5"
}
],
"size": 10
},
"Release date range": {
"date_ranges": [
{
"end": "2010-12-31T23:59:59Z",
"name": "2001 - 2010",
"start": "2001-01-01T00:00:00Z"
},
{
"end": "2020-12-31T23:59:59Z",
"name": "2011 - 2020",
"start": "2011-01-01T00:00:00Z"
}
],
"field": "release",
"size": 10
}
},
"fields": [
"*"
],
"from": 0,
"highlight": {
"fields": [
"name",
"description"
],
"style": "html"
},
"includeLocations": false,
"query": {
"query": "name:*"
},
"size": 10,
"sort": [
"-_score"
]
},
"status": {
"failed": 0,
"successful": 1,
"total": 1
},
"took": 446410,
"total_hits": 7
},
"success": true
}Also you can do above commands via HTTP REST API that listened port 10001 (address port + 1).
Putting a document via HTTP is as following:
$ curl -X PUT 'http://localhost:10002/rest/1?pretty-print=true' -d @./example/doc1.jsonYou can see the result in JSON format. The result of the above request is:
{
"success": true
}Getting a document via HTTP is as following:
$ curl -X GET 'http://localhost:10002/rest/1?pretty-print=true'You can see the result in JSON format. The result of the above request is:
{
"id": "1",
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"success": true
}Deleting a document via HTTP is as following:
$ curl -X DELETE 'http://localhost:10002/rest/1?pretty-print=true'You can see the result in JSON format. The result of the above request is:
{
"success": true
}Bulk update is as following:
$ curl -X POST 'http://localhost:10002/rest/_bulk?pretty-print=true' -d @./example/bulk_put_request.json
You can see the result in JSON format. The result of the above command is:
{
"put_count": 7,
"success": true
}Searching documents via HTTP is as following:
$ curl -X POST 'http://localhost:10002/rest/_search?pretty-print=true' -d @./example/search_request.jsonYou can see the result in JSON format. The result of the above request is:
{
"search_result": {
"facets": {
"Category count": {
"field": "category",
"missing": 0,
"other": 0,
"terms": [
{
"count": 4,
"term": "library"
},
{
"count": 3,
"term": "server"
}
],
"total": 7
},
"Popularity range": {
"field": "popularity",
"missing": 0,
"numeric_ranges": [
{
"count": 2,
"max": 4,
"min": 3,
"name": "more than or equal to 3 and less than 4"
},
{
"count": 2,
"min": 5,
"name": "more than or equal to 5"
},
{
"count": 1,
"max": 2,
"min": 1,
"name": "more than or equal to 1 and less than 2"
},
{
"count": 1,
"max": 3,
"min": 2,
"name": "more than or equal to 2 and less than 3"
},
{
"count": 1,
"max": 5,
"min": 4,
"name": "more than or equal to 4 and less than 5"
}
],
"other": 0,
"total": 7
},
"Release date range": {
"date_ranges": [
{
"count": 4,
"end": "2010-12-31T23:59:59Z",
"name": "2001 - 2010",
"start": "2001-01-01T00:00:00Z"
},
{
"count": 2,
"end": "2020-12-31T23:59:59Z",
"name": "2011 - 2020",
"start": "2011-01-01T00:00:00Z"
}
],
"field": "release",
"missing": 0,
"other": 0,
"total": 6
}
},
"hits": [
{
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"fragments": {
"description": [
"Bleve is a full-text search and indexing library for Go."
],
"name": [
"\u003cmark\u003eBleve\u003c/mark\u003e"
]
},
"id": "1",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"bleve": [
{
"array_positions": null,
"end": 5,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java.",
"name": "Lucene",
"popularity": 4,
"release": "2000-03-30T00:00:00Z"
},
"fragments": {
"description": [
"Apache Lucene is a high-performance, full-featured text search engine library written entirely in Java."
],
"name": [
"\u003cmark\u003eLucene\u003c/mark\u003e"
]
},
"id": "2",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"lucene": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python.",
"name": "Whoosh",
"popularity": 3,
"release": "2008-02-20T00:00:00Z"
},
"fragments": {
"description": [
"Whoosh is a fast, featureful full-text indexing and searching library implemented in pure Python."
],
"name": [
"\u003cmark\u003eWhoosh\u003c/mark\u003e"
]
},
"id": "3",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"whoosh": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Library",
"description": "Ferret is a super fast, highly configurable search library written in Ruby.",
"name": "Ferret",
"popularity": 2,
"release": "2005-10-01T00:00:00Z"
},
"fragments": {
"description": [
"Ferret is a super fast, highly configurable search library written in Ruby."
],
"name": [
"\u003cmark\u003eFerret\u003c/mark\u003e"
]
},
"id": "4",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"ferret": [
{
"array_positions": null,
"end": 6,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project.",
"name": "Solr",
"popularity": 5,
"release": "2006-12-22T00:00:00Z"
},
"fragments": {
"description": [
"Solr is an open source enterprise search platform, written in Java, from the Apache Lucene project."
],
"name": [
"\u003cmark\u003eSolr\u003c/mark\u003e"
]
},
"id": "5",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"solr": [
{
"array_positions": null,
"end": 4,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Elasticsearch is a search engine based on Lucene, written in Java.",
"name": "Elasticsearch",
"popularity": 5,
"release": "2010-02-08T00:00:00Z"
},
"fragments": {
"description": [
"Elasticsearch is a search engine based on Lucene, written in Java."
],
"name": [
"\u003cmark\u003eElasticsearch\u003c/mark\u003e"
]
},
"id": "6",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"elasticsearch": [
{
"array_positions": null,
"end": 13,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
},
{
"fields": {
"category": "Server",
"description": "Blast is a full-text search and indexing server written in Go, built on top of Bleve.",
"name": "Blast",
"popularity": 1,
"release": "2017-01-13T00:00:00Z"
},
"fragments": {
"description": [
"Blast is a full-text search and indexing server written in Go, built on top of Bleve."
],
"name": [
"\u003cmark\u003eBlast\u003c/mark\u003e"
]
},
"id": "7",
"index": "/tmp/blast/node0/index",
"locations": {
"name": {
"blast": [
{
"array_positions": null,
"end": 5,
"pos": 1,
"start": 0
}
]
}
},
"score": 0.12163776688600772,
"sort": [
"_score"
]
}
],
"max_score": 0.12163776688600772,
"request": {
"explain": false,
"facets": {
"Category count": {
"field": "category",
"size": 10
},
"Popularity range": {
"field": "popularity",
"numeric_ranges": [
{
"max": 1,
"name": "less than 1"
},
{
"max": 2,
"min": 1,
"name": "more than or equal to 1 and less than 2"
},
{
"max": 3,
"min": 2,
"name": "more than or equal to 2 and less than 3"
},
{
"max": 4,
"min": 3,
"name": "more than or equal to 3 and less than 4"
},
{
"max": 5,
"min": 4,
"name": "more than or equal to 4 and less than 5"
},
{
"min": 5,
"name": "more than or equal to 5"
}
],
"size": 10
},
"Release date range": {
"date_ranges": [
{
"end": "2010-12-31T23:59:59Z",
"name": "2001 - 2010",
"start": "2001-01-01T00:00:00Z"
},
{
"end": "2020-12-31T23:59:59Z",
"name": "2011 - 2020",
"start": "2011-01-01T00:00:00Z"
}
],
"field": "release",
"size": 10
}
},
"fields": [
"*"
],
"from": 0,
"highlight": {
"fields": [
"name",
"description"
],
"style": "html"
},
"includeLocations": false,
"query": {
"query": "name:*"
},
"size": 10,
"sort": [
"-_score"
]
},
"status": {
"failed": 0,
"successful": 1,
"total": 1
},
"took": 280479,
"total_hits": 7
},
"success": true
}Blast is easy to bring up the cluster. Blast node is already running, but that is not fault tolerant. If you need to increase the fault tolerance, bring up 2 more nodes like so:
$ ./bin/blast start --bind-addr=localhost:11000 --grpc-addr=localhost:11001 --http-addr=localhost:11002 --node-id=node1 --raft-dir=/tmp/blast/node1/raft --store-dir=/tmp/blast/node1/store --index-dir=/tmp/blast/node1/index --peer-grpc-addr=localhost:10001
$ ./bin/blast start --bind-addr=localhost:12000 --grpc-addr=localhost:12001 --http-addr=localhost:12002 --node-id=node2 --raft-dir=/tmp/blast/node2/raft --store-dir=/tmp/blast/node2/store --index-dir=/tmp/blast/node2/index --peer-grpc-addr=localhost:10001Above example shows each Blast node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host.
So you have a 3-node cluster. That way you can tolerate the failure of 1 node. You can check the peers with the following command:
$ ./bin/blast peers --grpc-addr=localhost:10001 --pretty-printYou can see the result in JSON format. The result of the above command is:
{
"peers": [
{
"address": "127.0.0.1:10000",
"leader": true,
"metadata": {
"grpc_address": "localhost:10001",
"http_address": "localhost:10002"
},
"node_id": "node0"
},
{
"address": "127.0.0.1:11000",
"metadata": {
"grpc_address": "localhost:11001",
"http_address": "localhost:11002"
},
"node_id": "node1"
},
{
"address": "127.0.0.1:12000",
"metadata": {
"grpc_address": "localhost:12001",
"http_address": "localhost:12002"
},
"node_id": "node2"
}
],
"success": true
}Recommend 3 or more odd number of nodes in the cluster. In failure scenarios, data loss is inevitable, so avoid deploying single nodes.
This tells each new node to join the existing node. Once joined, each node now knows about the key:
Following command puts a document to node0:
$ cat ./example/doc1.json | xargs -0 ./bin/blast put --grpc-addr=localhost:10001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"success": true
}So, you can get a document from node0 like following:
$ ./bin/blast get --grpc-addr=localhost:10001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"id": "1",
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"success": true
}Also, you can get same document from node1 (localhost:11001) like following:
$ ./bin/blast get --grpc-addr=localhost:11001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"id": "1",
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"success": true
}Lastly, you can get same document from node2 (localhost:12001) like following:
$ ./bin/blast get --grpc-addr=localhost:12001 --pretty-print 1You can see the result in JSON format. The result of the above command is:
{
"id": "1",
"fields": {
"category": "Library",
"description": "Bleve is a full-text search and indexing library for Go.",
"name": "Bleve",
"popularity": 3,
"release": "2014-04-18T00:00:00Z"
},
"success": true
}