这是indexloc提供的服务，不要输入任何密码
Skip to content
This repository was archived by the owner on Dec 10, 2021. It is now read-only.

Add GEO search example #65

Merged
merged 1 commit into from
Jun 21, 2019
Merged

Add GEO search example #65

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
69 changes: 53 additions & 16 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -249,7 +249,7 @@ $ ./bin/blastd \
--grpc-addr=:5001 \
--http-addr=:5002 \
--data-dir=/tmp/blast/indexer1 \
--index-mapping-file=./example/index_mapping.json \
--index-mapping-file=./example/wiki_index_mapping.json \
--index-type=upside_down \
--index-storage-type=boltdb
```
Expand All @@ -268,7 +268,7 @@ You can now put, get, search and delete the documents via CLI.
For document indexing, execute the following command:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 enwiki_1
$ cat ./example/wiki_doc_enwiki_1.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -303,7 +303,7 @@ You can see the result in JSON format. The result of the above command is:
Searching documents is as like following:

```bash
$ cat ./example/search_request.json | xargs -0 ./bin/blast search --grpc-addr=:5001
$ cat ./example/wiki_search_request.json | xargs -0 ./bin/blast search --grpc-addr=:5001
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -483,7 +483,7 @@ You can see the result in JSON format. The result of the above command is:
Indexing documents in bulk, run the following command:

```bash
$ cat ./example/bulk_index_wiki.json | xargs -0 ./bin/blast set document --grpc-addr=:5001
$ cat ./example/wiki_bulk_index.json | xargs -0 ./bin/blast set document --grpc-addr=:5001
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -498,7 +498,7 @@ You can see the result in JSON format. The result of the above command is:
Deleting documents in bulk, run the following command:

```bash
$ cat ./example/bulk_delete_wiki.json | xargs -0 ./bin/blast delete document --grpc-addr=:5001
$ cat ./example/wiki_bulk_delete.json | xargs -0 ./bin/blast delete document --grpc-addr=:5001
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -518,7 +518,7 @@ Also you can do above commands via HTTP REST API that listened port 5002.
Indexing a document via HTTP is as following:

```bash
$ curl -X PUT 'http://127.0.0.1:5002/documents/enwiki_1' -d @./example/doc_enwiki_1.json
$ curl -X PUT 'http://127.0.0.1:5002/documents/enwiki_1' -d @./example/wiki_doc_enwiki_1.json
```


Expand All @@ -536,7 +536,7 @@ $ curl -X GET 'http://127.0.0.1:5002/documents/enwiki_1'
Searching documents via HTTP is as following:

```bash
$ curl -X POST 'http://127.0.0.1:5002/search' -d @./example/search_request.json
$ curl -X POST 'http://127.0.0.1:5002/search' -d @./example/wiki_search_request.json
```


Expand All @@ -554,7 +554,7 @@ $ curl -X DELETE 'http://127.0.0.1:5002/documents/enwiki_1'
Indexing documents in bulk via HTTP is as following:

```bash
$ curl -X PUT 'http://127.0.0.1:5002/documents' -d @./example/bulk_index_wiki.json
$ curl -X PUT 'http://127.0.0.1:5002/documents' -d @./example/wiki_bulk_index.json
```


Expand All @@ -563,7 +563,7 @@ $ curl -X PUT 'http://127.0.0.1:5002/documents' -d @./example/bulk_index_wiki.js
Deleting documents in bulk via HTTP is as following:

```bash
$ curl -X DELETE 'http://127.0.0.1:5002/documents' -d @./example/bulk_delete_wiki.json
$ curl -X DELETE 'http://127.0.0.1:5002/documents' -d @./example/wiki_bulk_delete.json
```


Expand All @@ -583,7 +583,7 @@ $ ./bin/blastd \
--grpc-addr=:5001 \
--http-addr=:5002 \
--data-dir=/tmp/blast/indexer1 \
--index-mapping-file=./example/index_mapping.json \
--index-mapping-file=./example/wiki_index_mapping.json \
--index-type=upside_down \
--index-storage-type=boltdb
```
Expand Down Expand Up @@ -659,7 +659,7 @@ Recommend 3 or more odd number of nodes in the cluster. In failure scenarios, da
The following command indexes documents to any node in the cluster:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 enwiki_1
$ cat ./example/wiki_doc_enwiki_1.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 enwiki_1
```

So, you can get the document from the node specified by the above command as follows:
Expand Down Expand Up @@ -724,7 +724,7 @@ $ ./bin/blastd \
--grpc-addr=:15001 \
--http-addr=:15002 \
--data-dir=/tmp/blast/manager1 \
--index-mapping-file=./example/index_mapping.json \
--index-mapping-file=./example/wiki_index_mapping.json \
--index-type=upside_down \
--index-storage-type=boltdb

Expand Down Expand Up @@ -827,15 +827,15 @@ $ ./bin/blastd \
```

```bash
$ cat ./example/bulk_index_wiki.json | xargs -0 ./bin/blast set document --grpc-addr=:25001
$ cat ./example/wiki_bulk_index.json | xargs -0 ./bin/blast set document --grpc-addr=:25001
```

```bash
$ cat ./example/search_request.json | xargs -0 ./bin/blast search --grpc-addr=:25001
$ cat ./example/wiki_search_request.json | xargs -0 ./bin/blast search --grpc-addr=:25001
```

```bash
$ cat ./example/bulk_delete_wiki.json | xargs -0 ./bin/blast delete document --grpc-addr=:25001
$ cat ./example/wiki_bulk_delete.json | xargs -0 ./bin/blast delete document --grpc-addr=:25001
```


Expand Down Expand Up @@ -886,7 +886,7 @@ $ docker run --rm --name blast-indexer1 \
--grpc-addr=:5001 \
--http-addr=:5002 \
--data-dir=/tmp/blast/indexer1 \
--index-mapping-file=/opt/blast/example/index_mapping.json \
--index-mapping-file=/opt/blast/example/wiki_index_mapping.json \
--index-storage-type=leveldb
```

Expand Down Expand Up @@ -936,3 +936,40 @@ $ for FILE in $(find ~/tmp/enwiki -type f -name '*' | sort)
curl -s -X PUT -H 'Content-Type: application/json' "http://127.0.0.1:8080/documents" -d "${DOCS}"
done
```


## GEO example

This section explain how to index GEO data to Blast.

### Starting Indexer with GEO index mapping

```bash
$ ./bin/blastd \
indexer \
--node-id=indexer1 \
--bind-addr=:5000 \
--grpc-addr=:5001 \
--http-addr=:5002 \
--data-dir=/tmp/blast/indexer1 \
--index-mapping-file=./example/geo_index_mapping.json \
--index-type=upside_down \
--index-storage-type=boltdb
```

### Indexing example GEO data

```bash
$ cat ./example/geo_doc1.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc1
$ cat ./example/geo_doc2.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc2
$ cat ./example/geo_doc3.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc3
$ cat ./example/geo_doc4.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc4
$ cat ./example/geo_doc5.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc5
$ cat ./example/geo_doc6.json | xargs -0 ./bin/blast set document --grpc-addr=:5001 geo_doc6
```

### Searching example GEO data

```bash
$ cat ./example/geo_search_request.json | xargs -0 ./bin/blast search --grpc-addr=:5001
```
18 changes: 18 additions & 0 deletions example/geo_doc1.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
{
"name": "Brewpub-on-the-Green",
"city": "Fremont",
"state": "California",
"code": "",
"country": "United States",
"phone": "",
"website": "",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "",
"address": [],
"geo": {
"accuracy": "APPROXIMATE",
"lat": 37.5483,
"lon": -121.989
}
}
20 changes: 20 additions & 0 deletions example/geo_doc2.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "Capital City Brewing Company",
"city": "Washington",
"state": "District of Columbia",
"code": "20005",
"country": "United States",
"phone": "202.628.2222",
"website": "http://www.capcitybrew.com",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "Washington DC's first brewpub since prohibition, Capitol City Brewing Co. opened its doors in 1992. Our first location still stands in Downtown DC, at 11th and H St., NW. Our company policy is to bring the fine craft of brewing to every person who lives and visits our region, as well as treating them to a wonderful meal and a great experience.",
"address": [
"1100 New York Ave, NW"
],
"geo": {
"accuracy": "ROOFTOP",
"lat": 38.8999,
"lon": -77.0272
}
}
20 changes: 20 additions & 0 deletions example/geo_doc3.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "Firehouse Grill & Brewery",
"city": "Sunnyvale",
"state": "California",
"code": "94086",
"country": "United States",
"phone": "1-408-773-9500",
"website": "",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "",
"address": [
"111 South Murphy Avenue"
],
"geo": {
"accuracy": "RANGE_INTERPOLATED",
"lat": 37.3775,
"lon": -122.03
}
}
20 changes: 20 additions & 0 deletions example/geo_doc4.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "Hook & Ladder Brewing Company",
"city": "Silver Spring",
"state": "Maryland",
"code": "20910",
"country": "United States",
"phone": "301.565.4522",
"website": "http://www.hookandladderbeer.com",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "At Hook & Ladder Brewing we believe in great beer in the company of good friends, so we bring you three great beers for your drinking pleasure (please drink responsibly). Each of our beers is carefully crafted with the finest quality ingredients for a distinctive taste we know you will enjoy. Try one tonight, you just might get hooked. Through our own experiences in the fire and rescue service we have chosen the Hook & Ladder as a symbol of pride and honor to pay tribute to the brave men and women who serve and protect our communities.",
"address": [
"8113 Fenton St."
],
"geo": {
"accuracy": "ROOFTOP",
"lat": 38.9911,
"lon": -77.0237
}
}
20 changes: 20 additions & 0 deletions example/geo_doc5.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "Jack's Brewing",
"city": "Fremont",
"state": "California",
"code": "94538",
"country": "United States",
"phone": "1-510-796-2036",
"website": "",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "",
"address": [
"39176 Argonaut Way"
],
"geo": {
"accuracy": "ROOFTOP",
"lat": 37.5441,
"lon": -121.988
}
}
20 changes: 20 additions & 0 deletions example/geo_doc6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
{
"name": "Sweet Water Tavern and Brewery",
"city": "Sterling",
"state": "Virginia",
"code": "20121",
"country": "United States",
"phone": "(703) 449-1108",
"website": "http://www.greatamericanrestaurants.com/sweetMainSter/index.htm",
"type": "brewery",
"updated": "2010-07-22 20:00:20",
"description": "",
"address": [
"45980 Waterview Plaza"
],
"geo": {
"accuracy": "RANGE_INTERPOLATED",
"lat": 39.0324,
"lon": -77.4097
}
}
36 changes: 36 additions & 0 deletions example/geo_index_mapping.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,36 @@
{
"types": {
"brewery": {
"properties": {
"name": {
"fields": [
{
"include_term_vectors": true,
"include_in_all": true,
"index": true,
"store": true,
"analyzer": "keyword",
"type": "text"
}
],
"dynamic": true,
"enabled": true
},
"geo": {
"fields": [
{
"include_term_vectors": true,
"include_in_all": true,
"index": true,
"store": true,
"type": "geopoint"
}
],
"dynamic": true,
"enabled": true
}
}
}
},
"default_type": "brewery"
}
55 changes: 55 additions & 0 deletions example/geo_search_request.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
{
"query": {
"location": {
"lon": -122.107799,
"lat": 37.399285
},
"distance": "100mi",
"field": "geo"
},
"size": 10,
"from": 0,
"fields": [
"*"
],
"sort": [
{
"by": "geo_distance",
"field": "geo",
"unit": "mi",
"location": {
"lon": -122.107799,
"lat": 37.399285
}
}
],
"facets": {
"State count": {
"size": 10,
"field": "state"
},
"Updated range": {
"size": 10,
"field": "updated",
"date_ranges": [
{
"name": "2001 - 2010",
"start": "2001-01-01T00:00:00Z",
"end": "2010-12-31T23:59:59Z"
},
{
"name": "2011 - 2020",
"start": "2011-01-01T00:00:00Z",
"end": "2020-12-31T23:59:59Z"
}
]
}
},
"highlight": {
"style": "html",
"fields": [
"title",
"text"
]
}
}
0 example/bulk_delete_wiki.json → example/wiki_bulk_delete.json
View file
Open in desktop
File renamed without changes.
0 example/bulk_index_wiki.json → example/wiki_bulk_index.json
View file
Open in desktop
File renamed without changes.
0 example/doc_arwiki_1.json → example/wiki_doc_arwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_bgwiki_1.json → example/wiki_doc_bgwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_cawiki_1.json → example/wiki_doc_cawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_cswiki_1.json → example/wiki_doc_cswiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_dawiki_1.json → example/wiki_doc_dawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_dewiki_1.json → example/wiki_doc_dewiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_elwiki_1.json → example/wiki_doc_elwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_enwiki_1.json → example/wiki_doc_enwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_eswiki_1.json → example/wiki_doc_eswiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_fawiki_1.json → example/wiki_doc_fawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_fiwiki_1.json → example/wiki_doc_fiwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_frwiki_1.json → example/wiki_doc_frwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_gawiki_1.json → example/wiki_doc_gawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_glwiki_1.json → example/wiki_doc_glwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_guwiki_1.json → example/wiki_doc_guwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_hiwiki_1.json → example/wiki_doc_hiwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_huwiki_1.json → example/wiki_doc_huwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_hywiki_1.json → example/wiki_doc_hywiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_idwiki_1.json → example/wiki_doc_idwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_itwiki_1.json → example/wiki_doc_itwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_jawiki_1.json → example/wiki_doc_jawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_knwiki_1.json → example/wiki_doc_knwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_kowiki_1.json → example/wiki_doc_kowiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_mlwiki_1.json → example/wiki_doc_mlwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_nlwiki_1.json → example/wiki_doc_nlwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_nowiki_1.json → example/wiki_doc_nowiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_pswiki_1.json → example/wiki_doc_pswiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_ptwiki_1.json → example/wiki_doc_ptwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_rowiki_1.json → example/wiki_doc_rowiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_ruwiki_1.json → example/wiki_doc_ruwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_svwiki_1.json → example/wiki_doc_svwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_tawiki_1.json → example/wiki_doc_tawiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_tewiki_1.json → example/wiki_doc_tewiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_thwiki_1.json → example/wiki_doc_thwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_trwiki_1.json → example/wiki_doc_trwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/doc_zhwiki_1.json → example/wiki_doc_zhwiki_1.json
View file
Open in desktop
File renamed without changes.
0 example/index_mapping.json → example/wiki_index_mapping.json
View file
Open in desktop
File renamed without changes.
0 example/search_request.json → example/wiki_search_request.json
View file
Open in desktop
File renamed without changes.