这是indexloc提供的服务，不要输入任何密码
Skip to content
This repository was archived by the owner on Dec 10, 2021. It is now read-only.

Add cluster manager #48

Merged
merged 1 commit into from
Mar 28, 2019
Merged

Add cluster manager #48

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 72 additions & 13 deletions Gopkg.lock
View file
Open in desktop

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

30 changes: 15 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -165,7 +165,7 @@ You can see the binary file when build successful like so:

```bash
$ ls ./bin
blast-index
blast-indexer
```


Expand Down Expand Up @@ -235,7 +235,7 @@ $ make \
Running a Blast index node is easy. Start Blast data node like so:

```bash
$ ./bin/blast-index start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-mapping-file=./example/index_mapping.json
$ ./bin/blast-indexer start --node-id=index1 --data-dir=/tmp/blast/index1 --bind-addr=:6060 --grpc-addr=:5050 --http-addr=:8080 --index-mapping-file=./example/index_mapping.json
```

Please refer to following document for details of index mapping:
Expand All @@ -252,7 +252,7 @@ You can now put, get, search and delete the documents via CLI.
For document indexing, execute the following command:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050 --id=enwiki_1
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -269,7 +269,7 @@ You can see the result in JSON format. The result of the above command is:
Getting a document is as following:

```bash
$ ./bin/blast-index get --grpc-addr=:5050 --id=enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -290,7 +290,7 @@ You can see the result in JSON format. The result of the above command is:
Searching documents is as like following:

```bash
$ cat ./example/search_request.json | xargs -0 ./bin/blast-index search --grpc-addr=:5050
$ cat ./example/search_request.json | xargs -0 ./bin/blast-indexer search --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -462,7 +462,7 @@ Please refer to following document for details of search request and result:
Deleting a document is as following:

```bash
$ ./bin/blast-index delete --grpc-addr=:5050 --id=enwiki_1
$ ./bin/blast-indexer delete --grpc-addr=:5050 --id=enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -479,7 +479,7 @@ You can see the result in JSON format. The result of the above command is:
Indexing documents in bulk, run the following command:

```bash
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -496,7 +496,7 @@ You can see the result in JSON format. The result of the above command is:
Deleting documents in bulk, run the following command:

```bash
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-index delete --grpc-addr=:5050
$ cat ./example/docs_wiki.json | xargs -0 ./bin/blast-indexer delete --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -572,8 +572,8 @@ $ curl -X DELETE 'http://127.0.0.1:8080/documents' -d @./example/docs_wiki.json
Blast is easy to bring up the cluster. Blast data node is already running, but that is not fault tolerant. If you need to increase the fault tolerance, bring up 2 more data nodes like so:

```bash
$ ./bin/blast-index start --node-id=index2 --data-dir=/tmp/blast/index2 --bind-addr=:6061 --grpc-addr=:5051 --http-addr=:8081 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-index start --node-id=index3 --data-dir=/tmp/blast/index3 --bind-addr=:6062 --grpc-addr=:5052 --http-addr=:8082 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-indexer start --node-id=index2 --data-dir=/tmp/blast/index2 --bind-addr=:6061 --grpc-addr=:5051 --http-addr=:8081 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
$ ./bin/blast-indexer start --node-id=index3 --data-dir=/tmp/blast/index3 --bind-addr=:6062 --grpc-addr=:5052 --http-addr=:8082 --index-mapping-file=./example/index_mapping.json --join-addr=:5050
```

_Above example shows each Blast node running on the same host, so each node must listen on different ports. This would not be necessary if each node ran on a different host._
Expand All @@ -582,7 +582,7 @@ This instructs each new node to join an existing node, each node recognizes the
So you have a 3-node cluster. That way you can tolerate the failure of 1 node. You can check the peers with the following command:

```bash
$ ./bin/blast-index cluster --grpc-addr=:5050
$ ./bin/blast-indexer cluster --grpc-addr=:5050
```

You can see the result in JSON format. The result of the above command is:
Expand Down Expand Up @@ -621,13 +621,13 @@ Recommend 3 or more odd number of nodes in the cluster. In failure scenarios, da
The following command indexes documents to any node in the cluster:

```bash
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-index index --grpc-addr=:5050 enwiki_1
$ cat ./example/doc_enwiki_1.json | xargs -0 ./bin/blast-indexer index --grpc-addr=:5050 enwiki_1
```

So, you can get the document from the node specified by the above command as follows:

```bash
$ ./bin/blast-index get --grpc-addr=:5050 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5050 enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand All @@ -645,8 +645,8 @@ You can see the result in JSON format. The result of the above command is:
You can also get the same document from other nodes in the cluster as follows:

```bash
$ ./bin/blast-index get --grpc-addr=:5051 enwiki_1
$ ./bin/blast-index get --grpc-addr=:5052 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5051 enwiki_1
$ ./bin/blast-indexer get --grpc-addr=:5052 enwiki_1
```

You can see the result in JSON format. The result of the above command is:
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/cluster.go → cmd/blast-indexer/cluster.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,14 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/urfave/cli"
)

func execCluster(c *cli.Context) error {
grpcAddr := c.String("grpc-addr")

client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/delete.go → cmd/blast-indexer/delete.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
)
Expand Down Expand Up @@ -64,7 +64,7 @@ func execDelete(c *cli.Context) error {
}

// create client
client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/get.go → cmd/blast-indexer/get.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@ import (
"fmt"
"os"

"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/mosuka/blast/protobuf"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
Expand All @@ -38,7 +38,7 @@ func execGet(c *cli.Context) error {
Id: id,
}

client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
4 changes: 2 additions & 2 deletions cmd/blast-index/index.go → cmd/blast-indexer/index.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,7 +21,7 @@ import (
"os"

"github.com/golang/protobuf/ptypes/any"
"github.com/mosuka/blast/index"
"github.com/mosuka/blast/indexer"
"github.com/mosuka/blast/protobuf"
pbindex "github.com/mosuka/blast/protobuf/index"
"github.com/urfave/cli"
Expand Down Expand Up @@ -93,7 +93,7 @@ func execIndex(c *cli.Context) error {
}

// create gRPC client
client, err := index.NewGRPCClient(grpcAddr)
client, err := indexer.NewGRPCClient(grpcAddr)
if err != nil {
return err
}
Expand Down
Loading