A Synchronously replicated distributed log server.

This is a proof of concept implementation only. It is not intended for production use!!!

logsrd implements a system for high-availability log replication based entirely on synchronous replication without the use of quorum based consistency. For a more detailed write up on the approach see Synchronous Replication with High Availability — A New Approach

mkdir data
mkdir data/logs
node utils/create-data-dirs ./data/logs

npm run build

node build/logsrd

Logs define their own access control.

For operations that require a token an Authorization: Bearer TOKEN header must be set.

Create a new log.

POST body must be JSON conforming to the LogConfig schema.

logId must be empty. This value is generated by the server.

Append an entry to log.

POST body may be either binary or JSON depending on log type.

• lastEntryNum=ENTRY_NUMBER

Do conditional append only if there have been no appends after ENTRY_NUMBER.

Get most recent entry to log.

This may return either a log entry or the most recent log config if that is the most recent entry. Mixing config entries with other entries is a rather poor design so this will probably be changed

• meta=true

Return log entry as JSON object that includes entryNum, crc, and entry

This currently only works for JSON type logs

Get list of log entries.

• offset=ENTRY_NUM

Start list at ENTRY_NUM

• limit=NUMBER

Limit number of entries returned (Max 100).

• entryNums=ENTRY_NUM,ENTRY_NUM,...

Return list of specific entries

• meta=true

Return log entry as JSON object that includes entryNum, crc, and entry

This currently only works for JSON type logs

Get current log config

Set included properties to existing log config and store as new config.

Get version of logsrd.

After making a /client websocket connection commands are sent as text.

Subscribe to logId using token that allows read access to log if it is private.

Once subscribed all new log entries will be sent to socket as binary.

Unsubscribe from log

Commands will be replied to with a text response that is original command and either ok or err.

• Type: String
• Default: ""
• Description: Unique identifier for the log
• Required: Yes

• Type: String
• Enum:
  • binary
  • json
• Default: "json"
• Description: Specifies the log data format
• Required: Yes

• Type: String
• Default: ""
• Description: Primary/master log reference
• Required: Yes

• Type: Array
• Items:
  • Type: String
• Nullable: Yes
• Description: List of replica log references

• Type: Array
• Items:
  • Type: String
• Nullable: Yes
• Description: List of asynchronous replica log references

• Type: String
• Enum:
  • public
  • private
  • readOnly
  • writeOnly
• Default: "private"
• Description: Access control level for the log
• Required: Yes

• Type: String
• Enum:
  • token
  • jwt
• Default: "token"
• Description: Authentication method
• Required: Yes

• Type: String
• Nullable: Yes
• Description: General access token

• Type: String
• Nullable: Yes
• Description: Administrative access token

• Type: String
• Nullable: Yes
• Description: Read-only access token

• Type: String
• Nullable: Yes
• Description: Write-only access token

• Type: String
• Nullable: Yes
• Description: Superuser access token

• Type: Array
• Items:
  • Type: String
• Nullable: Yes
• Description: List of JWT properties when authType is "jwt"

• Type: String
• Nullable: Yes
• Description: Secret key for JWT authentication

• Type: Boolean
• Default: false
• Description: Indicates if the log is stopped
• Required: Yes

• Type: String
• Nullable: Yes
• Description: Address for configuration log

There are 3 types of operations that can be performed on logs.

Read data from log.

Write data to log.

Read/Write the config for log.

Logs have four access control modes:

Allow unauthenticated read/write access to log but not admin.

Do not allow any unauthenticated access to log.

Allow unauthenticated read to log.

Allow unauthenticated write to log.

Logs can use either token or jwt authentication.

Allows full read|write|admin access to log unless adminToken or superToken is set on log, in which case it only provides read|write access.

Allows read access to log.

Allows write access to log.

Allows admin access (but not read|write) to log.

Allows full read|write|admin access to log.

If jwt authentication is used then auth token must be HS256 and signed with jwtSecret.

Signed JWT Token must include allow property which contains one or more of admin|read|write privileges that are allowed by token.

Comma separated list of additional properties in the JWT Token which should be copied to the log entry when doing an append.

This is useful for client side applications where the authentication service wants to include user specific information (e.g. userId) in a log entry that is sent from client.