+
Skip to content

Experimental: Add Probabilistic Hashing in Physical Backup #304

@tanmoysrt

Description

@tanmoysrt

In Physical Backup, currently we use only file metadata for validating snapshot.

We need more validations.
But, sha256 like checksums can't be used

  • We can't read the whole file, because that will do high IO usage
  • cryptographic checksumming is slow on large files (5~10GB)

Solution :

  • Randomly pick segment of files and checksum those, so that we can validate later. 1MB read for 500MB file can be the target. (Need to do some math on how much reliable it can be)
  • Use non-cryptographic algo instead : https://xxhash.com/

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Todo

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions

    点击 这是indexloc提供的php浏览器服务,不要输入任何密码和下载