Project Wycheproof is a community managed repository of test vectors that can be used by cryptography library developers to test against known attacks, specification inconsistencies, and other various implementation bugs.
Test vectors are maintained as JSON test vector data, with accompanying JSON schema files that document the structure of the test vector data.
- Clone this repository. You may also want to integrate Wycheproof as a Git submodule or otherwise set up automation to keep track of changes over time.
- Write (or generate from the JSON schema files) code to load the vector data as appropriate for your implementation language/project.
- For each algorithm of interest, identify the inputs to your cryptography APIs, and the produced outputs, mapping back to what the test vectors provide.
- Iterate through applicable test vectors, ensuring that the results produced by your API when given the relevant input data matches the test vector expected results.
- For best results, integrate this process into your continuous integration (CI) process so tests are run for all new contributions/changes.
You may find it helpful to examine how other projects like pyca/cryptography have integrated Wycheproof's test vectors.
Project Wycheproof has test vectors for the most popular crypto algorithms, including
- AES-EAX
- AES-GCM
- ChaCha20-Poly1305
- DH
- DHIES
- DSA
- ECDH
- ECDSA
- EdDSA
- ECIES
- HKDF
- HMAC
- RSA
- X25519, X448
- ML-KEM (Kyber)
- ML-DSA (CRYSTALS-Dilithium)
The test vectors detect whether a library is vulnerable to many attacks, including
- Invalid curve attacks
- Biased nonces in digital signature schemes
- Of course, all Bleichenbacher’s attacks
- And many more -- we have over 80 test cases
We welcome contribution of new test vector data, and algorithms.
If you want to contribute, please read CONTRIBUTING and send us pull requests. You can also report bugs or request new tests as GitHub issues.
We're in the process of revitalizing development and maintenance of Project Wycheproof as a C2SP project with a renewed focus on the test vector data. Our immediate priorities are:
- Consolidating
testvectorsandtestvectors_v1into a single directory of test data. - Completing JSON schema descriptions of all test vectors.
- Improving documentation and support for external contributors to provide new test data.
- Developing a community of downstream consumers who can help sheppard maintenance and review of new test vector data.
- Adding additional algorithm and test case coverage to the test vector data.
Project Wycheproof is named after Mount Wycheproof, the smallest mountain in the world. The main motivation for the project at the time of its creation was to have a goal that is achievable. The smaller the mountain the more likely it is to be able to climb it.
Wycheproof test vectors are used in some form by a number of important cryptography projects and libraries. In no particular order these include:
- OpenSSL
- BoringSSL
- aws-lc
- LibreSSL
- NSS
- pyca/cryptography
- Botan
- Go cryptography
- swift-crypto
- RustCrypto
- Graviola
- Tink
- PyCryptdome
- OpenTitan
If your project uses test vectors from Wycheproof, feel free to open a PR to add it to the list above!
See docs/bugs.md for some notable historic bugs found using Wycheproof's test harnesses, or test vector data.
At the time of writing, projects should consider writing harness code to use
both vector data sources for maximum coverage. Some algorithms only have
coverage via testvectors_v1 (e.g. ML-KEM, ML-DSA) while others are only
covered by testvectors data (e.g. RsassaPkcs1Generate).
We understand this situation is not ideal and are prioritizing an effort to consolidate down to single source of test data. Stay tuned.
At the time of writing, the following testvectors_v1 files are missing schemas:
testvectors_v1/aes_ff1_base*_test.jsontestvectors_v1/aes_ff1_radix*_test.jsontestvectors_v1/ec_prime_order_curves_test.jsontestvectors_v1/ecdsa_secp256k1_sha256_bitcoin_test.jsontestvectors_v1/pbes2_hmacsha*_aes_*_test.jsontestvectors_v1/pbkdf2_hmacsha*_test.jsontestvectors_v1/rsa_pss_*_sha*_mgf*_params_test.jsontestvectors_v1/rsa_pss_misc_params_test.json
Contribution of schemas for the above vectors would be most welcome.
Some legacy documentation for files, formats and types are available, but not necessarily in-sync with the current test vector state.
In general, prefer referencing the schema files since these are tested in CI to ensure vector file contents match their advertised schema.
Historically Wycheproof also included test harnesses (e.g. for Java and
Javascript cryptography implementations) that tested a variety of attacks
directly against implementations. Since transitioning to community support
these harnesses have been removed (but still exist in git
history for interested parties at cd27d64). Our current focus is on
implementation-agnostic test vectors.
Testing 3rd party cryptography libraries directly means flaws are only uncovered after they have been committed, and potentially released, by the projects under test. Instead, we encourage downstream projects to regularly test their code using Wycheproof test vectors as part of their development process. This approach helps catch flaws before they can become CVEs, means new features get tested immediately, and helps distribute the maintenance burden. This allows the Wycheproof maintainers to focus on test vectors instead of tracking downstream development of many projects while simultaneously maintaining an ever-increasing number of language & project-specific test harnesses.
Parties interested in test harnesses may find continued work by Daniel Bleichenbacher in Rooterberg of interest.
Project Wycheproof was originally created and maintained by:
- Daniel Bleichenbacher
- Thai Duong
- Emilia Kasper
- Quan Nguyen
- Charles Lee