There is a HashAgility tool that allows for setting custom digest algorithms.

There might be a slight problem: inside init method it tries to call
DigestAlgorithmFactory.registerClass(customDigestAlgorithmClassName)
which tries to instantiate
CustomAlgorithmFactory(customDigestAlgorithmClassName)
which tries to instantiate CustomDigestAlgorithm via constructor that it gets from reflection using
this.classLoader

Problem seems to be that CustomAlgorithmFactory is loaded using node's core class loader that normally doesn't include any cordapp classes. So if CustomDigestAlgorithm is defined in a cordapp - it cannot be loaded. So one can only do hash agility with forking the Corda node which is not a desired way.

Also DigestAlgorithmFactory is sealed, and CustomAlgorithmFactory is private so the ability of overriding this behaviour is limited.

Potential solution might be to extend HashAgility and DigestAlgorithmFactory to be able to receive either the very class or just class loader.