Make SHA256 compression pluggable

[furszy]

Tackling the long-standing request #702.

Right now we ship our own SHA256 implementation, a standard baseline version that does not take advantage of any hardware-optimized instruction, and it cannot be accessed by the embedding application - it is for internal usage only.

This means embedding applications often have to implement or include a different version for their use cases, wasting space on constrained environments, and in performance-sensitive setups it forces them to use a slower path than what the platform provides. Many projects already rely on tuned SHA-NI / ARMv8 / or other hardware-optimized code, so always using the baseline implementation we ship within the library is not ideal.

This PR gives our users a way to bring their own SHA256 compression, either at build time or at runtime, while keeping the default behavior untouched for everyone else.

Compile-time: External SHA256 module

The built-in transform is moved out of hash_impl.hinto a small sha module with its own public header (secp256k1_sha.h). The idea is to stop treating the compression step as a private, hidden detail, and instead make it part of a public interface that users can swap out if they want to.
Both build systems gain optional support for pointing libsecp to an external implementation:

• Autotools: --with-external-sha256=<path>
• CMake: SECP256K1_EXTERNAL_SHA256_PATH

Note:
Due to our current limitation of disabling C++ builds (see this), the provided external implementation must be written in C. (A way to overcome this would be to compile and link this externally instead of doing it within the library).

Runtime: Context callback

For setups where we detect the available SHA256 compression function at runtime and cannot re-compile the sources, or when the function is not written in bare C89, there’s a new API:

secp256k1_context_set_sha256_transform_callback(ctx, fn_transform)

This lets users plug-in their hardware-optimized implementation into the secp256k1_context, which is required in all functions that compute sha256 hashes.
During the initial setting, as a sanity check, this mechanism verifies that the provided compression function is equivalent to the default one.

As a quick example, the changes required to implement this in Bitcoin-Core at runtime are very straightforward: furszy/bitcoin-core@f68bef0

Implementing this at compile time in Bitcoin Core is possible, but it would require changing the C version we build against (which is not recommended). For example, we cannot link the SHA-NI implementation we have in Core (even if rewritten in C89) because it depends on the arm_neon library, which requires C99.

[fjahr]

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs. Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

[furszy]

Thanks for the feedback fjahr!

I guess this is more of a draft for initial review but I will spell it out anyway: It's currently missing some CI coverage for building with an external library as well as some docs.

Yeah, can create a CI job testing the compile-time pluggable compression function very easily. Thanks for reminding that.

And about the missing docs; yeah. I didn't add it because we currently don't have much documentation outside configure.ac / CMakeLists.txt. Happy to create a file for it.

Curious what people think in terms of docs for something like this: Should there be extensive guidance in which context this is safe to use or would users be left on their own to try it out? Or are we assuming all users that go to this length are competent enough to judge if it's a good idea to bring their own sha256 or use the systems one?

It’s not that we’re letting people plug in whatever they want. Both introduced features have guardrails:

1. Compile-time: we run all current tests against the provided implementation, plus a runtime self-test.
2. Runtime: besides the runtime self-test, have introduced an "equivalence check" that hashes known inputs and compares them against the library’s internal implementation before accepting the external function.

So you can bring your own compression function, but it still has to prove it behaves exactly like ours bit-for-bit.

Also, the target user here is someone who's actually written a hardware-optimized SHA256. It’s not like they stumbled into this by accident.