Large performance regression 1.14.0 -> 1.14.1

[ctz]

Downstream in rustls we have some simple performance tests that show a 26000% regression in full handshake performance when taking the aws-lc-rs 1.14.0 -> 1.14.1 upgrade.

This is measured in valgrind instruction-equivalents, and those go from approx 1,500,000 insns (1.14.0) to 423,000,000 insns (1.14.1).

I haven't looked into this in detail yet, but a few observations so far:

• this is a non-FIPS use, so is jitterent the right thing to be using? I thought it was a legalistic work-around for FIPS requirements about having a standalone entropy source.
• SHA3 has generally poor CPU performance compared to SHA2 (especially when SHA2 has wide CPU support, but SHA3 does not yet), so is that a good choice for a default entropy source?
• It seems the jitter entropy code has its own SHA3 code? What's up with that?
• It seems the jitter entropy code is compiled with -O0. Intentional?

At this point it's unclear to me whether this is "real" or specific to how valgrind measures things.

[justsmth]

Starting with AWS-LC v1.60 our entropy source for both FIPS and non-FIPS builds switched to CPU jitter. We believe this is more secure, but it can add a few milliseconds of latency when (and only when) a new process is forked.

I checked our versioning to verify that it aligns with your observation -- aws-lc-rs v1.14.0 (i.e., aws-lc-sys v0.31.0) aligned with AWS-LC v1.59.0, while aws-lc-rs v1.14.1 (i.e., aws-lc-sys v0.32.x) aligns with AWS-LC v1.61.x.)

I apologize for not calling this out clearly in our release notes. In most real-world scenarios the effect should be minimal, but it could definitely affect benchmarking. To avoid the issue, something would need to trigger entropy collection prior to the benchmark beginning data collection.

Let us know your thoughts. We can discuss mitigations/alternatives for this if needed.

[justsmth]

It seems the jitter entropy code is compiled with -O0. Intentional?

Yeah, this is intentional. Unfortunately, it's required by the logic:

#ifdef __OPTIMIZE__
 #error "The CPU Jitter random number generator must not be compiled with optimizations. See documentation. Use the compiler switch -O0 for compiling jitterentropy.c."
#endif

[cpu]

Starting with AWS-LC v1.60 our entropy source for both FIPS and non-FIPS builds switched to CPU jitter. We believe this is more secure

Is the jitter entropy being hedged with another randomness source by passing it in as “uncredited” additional data or is the entirety of the entropy sourced from CPU jitter in both FIPS/non-FIPS mode?

[torben-hansen]

Starting with AWS-LC v1.60 our entropy source for both FIPS and non-FIPS builds switched to CPU jitter. We believe this is more secure

Is the jitter entropy being hedged with another randomness source by passing it in as “uncredited” additional data or is the entirety of the entropy sourced from CPU jitter in both FIPS/non-FIPS mode?

CPU Jitter is one of (possibly) 3 different sources. We always use two sources at a minimum: https://github.com/aws/aws-lc/blob/main/crypto/fipsmodule/rand/entropy/entropy_sources.c#L39-L57

[justsmth]

Is the jitter entropy being hedged with another randomness source by passing it in as “uncredited” additional data or is the entirety of the entropy sourced from CPU jitter in both FIPS/non-FIPS mode?

My understanding is that jitter entropy is used to "seed" the thread-local entropy pool. The thread-local entropy pool is periodically updated from more traditional sources (RD_RAND or /dev/urandom) when available, but I'll defer to @torben-hansen (the designer/author) for details about the implementation.

[justsmth]

Draft for a mitigation: aws/aws-lc#2733

I also want to clarify that jitter entropy is only collected once per process - the "entropy pool" is shared across all threads within a process. If you see jitter entropy being collected for every thread (or other per-thread latency) please let us know.

[djc]

FWIW, we mitigated this in the rustls benchmark setup here:

• ci-bench: "pre-warm" per-thread entropy source rustls/rustls#2683

(So I think we were seeing per-thread delays seemingly from entropy collection.)

[ctz]

(So I think we were seeing per-thread delays seemingly from entropy collection.)

I think that was a misunderstanding on my part, the delays were per-fork.

[justsmth]

We just released aws-lc-sys v0.32.3. With this release, if you have AWS_LC_SYS_NO_JITTER_ENTROPY=1 set in the build environment it will disable the build (and use) of CPU Jitter Entropy. This should eliminate the additional latency that you see when a process starts.

I'll leave this issue open for feedback/visibility. (We also need to update our User Guide to reflect this and other upcoming changes that we have planned.) Please let us know if you have any other issues with our library. Thanks!