Wrapping sensitive data in Mint

[a3kov]

According to EEF security guidance one of the best ways to protect sensitive data, such as security-related (tokens, passwords) or PII from accidental leakage is to wrap it in a closure. Other ways such as flagging the whole process as sensitive or post-processing stacktraces have significant downsides, limited effect and seem less optimal.

What if Mint allowed to pass some fields, such as headers, path (URLs may also contain sensitive info when we have to deal with 3rd party services outside of our control), etc in closures ? The closure could return {redacted, raw} pair (just an example, other wrappers could be used, but the raw value must be enclosed). Then

• Mint would use the raw value only when sending the request to the socket
• downstream libraries (Finch, Req and others) would obviously need to become aware of this feature, and use it accordingly
• logging/telemetry libraries would also need to adapt to it, and use redacted value everywhere where they previously used the raw value

What I also like about this approach is that all involved libraries don't need to know what kind of value it is and how to redact it. The caller provides both versions so no additional abstractions are needed.

[a3kov]

Maybe something like this ?

%Mint.SensitiveValue{
  redacted: "...",
  raw: fn -> "..." end
}

Looks like we could even have it as generic abstraction in Elixir for all libraries dealing with sensitive data. But Mint could start using this tomorrow without waiting for an ecosystem-wide solution.

[wojtekmach]

If anyone is interested, here’s a similar discussion in Req: wojtekmach/req#461

[whatyouhide]

Nice discussion thank you. I think we should do this in Mint if possible, and bubble it up to Finch/Req. I think the idea of a struct that

1. Implements String.Chars returning the original wrapped value
2. Implements Inspect and returns the redacted value

would already be a really good start? Then you'd just use it in place of strings:

Mint.HTTP.request(conn, "GET", "/user/:id", [{"Authorization", Mint.redact("Bearer foo")}])

Mint.redact("Foo")
#=> %Mint.RedactedString{value: "Foo"} (which implements protocols mentioned above)

Thoughts @wojtekmach @a3kov?

[wojtekmach]

I don't know if I can explain it but my gut reaction is to either do it higher level, in Req, or lower level, in Elixir. This is obviously a separate conversation, I believe the latter would have maximum community leverage and interoperability on one hand but higher barrier to entry and way slower rollout on the other. I haven't thought that through but keep wondering about a System.get_env_secret or something like that, to mark the secret as such when it enters the system for the first time. Otherwise, it'd be awkward to use Mint.redact/1 directly in library like Swoosh or Sentry that supports other http clients, however I suppose they'd be commonly configured using secrets as plain values and internally could call Mint.redact before passing it down. On the other hand, if it's just on the higher level, say in Req, then the plain value when passed down the stack could sneak into logs, telemetry events, etc.

As another data point, Ecto redacts fields when inspected but otherwise keeps them as is. See https://hexdocs.pm/ecto/Ecto.Schema.html#module-redacting-fields. Perhaps it's done like that to avoid overhead but more likely to stay within "Ecto types" like :string, defined on the schema.

In any case, if Mint supports this, I'll be glad to help with Finch and Req support! I think a %Mint.RedactedString{} with protocol implementations is a very practical solution for this.

[whatyouhide]

I think it's a practical solution that can evolve into a language feature eventually (Mint.redact would start returning the language struct instead of the Mint-specific one).

[a3kov]

@whatyouhide Usually when you deal with a secret you would want it to be wrapped as soon as it enters your app - when you load it from the database, for example.. So in case of Mint you wouldn't want to unwrap then ask Mind to rewrap - you would rather pass the already wrapped value. That is, of course, assuming that all libraries wrap it in the same way - with closure returning the value only.
I think we would want to support both passing the wrapped value and asking Mint to wrap it.

I agree that having this wrapper provided by Elixir would be the best way forward for the whole ecosystem. But adding it to the language and reaching the state where it's generally available could take a few years (not everyone can upgrade on a whim). And if a third-party library provided it, it would be much harder to reach wide adoption compared to "the official" solution from the language.
If, OTOH, this feature is provided by Mint it can be adopted very fast, and serve as a temporary solution in the meantime.

[whatyouhide]

Usually when you deal with a secret you would want it to be wrapped as soon as it enters your app - when you load it from the database, for example.. So in case of Mint you wouldn't want to unwrap then ask Mind to rewrap - you would rather pass the already wrapped value.

You could call Mint.redact/1 as soon as you read the secret though, no?

[a3kov]

In my case, I store security token in a custom encrypted Ecto field, and the field load callback wraps the value in a closure. I could probably call Mint.redact in the load callback too as I have no use for the token other than to send it in a request. But there can be different situations.
Maybe this unwrap-rewrap tango would be optimised by JIT anyway ?

[a3kov]

@whatyouhide One more note, your suggested Mint.redact/1 function implies Mint knows how to redact the value. In the original post I suggested that the caller provides both raw and redacted value. This is important detail as Mint can't possibly know how to redact different types of values, and there are many different ways you could do that.

[a3kov]

Perhaps as an optimization we could provide the option to store redact/1 function next to the wrapped raw value. This way we wouldn't need to waste memory on the redacted value all the time. Security tokens and passwords is one story, but there could be bigger sensitive payloads probably in some cases.
So we can offer both arguments - pre-redacted value or redact/1 function, and the caller can choose which one is more appropriate to the situation.

[whatyouhide]

If we define a Mint.Redactable protocol we could implement it for strings, functions, and %Mint.SensitiveValue{} and then call it on basically everything we think could be sensitive—headers, payloads, URIs, whatever. This way you can naturally pass fn -> secret_value end to stuff. You could also use Mint.redact/1 still as described above. This is however based on Mint knowing how to redact a value, you're right about that.

We could fix that by accepting a function like

Mint.redact(value, redact_fn)

and store that function alongside the struct. The protocol could define redact(value, fun) and basically default the function to fn _value -> "<redacted>" end or something.

Thoughts?

[a3kov]

I'm not sure about the protocol. If there's a struct for sensitive values, we already know that it must be redacted, and everything else should not be. With the struct approach the caller tells you what is sensitive by wrapping the value, and the struct doesn't care about the type of the value inside. With Mint.redact(value, redact_fn) the function would simply wrap it in the struct. Seems like the protocol is unnecessary step. What are you trying to achieve here ?

The redact/1 function is definitely the way to go. When a library simply replaces values with "[redacted]" is not good enough IMO.

[whatyouhide]

If you have

# In the Mint module
def redact(value) do
  fn -> value end
end

then you don't have the benefit of "custom redacting" that you talked about. The only thing you can do is get the value (fun.()) or inspect/print it as #Function<...>.

[a3kov]

Let's say you have a request header. The value of the header parameter will contain either a ready to use value, or the "sensitive value" struct.
In Mint you know what values can be sensitive (and it would be documented), and headers are in that list, so you pattern match on the sensitive value struct, with the struct containing the redact function. The redact function accepts the raw value which must be extracted by calling the closure - both enclosed raw value and the redact function are inside the struct. The caller can be a direct user of Mint, a downstream library, etc. so they call Mint to wrap the value (or wrap it manually) and pass the struct instead of the value.

%Mint.SensitiveValue{
  redact_fun: fn _x -> "[redacted]" end,
  value: fn -> "secretpassword" end
}

At least this is how it can work w.r.t. to requests. Now, if someone wanted to redact responses, I have no idea, as I haven't thought about that, and it may require a different approach.

[a3kov]

A special case is iolist/iodata, where we might want to redact piece by piece instead of the whole binary. And since redaction may require the context (you can't redact half of a word, you need the whole word to detect it), maybe it will involve a special encoding.