Cannot get mutual TLS to work with gRPC over Mint adapter, even though I can get it working with native Mint adapter. Syntax?

[vegabook]

Hello!

When I use elixir Mint directly using mutual TLS, it works fine, but if I try to use the Mint adapter with elixir-grpc, it doesn't.

This works:
https://github.com/user-attachments/assets/095e5450-ae2b-49e5-86fe-eb3f300fbc50
(gives no errors server side)

But this doesn't:
https://github.com/user-attachments/assets/0373023a-df53-40de-b4f0-0b9b9c056034
Server side error:
https://github.com/user-attachments/assets/f985d534-6964-45b9-b726-aef6e0ffa94d

I've tried putting the configs into my config.exs (duplicating because syntax is not obvious from docs):
https://github.com/user-attachments/assets/79be4544-938c-4f18-832f-a4afaca2aa7d

Any idea how I can get the working mint connection, working with elixir-grpc?

I have this working with mutual TLS in Python just for guide: https://github.com/vegabook/gBLP/blob/main/gBLP/client_gblp.py#L135-L166

Here is the text of the connection code:

channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, custom_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert.pem"]) |> elem(1)

I should add that the WRONG_VERSION_NUMBER error is the same one I was getting when my python code was not sending keys and certs properly, and is also what happens if you send no keys and certs.

Note that even if you do not provide any certs whatsoever, it still appears to connect:
https://github.com/user-attachments/assets/3c26c12e-91f0-49dc-a7c5-27a8ee511b4c

whereas if you do the same on Mint without gRPC you definitely get an error:

https://github.com/user-attachments/assets/aaa565cf-387b-422d-ae73-0bfe31991df0

How do I get the working Mint config to be passed via elixir-grpc? Or maybe Gun adapter can do this?

(just for guide if you want to generate a CA authority and server and client authority, this python code will do it)

[polvalente]

Hi, @vegabook. The prints are really hard for me to read.
Could you paste those as code blocks instead next time?

I believe the error stems from you using an invalid :custom_opts key to pass the credentials to the Stub adapter.
You might want adapter_opts: [transport_opts: ...] instead.

[polvalente]

I've edited the issue to remove the print renders, but they are viewable if you click the remaining links.

[vegabook]

I've edited the issue to remove the print renders, but they are viewable if you click the remaining links.

Apols yeah I've gone a bit crazy on wezterm but I thought the syntax highlighting might help.

Okay I've tried this:

iex(32)> channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert.pem"]) |> elem(1)
%GRPC.Channel{
  host: "signaliser.com",
  port: 50051,
  scheme: "http",
  cred: nil,
  adapter: GRPC.Client.Adapters.Mint,
  adapter_payload: %{conn_pid: #PID<0.1005.0>},
  codec: GRPC.Codec.Proto,
  interceptors: [],
  compressor: nil,
  accepted_compressors: [],
  headers: []
}

But server side I still get the error:

E0000 00:00:1732705949.860719    6576 ssl_transport_security.cc:1650] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

What I'll do is I'll try to run non-gRPC mTLS over Mint adapter that I am claiming works (because it doesn't produce the above error), and actually make an https request to ensure it is indeed working properly. Then at least I'll have ascertained that Mint is working correctly.

Another thing I"ll try is Linux <-> Linux as opposed to currently Linux client Windows server. Sometimes Windows is the problem. Unfortunately I do need the server to run on windows because Bloomberg only runs on windows, but at least that might further narrow down the issue or confirm that this is not the problem.

[polvalente]

FYI if you use ```elixir for the code block we do have syntax highlighting

[polvalente]

My recommendation is that you try and compare the options that Mint is receiving inside the adapter (change locally and remember to do mix deps.compile grpc afterwards) with the ones that you're passing in the standalone version.

[vegabook]

My recommendation is that you try and compare the options that Mint is receiving inside the adapter (change locally and remember to do mix deps.compile grpc afterwards) with the ones that you're passing in the standalone version.

Okay so I'm going to check which options Mint is receiving from elixir-grpc? How do I do that?

Also why do I need deps recompile? Doesn't it pick up the options from the config file and/or the passed transport_opts parameter?

[polvalente]

I mean that you can add dbg or similar calls inside the code in the deps folder and recompile as the changes are applied.

[vegabook]

Okay so I put an IO.inspect into Mint's deps/mint/lib/mint/http.ex connect function. It prints out some 111111s first.

  @spec connect(Types.scheme(), Types.address(), :inet.port_number(), keyword()) ::
          {:ok, t()} | {:error, Types.error()}
  def connect(scheme, address, port, opts \\ []) do
    IO.puts("111111111111111")
    IO.inspect(opts)
    case Keyword.fetch(opts, :proxy) do
      {:ok, {proxy_scheme, proxy_address, proxy_port, proxy_opts}} ->
        case Util.scheme_to_transport(scheme) do
          Transport.TCP ->
            proxy = {proxy_scheme, proxy_address, proxy_port}
            host = {scheme, address, port}
            opts = Keyword.merge(opts, proxy_opts)
            UnsafeProxy.connect(proxy, host, opts)

          Transport.SSL ->
            proxy = {proxy_scheme, proxy_address, proxy_port, proxy_opts}
            host = {scheme, address, port, opts}
            IO.putsct(opts)
            TunnelProxy.connect(proxy, host)
        end

      :error ->
        Mint.Negotiate.connect(scheme, address, port, opts)
    end
  end

Works fine on an unprotected endpoint (return certs redacted):

iex(10)> {:ok, channel} = GRPC.Stub.connect("signaliser.com:50052", adapter: GRPC.Client.Adapters.Mint)
111111111111111
[
  parent: #PID<0.430.0>,
  protocols: [:http2],
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[
  hostname: "signaliser.com",
  timeout: :infinity,
  alpn_advertised_protocols: ["h2"]
]
[
  parent: #PID<0.430.0>,
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
{:ok,
 %GRPC.Channel{
   host: "signaliser.com",
   port: 50052,
   scheme: "http",
   cred: nil,
   adapter: GRPC.Client.Adapters.Mint,
   adapter_payload: %{conn_pid: #PID<0.453.0>},
   codec: GRPC.Codec.Proto,
   interceptors: [],
   compressor: nil,
   accepted_compressors: [],
   headers: []
 }}
iex(11)> id = Bloomberg.KeyRequestId.new(id: "yada")
warning: Bloomberg.KeyRequestId.new/1 is deprecated. Build the struct by hand with %MyMessage{...} or use struct/2
└─ iex:11

%Bloomberg.KeyRequestId{id: "yada", __unknown_fields__: []}
iex(12)> Bloomberg.KeyManager.Stub.request_key(channel,id)
{:ok,
 %Bloomberg.KeyResponse{
   key: "-----BEGIN RSA PRIVATE KEY-----OzoE4dRgdIo3fYS4sBR9Z0jJlTs7\nWAEQu97Q7nllIJDkm5C+y+rpIGIwzCuD+TebIc09ZAVeGI3kuzHqtNAnmf4fXF06\nBiYRpzjoiLQjdalkPO1TpxLJryy8CVMuDLDbHDjrj2mUlzPqiMOMMYhuh6uv6jXH\n5IRwGwKBgQCqdYk1IFbhNFox71xZuCwMXwMmjvqXHBzRCi30ilWwxNO3QUHJ45Rx\n8tZcMF9qMS32CrVnGEo5tqRJmepzluNTtN6JZ+SMyNM17SpUKb4MK/4T4utfhQ8i\nIsa9jSneXTiWY3td0OSgvWDOsBFkkCgJuX7eO86PbJZ76S6KUMj+2g==\n-----END RSA PRIVATE KEY-----\n",
   cert: "-----BEGIN CERTIFICATE-----\nMIIDZjCCAk6gAwIBAgIUWecRA1ZJIR9FYhDG/mxE2H2c1aeZWCqSWAIoclk4BZ\nj3s7Aj2ZamV3lDuyCeeH0Jj8OEGpa1JHuR5QNK8LkauzwywTGCuG85hJZMSCn57h\n1WWYMV3bJTsQhg9uU7OzraVr567bJIwgan8xNWFdQoK1M9YM9v4QASQpo2wdZSgS\nVZEjpH65Mss979ISAVsFj7PrnBlv3vtlxuPRES17QY4RfyEvXIaFfQWqdXdfPSi0\naGCy0+KNVgwfiA==\n-----END CERTIFICATE-----\n",
   cacert: "-----BEGIN CERTIFICATE-----cHSUnzNccRfIQUK2RTi2m1hLVCyDvwfQzfVZH\ngd1UaGPoQsufFFFqo1MbpSTUltl6AeWBBg==\n-----END CERTIFICATE-----\n",
   authorised: true,
   reason: "",
   __unknown_fields__: []
 }}

So these keys have been saved cert: and key: merged into client_combined.pem and the cacert: saved into cacert.pem both in ~/scratch. The look like this (again, redacted):

[tbrowne@bee:~/scratch]$ cat cacert.pem
-----BEGIN CERTIFICATE-----
MIIDdTCCAl2gAwIBAgIUFsmob+ck2n86ELvxLrCELhKd9TEwDQYJKoZIhvcNAQEL
BQAwajELMAkGA1UEBhMCWkExEDAOBgNVBAgMB0dhdXRlbmcxFTATBgNVBAcMDEpv
aGFubmVzYnVyZzEfMB0GA1UECgwWWm9tYmllIENBIE9yZ2FuaXphdGlvbjERMA8G
A1UEAwwIWm9tYmllQ0EwHhcNMjQxMTI2MjIwNDE5WhcNMzQxMTI0MjIwNDE5WjBq
...
UxsnImTpotPcEdeF6JDCz7MhIwIoUZ0t/3QK/0Zj+lkYW1dEL0tIvTsItf32OPNx
frJbJOYWS5/igLXjrv3M/0hXB5FVGCSpeqr5VTfBa19AqbWByBxSuJXoseaTNzmE
XYwpDD5AuNtTFkr9oMFxedHBi5/cHSUnzNccRfIQUK2RTi2m1hLVCyDvwfQzfVZH
gd1UaGPoQsufFFFqo1MbpSTUltl6AeWBBg==
-----END CERTIFICATE-----

[tbrowne@bee:~/scratch]$ cat client_combined.pem
-----BEGIN CERTIFICATE-----
MIIDZjCCAk6gAwIBAgIUU8uCFVbJZYeOTkyqZUIwrZZW8ZcwDQYJKoZIhvcNAQEL
BQAwajELMAkGA1UEBhMCWkExEDAOBgNVBAgMB0dhdXRlbmcxFTATBgNVBAcMDEpv
aGFubmVzYnVyZzEfMB0GA1UECgwWWm9tYmllIENBIE9yZ2FuaXphdGlvbjERMA8G
A1UEAwwIWm9tYmllQ0EwHhcNMjQxMTI2MjMyNjM1WhcNMjUxMTI2MjMyNjM1WjBw
...
4laNdA9KV96pz0IBfvLii7UbgeJ8abM0hZH1itXRuNXgdkA+KjFJ/iGTZVGQGCZX
4TYaOt1gwUU6HE9Vd0WUvS5YibzisUpCpZ78XqHxTfnz8nt9kd0fW2wWXG8ohs7e
kHXQpXzI/2PLgHa0vgVOHWMJxbRqfdHR2UekxH/V51aPXj6n5MRJWrwv4cb/SX2/
4J4FPg4N/BIWUg==
-----END CERTIFICATE-----
-----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEAqS4r8SVYBqboDtPDANm/VwkAX309+vh8HU268dLT4xOELdcg
MIYnfKJnQ3/DXTFoptdwaqPPI7LEioAvxuSF8a5RFbJ7vImvGgT/ZY4+Cszcwfzh
gpqKxlS9+gkV2atRsyGCXuVxW0EYhjYxdP4Y9kq3V046mjPP21aIyEUx2uKx38fg
r9iW7b8VIEFSGiirT3J3SKemSOgBM3YwpLEijqZwsd9kKUS5PK4w1N1Kv7C8CcCu
...
n307ptx8TvYE+quPEyIQANlXZwpXjp8Lbws21Npe5DjLGqnatW8bQhn+nnn6FkC0
wKJ0jAf1IUQxXRNGcz5L+OuUqQSSscAZaNhvpA/cbX5vsO7GQfS9
-----END RSA PRIVATE KEY-----

Let's first connect with Mint. It takes a cacertfile option and this question on Elixir Forum suggests it also takes a certfile which I assume is the client cert.

iex(18)> Mint.HTTP.connect(:https, "signaliser.com", 50051, transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert.pem"])
111111111111111
[
  transport_opts: [
    certfile: "/home/tbrowne/scratch/client_combined.pem",
    cacertfile: "/home/tbrowne/scratch/cacert.pem"
  ]
]
{:ok,
 %Mint.HTTP2{
   transport: Mint.Core.Transport.SSL,
   socket: {:sslsocket, {:gen_tcp, #Port<0.31>, :tls_connection, :undefined},
    [#PID<0.480.0>, #PID<0.479.0>]},
   mode: :active,
   hostname: "signaliser.com",
   port: 50051,
   scheme: "https",
   authority: "signaliser.com:50051",
   state: :handshaking,
   buffer: "",
   window_size: 65535,
   encode_table: %HPAX.Table{
     max_table_size: 4096,
     entries: [],
     size: 0,
     length: 0
   },
   decode_table: %HPAX.Table{
     max_table_size: 4096,
     entries: [],
     size: 0,
     length: 0
   },
   ping_queue: {[], []},
   client_settings_queue: {[[]], []},
   next_stream_id: 3,
   streams: %{},
   open_client_stream_count: 0,
   open_server_stream_count: 0,
   ref_to_stream_id: %{},
   server_settings: %{
     enable_connect_protocol: false,
     enable_push: true,
     initial_window_size: 65535,
     max_concurrent_streams: 100,
     max_frame_size: 16384,
     max_header_list_size: :infinity
   },
   client_settings: %{
     enable_push: true,
     initial_window_size: 65535,
     max_concurrent_streams: 100,
     max_frame_size: 16384,
     max_header_list_size: :infinity
   },
   headers_being_processed: nil,
   proxy_headers: [],
   private: %{},
   log: false
 }}

We can see in the IO.inspect that we have those transport options in there. And the server does NOT complain, therefore I assume the connection is fine.

Now let's try this with elixir-grpc:

iex(19)> channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert."]) |> elem(1)
111111111111111
[
 parent: #PID<0.430.0>,
 protocols: [:http2],
 transport_opts: [timeout: :infinity],
 client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[
 hostname: "signaliser.com",
 timeout: :infinity,
 alpn_advertised_protocols: ["h2"]
]
[
 parent: #PID<0.430.0>,
 transport_opts: [timeout: :infinity],
 client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
%GRPC.Channel{
 host: "signaliser.com",
 port: 50051,
 scheme: "http",
 cred: nil,
 adapter: GRPC.Client.Adapters.Mint,
 adapter_payload: %{conn_pid: #PID<0.482.0>},
 codec: GRPC.Codec.Proto,
 interceptors: [],
 compressor: nil,
 accepted_compressors: [],
 headers: []
}

The transport_opts did not get passed through, and as expected, the server complains:

E0000 00:00:1732814795.640362   25944 ssl_transport_security.cc:1650] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

Maybe passing transport_opts via opts: will work:

channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, opts: [transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert."]]) |> elem(1)
111111111111111
[
  parent: #PID<0.430.0>,
  protocols: [:http2],
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[
  hostname: "signaliser.com",
  timeout: :infinity,
  alpn_advertised_protocols: ["h2"]
]
[
  parent: #PID<0.430.0>,
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
%GRPC.Channel{
  host: "signaliser.com",
  port: 50051,
  scheme: "http",
  cred: nil,
  adapter: GRPC.Client.Adapters.Mint,
  adapter_payload: %{conn_pid: #PID<0.484.0>},
  codec: GRPC.Codec.Proto,
  interceptors: [],
  compressor: nil,
  accepted_compressors: [],
  headers: []
}

Nope. Passed transport_opts nowhere to be seen. custom_opts maybe?

channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, custom_opts: [transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert."]]) |> elem(1)
111111111111111
[
  parent: #PID<0.430.0>,
  protocols: [:http2],
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[
  hostname: "signaliser.com",
  timeout: :infinity,
  alpn_advertised_protocols: ["h2"]
]
[
  parent: #PID<0.430.0>,
  transport_opts: [timeout: :infinity],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
%GRPC.Channel{
  host: "signaliser.com",
  port: 50051,
  scheme: "http",
  cred: nil,
  adapter: GRPC.Client.Adapters.Mint,
  adapter_payload: %{conn_pid: #PID<0.486.0>},
  codec: GRPC.Codec.Proto,
  interceptors: [],
  compressor: nil,
  accepted_compressors: [],
  headers: []
}

Same deal. And of course server complains both those times too:

E0000 00:00:1732814795.640362   25944 ssl_transport_security.cc:1650] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0000 00:00:1732814937.840075   25944 ssl_transport_security.cc:1650] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.
E0000 00:00:1732815017.610234    3228 ssl_transport_security.cc:1650] Handshake failed with fatal error SSL_ERROR_SSL: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER.

So let's put those files into config.exs Mint section:

config :grpc, GRPC.Client.Adapters.Mint, 
  transport_opts: [certfile: "/home/tbrowne/scratch/client_combined.pem", cacertfile: "/home/tbrowne/scratch/cacert.pem"]

Recompile. Now let's try:

iex(1)> channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint) |> elem(1)                                 111111111111111
[
  parent: #PID<0.634.0>,
  protocols: [:http2],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000],
  transport_opts: [
    certfile: "/home/tbrowne/scratch/client_combined.pem",
    cacertfile: "/home/tbrowne/scratch/cacert.pem"
  ]
]
[
  hostname: "signaliser.com",
  certfile: "/home/tbrowne/scratch/client_combined.pem",
  cacertfile: "/home/tbrowne/scratch/cacert.pem",
  alpn_advertised_protocols: ["h2"]
]
[
  parent: #PID<0.634.0>,
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000],
  transport_opts: [
    certfile: "/home/tbrowne/scratch/client_combined.pem",
    cacertfile: "/home/tbrowne/scratch/cacert.pem"
  ]
]
[error] unable to establish a connection. reason: :badarg
"Error while opening connection: {:error, :badarg}"

Now we get a :badarg.

So now they're there, in the transport_opts, as per the direct Mint connection at the beginning of this message, but we get a :badarg somehow.

As per gBLP, the repo I'm trying to make this work for, Python works fine python <-> python with mTLS and I'm generating my own ca cert authority and ca certs for both client and server, so I'm happy to fork gBLP and create a minimal python server that will serve you some certs, and give you a Ping-Pong grpc endpoint to try, if that helps? If you use Nix I can even turn the whole thing into a flake. But perhaps the above will help to nail this?

Very keen to bring the Bloomberg terminal into the Elixir NX ecosystem as an alternative to R or Python, and thereby maybe drag some of the quants in from there as a result.

EDIT

In fact if it helps I'm quite happy to put up a public server that does nothing but throw out certs via an unsecured gRPC endpoint, so that you can use the certs to connect to another, secure gRPC endpoint that will return "pong" from a gRPC "ping" if correctly authorised. Let me know if that would help.

[polvalente]

I think you need to pass transport_opts as a key inside the adapter_opts key.

Edit: I expect that to result in the same error as setting it in the config, but let's try that out first.

[vegabook]

I think you need to pass transport_opts as a key inside the adapter_opts key.

Edit: I expect that to result in the same error as setting it in the config, but let's try that out first.

Yes indeed.

iex(7)> channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, adapter_opts: [transport_opts: [certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem", cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem"]]) |> elem(1)
111111111111111
[
  parent: #PID<0.430.0>,
  protocols: [:http2],
  transport_opts: [
    timeout: :infinity,
    certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem",
    cacertfile: "/home/tbrowne/.config/suprabonds/cacert."
  ],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[
  hostname: "signaliser.com",
  timeout: :infinity,
  certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem",
  cacertfile: "/home/tbrowne/.config/suprabonds/cacert.",
  alpn_advertised_protocols: ["h2"]
]
[
  parent: #PID<0.430.0>,
  transport_opts: [
    timeout: :infinity,
    certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem",
    cacertfile: "/home/tbrowne/.config/suprabonds/cacert."
  ],
  client_settings: [initial_window_size: 8000000, max_frame_size: 8000000]
]
[error] unable to establish a connection. reason: :badarg
"Error while opening connection: {:error, :badarg}"

(please note that I moved the certs to ~/.config/suprabonds/ if you're comparing it to previous messages. But I did test again with plain mint and it works:

iex(6)> Mint.HTTP.connect(:https, "signaliser.com", 50051, transport_opts: [certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem", cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem"])
111111111111111
[
  transport_opts: [
    certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem",
    cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem"
  ]
]
{:ok,
 %Mint.HTTP2{
   transport: Mint.Core.Transport.SSL,
   socket: {:sslsocket, {:gen_tcp, #Port<0.24>, :tls_connection, :undefined},
    [#PID<0.449.0>, #PID<0.448.0>]},
   mode: :active,
   hostname: "signaliser.com",
   port: 50051,
   scheme: "https",
   authority: "signaliser.com:50051",
   state: :handshaking,
   buffer: "",
   window_size: 65535,
   encode_table: %HPAX.Table{
     max_table_size: 4096,
     entries: [],
     size: 0,
     length: 0
   },
   decode_table: %HPAX.Table{
     max_table_size: 4096,
     entries: [],
     size: 0,
     length: 0
   },
   ping_queue: {[], []},
   client_settings_queue: {[[]], []},
   next_stream_id: 3,
   streams: %{},
   open_client_stream_count: 0,
   open_server_stream_count: 0,
   ref_to_stream_id: %{},
   server_settings: %{
     enable_connect_protocol: false,
     enable_push: true,
     initial_window_size: 65535,
     max_concurrent_streams: 100,
     max_frame_size: 16384,
     max_header_list_size: :infinity
   },
   client_settings: %{
     enable_push: true,
     initial_window_size: 65535,
     max_concurrent_streams: 100,
     max_frame_size: 16384,
     max_header_list_size: :infinity
   },
   headers_being_processed: nil,
   proxy_headers: [],
   private: %{},
   log: false
 }}

I'm going to put up a public insecure dummy cert gRPC endpoint, whose certs will allow connection to a secure ping gRPC endpoint, for testing, in a few hours.

[vegabook]

Okay I've setup a basic grpc server at rixtract.com ports 50051 and 50052.

50051 requires mtls
50052 is completely open and will provide certs

So clone this repo: https://github.com/vegabook/suprabonds
install the deps, iex -S mix, then this code will get you certs:

{:ok, channel} = GRPC.Stub.connect("rixtract.com:50052", adapter: GRPC.Client.Adapters.Mint)
 id = Bloomberg.KeyRequestId.new(id: "yada")
 Bloomberg.KeyManager.Stub.request_key(channel,id)

Then you can merge the returned :cert and :key into a certfile, and use the :cacert and as the certificate authority certificate.

Here it is working but I've obfuscated the returned keys (even though the server is open lol) but at least it puts a mild hurdle for scrapers.

iex(3)> {:ok, channel} = GRPC.Stub.connect("rixtract.com:50052", adapter: GRPC.Client.Adapters.Mint)
{:ok,
 %GRPC.Channel{
   host: "rixtract.com",
   port: 50052,
   scheme: "http",
   cred: nil,
   adapter: GRPC.Client.Adapters.Mint,
   adapter_payload: %{conn_pid: #PID<0.768.0>},
   codec: GRPC.Codec.Proto,
   interceptors: [],
   compressor: nil,
   accepted_compressors: [],
   headers: []
 }}
iex(4)> id = Bloomberg.KeyRequestId.new(id: "yada")                                                                                        warning: Bloomberg.KeyRequestId.new/1 is deprecated. Build the struct by hand with %MyMessage{...} or use struct/2
└─ iex:4

%Bloomberg.KeyRequestId{id: "yada", __unknown_fields__: []}
iex(5)> Bloomberg.KeyManager.Stub.request_key(channel,id)                                                                                  {:ok,
 %Bloomberg.KeyResponse{
   key: "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEAx0/sX2ezjcH4Q3ezWbG2gn6FKX/2oGhkbR6SmVPrTYJdA8cn\nlUPYGfVjb1go9asFDVxcO7Z7h3Ev0x5KDQRj9Wgocuv3v6cEjDxQ9zFyjAvLKxQW\nWeXf3PaujqWVLHbs6Jnj7ZZLAIL2/DGw7LFFgXAe/oE4xijuEC3sDDDDRSzegsf1\nywBTcVNZJ1vCpN24WQzIzdc4veKe4SH+/TJv5BErCq8TF+Hwh3TMCMeNfAaOKhj5\nmJ3W5jcnU9d5hBsW0Pdxx/maK1xeq6udUkihkkbelqfocPrSqBZELQaGIEy4Gora\n5QYlYCMp6NWLhE8Q44ikwu+kkFTbNQ6b8LMj/wIDAQABAoIBAFSbsHdVkcq5Dzcw\nvqFyR8xAgi26RQsWFZ+ezVOqZKZcEzkx1zNNgBSQ7zuKfKTBCx1/WiX9EpZN7aPc\n5Ul+ViKMqzMRyXOBpnQTfLlas8oQjZgFaHGUHEJ0M2iZkOg1ud8JyNCILuGSNt+b\nBoSLwjouyvF2EJBKOrkCjYQlRRd9M0AF+EV2v2XxG98sLzBYLzyB6ZTF9npdYYKf\ntRQ5HWILZOBg3C/IRb1GeLcjMF0c2iVwPqlQFzisABeTHrTEs93SJ+EotEOvJo5a\nQfkwJFKEMMTLXQCKeAecotGy2meYaolbE1GNISXYBarh9togjWgNAFKFG3AS3+1V\nZcvMgBkCgYEA6M5VvqJ7fZ6qpdcg4M2K6q/DB93BDzmlozkHmI9P5QXheQpqD02R\nciJNZsh3wu/5oodtktuNPEG51L83EzmH2QOFPo/3Px8SM4pDlwanWNOPNLQg7irf\n1xkCVaseVUQEA0mXNgSRxbhiO8rPjI0BHuPaA2GQIaif95vzfbv8PQcCgYEA2ytW\nK4z+YjttMssPc2Qc5fBHa5Dknyi+2+tgNnazb7HGneFUQXyZR7hC/0oqxDB63b3yMh9Nn2UBRWx6Ga\nTExjV5PqI2xeNYqc6OSA+v3tn+MYQ6SOoWyaYO6HGFwD9ZHomJRyRN7dK6IaN8hK\nO5d2QQKBgQCiHyQc8neGoustlJmTEmEmF1cUeBfOIc72kgk29aR3/Y0M/xUE5hpR\nW9AaazuStET7K98HuFiMDa17V7xppdILjBMq2yNJPucBD6IYRP+PLCuB/n5N539X\nF28bDFxXnCCMveWuTdwAkCk5CNajQhS3ZvuTwuYzdK04wp6bbO+Oxw==\n-----END RSA PRIVATE KEY-----\n",
   cert: "-----BEGIN CERTIFICATE-----\nMIIDZDCCAkygAwIBAgIUWooV9n95Mfa9a8Xbk+NksyA9lZgwDQYJKoZIhvcNAQEL\nBQAwajELMAkGA1UEBhMCWkExEDAOBgNVBAgMB0dhdXRlbmcxFTATBgNVBAcMDEpv\naGFubmVzYnVyZzEfMB0GA1UECgwWWm9tYmllIENBIE9yZ2FuaXphdGlvbjERMA8G\nA1UEAwwIWm9tYmllQ0EwHhcNMjQxMTI5MjAxNDAyWhcNMjUxMTI5MjAxNDAyWjBu\nMQswCQYDVQQGEwJaQTEQMA4GA1UECAwHR2F1dGVuZzEVMBMGA1UEBwwMSm9oYW5u\nZXNidXJnMR8wHQYDVQQKDBZab21iaWUgQ0EgT3JnYW5pemF0aW9uMRUwEwYDVQQD\nDAxyaXh0cmFjdC5jb20wggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEKAoIBAQDH\nT+x0CXZy59im0qeglGlYoFP+Hv\ngkBIWqh6abCwFCHd//ILqfKuGbAjkOseI5430zYwatnSIzWDKzmtpyKatcmsdHv+\nt5jwWDiOM11nIpWX61MFPnnfnf8wV0gv/Mgt97gklCWNecvt7T54dvquoKDlY8hS\nAYmRTAIpXotB97kOx4cM9nO2wVPPEeVAPItQDBf6nfqtRK0r5JFCM2++2U83lt5B\nBbfDFMjRZwY=\n-----END CERTIFICATE-----\n",
   cacert: "-----BEGIN CERTIFICATE-----\nMIIDdTCCAl2gAwIBAgIUfUEZZSg7gEf8dtmzvDa+dFS5uAIwDQYJKoZIhvcNAQEL\nBQAwajELMAkGA1UEBhMCWkExEDAOBgNVBAgMB0dhdXRlbmcxFTATBgNVBAcMDEpv\naGFubmVzYnVyZzEfMB0GA1UECgwWWm9tYmllIENBIE9yZ2FuaXphdGlvbjERMA8G\nA1UEAwwIWm9tYmllQ0EwHhcNMjQxMTI5MTkxMjUwWhcNMzQxMTI3MTkxMjUwWjBq\nMQswCQYDVQQGEwJaQTEQMA4GA1UECAwHR2F1dGVuZzEVMBMGA1UEBwwMSm9oYW5u\nZXNidXJnMR8wHQYDVQQKDBZab21iaWUgQ0EgT3JnYW5pemF0aW9uMREwDwYDVQQD\nDAhab21iaWVDQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBALxNepRk\n+xxtyT/KYiaeQdVBwXe1TSaXC1TUwqbchUNqVKoFq+GZNcmicoJPfMEBXJZb2PRr\nGnfG7qNiMkqyqLXsp8krM9k4Uq4m3qWzNQhgvg2fuWjhxEwrnYWcqJa\nZh5oJVyKXaqyd9QsrNXzpf8TTg0zhCpbSax6szgI5NsElCdHzdKoDctbiKhkkWgL\npxC4ityPutm3lAS9JJcAofJ7lios0WR3ntJHePXpUKQ4nu6hOEvt+cJzss2C8YQc\naTRmzOmiwwk3ktJIpVmKkXaCbrtyd7h74SzwnTul6GCmib5HCCoS+E1eiQ94JWK5\nwtMLnvmcvrmYfWLxjWJ8X8vYQAH1JL5IdQ==\n-----END CERTIFICATE-----\n",
   authorised: true,
   reason: "",
   __unknown_fields__: []
 }}

If you want to test it with python, you can clone this repo: https://github.com/vegabook/gBLP, then checkout branch elixir_test.

if you use nix, then nix develop will sort everything out, otherwise there's a pyproject.toml there for poetry which should work. Then you can:

[tbrowne@bee:~/code/gBLP/gBLP]$ python client_gblp.py --grpchost rixtract.com
/home/tbrowne/code/gBLP/gBLP/bloomberg_pb2_grpc.py:21: RuntimeWarning: The grpc package installed is at version 1.64.1, but the generated code in bloomberg_pb2_grpc.py depends on grpcio>=1.65.1. Please upgrade your grpc module to grpcio>=1.65.1 or downgrade your generated code using grpcio-tools<=1.64.1. This warning will become an error in 1.66.0, scheduled for release on August 6, 2024.
  warnings.warn(
┌───────────────────────────────────────────────────────────────── BETA ──────────────────────────────────────────────────────────────────┐
│ This software is currently in beta testing. All features are working and tested, but there may still be bugs or issues that have not    │
│ been discovered. https://github.com/vegabook/gBLP/issues                                                                                │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
2024-11-29 20:22:23,484 - asyncio - DEBUG - Using selector: EpollSelector
2024-11-29 20:22:23,485 - __main__ - INFO - Connecting to rixtract.com:50051...
2024-11-29 20:22:23,485 - grpc._cython.cygrpc - DEBUG - Using AsyncIOEngine.POLLER as I/O engine
Pinging...
2024-11-29 20:22:23,489 - __main__ - INFO - Pinging server with message mocilo940
---------------------
pong timestamp: seconds: 1732911743
nanos: 785104000
 message: mocilo940 Pong
Pinging...
2024-11-29 20:22:24,819 - __main__ - INFO - Pinging server with message finulo030
---------------------
pong timestamp: seconds: 1732911744
nanos: 868675000
 message: finulo030 Pong
Pinging...
2024-11-29 20:22:25,902 - __main__ - INFO - Pinging server with message culebu373
---------------------
pong timestamp: seconds: 1732911745
nanos: 952292000
 message: culebu373 Pong
^CTraceback (most recent call last):
  File "/home/tbrowne/code/gBLP/gBLP/client_gblp.py", line 308, in <module>
    time.sleep(1)
KeyboardInterrupt
^CException ignored in: <module 'threading' from '/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/threading.py'>
Traceback (most recent call last):
  File "/nix/store/h3i0acpmr8mrjx07519xxmidv8mpax4y-python3-3.12.5/lib/python3.12/threading.py", line 1624, in _shutdown
    lock.acquire()
KeyboardInterrupt:

And you'll see that with python and keys, we're getting pongs on the mTLS endpoint 50051

I've put a 1 second delay between cert requests in case someone tries to ddos and if you have any problems let me know.

[vegabook]

Okay so just to be really sure, I created a connection to the mtls port 50052 using Mint directly, and tried to do a get request on it, which does work fine:

iex(1)> MTLSTest.start()
[info] Received status: 200
[info] Received headers: [{"content-type", "application/grpc"}, {"grpc-status", "2"}, {"grpc-message", "Bad method header"}]
[info] Stream done

As you can see the transport is working with status 200, but of course the grpc server itself is complaining about bad method headers. I have also confirmed that mtls is occurring by changing single characters in the client_combined.pem and cacert.pem separately, and in both cases the connection is refused. In other words, it's all good on the Mint side. We have operational mTLS.

If you have further hints on where I should look inside elixir-grpc then I can try to see if I can fix it.

[vegabook]

yabadabadoo

 channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, adapter_opts: [transport_opts: [cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem", certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem"]], scheme: :https) |> elem(1)
[(grpc 0.9.0) lib/grpc/client/adapters/mint.ex:42: GRPC.Client.Adapters.Mint.connect/2]
[t: channel] #=> [
  t: %GRPC.Channel{
    host: "signaliser.com",
    port: 50051,
    scheme: "http",
    cred: nil,
    adapter: GRPC.Client.Adapters.Mint,
    adapter_payload: nil,
    codec: GRPC.Codec.Proto,
    interceptors: [],
    compressor: nil,
    accepted_compressors: [],
    headers: []
  }
]

-----------
%GRPC.Channel{
  host: "signaliser.com",
  port: 50051,
  scheme: "http",
  cred: nil,
  adapter: GRPC.Client.Adapters.Mint,
  adapter_payload: nil,
  codec: GRPC.Codec.Proto,
  interceptors: [],
  compressor: nil,
  accepted_compressors: [],
  headers: []
}
-----------
============
%GRPC.Channel{
  host: "signaliser.com",
  port: 50051,
  scheme: "https",
  cred: nil,
  adapter: GRPC.Client.Adapters.Mint,
  adapter_payload: nil,
  codec: GRPC.Codec.Proto,
  interceptors: [],
  compressor: nil,
  accepted_compressors: [],
  headers: []
}
============

That's a success. No more bad args. Same return as the Mint direct call.

The key here is the "scheme" line. I forced it to "https". Because before being forced it was "http", and this meant the erlang :ssl transport was never being used and so it didn't use the certfile and cacertfile arguments thinking they were :badargs because it was going through the "http" codepath.

Here is where I forced https:

defmodule GRPC.Client.Adapters.Mint do
  @moduledoc """
  A client adapter using Mint.
  """

  alias GRPC.Channel
  alias GRPC.Client.Adapters.Mint.ConnectionProcess
  alias GRPC.Client.Adapters.Mint.StreamResponseProcess
  alias GRPC.Credential

  @behaviour GRPC.Client.Adapter

  @default_connect_opts [
    protocols: [:http2]
  ]
  @default_client_settings [
    initial_window_size: 8_000_000,
    max_frame_size: 8_000_000
  ]
  @default_transport_opts [timeout: :infinity]

  @doc """
  Connects using Mint based on the provided configs. Options
    * `:transport_opts`: Defaults to `[timeout: :infinity]`, given the nature of H2 connections (with support to
      long-lived streams) this default is set to avoid timeouts while waiting for server streams to complete. The other
      options may vary based on the transport used for this connection (tcp or ssl). Check [Mint.HTTP.connect/4](https://hexdocs.pm/mint/Mint.HTTP.html#connect/4)
    * `:client_settings`: Defaults to `[initial_window_size: 8_000_000, max_frame_size: 8_000_000]`, a larger default
      window size ensures that the number of packages exchanges is smaller, thus speeding up the requests by reducing the
      amount of networks round trip, with the cost of having larger packages reaching the server per connection.
      Check [Mint.HTTP2.setting() type](https://hexdocs.pm/mint/Mint.HTTP2.html#t:setting/0) for additional configs.
  """
  @impl true
  def connect(%{host: host, port: port} = channel, opts \\ []) do
    # Added :config_options to facilitate testing.


    {config_opts, opts} = Keyword.pop(opts, :config_options, [])
    module_opts = Application.get_env(:grpc, __MODULE__, config_opts)

    opts = connect_opts(channel, opts) |> merge_opts(module_opts)

    dbg(t: channel)
    Process.flag(:trap_exit, true)
    IO.puts("-----------")
    IO.inspect(channel)
    IO.puts("-----------")

    channel = %{channel | scheme: "https"}
    IO.puts("============")
    IO.inspect(channel)
    IO.puts("============")

See the 4th line from the bottom. Note that whether I do that or not, it appears to create a channel. It's just that without the http: force, the channel will not be secure, and the server will refuse to serve it.

basically, the mint_scheme function is correctly parsing "https" in the url if I call the connection like this (note the https in the url).

iex(5)> channel = GRPC.Stub.connect("https://signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, adapter_opts: [transport_opts: [cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem", certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem"]]) |> elem(1)

but then I get this error:

[error] unable to establish a connection. reason: %Mint.TransportError{reason: {:options, {:socket_options, [cacert_file: "/home/tbrowne/code/suprabonds/_build/dev/lib/castore/priv/cacerts.pem", packet_size: 0, packet: 0, header: 0, active: false, mode: :binary]}}}
"Error while opening connection: {:error, %Mint.TransportError{reason: {:options, {:socket_options, [cacert_file: \"/home/tbrowne/code/suprabonds/_build/dev/lib/castore/priv/cacerts.pem\", packet_size: 0, packet: 0, header: 0, active: false, mode: :binary]}}}}"

you can see that it's trying to set the ca_certfile: to _build/dev/lib/castore/priv/cacerts.pem and ignoring the cacertfile: parameter that I specified, parameter which does work when using Mint directly.

However if I force https with the line as above namely channel = %{channel | scheme: "https"} in client/adapters/mint.ex, and then call the connect function without https//

iex(8)> channel = GRPC.Stub.connect("signaliser.com:50051", adapter: GRPC.Client.Adapters.Mint, adapter_opts: [transport_opts: [cacertfile: "/home/tbrowne/.config/suprabonds/cacert.pem", certfile: "/home/tbrowne/.config/suprabonds/client_combined.pem"]]) |> elem(1)

Then it all works beautifully and I can make gRPC calls

iex(8)> Bloomberg.Bbg.Stub.intraday_bar_request(channel,  %Bloomberg.Topic{topic: "USDZAR Curncy", topictype: 1}, metadata: [client: "yada"])                                                                                                                                         {:ok,
 %Bloomberg.IntradayBarResponse{
   responseError: %Bloomberg.ErrorInfo{
     source: "bbdbl14",
     code: 10,
     category: "BAD_SEC",
     message: "Unknown/Invalid security [nid:3121] ",
     subcategory: "INVALID_SECURITY",
     __unknown_fields__: []
   },
   bars: [],
   __unknown_fields__: []
 }}

(okay I have a topic error but that's not a problem. The server received the call and sent back an error).

So, something weird is going on when the URL is parsed with "https" in it. Settings or options somehow get messed up. However if I just pass signaliser.com:50051 then sure it thinks it's normal http, and fails unless I force :https as above, but at least I don't get all the other errors and I can connect. Maybe I'm supposed to set :https explicitly in my call somewhere (although It does seem that URI.parse is supposed to be handling that, but it's also doing something else that's breaking the https connection).

[vegabook]

INTERIM SUMMARY CONCLUSION

From all of the above, it seems to me that when specifiying an https://xxx.yyy host, then a codepath is taken which ignores the cacertfile or cacert options in transport_opts (these work when used directly with Mint). This is not the case if we only pass the authority namely xxx.yyy, because elixir-grpc then takes a plain http codepath, which paradoxically does take into account cacertfile and cacert. However we then later have to hack back to https otherwise those options are meaningless.

It's possible (probable?) that there is some combination of options that will avoid this problem, that I haven't found yet. But it's not obvious from the docs (we're told to got Mint docs, which tells us to go to its transport options section, which mentions ca certs but not client certs, for which we must then go to Erlang's ssl docs, where there certfile makes an appearance but is apparently deprecated, although it works with Mint, and the alternative, certs_keys doesn't work). So it's kind of a labyrinth that I've only been able to navigate so far by hacking through one of the walls, and while it seems that it might just be a case of finding the right combination of options, I can't rule out the possibility that it's a bug.