Skip to content

Starting two GRPC.Client.Supervisors prevents the VM from shutting down gracefully #479

@ollien

Description

@ollien

Describe the bug
If there are two GRPC.Client.Supervisors running, with default OTP configurations, the VM will never shut down.

I believe this is caused by #466 - it allows for two unrelated supervisors to supervise the same pid, which leads to this unfortunate situation.

To Reproduce
Create a mix project with this Application. If you run mix run --no-halt, and then send a SIGTERM to the VM process (kill -s TERM $(pgrep beam)), the process will log [notice] SIGTERM received - shutting down, and then hang forever. If you remove one of the children of the supervisor (e.g. sup2 below), the VM shuts down rather quickly after logging the SIGTERM message.

This application is obviously a contrived example, but it can be maddeningly hard to debug if, for instance, two dependency's OTP applications start GRPC.Client.Supervisors for you.

This can be worked around if you set the shutdown_timeout parameter to some reasonable timeout before sending the SIGTERM (e.g. Application.put_env(:kernel, :shutdown_timeout, 1000)), this causes the VM to brutally kill any process that doesn't shut down in time, which is likely not a good long-term fix

defmodule GrpcDualSupTest.Application do
  @moduledoc false

  use Application

  defmodule TestSupervisor do
    use Supervisor

    def start_link(arg) do
      Supervisor.start_link(__MODULE__, arg)
    end

    @impl true
    def init(_) do
      children = [
        {GRPC.Client.Supervisor, []}
      ]

      Supervisor.init(children, strategy: :one_for_one)
    end
  end

  @impl true
  def start(_type, _args) do
    children = [
      Supervisor.child_spec({TestSupervisor, []}, id: :sup1),
      Supervisor.child_spec({TestSupervisor, []}, id: :sup2)
    ]

    opts = [strategy: :one_for_one, name: GrpcDualSupTest.Supervisor]
    Supervisor.start_link(children, opts)
  end
end

Expected behavior
The client supervisor should not prevent the VM from being shutdown.

Versions:

  • OS: Ubuntu 22.04
  • Erlang: OTP 25
  • Elixir: Elixir 1.15.5
  • mix.lock(grpc, gun, cowboy, cowlib):
mix.lock
%{
  "cowboy": {:hex, :cowboy, "2.14.2", "4008be1df6ade45e4f2a4e9e2d22b36d0b5aba4e20b0a0d7049e28d124e34847", [:make, :rebar3], [{:cowlib, ">= 2.16.0 and < 3.0.0", [hex: :cowlib, repo: "hexpm", optional: false]}, {:ranch, ">= 1.8.0 and < 3.0.0", [hex: :ranch, repo: "hexpm", optional: false]}], "hexpm", "569081da046e7b41b5df36aa359be71a0c8874e5b9cff6f747073fc57baf1ab9"},
  "cowlib": {:hex, :cowlib, "2.16.0", "54592074ebbbb92ee4746c8a8846e5605052f29309d3a873468d76cdf932076f", [:make, :rebar3], [], "hexpm", "7f478d80d66b747344f0ea7708c187645cfcc08b11aa424632f78e25bf05db51"},
  "flow": {:hex, :flow, "1.2.4", "1dd58918287eb286656008777cb32714b5123d3855956f29aa141ebae456922d", [:mix], [{:gen_stage, "~> 1.0", [hex: :gen_stage, repo: "hexpm", optional: false]}], "hexpm", "874adde96368e71870f3510b91e35bc31652291858c86c0e75359cbdd35eb211"},
  "gen_stage": {:hex, :gen_stage, "1.3.2", "7c77e5d1e97de2c6c2f78f306f463bca64bf2f4c3cdd606affc0100b89743b7b", [:mix], [], "hexpm", "0ffae547fa777b3ed889a6b9e1e64566217413d018cabd825f786e843ffe63e7"},
  "googleapis": {:hex, :googleapis, "0.1.0", "13770f3f75f5b863fb9acf41633c7bc71bad788f3f553b66481a096d083ee20e", [:mix], [{:protobuf, "~> 0.12", [hex: :protobuf, repo: "hexpm", optional: false]}], "hexpm", "1989a7244fd17d3eb5f3de311a022b656c3736b39740db46506157c4604bd212"},
  "grpc": {:hex, :grpc, "0.11.5", "5dbde9420718b58712779ad98fff1ef50349ca0fa7cc0858ae0f826015068654", [:mix], [{:castore, "~> 0.1 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:cowboy, "~> 2.10", [hex: :cowboy, repo: "hexpm", optional: false]}, {:cowlib, "~> 2.12", [hex: :cowlib, repo: "hexpm", optional: false]}, {:flow, "~> 1.2", [hex: :flow, repo: "hexpm", optional: false]}, {:googleapis, "~> 0.1.0", [hex: :googleapis, repo: "hexpm", optional: false]}, {:gun, "~> 2.0", [hex: :gun, repo: "hexpm", optional: false]}, {:jason, ">= 0.0.0", [hex: :jason, repo: "hexpm", optional: false]}, {:mint, "~> 1.5", [hex: :mint, repo: "hexpm", optional: false]}, {:protobuf, "~> 0.14", [hex: :protobuf, repo: "hexpm", optional: false]}, {:telemetry, "~> 1.0", [hex: :telemetry, repo: "hexpm", optional: false]}], "hexpm", "0a5d8673ef16649bef0903bca01c161acfc148e4d269133b6834b2af1f07f45e"},
  "gun": {:hex, :gun, "2.2.0", "b8f6b7d417e277d4c2b0dc3c07dfdf892447b087f1cc1caff9c0f556b884e33d", [:make, :rebar3], [{:cowlib, ">= 2.15.0 and < 3.0.0", [hex: :cowlib, repo: "hexpm", optional: false]}], "hexpm", "76022700c64287feb4df93a1795cff6741b83fb37415c40c34c38d2a4645261a"},
  "hpax": {:hex, :hpax, "1.0.3", "ed67ef51ad4df91e75cc6a1494f851850c0bd98ebc0be6e81b026e765ee535aa", [:mix], [], "hexpm", "8eab6e1cfa8d5918c2ce4ba43588e894af35dbd8e91e6e55c817bca5847df34a"},
  "jason": {:hex, :jason, "1.4.4", "b9226785a9aa77b6857ca22832cffa5d5011a667207eb2a0ad56adb5db443b8a", [:mix], [{:decimal, "~> 1.0 or ~> 2.0", [hex: :decimal, repo: "hexpm", optional: true]}], "hexpm", "c5eb0cab91f094599f94d55bc63409236a8ec69a21a67814529e8d5f6cc90b3b"},
  "mint": {:hex, :mint, "1.7.1", "113fdb2b2f3b59e47c7955971854641c61f378549d73e829e1768de90fc1abf1", [:mix], [{:castore, "~> 0.1.0 or ~> 1.0", [hex: :castore, repo: "hexpm", optional: true]}, {:hpax, "~> 0.1.1 or ~> 0.2.0 or ~> 1.0", [hex: :hpax, repo: "hexpm", optional: false]}], "hexpm", "fceba0a4d0f24301ddee3024ae116df1c3f4bb7a563a731f45fdfeb9d39a231b"},
  "protobuf": {:hex, :protobuf, "0.15.0", "c9fc1e9fc1682b05c601df536d5ff21877b55e2023e0466a3855cc1273b74dcb", [:mix], [{:jason, "~> 1.2", [hex: :jason, repo: "hexpm", optional: true]}], "hexpm", "5d7bb325319db1d668838d2691c31c7b793c34111aec87d5ee467a39dac6e051"},
  "ranch": {:hex, :ranch, "2.2.0", "25528f82bc8d7c6152c57666ca99ec716510fe0925cb188172f41ce93117b1b0", [:make, :rebar3], [], "hexpm", "fa0b99a1780c80218a4197a59ea8d3bdae32fbff7e88527d7d8a4787eff4f8e7"},
  "telemetry": {:hex, :telemetry, "1.3.0", "fedebbae410d715cf8e7062c96a1ef32ec22e764197f70cda73d82778d61e7a2", [:rebar3], [], "hexpm", "7015fc8919dbe63764f4b4b87a95b7c0996bd539e0d499be6ec9d7f3875b79e6"},
}

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions