Worker is trying to rescue jobs from queues that are not assigned to it

[NanoBjorn]

A bit of context

We have two kinds of workers at the moment, sharing the same database, let's say worker-1 is running jobs from queue-1 and worker-2 is running jobs from queue-2. However, we are using the same binary for the worker and we recently tried registering jobs conditionally, basically registering only those jobs that are supposed to be executed by this worker based on queue selection.

Another thing to mention is that our RescueStuckJobsAfter is configured to be pretty small (30s) whereas timeouts on the jobs are minutes.

An issue

After introducing this conditional registration, we observed the following behaviour:

• worker-1 is fetching all jobs (for both queues) to rescue:

  river/riverdriver/riverpgxv5/internal/dbsqlc/river_job.sql

  Lines 257 to 263 in eb0b985

  	-- name: JobGetStuck :many
  	SELECT *
  	FROM /* TEMPLATE: schema */river_job
  	WHERE state = 'running'
  	AND attempted_at < @stuck_horizon::timestamptz
  	ORDER BY id
  	LIMIT @max;

• amongst others, it can fetch jobs dedicated for queue-2 and they are not registered in the worker-1, so makeRetryDecision discards the job:

  river/internal/maintenance/job_rescuer.go

  Lines 302 to 307 in eb0b985

  	workUnitFactory := s.Config.WorkUnitFactoryFunc(job.Kind)
  	if workUnitFactory == nil {
  	s.Logger.ErrorContext(ctx, s.Name+": Attempted to rescue unhandled job kind, discarding",
  	slog.String("job_kind", job.Kind), slog.Int64("job_id", job.ID))
  	return jobRetryDecisionDiscard, time.Time{}
  	}

Checked documentation if we are missing something, and both pages that could have mention any details of this behaviour are not saying anything about our weird case:

• Inserting and working jobs does not mention if all jobs from a single database have to be registered in all workers
• Multiple queues only says but workers will only select jobs to work for queues that they're configured to handle, but still it is unclear if we should register all jobs or not

A very bad side effect of this for us was that discarded job continued execution (context wasn't cancelled) and another job with the same unique_key was scheduled, violating unique constraint that we are relying on for correctness, but I am not blaming this on river, this is a consequence of our ignorance of rescue mechanics nuances.

---

Short-term we fixed our issue by again registering all jobs in the worker binary regardless its configuration, but it would be valuable to have an answer if this is a designed behaviour or a bug?

[brandur]

Yeah, for better or worse, this is currently implemented as originally designed.

I've thought about modifying the various maintenance services so they only work on jobs that their associated client knows about, thereby making isolation possible between multiple clients trying to make separate use of the same schema. I think this isn't a bad idea, but it does have other downsides. For example, jobs and/or queues which have become "orphaned" (i.e. they were in use previously, but no client is configured with them anymore) will not get any attention from any clients. It's also a bit of a non-trivial change because we'd have to account for multiple leadership mechanisms that can be configured independently.

For now, a quick fix for what you're trying to do is to switch to using multiple schemas.

I'll flag this one for discussion with @bgentry in case there's anything else we could potentially be doing here.

[NanoBjorn]

"orphaned" will not get any attention from any clients

Yeah, that makes sense, I was also thinking about it.

Then I think action items on this one could be:

1. Mention this in both documentation pages on registering jobs and multiple queues
2. Cancel context by the job itself if the job was marked as discarded in the db (I am not sure though if there is any active polling from the side of running job now)