Shoryuken options

Shoryuken options can be set via CLI:

shoryuken -q queue1 -l ./shoryuken.log -p ./shoryuken.pid

or via configuration file:

# shoryuken.yml
logfile: ./shoryuken.log
pidfile: ./shoryuken.pid
queues: 
  - queue1

When using a configuration file, you must set via CLI shoryuken -C ./shoryuken.yml, otherwise Shoryuken won't use it.

Some options available in the configuration file, are not available in the CLI. For checking all options available in the CLI: shoryuken help start.

delay

Delay is the number of seconds to pause fetching from when an empty queue.

Given this configuration:

delay: 25
queues:
  - queue1
  - queue2

If Shoryuken tries to fetch messages from queue1 and it has no messages, Shoryuken will pause fetching from queue1 for 25 seconds.

Usually having a delay is more cost efficient, but if you want to consume messages as soon as they get in the queue, I would recommend setting delay: 0 (default value) to stop pausing empty queues.

Check the AWS SQS pricing page for more information.

queues

A single Shoryuken process can consume messages from multiple queues. You can define your queues as follows:

queues:
  - queue1
  - queue2

Load balancing

Supposing you a queue1, which you would like to fetch messages twice as much as queue2, you can configure that as follows:

queues:
  - [queue1, 8]
  - [queue2, 4]
  - [queue3, 1]

The setup above will cause Shoryuken to fetch messages in cycles of queue1 8 times, then queue2 4 times, then queue3 once, then repeat.

Note: Each fetch can fetch up to 10 messages at time (SQS limitation). The fetch size will also depend on the available workers at the time of the fetch.

Note: The queue priorities starts always at 1, and increases one-by-one until it reaches the max configured for every successful fetch returning messages. Similar happens when a queue reaches its maximum then gets empty, Shoryuken will decrease its priority one-by-one until starts to receive messages.