Skip to content

Finish the transition to asyncio #122

@bpiwowar

Description

@bpiwowar

asyncio has been introduced to reduce the complexity of the scheduler – and to avoid many deadlocks. To finish the transitions, the following would be needed:

  1. Job dependencies should be handled directly through asyncio futures; a job basically loops over all its dependencies and wait for them to be finished (instead of relying on asyncio-based updates)
  2. (most important) Token dependencies should also be handled this way, since this would prevent current (rare) deadlocks

Roadmap:

-[x] SchedulerCentral should have a unique instance with a event loop that manages all the jobs – and we should merge SchedulerCentral into Scheduler (since they will mostly do the same thing). This will make the distinction between experiment and scheduler clearer: the experiment uses a unique scheduler;
-[ ] Use asyncio for the locking mechanism of tokens
-[ ] Create a TokenResourceLocker that handles the task of locking all the tokens at once
-[ ] Starting jobs should be directly done in Job

Job handling

Job launching sequence:

  1. (to adapt) Checking that the task is not already registered in a scheduler (otherwise, return the scheduled job);
  2. (to adapt) Check that the job is not already done or running
  3. Wait that all job dependencies are completed (simple loop over futures) – instead of monitor the jobs.
  4. Lock all the token dependencies (if one lock fails, just put this token first and start again)
  5. (to adapt) Create the directory and other resources needed to actually run the script.
  6. Run and monitor
  7. Resolve the future

Sub-issues

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

Status

Done

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions