Add automated S3 workset monitor #161
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting
| LOGGER.info("Validating stage samples manifest %s", samples_path) | ||
| with samples_path.open("r", encoding="utf-8") as handle: | ||
| reader = csv.DictReader(handle, delimiter="\t") | ||
| for row in reader: | ||
| for value in row.values(): | ||
| if not value: | ||
| continue | ||
| if value.startswith("s3://"): | ||
| self._assert_s3_object_exists(value) | ||
| else: | ||
| candidate = samples_path.parent / value | ||
| if not candidate.exists(): | ||
| raise MonitorError(f"Referenced file does not exist: {candidate}") |
There was a problem hiding this comment.
The new _validate_stage_samples helper treats every nonempty TSV field as a file path and requires it either to exist locally or in S3. Real workset manifests typically contain many metadata columns (sample IDs, library names, read group labels, etc.) whose values are not paths, so this logic raises MonitorError for legitimate manifests by looking for files like …/S001 or …/Tumor. The monitor will therefore fail before staging for most worksets. Validation should be limited to columns known to contain paths instead of applying it to all values.
Useful? React with 👍 / 👎.
Summary
bin/daylily-monitor-worksetsCLI and document configuration/usage indocs/workset-monitor.mdTesting
https://chatgpt.com/codex/tasks/task_e_68f0bb25aab88331b3027151f0f5c0fc