feat: Splittable fastq reading for comet-bio #12
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…boundaries - Fix LimitedRangeFile to properly enforce byte range limits during reading - Add proper FASTQ record boundary detection for splits - Implement LimitedRangeFile wrapper that stops reading at end offset - Add FastqLocalReader::PlainRanged variant for byte range reading - Add comprehensive unit tests for byte range functionality - Fix missing benchmark_bgzf_threads.rs reference in bio-format-gff This resolves the issue where FASTQ readers were reading entire files instead of respecting byte ranges, causing poor parallelization performance. Tests verify: - Proper byte range enforcement - FASTQ record boundary handling - Various split scenarios (start, middle, end) - Integration with FastqLocalReader - Error handling for invalid ranges
The previous implementation only checked if a line starts with '@', which fails because FASTQ quality scores can contain '@' characters (ASCII 64 = Q31). This caused false positive matches and incorrect partition boundaries. Changes: - Implement sliding window approach that checks 4-line patterns - Validate complete FASTQ records: @Header, sequence, +separator, quality - Add strong validation: sequence length must equal quality length - Search up to 2000 bytes beyond partition end to find valid boundary - Skip initial partial line before starting pattern search This ensures all partitions find correct record boundaries, fixing the issue where some partitions would return 0 records due to false '@' matches in quality scores. 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude <noreply@anthropic.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
No description provided.