feat: Streaming VCF reader (#2)

* WIP: VCF

* Refactor

* Parsing info draft

* Working parser

* Refactoring infos

* async-trait downgrade

* Reverting to 0-based

* Renamed columns

* Add retry go operator

* Add IOTimeout

* Fixing streams

* Adding s3

* fix: Basic fields

* Adding support for remote reading of uncompressed VCFs

* Fix header

* Optimize variant_end

* Enabling projection

* Fixing local vcf without compression and gcs reads optimization

* Fixing local vcf reading with no compression

* Describe VCF

* fix: Tag case sensitive

* add performance/time measurement for batch processing vcf with noodles

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* add retry mechanism and adjust chunk size along with minimal concurrent fetches

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* Refactor scan to separate projected schema computation and use Field::nullable flag

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* propagate builders errors

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* Optimize OptionalField::new() to use with_capacity

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* Improve info_to_arrow_type logic

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* refactor format fields and cleanup code

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* complete bgzf compressed files format ingestion tests

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* add bgzf test in similar format to test_noodles.rs

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* add docker-compose for testing iceberg

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* add simple github workflows CI

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>

* Cleanup a few warnings

* Bump runner image

---------

Signed-off-by: Piotr Dębski <ppdebski@interia.eu>
Co-authored-by: Piotr Dębski <ppdebski@interia.eu>

Author: mwiewior

.github/workflows/ci.yml

@@ -0,0 +1,44 @@
+name: CI
+
+on:
+  push:
+    branches:
+      - main
+  pull_request:
+
+jobs:
+  build-test:
+    runs-on: ubuntu-22.04
+    concurrency:
+      group: ${{ github.workflow }}-${{ github.ref }}
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+        with:
+          submodules: "recursive"
+          fetch-depth: 1
+
+      - name: Setup Rust
+        uses: actions-rust-lang/setup-rust-toolchain@v1
+        with:
+          toolchain: '1.85.0'
+          components: 'clippy, rustfmt'
+
+      - name: Cache Cargo registry and build
+        uses: actions/cache@v3
+        with:
+          path: |
+            ~/.cargo/registry
+            ~/.cargo/git
+            datafusion/vcf/target
+          key: ${{ runner.os }}-cargo-${{ hashFiles('datafusion/vcf/Cargo.lock') }}
+          restore-keys: |
+            ${{ runner.os }}-cargo-
+
+      - name: Check formatting
+        working-directory: datafusion/vcf
+        run: cargo fmt --all -- --check
+
+      - name: Run tests
+        working-directory: datafusion/vcf
+        run: cargo test

.gitignore

@@ -18,4 +18,4 @@ Cargo.lock
 #  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore
 #  and can be added to the global gitignore or merged into this file.  For a more nuclear
 #  option (not recommended) you can uncomment the following to ignore the entire idea folder.
-#.idea/
+.idea/

.pre-commit-config.yaml

@@ -0,0 +1,8 @@
+repos:
+  - repo: https://github.com/doublify/pre-commit-rust
+    rev: v1.0
+    hooks:
+      - id: fmt
+        args: ["--all", "--"]
+      - id: cargo-check
+