Skip to content

Cannot infer storage location from: Unknown scheme: s3. Known schemes: file,memory #330

@debajyoti-truefoundry

Description

@debajyoti-truefoundry
cargo install datafusion-dft     --git https://github.com/datafusion-contrib/datafusion-dft     --rev b1f1011d9b1070eec658d51992ab717709e355f0     --features "deltalake s3 functions-json functions-parquet"
    Updating git repository `https://github.com/datafusion-contrib/datafusion-dft`
     Ignored package `datafusion-dft v0.2.2 (https://github.com/datafusion-contrib/datafusion-dft?rev=b1f1011d9b1070eec658d51992ab717709e355f0#b1f1011d)` is already installed, use --force to override

I have set up S3 as an object store.

2025-06-03T12:02:55Z DEBUG datafusion_dft::config] Config exists
[2025-06-03T12:02:55Z DEBUG datafusion_dft::config] Parsed config: AppConfig { shared: ExecutionConfig { object_store: Some(ObjectStoreConfig { s3: Some([S3Config { bucket_name: "bucket", object_store_url: Some("s3://bucket"), aws_access_key_id: Some("redact"), aws_secret_access_key: Some("redact"), _aws_default_region: None, aws_endpoint: Some("https://s3.us-east-1.amazonaws.com"), aws_session_token: None, aws_allow_http: None }]) }), ddl_path: Some("/home/jovyan/.config/dft/ddl.sql"), benchmark_iterations: 10, datafusion: None, dedicated_executor_enabled: false, dedicated_executor_threads: 8, catalog: CatalogConfig { name: "dft" } }, cli: CliConfig { execution: ExecutionConfig { object_store: Some(ObjectStoreConfig { s3: Some([S3Config { bucket_name: "bucket", object_store_url: Some("s3://bucket"), aws_access_key_id: Some("redact"), aws_secret_access_key: Some("redact"), _aws_default_region: None, aws_endpoint: Some("https://s3.us-east-1.amazonaws.com"), aws_session_token: None, aws_allow_http: None }]) }), ddl_path: Some("/home/jovyan/.config/dft/ddl.sql"), benchmark_iterations: 10, datafusion: None, dedicated_executor_enabled: false, dedicated_executor_threads: 8, catalog: CatalogConfig { name: "dft" } } }, tui: TuiConfig { execution: ExecutionConfig { object_store: None, ddl_path: Some("/home/jovyan/.config/dft/ddl.sql"), benchmark_iterations: 10, datafusion: None, dedicated_executor_enabled: false, dedicated_executor_threads: 8, catalog: CatalogConfig { name: "dft" } }, display: DisplayConfig { frame_rate: 30.0 }, interaction: InteractionConfig { mouse: false, paste: false }, editor: EditorConfig { experimental_syntax_highlighting: false } }, db: DbConfig { path: Url { scheme: "file", cannot_be_a_base: false, username: "", password: None, host: None, port: None, path: "/home/jovyan/.local/share/dft/", query: None, fragment: None } } }
[2025-06-03T12:02:55Z INFO  datafusion_app::extensions::s3] S3 configs exists
[2025-06-03T12:02:55Z INFO  object_store::aws::builder] Using Static credential provider
[2025-06-03T12:02:55Z DEBUG datafusion_app::extensions::s3] created object store: AmazonS3(bucket)
[2025-06-03T12:02:55Z INFO  datafusion_app::extensions::s3] registered s3 object store at s3://bucket

But by DDL query is failing

[2025-06-03T12:02:55Z INFO  datafusion_app::local] Loading DDL from: Some("/home/jovyan/.config/dft/ddl.sql")
[2025-06-03T12:02:55Z DEBUG datafusion_app::local] Executing DDL statement: "CREATE EXTERNAL TABLE tbl \nSTORED AS DELTATABLE \nLOCATION 's3://bucket/delta_traces_3/otel_traces/'"
[2025-06-03T12:02:55Z ERROR datafusion_app::local] Error executing DDL statement: External error: Cannot infer storage location from: Unknown scheme: s3. Known schemes: file,memory

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions