Support Vortex as a file format

[AdamGS]

Vortex is a new open-source columnar file format, it already shows significant performance benefits while still being relatively young. Would you consider supporting it in dft?

We provide a FileSource-based interface that's already seeing some usage, which would basically make it a drop-in replacement for Parquet (datafusion wise) and would love to also provide users with a datafusion-terminal based interface (we currently have to point them at DuckDB for that).

I would obviously do all the work of doing the initial PR and maintaining it, which hopefully wouldn't be too hard as DataFusion is becoming more stable with every release.

[matthewmturner]

Sorry for the delay getting back to you on this @AdamGS. We actually have been trying to use Vortex where I work (I work with @xudong963 who has raised some issues in the Vortex repo) and we are still hopeful that we'll be able to use it.

I am definitely supportive of getting Vortex support added to dft. I just need to do some catching up (mainly upgrading to latest df versions). I will take care of that and then will follow up here so we can coordinate the actual work needed to integrate Vortex.

One question I have that comes to mind - I frequently use a parquet_metadata function for debugging parquet files. Is there an existing ability for doing something similar with Vortex that we could use for creating a vortex_metadata UDTF?

[AdamGS]

We don't have that sort of UDF, but mostly because no one asked for it! I'll put something together, seems very useful to have as part of the DataFusion integration, we've been thinking about multiple other specialized things we can include there, so it fits.

[AdamGS]

We also release a tui tool that makes debugging files pretty nice, but isn't as scalable if you're trying to look at many files at once.

[matthewmturner]

We don't have that sort of UDF, but mostly because no one asked for it! I'll put something together, seems very useful to have as part of the DataFusion integration, we've been thinking about multiple other specialized things we can include there, so it fits.

Okay great! I think that would be very helpful on our side as we compare to Parquet.

Also, I had started to explore alternative ways to show execution plan statistics that was more focused on providing an actionable summary (as opposed to having to find the relevant details from a tree of stats). Maybe you would be interested in having those specialized statistics show in something like that (forewarning, that feature has not had much love since the initial work so it may be rough around the edges).

I've updated to DF50, which is the version i saw vortex-datafusion was on. I'll start looking into how we can integrate it now. Presumably it will be adding a vortex feature to the datafusion-app crate. Let me know if you would like to drive that or want me to work on it :)

[matthewmturner]

Was thinking on this - right now there is a generate-tpch command that creates parquet files for the tpch dataset. Would be cool if we allowed that to optionally be written as vortex.

[AdamGS]

I would have to take a look at everything to figure what how to integrate it, if you want to get started go ahead I'm spread a bit thin at the moment, but glad to help with everything or just get around it it once I clear my plate a bit.

[matthewmturner]

Got an initial PR in. Maybe I did something stupid but out of the box the perf was much worse than parquet on a SELECT * query. Anything stick out to you?

[matthewmturner]

Also didnt seem to do as well on query with filter - but its much closer in this case. Sorry if im holding it wrong! This is the first time im actually using it.

[AdamGS]

The code seems very straight forward, I don't think there's anything wrong there. If there's an issue here its on the Vortex side. I would have to take a deeper look, this seems like a query Vortex should be pretty good at as compressing and operating over something like a year is usually very effective.

[AdamGS]

Its been a few weeks since our last release, I hope we can get one out today/early next week and benchmark again. Is this data that you're able to share? Mostly curious about its overall shape.

[matthewmturner]

Let me find out if we can share, I think we have different rights depending on the dataset. If not the one from the screenshot i should be able to find one i can share though.

[matthewmturner]

Also FYI it looks like writing was slower

[matthewmturner]

On the trades dataset vortex did much better than parquet

[matthewmturner]

Ah but only when theres a ticker filter, full scan it was back to slower