feat(cubestore): Upgrade to DataFusion 46.0.1#10215

Open

waralexrom wants to merge 153 commits intomasterfrom

cubestore-df-46.0.1-rebased

Commits on Dec 9, 2025

feat(cubestore): Upgrade to DF 42.2.0

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix couple decimal tests

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix info schema table providers

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix ordering issues

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix create table with location tests

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix filter pushdown to CubeTable

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix partial aggregate not pushed under ClusterSend

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix join requirement extraction and PlanProperties for ClusterSend

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix nested_union_empty_tables test

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix limit pushdown

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix limit pushdown for LastRowByKey

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix divide by zero error message

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: upgrade HllCardinality ScalarUDF implementation

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix HLLMergeUDF implementation

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix aggregate index hll tests

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: apply some hll aggregate index fixes to other aggregation types

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Use lowercase names for UDAF registry

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Implement DATE_BIN with MonthDayNano support

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Implement DATE_ADD and DATE_SUB by invoking DF arithmetic operator behavior

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Remove commented now() UDF and MaterializeNow rewrite

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: remove coalesce UDF, make coalesce test fixes, handle DataType::Null in batches_to_dataframe

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix UDF style and organization, lowercase names

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pass physical predicate for Parquet row group pruning

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: register unix_timestamp as a ScalarUDF

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pass physical predicate to second ParquetExecBuilder

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix usage of MetadataCacheFactory and CubestoreParquetMetadataCache

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix various problems with compaction.

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix compaction merge_chunks in unique_key_columns case

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Revert to capitalized table aliases in HLL tests

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Revert lowercasing in InlineTable::New, fix tests

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Implement PanicWorkerNode

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Reenable three_tables_join tests

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Split SerializedPlan type into PreSerializedPlan

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Reimplement and use PreSerializedPlan::remove_unused_tables

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fully implement remove_unused_tables

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: post_process_columns aggregate index maintaining sort order

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make ilike test expect different, correct SQL string escaping behavior

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Update datafusion dependency pointer

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Rewrite InList expression type conversion when list is literals

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Implement convert_tz

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix cast_timestamp_to_utf8 test

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Factor out QueryPlannerImpl::make_execution_context

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix bugs in partition_filter::Builder::extract_filter

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Keep necessary EnforceSorting optimizer rule

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Update datafusion repo pointer

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make Int96 and Decimal96 reading and migration working

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix decimal_partition_pruning test

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix table::parquet::tests::column_statistics test

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Kafka-related fixes

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix create_table_test and create_table_test_seal_at

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: fix streaming_projection_kafka_timestamp_ops

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Avoid FinalPartitioned when pushing aggregate to workers

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: use correct input ordering trait impls on ClusterSendExec and WorkerExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: backport rolling window implementation and allow multiple ClusterSend nodes within plan to support multi-stage aggregations

paveltiunov
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix planning tests, improve pretty-printing, CoalescePartitionsExec output hints

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Construct SessionConfig, update datafusion interfaces

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Parse SQL in Cubestore with string literal backslash escapes

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Revert "Make ilike test expect different, correct SQL string escaping behavior"

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Implement top-k aggregation, still with slow Accumulators

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Use max_batch_rows on Worker

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Bugfix from topk: Correct compute_properties in WorkerExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Treat unquoted schema/table names case sensitively as before

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Disable ident normalization, remove lowercase normalization in some lookups

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Split topk logical node into two parts, avoiding need for DF type_coercion changes

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: pretty_printer adjustments: show_partitions, show_schema

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Test MergeSort node present when ClusterSend has multiple partitions with sorted aggregate

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Update Arrow with Decimal64 backwards compatibility fix

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Partitioned index support

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix suboptimal query plan detection

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pass tracing spans through spawned tasks in ExecutionPlan execution

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Avoid needless Arc in DataFrame

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Put tracing instrumentation back into datafusion

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Reduce the amount of redundant planning and optimization

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add distribution and input order requirement to LastRowByUniqueKeyExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make columns_vec_buffer_size use min_credited_buffer_size

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Use DF 46.0.1

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix ilike failure

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix rolling window optimization

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Retain sort information with Projection pushdown through MemorySourceConfig

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Let DATE_ADD and DATE_SUB tolerate time zones

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make TopicTableProvider udf_names implementation return three names

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make default UDFs working in Kafka streaming

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make DataFrame rendering use correct decimal scale

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Address low-hanging warnings

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make Kafka plan error messages display plan

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Tolerate SubqueryAlias in plans for kafka streaming

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix intermittent failures with streaming_filter_kafka and streaming_filter_kafka_concat tests

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pass customizer more completely and avoid ParquetExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add `XIRR` aggregate function to Cube Store

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pretty-printing improvements

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix unnested union deserialization and use string->number comparison coercion

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make DF optimization propagate_empty_relation handle unions properly

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make remove_unused_tables handle Union output case correctly

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make kafka streaming tolerate compatible projection plans with non-nullable fields in the schema

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix concat_batches schema in KafkaPostProcessPlan::apply

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add migration testing form of cubesql test cases

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make query executor show metrics when logging slow queries

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Have SessionContext from kafka using metadata_cache_factory

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make HLLUnionDataSketch::allocated_size compute a reasonable value

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make DataFusionError -> CubeError pass through panic case

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Remove the unused FlattenUnion

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fixes or cleanup for udf, udaf, udwf lookups

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Return finer EmissionType values

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Avoid or mark allowed deprecated MemoryExec and ParquetExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Avoid deprecated NaiveDateTime::from_timestamp_opt, from_ymd, and_hms, DateTime::from_utc

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: PPOptions::show_most and pretty printing cleanup

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Small code dedup of router_context and worker_context

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Regroup batches in compaction to avoid i32 offset overflow

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Return correct EmissionType values in Cube ExecutionPlans

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: In compaction, limit batch size to 2048

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Limit compact() row group size to 2048, and log over-large columns

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix panics involving TableValue::Decimal96

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix bug in RollingWindowAggregate partial_cmp implementation

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Remove, silence, or fix unused imports, variables, deprecation warnings

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make string/boolean comparisons coerce to boolean

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add planning metrics

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Fix pretty-printing of physical plan Scan predicates

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: finer grained app_metrics in planning

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Write PARE magic and add datafusion tracing features

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: More fine-grained planning metrics

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make test_queue_item_sort pass consistently

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add CUBESTORE_DATAFUSION_LOG_LEVEL and CUBESTORE_GLOBAL_LOG_LEVEL

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Set compaction row group size back to 16384

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Revert pass-by-pass logical plan optimization tracing

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add compaction operation app_metrics

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make tokio max_blocking_threads configurable from the env

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add choose_index_ext metrics

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: logical plan optimization for Limit>Sort push down in one pass

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: eliminate_nested_union and optimize_projections performance improvements

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Add data frame serialization time metrics

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Disable CoalesceBatches

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Disable parallelization of SortPreservingMergeExec inputs

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Remove redundant RewriteInListLiterals analysis pass

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Deduplicate and simplify SessionContext creation

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make add_limit_to_workers behave the same on router nodes as worker nodes

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Make ChooseIndex PlanWriter handle skips

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Push down Sort,fetch at logical plan phase

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Log query that produces error processing HTTP command

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Pretty-print sort column improvements for MergeSort and TopK Aggregates

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: push down GlobalLimit after adding it to workers

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: replace suboptimal SortPreservingMergeExec nodes underneath SortExec

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Backport DF 48 patch for boxed WindowFunction and smaller Expr struct

srh
authored and
waralexrom
committed
chore(cubestore): Upgrade DF: Use split_row_group_reads with consistent 4096 batch size

srh
authored and
waralexrom
committed
fix(cubestore): Allowing window functions in cluster send planning
waralexrom
committed
feat(cubestore): Inline aggregate
waralexrom
committed
chore(cubestore): New cubestore rebased to master
waralexrom
committed
fix
waralexrom
committed
fix
waralexrom
committed
fix
waralexrom
committed
win compilation fix
waralexrom
committed
change env
waralexrom
committed
unit tests fix
waralexrom
committed
cubestore artifact name change
waralexrom
committed