feat: Add schema diff data structures and foundation (1/5) #1477
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Introduces core data structures for schema diffing: - SchemaDiff, FieldChange, FieldUpdate types - FieldChangeType enum for classifying changes - SchemaDiffError for validation errors - ColumnName::parent() helper method This is part 1/5 of the schema diffing feature implementation. The actual diffing algorithm will be added in PR 2. Note: This PR includes a temporary stub for compute_schema_diff() to allow basic tests to compile. The full implementation from the original PR delta-io#1346 will be copied exactly in PR 2. Related to delta-io#1346
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #1477 +/- ##
==========================================
+ Coverage 84.84% 84.89% +0.05%
==========================================
Files 120 121 +1
Lines 32103 32262 +159
Branches 32103 32262 +159
==========================================
+ Hits 27238 27390 +152
- Misses 3542 3547 +5
- Partials 1323 1325 +2 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
This was referenced Nov 14, 2025
murali-db
force-pushed
the
murali-db/schema-diff-pr1
branch
from
9df62ad to
4dd5d99
Compare
Adds a comprehensive unit test that exercises the filtering logic by manually constructing a SchemaDiff with both top-level and nested field changes. This test verifies that the methods correctly filter fields by path depth (length 1 vs length > 1). The test improves code coverage for these methods from 0% to full coverage of the filtering logic, addressing CI coverage requirements.
murali-db
force-pushed
the
murali-db/schema-diff-pr1
branch
from
4dd5d99 to
31cee42
Compare
OussamaSaoudi
approved these changes
Nov 15, 2025
Collaborator
zachschuermann
left a comment
zachschuermann
left a comment
There was a problem hiding this comment.
few comments but generally looks good!
| /// Feature gate for schema diff functionality. | ||
| /// Set to `false` to ensure incomplete implementations don't activate until all PRs are merged. | ||
| /// Will be removed in the final PR when all tests and implementation are complete. | ||
| const SCHEMA_DIFF_ENABLED: bool = false; |
Collaborator
There was a problem hiding this comment.
hm. we could just do a feature flag if we are really concerned about this? prefer feature flag over a constant I think
Comment on lines
+20
to
+34
| /// Arguments for computing a schema diff | ||
| #[derive(Debug, Clone)] | ||
| pub(crate) struct SchemaDiffArgs<'a> { | ||
| /// The before/original schema | ||
| pub before: &'a StructType, | ||
| /// The after/new schema to compare against | ||
| pub after: &'a StructType, | ||
| } | ||
|
|
||
| impl<'a> SchemaDiffArgs<'a> { | ||
| /// Compute the difference between the two schemas | ||
| pub(crate) fn compute_diff(self) -> Result<SchemaDiff, SchemaDiffError> { | ||
| compute_schema_diff(self.before, self.after) | ||
| } | ||
| } |
Collaborator
There was a problem hiding this comment.
thoughts on just leaving these as parameters? we could just have something like SchemaDiff::new(before, after)?
| /// Fields that were modified between schemas | ||
| pub updated_fields: Vec<FieldUpdate>, | ||
| /// Whether the diff contains breaking changes (computed once during construction) | ||
| has_breaking_changes: bool, |
Collaborator
There was a problem hiding this comment.
nit: maybe more concise?
Suggested change
| has_breaking_changes: bool, | |
| breaking_changes: bool, |
| #[derive(Debug, Clone)] | ||
| pub(crate) struct SchemaDiffArgs<'a> { | ||
| /// The before/original schema | ||
| pub before: &'a StructType, |
Collaborator
There was a problem hiding this comment.
not sure how others feel but generally for pub(crate) fields i like the clarity of making each one pub(crate) instead of pub (which is the same as pub(crate) as long as the struct itself is pub(crate)`)
Comment on lines
+192
to
+213
| // Feature gate check - prevents activation of incomplete implementation in production | ||
| // This gate will be removed in the final PR when all functionality is complete | ||
| // Note: Gate is bypassed in test builds to allow tests to validate the implementation | ||
| #[cfg(not(test))] | ||
| { | ||
| if !SCHEMA_DIFF_ENABLED { | ||
| return Ok(SchemaDiff { | ||
| added_fields: Vec::new(), | ||
| removed_fields: Vec::new(), | ||
| updated_fields: Vec::new(), | ||
| has_breaking_changes: false, | ||
| }); | ||
| } | ||
| } | ||
|
|
||
| // Stub implementation - actual implementation will be added in PR 2 | ||
| Ok(SchemaDiff { | ||
| added_fields: Vec::new(), | ||
| removed_fields: Vec::new(), | ||
| updated_fields: Vec::new(), | ||
| has_breaking_changes: false, | ||
| }) |
Collaborator
There was a problem hiding this comment.
I think we can either make this todo!() or just always return Err::unsupported
| } | ||
|
|
||
| /// Get all changes at the top level only (fields with path length of 1) | ||
| pub(crate) fn top_level_changes( |
Collaborator
There was a problem hiding this comment.
will we not want something that does both top-level and nested?
Closed
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is part 1 of 5 PRs that implement schema diffing for Delta Kernel Rust.
What's in this PR
SchemaDiff,FieldChange,FieldUpdateFieldChangeTypeenum for classifying changesSchemaDiffErrorfor validation errorsColumnName::parent()helper methodWhat's NOT in this PR
Feature Gating
This PR includes
const SCHEMA_DIFF_ENABLED: bool = falsewith a#[cfg(not(test))]guard.Part of #1346