Skip to content

Conversation

@ypeckstadt
Copy link
Contributor

@ypeckstadt ypeckstadt commented Nov 18, 2025

Description

This PR refactors the data loader architecture to decouple the usage of mode in the core module from the CLI-specific mode that is still being used but does not really match what is happening in the core code.

The core module now uses a new TransactionMode enum with more descriptive terminology that accurately reflects the underlying transaction behavior (SINGLE_CRUD and CONSENSUS_COMMIT), while the CLI maintains backward compatibility with the existing ScalarDbMode enum, for now. This enum will be completely removed once the ScalarDB Cluster config is exposed to the client as well.

This PR is not high priority and is ok if cant be included with 3.17 release.

Related issues and/or PRs

NA

Changes made

Core Module (data-loader/core)

  • Added new TransactionMode enum with two values:
    • SINGLE_CRUD: Replaces STORAGE mode
    • CONSENSUS_COMMIT: Replaces TRANSACTION mode
  • Updated all core classes to use TransactionMode instead of ScalarDbMode
  • Updated all test files:
  • Removed ScalarDbMode.java from the core module
  • Renamed the method names in ImportProcessor to use clear terminogly

CLI Module (data-loader/cli)

  • Retained ScalarDbMode enum for backward compatibility with existing CLI users
  • Marked as deprecated ScalarDbMode enum and all its values:
  • Added convertToTransactionMode() method in ImportCommand to translate from ScalarDbMode to transaction mode

Checklist

The following is a best-effort checklist. If any items in this checklist are not applicable to this PR or are dependent on other, unmerged PRs, please still mark the checkboxes after you have read and understood each item.

  • I have commented my code, particularly in hard-to-understand areas.
  • I have updated the documentation to reflect the changes.
  • I have considered whether similar issues could occur in other products, components, or modules if this PR is for bug fixes.
  • Any remaining open issues linked to this PR are documented and up-to-date (Jira, GitHub, etc.).
  • Tests (unit, integration, etc.) have been added for the changes.
  • My changes generate no new warnings.
  • Any dependent changes in other PRs have been merged and published.

Additional notes (optional)

NA

Release notes

Replaced ScalarDbMode with core TransactionMode for clarity

Copilot AI review requested due to automatic review settings November 18, 2025 03:29
@gemini-code-assist
Copy link
Contributor

Summary of Changes

Hello @ypeckstadt, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly refactors the data loader architecture by decoupling its core logic from CLI-specific transaction mode definitions. It introduces a new, more descriptive TransactionMode enum in the core module, which clarifies the underlying transaction behaviors as SINGLE_CRUD and CONSENSUS_COMMIT. While the CLI maintains backward compatibility with the older ScalarDbMode (now deprecated), the internal implementation uniformly uses the new TransactionMode and DistributedTransactionManager for all data operations, simplifying the codebase and preparing for future removal of the legacy enum. Additionally, a deprecated --include-metadata option in the export command has been removed, further streamlining the data export process.

Highlights

  • Core Module Refactoring: Introduced a new TransactionMode enum (SINGLE_CRUD, CONSENSUS_COMMIT) in the core module to replace the CLI-specific ScalarDbMode, enhancing clarity and decoupling. All core classes have been updated to use this new enum.
  • CLI Backward Compatibility: The ScalarDbMode enum is retained in the CLI module, marked as @Deprecated, to ensure backward compatibility for existing users. An internal conversion method (convertToTransactionMode()) has been added to translate ScalarDbMode values to the new TransactionMode.
  • Simplified Export Logic: The --include-metadata option for data export has been deprecated and its related logic removed from the core ExportManager and ProducerTask classes, streamlining export functionality and reducing complexity.
  • Unified Data Access: The ScalarDbDao and related import/export managers now consistently use DistributedTransactionManager for both transactional and single CRUD operations. This change removes the separate DistributedStorage interface and its manager classes (ScalarDbStorageManager, ScalarDbTransactionManager), leading to a more unified data access layer.
  • Improved Method Terminology: Method names within ImportProcessor have been updated to align with the new TransactionMode terminology (e.g., processStorageRecord was renamed to processSingleCrudRecord), improving code readability and consistency.
Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature Command Description
Code Review /gemini review Performs a code review for the current pull request in its current state.
Pull Request Summary /gemini summary Provides a summary of the current pull request in its current state.
Comment @gemini-code-assist Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help /gemini help Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

  1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution.

@ypeckstadt ypeckstadt changed the base branch from master to feat/data-loader/import-replace-storage November 18, 2025 03:30
@ypeckstadt ypeckstadt self-assigned this Nov 18, 2025
Copy link
Contributor

@gemini-code-assist gemini-code-assist bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request is a significant refactoring that improves the clarity and architecture of the data loader. By replacing the CLI-specific ScalarDbMode with a more descriptive core TransactionMode enum, you've successfully decoupled the core module and made the transaction behavior more explicit. The simplification of using DistributedTransactionManager for all database interactions, including single CRUD operations, is a great improvement that streamlines the data access layer. The changes are consistent across the codebase, including updates to tests and deprecation of old options. I have a few suggestions to further enhance the clarity of the new terminology in Javadocs and class names.

Copilot finished reviewing on behalf of ypeckstadt November 18, 2025 03:33
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR refactors the data loader architecture by replacing the ScalarDbMode enum with a new TransactionMode enum in the core module, providing more descriptive terminology that accurately reflects the underlying transaction behavior. The CLI module maintains the deprecated ScalarDbMode for backward compatibility with existing users.

  • Introduced TransactionMode enum with SINGLE_CRUD and CONSENSUS_COMMIT values
  • Updated all core classes and tests to use the new TransactionMode
  • Moved and deprecated ScalarDbMode to the CLI package
  • Renamed methods for consistency (e.g., processStorageRecordprocessSingleCrudRecord)

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated no comments.

Show a summary per file
File Description
data-loader/core/src/main/java/com/scalar/db/dataloader/core/TransactionMode.java New enum defining transaction modes with clear documentation for SINGLE_CRUD and CONSENSUS_COMMIT
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/processor/ImportProcessorParams.java Updated to use TransactionMode instead of ScalarDbMode with improved field documentation
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/processor/ImportProcessor.java Method names updated for clarity and documentation revised to use new terminology
data-loader/core/src/main/java/com/scalar/db/dataloader/core/dataimport/ImportManager.java Updated field and parameter to use TransactionMode
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataimport/processor/JsonLinesImportProcessorTest.java Test methods renamed and updated to use TransactionMode
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataimport/processor/JsonImportProcessorTest.java Test methods renamed and updated to use TransactionMode
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataimport/processor/ImportProcessorTest.java Test methods renamed and updated to use TransactionMode with improved naming conventions
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataimport/processor/CsvImportProcessorTest.java Test methods renamed and updated to use TransactionMode
data-loader/core/src/test/java/com/scalar/db/dataloader/core/dataimport/ImportManagerTest.java Test updated to use TransactionMode
data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/ScalarDbMode.java Moved from core to cli package and marked as deprecated with proper annotations
data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/command/dataimport/ImportCommand.java Added conversion method from ScalarDbMode to TransactionMode for backward compatibility
data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/command/dataimport/ImportCommandOptions.java Updated import path to reference cli package ScalarDbMode
Comments suppressed due to low confidence (1)

data-loader/cli/src/main/java/com/scalar/db/dataloader/cli/ScalarDbMode.java:27

  • Missing period after "3.17.0" in the deprecation notice. Should be "As of release 3.17.0." to be consistent with line 18 and line 7.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

@ypeckstadt ypeckstadt requested review from a team, Torch3333, brfrn169, feeblefakie, komamitsu and thongdk8 and removed request for a team November 19, 2025 02:02
Copy link
Collaborator

@brfrn169 brfrn169 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

Base automatically changed from feat/data-loader/import-replace-storage to master November 19, 2025 05:48
Copy link
Contributor

@feeblefakie feeblefakie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

Copy link
Contributor

@thongdk8 thongdk8 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Thank you!

Copy link
Contributor

@Torch3333 Torch3333 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you!

@feeblefakie
Copy link
Contributor

@komamitsu Can you take a quick look?

Copy link
Contributor

@komamitsu komamitsu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 👍

@brfrn169 brfrn169 merged commit 38b2009 into master Nov 21, 2025
139 of 140 checks passed
@brfrn169 brfrn169 deleted the ref/data-loader-mode branch November 21, 2025 04:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants