Backport to branch(3) : Refactor Data Loader Import to Use DistributedTransactionManager for Storage Mode

[feeblefakie]

This is an automated backport of the following:

• Original PR: Refactor Data Loader Import to Use DistributedTransactionManager for Storage Mode #3147
• Backported commit: ccbd39f

Please merge this PR after all checks have passed.

[gemini-code-assist]

Summary of Changes

Hello @feeblefakie, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This backport refactors the ScalarDB Data Loader to standardize its interaction with ScalarDB. For import operations, it unifies the data access mechanism by exclusively utilizing DistributedTransactionManager, thereby simplifying the codebase and removing the explicit distinction between TRANSACTION and STORAGE modes at the DAO level. Concurrently, the --include-metadata command-line option for data export has been deprecated and is now ignored, with a warning provided to users; its functionality is now controlled solely by a configuration property. These changes aim to improve consistency and maintainability of the Data Loader.

Highlights

• Unified Data Access for Import: The Data Loader's import functionality has been refactored to consistently use DistributedTransactionManager for all data operations, regardless of whether ScalarDB is configured in TRANSACTION or STORAGE mode. This simplifies the internal data access layer by removing the need for separate DistributedStorage handling.
• Deprecation of Export --include-metadata Option: The command-line option --include-metadata (and its short form -m) for the ExportCommand is now deprecated and ignored. Its functionality is replaced by the scalar.db.consensus_commit.include_metadata.enabled configuration property in the ScalarDB properties file.
• User Warning for Deprecated Option: A warning message is now displayed to users who attempt to use the deprecated --include-metadata option, informing them of its removal and directing them to the correct configuration property.
• Removal of Dedicated Storage/Transaction Managers: The ScalarDbStorageManager and ScalarDbTransactionManager utility classes have been removed, streamlining the management of ScalarDB instances within the Data Loader.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request is a backport that refactors the Data Loader to consistently use DistributedTransactionManager for both transaction and storage modes, removing the dependency on DistributedStorage. This is a significant improvement that simplifies the codebase by unifying the data access layer. The changes are well-implemented, removing redundant classes like ScalarDbStorageManager and ScalarDbTransactionManager and simplifying the logic in ImportCommand and ImportManager.

Additionally, the --include-metadata command-line option for data export is correctly deprecated. A warning is now shown to users, guiding them to use the corresponding ScalarDB configuration property instead. The related logic for handling metadata columns has been removed from the data loader, delegating this responsibility to the ScalarDB core, which is a good design choice.

The tests have been updated thoroughly to reflect these changes, including new tests for the deprecated option and updated assertions for expected output. Overall, the changes are clean, consistent, and improve the maintainability of the code. I have reviewed the pull request and found no issues.