Add comprehensive documentation for Database Swap, including guides on getting started, configuration, CLI commands, architecture, and troubleshooting.

Author: zaber-dev

Learn.md

@@ -0,0 +1,89 @@
+# Learn Database Swap
+
+A guided path to learn and use Database Swap effectively. This page links to focused docs so you can ramp up quickly and deepen as needed.
+
+- What it is: A Python tool to migrate data between SQLite, MySQL, and MongoDB with validation, rate limiting, and progress tracking.
+- Who it’s for: Developers and data engineers moving datasets across database engines reliably.
+
+## Start here
+
+1) Getting started (install + first migration)
+- Read: docs/getting-started.md
+
+2) Understand the configuration
+- Read: docs/configuration.md
+
+3) Use the CLI efficiently
+- Read: docs/cli.md
+
+4) Know your database adapters
+- Read: docs/adapters.md
+
+5) How it works under the hood
+- Read: docs/architecture.md
+
+6) Extend to new databases
+- Read: docs/extending.md
+
+7) Troubleshoot common issues
+- Read: docs/troubleshooting.md
+
+8) Recipes and examples
+- Read: docs/recipes.md
+
+9) FAQ
+- Read: docs/faq.md
+
+## Quick cheat sheet
+
+Installation (from source)
+
+```powershell
+pip install -r requirements.txt
+pip install -e .
+```
+
+Available CLI commands
+
+```powershell
+# Initialize a config file in the current folder
+database-swap init-config -o config.yaml
+
+# Test a connection
+database-swap test-connection --db-type sqlite --database .\source.db
+
+# Analyze a database (structure + counts)
+database-swap analyze --db-type sqlite --database .\source.db
+
+# Migrate from SQLite to MySQL (dry run first)
+database-swap migrate --dry-run `
+  --source-type sqlite --source-database .\source.db `
+  --target-type mysql --target-host localhost --target-database target_db --target-username root
+
+# Run the actual migration
+database-swap migrate `
+  --source-type sqlite --source-database .\source.db `
+  --target-type mysql --target-host localhost --target-database target_db --target-username root
+```
+
+Configuration keys you’ll use most
+
+- source.* and target.*: database types and connection info
+- migration.batch_size: records per write
+- migration.rate_limit_delay: seconds to wait between batches
+- migration.tables: subset of tables to migrate
+- validation.strict_mode: fail fast on invalid data
+
+See docs/configuration.md for a full reference.
+
+## Examples in repo
+
+- examples/sqlite-to-sqlite.yaml
+- examples/mysql-to-mongodb.yaml
+
+These are great starting points for your own config files.
+
+## Contributing
+
+- Open issues and PRs are welcome.
+- See docs/extending.md to add new database adapters.

docs/adapters.md

@@ -0,0 +1,40 @@
+# Adapters
+
+Database adapters implement a common interface (`DatabaseAdapter`) to interact with each database engine.
+
+Built-in adapters
+
+- SQLite (`database_swap.adapters.sqlite.SQLiteAdapter`)
+- MySQL (`database_swap.adapters.mysql.MySQLAdapter`) – requires mysql-connector-python
+- MongoDB (`database_swap.adapters.mongodb.MongoDBAdapter`) – requires pymongo
+
+The adapter factory (`database_swap.adapters.get_adapter`) resolves a type string to the adapter class if its dependency is available.
+
+## Required connection fields
+
+- SQLite: database (file path)
+- MySQL: host, database, username (password typically required)
+- MongoDB: host, database (username/password optional)
+
+## Common capabilities
+
+- connect(), disconnect(), test_connection()
+- get_tables(), get_table_schema(name), get_table_count(name)
+- read_data(name, batch_size): yields batches of dict rows
+- write_data(name, data, create_table)
+- create_table(name, schema), drop_table(name)
+
+Schemas are lightweight dicts with a `columns` map, optional `primary_keys`, and `indexes`.
+
+```python
+schema = {
+  'columns': {
+    'id': { 'type': 'INTEGER', 'nullable': False },
+    'name': { 'type': 'TEXT', 'nullable': True }
+  },
+  'primary_keys': ['id'],
+  'indexes': []
+}
+```
+
+MongoDB schemas are inferred from sample documents and are only advisory.

docs/architecture.md

@@ -0,0 +1,39 @@
+# Architecture overview
+
+This project is organized into modular layers for clarity and extensibility.
+
+- CLI: `database_swap/cli/interface.py` – user entry point and argument parsing
+- Config: `database_swap/config/settings.py` – YAML loading, defaults, overrides
+- Core: `database_swap/core/` – `migrator.py`, `rate_limiter.py`, `validator.py`
+- Adapters: `database_swap/adapters/` – DB-specific implementations
+- Utils: `database_swap/utils/` – logging, helpers, progress, timers
+
+## Data flow
+
+1) CLI parses args and loads config
+2) Migrator is created with config
+3) Migrator resolves source/target adapters and connects
+4) For each table/collection:
+   - Read in batches from source
+   - Validate and convert data types (`utils.helpers.convert_data_types`)
+   - Rate limit writes (`core.rate_limiter.AdaptiveRateLimiter`)
+   - Write to target and track stats
+5) Final stats and logs are emitted
+
+## Key components
+
+- DatabaseMigrator: Orchestrates migration, batching, and stats
+- RateLimiter / AdaptiveRateLimiter: Applies delays based on error rate
+- DataValidator / SchemaValidator: Optional validation of rows and schemas
+- MigrationStats: Aggregates counters and durations
+
+## Error handling
+
+- Retries for failed batch writes (configurable `max_retries`)
+- Exponential backoff per batch attempt
+- Non-fatal errors are collected and surfaced at the end
+
+## Logging
+
+- File and colored console logging via `utils.logger.setup_logging`
+- Progress logs per table via `ProgressLogger`

docs/cli.md

@@ -0,0 +1,96 @@
+# CLI reference
+
+The `database-swap` CLI exposes four main commands defined in `database_swap/cli/interface.py`.
+
+- init-config: Create a starter configuration file
+- test-connection: Check that you can connect to a database
+- analyze: Inspect tables/collections and record counts
+- migrate: Migrate data between source and target
+
+All commands accept a global `--config` (-c) to load settings from YAML and `--verbose` (-v) for DEBUG logging.
+
+## init-config
+
+Create a `config.yaml` in the current directory (or a custom output path).
+
+```powershell
+database-swap init-config -o config.yaml
+```
+
+## test-connection
+
+Test connectivity to one database.
+
+```powershell
+# SQLite
+database-swap test-connection --db-type sqlite --database .\source.db
+
+# MySQL
+database-swap test-connection --db-type mysql --host localhost --database mydb --username root --password
+
+# MongoDB
+database-swap test-connection --db-type mongodb --host localhost --database mydb
+```
+
+Options
+
+- --db-type: sqlite | mysql | mongodb (required)
+- --host, --port
+- --database (required)
+- --username, --password
+
+## analyze
+
+Summarize tables/collections, schemas, and counts.
+
+```powershell
+database-swap analyze --db-type sqlite --database .\source.db
+```
+
+Options
+
+- --db-type: sqlite | mysql | mongodb (required)
+- --host, --port
+- --database (required)
+- --username, --password
+- --table: analyze one table/collection only
+
+## migrate
+
+Perform the migration. You can pass connection details via flags or a YAML config.
+
+```powershell
+# Dry run (no writes)
+database-swap migrate --dry-run `
+  --source-type sqlite --source-database .\source.db `
+  --target-type mysql --target-host localhost --target-database target_db --target-username root
+
+# Actual run
+database-swap migrate `
+  --source-type sqlite --source-database .\source.db `
+  --target-type mysql --target-host localhost --target-database target_db --target-username root
+```
+
+Options
+
+- Source: --source-type, --source-host, --source-port, --source-database, --source-username, --source-password
+- Target: --target-type, --target-host, --target-port, --target-database, --target-username, --target-password
+- Scope: --tables "a,b,c" (optional)
+- Performance: --batch-size, --rate-limit-delay, --max-retries
+- Safety: --dry-run
+
+## Using a config file
+
+All flags can be provided through YAML and overridden on the CLI.
+
+```powershell
+database-swap migrate --config .\config.yaml
+```
+
+Global flags can precede any subcommand:
+
+```powershell
+database-swap -c .\config.yaml -v migrate
+```
+
+See docs/configuration.md for the YAML schema.

docs/configuration.md

@@ -0,0 +1,65 @@
+# Configuration reference
+
+Database Swap loads YAML from `--config` or searches: `config.yaml`, `config.yml`, `%USERPROFILE%\\.database_swap.yaml`, `/etc/database_swap.yaml`.
+
+The default shape (see `database_swap/config/settings.py`):
+
+```yaml
+source:
+  type: sqlite | mysql | mongodb
+  connection:
+    host: localhost
+    port: null
+    database: source.db
+    username: null
+    password: null
+
+target:
+  type: sqlite | mysql | mongodb
+  connection:
+    host: localhost
+    port: null
+    database: target.db
+    username: null
+    password: null
+
+migration:
+  batch_size: 1000
+  rate_limit_delay: 0.1
+  max_retries: 3
+  timeout: 30
+  tables: null   # null = all tables/collections
+
+validation:
+  strict_mode: true
+  data_type_validation: true
+  foreign_key_validation: false
+
+logging:
+  level: INFO
+  file: database_swap.log
+  console: true
+```
+
+Notes
+
+- For SQLite, only `connection.database` is required.
+- For MySQL, required: host, database, username; password typically too.
+- For MongoDB, required: host, database. Username/password optional.
+- `tables`: set a list like ["users", "orders"] to migrate a subset.
+- `rate_limit_delay`: seconds to wait between batch writes (adaptive during run).
+
+## Override with CLI
+
+Any key can be overridden via flags used by `Config.update_from_args`:
+
+- Source: --source-type, --source-host, --source-port, --source-database, --source-username, --source-password
+- Target: --target-type, --target-host, --target-port, --target-database, --target-username, --target-password
+- Migration: --batch-size, --rate-limit-delay, --max-retries, --timeout, --tables
+- Logging: implied via `-v` (DEBUG) or by editing YAML
+
+Example
+
+```powershell
+database-swap migrate -c .\config.yaml --batch-size 500 --tables "users,orders"
+```

docs/extending.md

@@ -0,0 +1,42 @@
+# Extending Database Swap
+
+You can add support for a new database by implementing `DatabaseAdapter` and registering it.
+
+## 1) Create an adapter class
+
+Create `database_swap/adapters/postgresql.py` (as an example):
+
+```python
+from .base import DatabaseAdapter
+
+class PostgreSQLAdapter(DatabaseAdapter):
+    def connect(self) -> bool: ...
+    def disconnect(self) -> None: ...
+    def test_connection(self) -> bool: ...
+    def get_tables(self) -> list[str]: ...
+    def get_table_schema(self, table_name: str) -> dict: ...
+    def get_table_count(self, table_name: str) -> int: ...
+    def read_data(self, table_name: str, batch_size: int = 1000, offset: int = 0): ...
+    def write_data(self, table_name: str, data: list[dict], create_table: bool = True) -> bool: ...
+    def create_table(self, table_name: str, schema: dict) -> bool: ...
+    def drop_table(self, table_name: str) -> bool: ...
+```
+
+Reuse patterns from `sqlite.py`, `mysql.py`, and `mongodb.py`.
+
+## 2) Register in the factory
+
+Edit `database_swap/adapters/__init__.py` to import conditionally and expose your adapter when its dependency is installed. Follow the MySQL/MongoDB examples.
+
+## 3) Map types and conversions
+
+If your engine needs special conversions, update `utils.helpers.convert_value` as needed to ensure cross-engine safety.
+
+## 4) Tests and docs
+
+- Add sample connection details and minimal test DB
+- Document required fields in `docs/adapters.md`
+
+## 5) Packaging
+
+- Add the dependency to `requirements.txt` (or make it optional and document installation)