Add production readiness review document

PRODUCTION_READINESS.md

@@ -0,0 +1,50 @@
+# SmartChunk Production Readiness Review
+
+## Overview
+This document captures an engineering review of SmartChunk's current codebase and command-line interface with a focus on production/startup readiness.
+
+## Packaging & Distribution
+- **PyProject configuration**: Uses `setuptools` with an explicit `src` layout, includes typed package marker, and exports the `smartchunk` console script via `Typer`. Dependencies cover CLI UX (`rich`, `typer`), parsing (`beautifulsoup4`, `markdown-it-py`), HTTP (`requests`), and ML (`numpy`, `scikit-learn`, `sentence-transformers`). Optional `tiktoken` dependency is isolated for token counting. Python `>=3.10` target is aligned with modern runtimes.
+- **Versioning**: Currently set to `0.1.2`; recommend adopting semantic versioning with release notes and automated builds (GitHub Actions / PyPI publish).
+- **Type support**: `py.typed` is included, but the codebase lacks type annotations on many callsites (e.g., helper functions). Adopting `mypy` or `pyright` gates would increase safety.
+
+## Runtime Dependencies & Optional Models
+- **Sentence Transformers**: Semantic splitting lazily loads `SentenceTransformer`; errors are surfaced clearly if the optional dependency is missing. For production, ship a default lightweight model or document the cold-start impact. Consider caching embeddings model across CLI invocations in long-running contexts.
+- **Token counting**: Falls back to heuristic when `tiktoken` missing. Document the accuracy trade-offs and expose configuration to disable heuristics in deterministic environments.
+
+## Core Chunking Engine (`smartchunk/chunker.py`)
+- **Structure awareness**: Detects markdown headers, code fences, and lists before segment packing. Overlap logic avoids duplication across unrelated sections and preserves code fences. Tests cover multi-section documents and mixed content.
+- **Semantic segmentation**: Splits long segments by sentence-level cosine similarity using embeddings; handles tensor→NumPy conversion explicitly. Add guardrails for GPU availability (currently assumes `.cpu()` works) and allow injecting an embeddings interface for unit tests.
+- **Edge cases**: `_too_big` relies on character count when `max_tokens` is `None`. Ensure documentation clarifies precedence. Consider exposing chunk ID prefix customization for downstream integration.
+
+## CLI Surface (`smartchunk/cli.py`)
+- **Commands**: `fetch`, `chunk`, `compare`, and `stream` share normalization helpers. Output supports table/JSON/JSONL with friendly Rich formatting. Log level flag sets global logging configuration. Provide `--version` and `--list-models` flags for parity with other CLIs.
+- **Fetch pipeline**: `fetch` command pulls HTML, normalizes via parser, and chunks with identical options to the local `chunk`. For production use, add rate-limiting/backoff indicators and friendly errors when BeautifulSoup or network dependencies missing.
+- **Streaming**: Maintains carry-over buffer to avoid mid-sentence emissions; flush factor is configurable. Consider adding heartbeat logging for long-running pipes and tests for interactive scenarios.
+
+## Fetcher (`smartchunk/fetcher.py`)
+- Uses a shared `requests.Session` with retry/backoff. BeautifulSoup heuristics target `<article>`/`<main>` fallback to paragraph density. Add timeout configuration flags and surface HTTP status info in CLI errors.
+
+## Parsers (`smartchunk/parsers.py`)
+- HTML parser transforms DOM to Markdown-like text while preserving structure, removing non-content tags, converting lists/tables/code blocks, and normalizing whitespace. Ensure unit tests cover nested lists and mixed content (currently limited).
+
+## Utilities (`smartchunk/utils.py`)
+- Provides `Chunk` dataclass and token counting. Consider storing chunk length metadata (tokens/chars) directly to avoid recomputation downstream.
+
+## Testing & Quality
+- `pytest` suite passes (5 tests) covering chunker, CLI, and parser behavior. Expand coverage for streaming and fetch commands (mocked HTTP). Introduce lint/type checks (ruff, mypy) and continuous integration pipeline.
+
+## Operational Considerations
+- **Logging**: CLI relies on Rich console messages; structured logging absent. For production pipelines, add JSON logging or allow non-TTY output mode.
+- **Error handling**: Most commands raise `typer.Exit` on fatal issues, but deeper layers return empty strings. Standardize exceptions and bubble up actionable messages.
+- **Security**: No sandboxing when fetching arbitrary HTML—document risks, sanitize output, and consider allow-listing schemes.
+
+## Recommendations for Startup Readiness
+1. Add automated CI (tests + lint + type check) and packaging workflows.
+2. Harden CLI UX: add `--version`, verbose flag, configurable timeouts, and helpful error codes.
+3. Improve semantic model management: allow offline caching and configuration of device (CPU/GPU).
+4. Expand documentation with architecture overview and examples for embedding integration.
+5. Consider modular API surface (e.g., `smartchunk.api` functions) for easier library consumption beyond CLI.
+
+## Summary
+The current codebase is clean, modular, and feature-complete for early adopters. With CI, extended typing, and operational hardening, SmartChunk can reach production-grade reliability for startup use cases.