Skip to content

Commit fa5dde6

Browse files
hoangsonwwhoangsonww
authored
Merge pull request #6 from hoangsonww/claude/expand-database-operations-01FnLMTtwp5ZiQGJPn5NfcNs
Expand database operations
2 parents 849bd7d + 28fd3d8 commit fa5dde6

17 files changed

+2814
-812
lines changed

‎CHANGELOG.md‎

Lines changed: 163 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,169 @@ All notable changes to this project will be documented in this file.
55
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
66
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
77

8+
## [2.1.0] - 2025-01-16
9+
10+
### Added
11+
12+
#### Comprehensive SQL Operation Support
13+
- **INSERT Operations**:
14+
- Single row inserts with column specifications
15+
- Bulk inserts (multiple VALUES)
16+
- INSERT without column names (implicit ordering)
17+
- Bidirectional conversion: SQL INSERT ↔ MongoDB insertOne/insertMany
18+
19+
- **UPDATE Operations**:
20+
- UPDATE with SET clause (single/multiple columns)
21+
- Conditional updates with WHERE clause
22+
- Bulk updates without WHERE
23+
- Bidirectional conversion: SQL UPDATE ↔ MongoDB updateMany with $set
24+
25+
- **DELETE Operations**:
26+
- Conditional DELETE with WHERE clause
27+
- Bulk DELETE without conditions
28+
- Bidirectional conversion: SQL DELETE ↔ MongoDB deleteMany
29+
30+
- **JOIN Operations**:
31+
- INNER JOIN support with MongoDB `$lookup` aggregation
32+
- LEFT JOIN support with `$lookup` preserving unmatched documents
33+
- Multi-table joins with ON conditions
34+
- Proper field aliasing with table prefixes (e.g., u.name, o.order_id)
35+
36+
- **CREATE Operations**:
37+
- CREATE TABLE with column definitions
38+
- Schema validation with BSON type mapping (INT→int, VARCHAR→string, FLOAT→double, etc.)
39+
- CREATE INDEX with single/multiple columns
40+
- Index sort order support (ASC→1, DESC→-1)
41+
- Bidirectional conversion support
42+
43+
- **DROP Operations**:
44+
- DROP TABLE → MongoDB collection drop
45+
- DROP INDEX → MongoDB dropIndex
46+
- Safety: Requires `allow_mutations=True` flag
47+
48+
#### Advanced SELECT Features
49+
- **DISTINCT Queries**:
50+
- Single field: `SELECT DISTINCT field FROM table`
51+
- Multiple fields with aggregation pipeline
52+
- Proper deduplication using MongoDB distinct() or $group
53+
54+
- **HAVING Clause**:
55+
- Post-aggregation filtering
56+
- Works with GROUP BY and aggregation functions
57+
- Converted to `$match` stage after `$group` in aggregation pipeline
58+
59+
- **Aggregation Functions**:
60+
- COUNT(*) and COUNT(field)
61+
- SUM(field)
62+
- AVG(field)
63+
- MIN(field)
64+
- MAX(field)
65+
- Proper integration with GROUP BY/HAVING
66+
67+
#### Advanced WHERE Clause Operators
68+
- **BETWEEN Operator**:
69+
- Syntax: `field BETWEEN val1 AND val2`
70+
- Converts to: `{field: {$gte: val1, $lte: val2}}`
71+
- Smart AND parsing to avoid splitting BETWEEN's internal AND
72+
73+
- **LIKE Operator with Wildcards**:
74+
- `%` wildcard → `.*` regex pattern
75+
- `_` wildcard → `.` regex pattern
76+
- Case-insensitive matching with `$options: "i"`
77+
- Example: `name LIKE 'John%'` → `{name: {$regex: "John.*", $options: "i"}}`
78+
79+
- **IN and NOT IN Operators**:
80+
- `field IN (val1, val2, ...)` → `{field: {$in: [val1, val2, ...]}}`
81+
- `field NOT IN (...)` → `{field: {$nin: [...]}}`
82+
- Proper list parsing with quotes and commas
83+
84+
- **IS NULL and IS NOT NULL**:
85+
- `field IS NULL` → `{field: None}` or `{field: {$eq: None}}`
86+
- `field IS NOT NULL` → `{field: {$ne: None}}`
87+
88+
- **OR Operator**:
89+
- `condition1 OR condition2` → `{$or: [{...}, {...}]}`
90+
- Nested OR conditions with proper precedence
91+
- Works with complex conditions
92+
93+
- **NOT Operator**:
94+
- `NOT condition` → `{$not: {...}}`
95+
- Handles NOT with IN, LIKE, and other operators
96+
- Proper precedence with AND/OR
97+
98+
#### Parser Improvements
99+
- **Enhanced WHERE Clause Parser**:
100+
- Regex-based condition detection for complex patterns
101+
- Smart AND splitting that preserves BETWEEN clauses
102+
- Recursive parsing for nested conditions
103+
- Proper operator precedence handling
104+
105+
- **sqlparse Token Handling**:
106+
- Fixed Function object detection for INSERT and CREATE INDEX
107+
- Improved JOIN parsing with enumeration-based approach
108+
- Better handling of Identifier vs Function tokens
109+
- Robust parenthesis and quote parsing
110+
111+
- **Aggregation Pipeline Builder**:
112+
- Dynamic pipeline construction based on query features
113+
- Stages: $match, $group, $lookup, $project, $sort, $limit, $skip
114+
- Proper stage ordering for optimal query execution
115+
- Support for complex GROUP BY with multiple aggregations
116+
117+
#### Validator Enhancements
118+
- **Keyword Separation**:
119+
- `MUTATION_KEYWORDS`: INSERT, UPDATE, DELETE, CREATE (allowed with `allow_mutations=True`)
120+
- `DANGEROUS_KEYWORDS`: DROP, TRUNCATE, ALTER, EXEC (require explicit permission)
121+
- Better security model for write operations
122+
123+
- **MongoDB Operator Validation**:
124+
- Added update operators: $set, $inc, $unset, $push, $pull
125+
- Aggregation operators: $match, $group, $lookup, $project, $sort, $limit
126+
127+
### Changed
128+
- **converter.py**: Enhanced routing logic for all operation types
129+
- **sql_to_mongo.py**:
130+
- Expanded from ~200 lines to 400+ lines
131+
- Added 8+ new parsing functions
132+
- Improved WHERE clause parsing with 200+ lines of new logic
133+
- **mongo_to_sql.py**: Added reverse conversion functions for INSERT, UPDATE, DELETE
134+
- **validator.py**: Separated mutation keywords from dangerous keywords
135+
136+
### Improved
137+
- **Test Coverage**: From 58.55% to 59.27%
138+
- **Test Count**: From 70 tests to 103 tests (+33 new tests)
139+
- **Error Handling**: Better error messages for unsupported operations
140+
- **Type Mapping**: Comprehensive SQL→BSON type conversion
141+
- **Documentation**: Extensive README updates with examples
142+
143+
### Fixed
144+
- **INSERT Parsing**: Fixed sqlparse treating `table(cols)` as Function object
145+
- **JOIN Parsing**: Fixed token iteration issues with while loop
146+
- **CREATE INDEX Parsing**: Fixed Function object detection
147+
- **BETWEEN Clause**: Fixed AND splitting within BETWEEN
148+
- **NOT IN Parsing**: Fixed regex capturing NOT separately from IN
149+
- **Test Validator**: Updated DELETE test expectation (write operation vs dangerous)
150+
151+
### Technical Details
152+
- **New Test Files**:
153+
- `test_new_operations.py`: 33 tests for CRUD, JOIN, CREATE, DROP operations
154+
155+
- **Code Metrics**:
156+
- Total tests: 103 passing
157+
- Code coverage: 59.27%
158+
- New functions: 15+
159+
- Enhanced functions: 10+
160+
161+
- **Performance**:
162+
- All conversions maintain O(n) complexity
163+
- Aggregation pipeline generation is optimized
164+
- No performance degradation from new features
165+
166+
### Breaking Changes
167+
None - all changes are backward compatible. Existing v2.0.0 code will work with v2.1.0.
168+
169+
---
170+
8171
## [2.0.0] - 2025-01-16
9172

10173
### Added

0 commit comments

Comments
 (0)