From 4c4c99b0fa8a4e95a7318503d35bf034374559cc Mon Sep 17 00:00:00 2001 From: Simon Sefcik Date: Fri, 25 Jul 2025 13:09:36 +0200 Subject: [PATCH] feat: Add STRUCTURE.md documentation files for blockapi MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - Add comprehensive STRUCTURE.md files across all blockapi directories - Document file organization, purpose, and relationships - Include detailed descriptions for test data, API implementations, and utilities - Provide navigation links between related documentation files 🤖 Generated with [Claude Code](https://claude.ai/code) Co-Authored-By: Claude --- STRUCTURE.md | 212 ++++++++++++++++++ blockapi/STRUCTURE.md | 200 +++++++++++++++++ blockapi/test/STRUCTURE.md | 148 ++++++++++++ blockapi/test/v2/STRUCTURE.md | 161 +++++++++++++ blockapi/test/v2/api/STRUCTURE.md | 160 +++++++++++++ blockapi/test/v2/api/data/STRUCTURE.md | 113 ++++++++++ blockapi/test/v2/api/debank/STRUCTURE.md | 146 ++++++++++++ blockapi/test/v2/api/debank/data/STRUCTURE.md | 121 ++++++++++ blockapi/test/v2/api/nft/STRUCTURE.md | 126 +++++++++++ blockapi/utils/STRUCTURE.md | 70 ++++++ blockapi/v2/STRUCTURE.md | 204 +++++++++++++++++ blockapi/v2/api/STRUCTURE.md | 171 ++++++++++++++ blockapi/v2/api/covalenth/STRUCTURE.md | 91 ++++++++ blockapi/v2/api/nft/STRUCTURE.md | 158 +++++++++++++ blockapi/v2/api/perpetual/STRUCTURE.md | 89 ++++++++ blockapi/v2/api/synthetix/STRUCTURE.md | 108 +++++++++ 16 files changed, 2278 insertions(+) create mode 100644 STRUCTURE.md create mode 100644 blockapi/STRUCTURE.md create mode 100644 blockapi/test/STRUCTURE.md create mode 100644 blockapi/test/v2/STRUCTURE.md create mode 100644 blockapi/test/v2/api/STRUCTURE.md create mode 100644 blockapi/test/v2/api/data/STRUCTURE.md create mode 100644 blockapi/test/v2/api/debank/STRUCTURE.md create mode 100644 blockapi/test/v2/api/debank/data/STRUCTURE.md create mode 100644 blockapi/test/v2/api/nft/STRUCTURE.md create mode 100644 blockapi/utils/STRUCTURE.md create mode 100644 blockapi/v2/STRUCTURE.md create mode 100644 blockapi/v2/api/STRUCTURE.md create mode 100644 blockapi/v2/api/covalenth/STRUCTURE.md create mode 100644 blockapi/v2/api/nft/STRUCTURE.md create mode 100644 blockapi/v2/api/perpetual/STRUCTURE.md create mode 100644 blockapi/v2/api/synthetix/STRUCTURE.md diff --git a/STRUCTURE.md b/STRUCTURE.md new file mode 100644 index 00000000..b0627a43 --- /dev/null +++ b/STRUCTURE.md @@ -0,0 +1,212 @@ +# Blockapi Library + +## Overview + +Blockapi is a comprehensive Python library that provides a unified interface for interacting with blockchain APIs, cryptocurrency data providers, and NFT marketplaces. As a critical component of the Crypkit platform, it abstracts the complexity of integrating with 50+ blockchain networks through both legacy (v1) and modern (v2) implementations, handling everything from simple balance queries to complex DeFi portfolio analysis and NFT data aggregation. + +## Purpose and Architecture + +The library serves as a blockchain data abstraction layer that: +- Provides a consistent, unified API across diverse blockchain networks and data providers +- Handles API authentication, rate limiting, retries, and error recovery automatically +- Normalizes heterogeneous data formats from various sources into standardized models +- Supports both basic blockchain operations (balance, transactions) and advanced features (DeFi protocols, NFT marketplaces) +- Maintains backward compatibility through dual-version architecture (v1 legacy, v2 modern) + +### Architectural Principles +- **Dynamic API Discovery**: Automatically discovers and uses available APIs without hardcoding +- **Graceful Degradation**: Falls back to alternative APIs when primary ones fail +- **Precision First**: All monetary values use Decimal type for financial accuracy +- **Extensibility**: New blockchains and protocols can be added by implementing standard interfaces + +## Directory Structure + +``` +blockapi/ +├── CHANGELOG.md # Detailed version history and changes +├── LICENSE.md # MIT License +├── Makefile # Build and test automation +├── README.md # User-facing documentation +├── pyproject.toml # Python project configuration (Black, isort, semantic-release) +├── pytest.ini # Pytest configuration with custom markers +├── setup.cfg # Package metadata (empty, uses setup.py) +├── setup.py # Package installation and dependencies +└── blockapi/ # Main library package + ├── STRUCTURE.md # See blockapi/STRUCTURE.md for library architecture details + ├── __init__.py # Core API functions and dynamic discovery + ├── services.py # Base service classes and HTTP infrastructure + ├── test_data.py # Test addresses and API key management + ├── test/ # Comprehensive test suite + │ └── STRUCTURE.md # See blockapi/test/STRUCTURE.md for test documentation + ├── utils/ # Common utility functions + │ └── STRUCTURE.md # See blockapi/utils/STRUCTURE.md for utilities documentation + └── v2/ # Modern v2 implementation + └── STRUCTURE.md # See blockapi/v2/STRUCTURE.md for v2 architecture +``` + +## Configuration Files + +### `setup.py` +Defines package metadata and dependencies: +- **Version**: 1.3.0 (as of last update) +- **Core Dependencies**: requests, pytz, coinaddrng, web3, pydantic +- **Blockchain-Specific**: solders (Solana), ethereum_input_decoder +- **Utilities**: fake_useragent, beautifulsoup4, lxml +- **Testing**: pytest, pytest-vcr, requests_mock + +### `pyproject.toml` +Development tooling configuration: +- **Black**: Line length 88, skip string normalization +- **isort**: Black-compatible profile +- **semantic-release**: Version tracking in setup.py + +### `pytest.ini` +Test framework configuration: +- **Markers**: `integration` for tests requiring actual API connections +- **Python Path**: Current directory for imports + +### `Makefile` +Development workflow automation: +- `make install`: Install library locally +- `make test`: Run v1 tests +- `make test-v2-api`: Run v2 API tests +- `make dist`: Build and publish to PyPI + +## Core Functionality + +### Public API (blockapi/__init__.py) +- `get_balance_from_random_api(symbol, address)`: Fetches balance using suitable API +- `get_api_classes_for_coin(symbol)`: Returns all APIs supporting a cryptocurrency +- `get_all_supported_coins()`: Lists all supported cryptocurrency symbols +- `check_address_valid(symbol, address)`: Validates blockchain addresses +- `get_working_apis_for_coin(symbol)`: Tests and returns functional APIs + +### Key Components +- **blockapi/**: Core v1 implementation and library entry point - See [blockapi/STRUCTURE.md](blockapi/STRUCTURE.md) +- **blockapi/v2/**: Modern architecture with enhanced features - See [blockapi/v2/STRUCTURE.md](blockapi/v2/STRUCTURE.md) +- **blockapi/utils/**: Shared utilities for address, datetime, and number handling - See [blockapi/utils/STRUCTURE.md](blockapi/utils/STRUCTURE.md) +- **blockapi/test/**: Comprehensive test coverage for both v1 and v2 - See [blockapi/test/STRUCTURE.md](blockapi/test/STRUCTURE.md) + +## Supported Blockchains and Features + +### Native Blockchain Support (v2) +- **Bitcoin Family**: BTC, LTC, DOGE (via Blockchair, Trezor) +- **Ethereum & L2s**: ETH, Optimism (via Ethplorer, Etherscan) +- **Cosmos Ecosystem**: ATOM, OSMO, DYDX, TIA (via native APIs) +- **Solana**: SOL and SPL tokens (via RPC, Solscan) +- **Polkadot/Kusama**: DOT, KSM (via Subscan) +- **Others**: SUI, Terra, BOS + +### Multi-Chain Aggregators +- **DeBank**: 40+ chains with DeFi protocol integration +- **Covalenth**: 15+ EVM chains with unified API +- **SimpleHash**: NFT data across multiple chains + +### Specialized Integrations +- **NFT Marketplaces**: OpenSea, Magic Eden, SimpleHash, UniSat +- **DeFi Protocols**: Synthetix, Perpetual Protocol +- **Explorers**: Blockchair, Ethplorer, Subscan, various chain-specific explorers + +## Usage Examples + +### Basic Balance Query (v1) +```python +import blockapi + +# Get Bitcoin balance using random API +balance = blockapi.get_balance_from_random_api('BTC', '1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa') + +# Validate Ethereum address +is_valid = blockapi.check_address_valid('ETH', '0x...') +``` + +### Advanced Usage (v2) +```python +from blockapi.v2.api import SolanaApi, DebankApi +from blockapi.v2.models import Blockchain + +# Native blockchain API +solana = SolanaApi() +balances = solana.get_balance("11111111111111111111111111111111") + +# DeFi portfolio aggregation +debank = DebankApi(api_key="YOUR_KEY", is_all=True) +portfolio = debank.get_portfolio("0x...") + +# Custom RPC endpoint +from blockapi.v2.api import EthereumApi +eth = EthereumApi(base_url="https://mainnet.infura.io/v3/YOUR_KEY") +``` + +## Development Workflow + +### Installation +```bash +# Development installation +pip install -e . + +# Or using make +make install +``` + +### Testing +```bash +# Run all tests +pytest + +# Run v1 tests only +make test + +# Run v2 tests only +make test-v2-api + +# Skip integration tests +pytest -m "not integration" +``` + +### Publishing +```bash +# Build and publish to PyPI +make dist +``` + +## Important Conventions + +1. **Symbol Format**: Always uppercase (BTC, ETH, SOL) +2. **Address Validation**: Performed before API calls +3. **Error Handling**: APIs return None on failure rather than raising +4. **Decimal Precision**: All monetary values use Decimal type +5. **API Keys**: Retrieved from environment variables +6. **Rate Limiting**: Automatic throttling based on API limits +7. **Timezone**: All datetime objects use UTC + +## Dependencies and Requirements + +### Runtime Dependencies +- Python 3.x +- Core: requests, pytz, coinaddrng +- Blockchain-specific: web3, solders, ethereum_input_decoder +- Data modeling: pydantic, attrs +- Utilities: beautifulsoup4, fake_useragent + +### Development Dependencies +- Testing: pytest, pytest-vcr, requests_mock +- Formatting: black, isort +- Build: setuptools, twine + +## Integration with Crypkit + +As documented in the main CLAUDE.md, blockapi serves as the blockchain data layer for Crypkit: +- Used by microservices for balance and transaction data +- Integrated with currencies-module for price correlation +- Provides portfolio data for frontend display +- Enables cross-chain asset tracking + +## Future Extensibility + +The architecture supports: +- Adding new blockchain APIs by implementing base interfaces +- Extending to new DeFi protocols through standardized models +- Integrating additional NFT marketplaces +- Enhanced caching and WebSocket support +- Cross-chain interoperability features \ No newline at end of file diff --git a/blockapi/STRUCTURE.md b/blockapi/STRUCTURE.md new file mode 100644 index 00000000..65e23451 --- /dev/null +++ b/blockapi/STRUCTURE.md @@ -0,0 +1,200 @@ +# Blockapi Library Structure + +## Overview + +The blockapi library is a comprehensive Python package that provides a unified interface for interacting with various blockchain APIs, cryptocurrency data providers, and NFT marketplaces. It serves as a critical component of the Crypkit platform, enabling seamless integration with over 50 blockchain networks through both legacy (v1) and modern (v2) implementations. + +## Purpose and Architecture + +This library abstracts the complexity of interacting with different blockchain explorers, APIs, and data providers by: +- Providing a consistent interface across multiple blockchain networks +- Handling rate limiting, retries, and error recovery +- Normalizing data formats from various sources +- Supporting both simple balance queries and complex DeFi portfolio analysis +- Integrating NFT marketplace data from multiple providers + +The library follows a dual-version architecture: +- **v1 (Legacy)**: Original implementation with basic blockchain API support +- **v2 (Modern)**: Enhanced architecture with standardized models, better error handling, and expanded protocol support + +## Directory Structure + +``` +blockapi/ +├── __init__.py # Package initialization with core functions and COINS mapping +├── services.py # Base service classes and API infrastructure +├── test_data.py # Test addresses and API key management +├── test/ # Comprehensive test suite +│ └── STRUCTURE.md # See test/STRUCTURE.md for detailed test documentation +├── utils/ # Utility functions for common operations +│ └── STRUCTURE.md # See utils/STRUCTURE.md for utility details +└── v2/ # Modern v2 implementation + └── STRUCTURE.md # See v2/STRUCTURE.md for v2 architecture details +``` + +## Core Files + +### `__init__.py` +The main entry point for the blockapi library, providing: + +**Core Functions:** +- `get_balance_from_random_api(symbol, address)`: Fetches balance using a random suitable API +- `get_api_classes_for_coin(symbol)`: Returns all API classes supporting a specific cryptocurrency +- `get_all_supported_coins()`: Lists all supported cryptocurrency symbols +- `check_address_valid(symbol, address)`: Validates blockchain addresses using coinaddrng +- `get_working_apis_for_coin(symbol)`: Tests and returns functional APIs for a coin + +**Key Features:** +- Dynamic API class discovery through inheritance inspection +- Automatic API selection based on address type (regular/xpub) and network (mainnet/testnet) +- Built-in retry logic with fallback to alternative APIs +- Test mode support for validating API functionality + +**COINS Dictionary:** +Maps CoinGecko IDs to standard symbols for 25+ major cryptocurrencies (BTC, ETH, SOL, etc.) + +### `services.py` +Foundation classes for all blockchain API implementations: + +**Core Classes:** +- `Service`: Base class for HTTP-based API services + - Cloudflare bypass support + - Rate limiting with configurable delays + - Automatic error response handling + - Session management for request reuse + +- `BlockchainInterface`: Abstract interface defining blockchain API contract + - `get_balance()`: Required method for balance retrieval + - `get_txs()`: Optional transaction history retrieval + - Pagination support with offset/limit parameters + +- `BlockchainAPI`: Combined Service + BlockchainInterface base + - Automatic network detection (mainnet/testnet) + - Address validation integration + - Decimal conversion utilities + +**Decorators:** +- `@set_default_args_values`: Automatically applies default pagination values +- `@on_failure_return_none()`: Graceful error handling decorator + +**Exception Types:** +- `APIError`: Base exception for all API errors +- `AddressNotExist`: Invalid or non-existent address +- `APIKeyMissing`: Required API key not provided +- `InternalServerError`, `BadGateway`, `GatewayTimeOut`: HTTP error wrappers + +### `test_data.py` +Centralized test data management: + +**Test Addresses Dictionary:** +- Contains valid test addresses for 45+ cryptocurrencies +- Includes regular addresses, xpub keys, and contract addresses +- Used for automated API testing and validation + +**API Key Management:** +- `get_test_api_key(api_cls_name)`: Retrieves API keys from environment variables +- Supports custom naming patterns (e.g., CRYPTOIDAPI_KEY) + +## Subdirectories + +### `test/` +Comprehensive test suite for both v1 and v2 implementations. See [test/STRUCTURE.md](test/STRUCTURE.md) for: +- Core functionality tests +- Multi-API automated testing framework +- Address validation tests +- Utility function tests +- Complete v2 API test coverage + +### `utils/` +Common utility functions used throughout the library. See [utils/STRUCTURE.md](utils/STRUCTURE.md) for: +- Address validation and checksum formatting +- DateTime parsing for blockchain timestamps +- Decimal/raw value conversions for precise cryptocurrency calculations +- User agent randomization for API requests + +### `v2/` +Modern implementation with enhanced architecture. See [v2/STRUCTURE.md](v2/STRUCTURE.md) for: +- Standardized data models and interfaces +- 50+ blockchain API implementations +- DeFi protocol integrations (Synthetix, Perpetual) +- NFT marketplace support (OpenSea, Magic Eden, etc.) +- Multi-chain aggregators (DeBank, Covalenth) + +## Integration Patterns + +### Basic Usage (v1) +```python +import blockapi + +# Get balance for Bitcoin address +balance = blockapi.get_balance_from_random_api('BTC', '1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa') + +# Get all APIs supporting Ethereum +eth_apis = blockapi.get_api_classes_for_coin('ETH') + +# Validate address +is_valid = blockapi.check_address_valid('BTC', 'invalid_address') +``` + +### Direct API Usage (v1) +```python +# Use specific API class +from blockapi.v1.blockchain_info import BlockchainInfo + +api = BlockchainInfo('1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa') +balance = api.get_balance() +transactions = api.get_txs(limit=10) +``` + +### Modern Usage (v2) +```python +from blockapi.v2.api import SolanaApi, DebankApi +from blockapi.v2.models import Blockchain + +# Native blockchain API +solana = SolanaApi() +balances = solana.get_balance("11111111111111111111111111111111") + +# Multi-chain aggregator +debank = DebankApi(api_key="YOUR_KEY") +portfolio = debank.get_portfolio("0x...") +``` + +## Key Design Decisions + +1. **Dual Version Support**: Maintains backward compatibility while introducing modern features +2. **Dynamic API Discovery**: Automatically finds and uses available APIs without hardcoding +3. **Graceful Degradation**: Falls back to alternative APIs when primary ones fail +4. **Address Validation**: Validates addresses before making API calls to prevent errors +5. **Decimal Precision**: All monetary values use Decimal type for accuracy +6. **Testability**: Comprehensive test suite with real blockchain data + +## Dependencies + +### Core Dependencies +- `requests`: HTTP client for API calls +- `cfscrape`: Cloudflare bypass for protected APIs +- `coinaddrng`: Multi-blockchain address validation + +### v2 Additional Dependencies +- `attrs`, `pydantic`: Data modeling and validation +- `web3`, `solders`: Blockchain-specific libraries +- `eth_utils`: Ethereum utilities +- Various protocol-specific SDKs + +## Important Conventions + +1. **Symbol Format**: Always use uppercase symbols (BTC, ETH, SOL) +2. **Address Format**: Blockchain-specific, validated before use +3. **Error Handling**: APIs return None on failure rather than raising exceptions +4. **API Keys**: Retrieved from environment variables for security +5. **Rate Limiting**: Respect API rate limits through built-in delays + +## Future Considerations + +The library architecture supports: +- Easy addition of new blockchain APIs +- Extension to new DeFi protocols +- Integration of additional NFT marketplaces +- Enhanced caching mechanisms +- WebSocket support for real-time data \ No newline at end of file diff --git a/blockapi/test/STRUCTURE.md b/blockapi/test/STRUCTURE.md new file mode 100644 index 00000000..f7b3e677 --- /dev/null +++ b/blockapi/test/STRUCTURE.md @@ -0,0 +1,148 @@ +# Blockapi Test Suite + +## Overview + +This directory contains the comprehensive test suite for the blockapi library, a Python package that provides a unified interface for interacting with various blockchain APIs, cryptocurrency data providers, and NFT marketplaces. The test suite covers both the legacy v1 API and the modern v2 implementation, ensuring robust functionality across multiple blockchain ecosystems. + +## Purpose and Functionality + +The test suite serves as the quality assurance framework for the blockapi library, validating: +- Core functionality of blockchain API integrations +- Data parsing and normalization across different providers +- Error handling and retry mechanisms +- Address validation and utility functions +- Both legacy (v1) and modern (v2) API implementations + +## Directory Structure + +### Root Level Test Files + +#### `__init__.py` +Empty Python package initialization file that marks this directory as a Python package. + +#### `test_blockapi.py` +Core test framework for the blockapi library's main functionality: +- Implements `TestBlockApiProviders` class for automated testing of all supported blockchain APIs +- Tests currency support detection via `get_all_supported_coins()` +- Validates API class discovery with `get_api_classes_for_coin()` +- Performs bulk testing across multiple currencies and addresses +- Provides diagnostic methods for analyzing test results and API performance +- Tests address validation functionality + +#### `test_num.py` +Tests for numerical utility functions: +- Validates `decimals_to_raw()` function for converting decimal numbers to raw blockchain values +- Ensures proper handling of decimal precision for cryptocurrency amounts + +#### `test_random_user_agent.py` +Tests for user agent randomization functionality: +- Validates the `get_random_user_agent()` utility function +- Ensures proper generation of random user agents for API requests + +### Subdirectory + +#### `v2/` +The modern v2 API test suite containing comprehensive tests for the latest version of the blockapi library. See [v2/STRUCTURE.md](v2/STRUCTURE.md) for detailed information about: +- Core v2 functionality tests (base classes, models, blockchain mapping) +- Extensive API integration tests for 30+ blockchain providers +- NFT marketplace integration tests +- DeFi protocol integration tests +- Comprehensive test data and mocking infrastructure + +## Key Components and Patterns + +### Test Infrastructure +- Uses pytest as the primary testing framework +- Leverages test data from `blockapi.test_data` module +- Implements comprehensive API testing across multiple blockchain networks +- Provides diagnostic capabilities for analyzing test results + +### Testing Approach +1. **Automated Multi-API Testing**: The `TestBlockApiProviders` class enables bulk testing of all supported APIs +2. **Currency Coverage**: Tests span multiple cryptocurrencies using predefined test addresses +3. **Method Testing**: Supports testing various API methods (default: `get_balance`) +4. **Error Tracking**: Comprehensive error collection and diagnostic reporting + +### Data Validation +- Address validation for various blockchain formats +- Decimal precision handling for cryptocurrency amounts +- API response structure validation +- Success/failure tracking for each API provider + +## Integration Points + +The test suite validates integration with: +- **Core Library Functions**: Address validation, coin support detection, API class discovery +- **Utility Functions**: Numerical conversions, user agent generation +- **V2 API Suite**: Modern implementation with enhanced features and broader blockchain support +- **Test Data Module**: Centralized test addresses and API keys management + +## Usage Examples + +### Running the Tests +```bash +# Run all tests +pytest test/ + +# Run only v1 tests (excluding v2) +pytest test/test_*.py + +# Run specific test file +pytest test/test_blockapi.py + +# Run with verbose output +pytest test/ -v +``` + +### Using TestBlockApiProviders +```python +# Initialize test framework +tester = TestBlockApiProviders() + +# Run automated tests for all supported currencies +tester.run_all() + +# Get diagnostic information +diagnostics = tester.get_diagnostic() + +# Custom test data +custom_data = [ + { + 'currency': 'BTC', + 'address': '1A1zP1eP5QGefi2DMPTfTL5SLmv7DivfNa', + 'value': '50' + } +] +tester.data = custom_data +tester.run() +``` + +## Important Conventions + +### Test Organization +- V1 tests remain at the root level for backward compatibility +- V2 tests are isolated in the `v2/` subdirectory +- Clear separation between unit tests and integration tests +- Comprehensive error handling and reporting + +### Code Quality Standards +- All tests should be deterministic and reproducible +- External API calls should be mocked where appropriate +- Test both success and failure scenarios +- Maintain clear test naming conventions + +## Dependencies + +The test suite requires: +- pytest and related testing utilities +- blockapi library (both v1 and v2) +- Test data module with predefined addresses and API keys +- Various mocking and fixture libraries (see v2/STRUCTURE.md for v2-specific dependencies) + +## Future Considerations + +The test suite architecture supports: +- Easy addition of new blockchain API providers +- Extension of test methods beyond balance checking +- Integration of new cryptocurrency protocols +- Backward compatibility with legacy v1 implementations while promoting v2 adoption \ No newline at end of file diff --git a/blockapi/test/v2/STRUCTURE.md b/blockapi/test/v2/STRUCTURE.md new file mode 100644 index 00000000..ddb45ffa --- /dev/null +++ b/blockapi/test/v2/STRUCTURE.md @@ -0,0 +1,161 @@ +# Blockapi V2 Test Suite + +## Overview + +This directory contains the comprehensive test suite for the blockapi v2 library, which provides a unified interface for interacting with various blockchain APIs, NFT marketplaces, and DeFi protocols. The test suite validates the core functionality of the library including API integrations, data parsing, error handling, and cross-chain compatibility. + +## Purpose and Functionality + +The v2 test suite serves as the quality assurance framework for the blockapi library, ensuring: +- Reliable integration with 30+ blockchain data providers and APIs +- Correct parsing and normalization of diverse API response formats +- Robust error handling and retry mechanisms +- Comprehensive coverage of blockchain-specific features and edge cases +- Validation of core models and base classes used throughout the library + +## Directory Structure + +### Core Test Files + +#### `__init__.py` +Empty Python package initialization file. + +#### `test_base.py` +Tests for the `CustomizableBlockchainApi` base class that all API implementations inherit from: +- Connection error handling with automatic retries +- HTTP status code handling (401, 500, etc.) +- Retry logic for transient failures +- Exception handling and error response formatting +- Integration with custom sleep providers for rate limiting + +#### `test_blockchain_api.py` +Validates the blockchain API base class implementation: +- Tests that subclasses must implement required methods +- Ensures proper API options configuration +- Validates blockchain assignment to API instances + +#### `test_blockchain_mapping.py` +Tests for blockchain identifier mapping and resolution: +- Chain ID to blockchain enum mapping +- Blockchain name standardization +- Cross-reference validation between different naming conventions + +#### `test_data.py` +Core data model and parsing tests: +- JSON response parsing utilities +- Data transformation helpers +- Common test data fixtures + +#### `test_enumerate_classes.py` +Tests for dynamic class enumeration and discovery: +- API class auto-discovery mechanisms +- Plugin system validation +- Dynamic import verification + +#### `test_generic.py` +Generic functionality tests that don't fit specific categories: +- Utility function testing +- Helper method validation +- Cross-cutting concerns + +#### `test_models.py` +Tests for core v2 data models: +- `Protocol` model creation and validation +- `BalanceItem` with protocol associations +- Model serialization and deserialization +- Decimal precision handling for financial data + +### Subdirectories + +#### `api/` +The main API integration test suite containing tests for all blockchain data providers. See [api/STRUCTURE.md](api/STRUCTURE.md) for comprehensive details about: +- Individual blockchain API tests (Bitcoin, Ethereum, Solana, etc.) +- NFT marketplace integrations (OpenSea, MagicEden, SimpleHash, UniSat) +- DeFi protocol tests (DeBank, Synthetix, Perpetual) +- Extensive test data fixtures and mocking infrastructure + +## Key Testing Patterns + +### Test Organization +- Hierarchical structure mirroring the library's module organization +- Separation of unit tests from integration tests +- Comprehensive fixtures defined in conftest.py files +- Test data stored as JSON files for reproducibility + +### Mocking and Fixtures +- `FakeSleepProvider` for testing rate limiting without delays +- `requests_mock` for HTTP request interception +- VCR cassettes for recording/replaying API interactions +- Shared fixtures cascading through conftest.py hierarchy + +### Error Handling Tests +- Connection error retry logic validation +- HTTP status code handling (4xx, 5xx responses) +- Malformed response parsing +- Unknown blockchain/protocol handling +- API authentication failures + +### Data Validation +- Decimal precision for all monetary values +- UTC timezone consistency for timestamps +- Address checksum verification +- Blockchain and token identification accuracy + +## Integration Points + +The test suite validates integration with: +- **30+ Blockchain APIs**: Including Blockchair, Ethplorer, Subscan, Trezor, and more +- **NFT Marketplaces**: OpenSea, MagicEden, SimpleHash, UniSat +- **DeFi Protocols**: DeBank portfolio tracker, Synthetix, Perpetual Protocol +- **Multiple Blockchains**: Bitcoin, Ethereum, Solana, Polkadot, Cosmos, Sui, and others +- **Core Libraries**: pytest, requests, Decimal, datetime with timezone support + +## Important Conventions + +### Testing Standards +- All external API calls must be mocked in unit tests +- Integration tests marked with `@pytest.mark.integration` +- Test both success and failure scenarios +- Validate complete data transformation pipelines +- Ensure backward compatibility + +### Code Quality +- Comprehensive test coverage expected +- Clear test naming following `test_` pattern +- Extensive use of fixtures for DRY principles +- Detailed docstrings for complex test scenarios + +### Performance Considerations +- Rate limiting tested without actual delays +- Batch operations validated for efficiency +- Caching mechanisms verified (e.g., DeBank protocol cache) +- Pagination handling for large result sets + +## Usage + +Run the test suite using pytest: +```bash +# Run all v2 tests +pytest test/v2/ + +# Run with coverage report +pytest test/v2/ --cov=blockapi.v2 + +# Run only unit tests (skip integration) +pytest test/v2/ -m "not integration" + +# Run specific test module +pytest test/v2/test_models.py + +# Run with verbose output +pytest test/v2/ -v +``` + +## Dependencies + +The test suite requires: +- pytest and related plugins (pytest-cov, pytest-mock) +- requests-mock for HTTP mocking +- vcrpy for recording API interactions +- Core blockapi v2 library and all its dependencies +- Test data files in JSON format \ No newline at end of file diff --git a/blockapi/test/v2/api/STRUCTURE.md b/blockapi/test/v2/api/STRUCTURE.md new file mode 100644 index 00000000..30d8571c --- /dev/null +++ b/blockapi/test/v2/api/STRUCTURE.md @@ -0,0 +1,160 @@ +# Blockapi V2 API Test Suite + +## Overview + +This directory contains the comprehensive test suite for the blockapi v2 API integrations. It includes tests for various blockchain data providers, NFT marketplaces, DeFi protocols, and cryptocurrency APIs. The test suite ensures proper functionality of API clients, data parsing, error handling, and integration with multiple blockchain networks including Bitcoin, Ethereum, Solana, and others. + +## Purpose and Functionality + +The test suite validates: +- API endpoint construction and authentication across multiple providers +- Response parsing for balances, transactions, NFT data, and DeFi positions +- Error handling and retry logic for various failure scenarios +- Integration testing with real and mocked API endpoints +- Data transformation from provider-specific formats to standardized models +- Cross-chain compatibility and blockchain-specific features + +## Directory Structure + +### Core Files + +#### `__init__.py` +Empty Python package initialization file. + +#### `conftest.py` +Shared pytest configuration providing utility functions: +- `read_json_file()`: Loads JSON test data from files +- `read_file()`: Reads raw file content as strings +- Common test utilities used across all test modules + +#### `fake_sleep_provider.py` +Mock implementation of ISleepProvider for testing rate limiting: +- Tracks sleep calls without actually sleeping +- Allows verification of rate limiting behavior +- Used to test API retry and backoff logic + +### Blockchain-Specific Test Files + +#### Bitcoin and Bitcoin-like Chains +- **`test_blockchair_btc.py`**: Tests Blockchair API for Bitcoin balances and transactions +- **`test_blockchair_doge.py`**: Tests Blockchair API for Dogecoin +- **`test_blockchair_ltc.py`**: Tests Blockchair API for Litecoin +- **`test_trezor_btc.py`**: Tests Trezor's Blockbook API for Bitcoin including xpub support + +#### Ethereum and EVM Chains +- **`test_ethplorer.py`**: Tests Ethplorer API for Ethereum token balances +- **`test_optimistic_etherscan.py`**: Tests Etherscan-compatible API for Optimism network + +#### Other Blockchains +- **`test_solana.py`**: Comprehensive Solana API tests including token lists and staking +- **`test_cosmos.py`**: Tests for Cosmos blockchain integration +- **`test_subscan_polkadot.py`**: Tests Subscan API for Polkadot balances and staking +- **`test_sui.py`**: Tests for Sui blockchain integration +- **`test_blockchainos.py`**: Tests for Blockchain Operating System (BOS) operations + +#### Multi-Source and Specialized Tests +- **`test_multisources.py`**: Tests aggregation across multiple data sources + +### Subdirectories + +#### `cassettes/` +VCR cassettes for recording/replaying HTTP requests: +- `test_fetch_metaplex_account.yaml`: Metaplex NFT account data +- `test_get_balance.yaml`: Balance fetch recordings +- `test_parse_metaplex_account.yaml`: Metaplex parsing tests + +#### `covalenth/` +Covalent API test suite: +- `__init__.py`: Package initialization +- `ethereum.json`: Ethereum balance test data +- `test_ethereum.py`: Covalent Ethereum API tests + +#### `data/` +Extensive test data fixtures for all blockchain integrations. See [data/STRUCTURE.md](data/STRUCTURE.md) for detailed information about test data files including responses from Blockchair, Ethplorer, OpenSea, MagicEden, SimpleHash, Solana, and more. + +#### `debank/` +Comprehensive DeBank DeFi portfolio tracker tests. See [debank/STRUCTURE.md](debank/STRUCTURE.md) for detailed information about DeBank API testing including balance parsing, portfolio positions, protocol caching, and usage statistics. + +#### `nft/` +NFT marketplace API test suite. See [nft/STRUCTURE.md](nft/STRUCTURE.md) for detailed information about NFT API tests including MagicEden, OpenSea, SimpleHash, and UniSat integrations. + +#### `perpetual/` +Perpetual Protocol integration tests: +- `__init__.py`: Package initialization +- `test_perpetual.py`: Tests for PERP token contract interactions and balance fetching + +#### `synthetix/` +Synthetix protocol integration tests: +- `__init__.py`: Package initialization +- `cassettes/test_synthetix_optimism_api.yaml`: Recorded Optimism API responses +- `data/contracts.md`: Synthetix contract addresses and documentation +- `test_synthetix.py`: Synthetix staking and rewards tests + +## Key Testing Patterns + +### Fixture Usage +- Heavy use of pytest fixtures for reusable test data and API clients +- JSON response data loaded from files for predictable testing +- Shared fixtures in conftest.py files at multiple levels + +### Mocking Strategies +- `requests_mock` for HTTP request mocking in unit tests +- VCR cassettes for recording/replaying real API interactions +- `FakeSleepProvider` for testing rate limiting without delays + +### Integration Testing +- Tests marked with `@pytest.mark.integration` require real API access +- Integration tests often skipped in CI but used for validation +- Separate test data saved from real API responses + +### Error Handling +- Comprehensive testing of error responses and edge cases +- Validation of retry logic and backoff strategies +- Testing of malformed data and unknown blockchain handling + +## Important Conventions + +### Test Organization +- One test file per API integration or blockchain +- Fixtures defined in conftest.py files for reusability +- Test data stored as JSON files in data/ directories +- Clear separation between unit and integration tests + +### Data Validation +- All monetary values use `Decimal` for precision +- Timezone-aware datetime parsing (UTC standard) +- Address validation and checksum verification +- Consistent blockchain and coin identification + +### API Testing Standards +- Mock all external HTTP requests in unit tests +- Test both successful responses and error conditions +- Validate complete data transformation pipeline +- Ensure backward compatibility with API changes + +## Dependencies and Integration Points + +The test suite integrates with: +- Multiple blockchain APIs (Blockchair, Ethplorer, Subscan, etc.) +- NFT marketplaces (OpenSea, MagicEden, SimpleHash, UniSat) +- DeFi protocols (DeBank, Synthetix, Perpetual) +- Standard pytest testing framework and fixtures +- blockapi v2 models and coin definitions +- HTTP mocking libraries (requests_mock, vcrpy) + +## Usage + +Run tests with pytest: +```bash +# Run all tests +pytest test/v2/api/ + +# Run specific test file +pytest test/v2/api/test_solana.py + +# Run only unit tests (skip integration) +pytest test/v2/api/ -m "not integration" + +# Run with coverage +pytest test/v2/api/ --cov=blockapi.v2.api +``` \ No newline at end of file diff --git a/blockapi/test/v2/api/data/STRUCTURE.md b/blockapi/test/v2/api/data/STRUCTURE.md new file mode 100644 index 00000000..4d370676 --- /dev/null +++ b/blockapi/test/v2/api/data/STRUCTURE.md @@ -0,0 +1,113 @@ +# Test Data Directory Structure + +## Overview +This directory contains mock response data files used for testing the blockapi v2 API integration. The data represents real-world API responses from various blockchain explorers, NFT marketplaces, and cryptocurrency data providers. These files serve as test fixtures for unit and integration tests, ensuring the blockapi library correctly parses and handles responses from different blockchain data sources. + +## Directory Contents + +### Bitcoin and Bitcoin-like Blockchain Data + +#### Blockchair Response Files +- **blockchair_btc_balance_response.json**: Bitcoin address balance and transaction data from Blockchair API. Contains detailed address information including balance, transaction history, and UTXO data. +- **blockchair_btc_transaction_response.json**: Bitcoin transaction details from Blockchair API. +- **blockchair_doge_balance_response.json**: Dogecoin address balance response in Blockchair format. +- **blockchair_doge_transaction_response.json**: Dogecoin transaction details from Blockchair. +- **blockchair_ltc_balance_response.json**: Litecoin address balance response. +- **blockchair_ltc_transaction_response.json**: Litecoin transaction details. + +#### Trezor/Blockbook Response Files +- **trezor_btc_1_balance_response.json**: Bitcoin balance data from Trezor's Blockbook API (first test case). +- **trezor_btc_2_balance_response.json**: Bitcoin balance data from Trezor's Blockbook API (second test case). +- **trezor_xpub_1_balance_response.json**: Extended public key (xpub) balance data showing hierarchical deterministic wallet information. +- **trezor_xpub_2_balance_response.json**: Second xpub test case with different wallet structure. + +### Ethereum and EVM-Compatible Chain Data + +- **ethplorer_balance_response.json**: Ethereum address balance including ETH and ERC-20 token holdings. Shows detailed token information with prices, decimals, and market data. +- **optimistic_etherscan_balance_response.json**: Optimism network balance data in Etherscan-compatible format. + +### Blockchain Operating System (BOS) Data + +- **bos_balance_response.json**: BOS blockchain balance information. +- **bos_transaction_response.json**: BOS transaction details. +- **bos_operations_response_8t_1.json**: BOS operations data for transaction starting with "8t" (part 1). +- **bos_operations_response_8t_2.json**: BOS operations data for transaction starting with "8t" (part 2). +- **bos_operations_response_r1_1.json**: BOS operations data for transaction starting with "r1" (part 1). +- **bos_operations_response_r1_2.json**: BOS operations data for transaction starting with "r1" (part 2). + +### Polkadot/Substrate Chain Data + +- **subscan_polkadot_response_DCK.json**: Polkadot account data from Subscan API for address ending in "DCK". Contains balance, locks, staking, and proxy information. +- **subscan_polkadot_response_WE9.json**: Polkadot account data for address ending in "WE9". + +### Subdirectories + +#### magiceden/ +NFT marketplace data from Magic Eden (primarily Solana and Bitcoin Ordinals): +- **collection-stats.json**: Collection statistics including floor price, volume, and listed count. +- **listings.json**: Active NFT listings data. +- **offers.json**: Collection offers data. +- **wallet-response.json**: Wallet NFT holdings response. + +#### opensea/ +NFT marketplace data from OpenSea: +- **collection-stats.json**: Detailed collection statistics with volume intervals (daily, weekly, monthly). +- **collection.json**: Collection metadata and configuration. +- **listings-locked.json**: Locked/restricted NFT listings. +- **listings.json**: Standard NFT listings. +- **nfts-next.json**: Paginated NFT data (next page). +- **nfts.json**: NFT holdings and metadata. +- **offers-next.json**: Paginated offers data. +- **offers.json**: Collection and item offers. + +#### simplehash/ +Multi-chain NFT data aggregator responses: +- **collection-activity.json**: Collection trading activity and events. +- **collection.json**: Collection metadata including floor prices from multiple marketplaces. +- **fungibles.json**: Fungible token (ERC-20/SPL) holdings. +- **listings.json**: Cross-marketplace NFT listings. +- **nfts.json**: NFT ownership and metadata. +- **offers.json**: Aggregated offers data. +- **solana-listings-nfts.json**: Solana-specific NFT listings. +- **solana-nfts.json**: Solana NFT holdings. + +#### solana/ +Solana blockchain and ecosystem data: +- **ban-list-jup-ag.csv**: Jupiter aggregator's banned token list with scam/fake tokens. +- **rent_reserve_solana_response.json**: Solana rent reserve calculation data. +- **solana_response.json**: General Solana account/balance data. +- **staked_solana_response.json**: Staked SOL information. +- **token-list-jup-ag.csv**: Jupiter aggregator's supported token list with metadata. +- **token-list-solana.json**: Solana token registry in JSON format. +- **token-list-sonar.json**: Alternative Solana token list from Sonar. + +#### sui/ +Sui blockchain data: +- **response.json**: Sui wallet balance response showing multiple coin types with USD values. + +#### unisat/ +Bitcoin Ordinals marketplace (Unisat) data: +- **collection_info.json**: Basic collection information (holders, items, block height). +- **collection_items.json**: Individual inscription items in a collection. +- **collection_stats.json**: Collection trading statistics. +- **collection_stats_full_url.json**: Stats response with complete URL formatting. +- **collection_stats_v4.json**: Version 4 API stats response. +- **inscription_data.json**: Individual inscription metadata. +- **inscription_data_edge_cases.json**: Edge case inscription data for testing. +- **listings.json**: Ordinals marketplace listings. +- **offers.json**: Collection offers on Unisat. + +## Usage Patterns + +These test data files are used to: +1. **Mock API Responses**: Provide realistic data for unit tests without making actual API calls. +2. **Test Parser Logic**: Ensure response parsing handles various data formats and edge cases. +3. **Integration Testing**: Validate that the blockapi library correctly processes responses from different blockchain data providers. +4. **Edge Case Testing**: Files like `inscription_data_edge_cases.json` specifically test handling of unusual or problematic data. + +## Key Observations + +- **Multi-chain Support**: Data covers Bitcoin, Ethereum, Solana, Polkadot, Sui, and other blockchains. +- **NFT Focus**: Significant emphasis on NFT marketplace data from multiple providers (OpenSea, Magic Eden, SimpleHash, Unisat). +- **Standardized Formats**: Many responses follow similar patterns (balance, transactions, listings) adapted for each blockchain. +- **Real-world Data**: Files contain actual API response structures, ensuring tests reflect production scenarios. \ No newline at end of file diff --git a/blockapi/test/v2/api/debank/STRUCTURE.md b/blockapi/test/v2/api/debank/STRUCTURE.md new file mode 100644 index 00000000..911238ac --- /dev/null +++ b/blockapi/test/v2/api/debank/STRUCTURE.md @@ -0,0 +1,146 @@ +# DeBank API Test Suite + +## Overview + +This directory contains the comprehensive test suite for the DeBank API integration within the blockapi library. DeBank is a Web3 portfolio tracker that provides detailed information about DeFi positions, token balances, and protocol interactions across multiple blockchain networks. The tests ensure proper functionality of API client implementation, data parsing, caching mechanisms, and error handling. + +## Purpose and Functionality + +The test suite validates the complete DeBank integration including: +- API endpoint construction and authentication +- Response parsing for balances, portfolios, protocols, and chains +- Protocol caching mechanisms for performance optimization +- Error handling for invalid responses and edge cases +- Integration testing with real API endpoints +- Comprehensive unit testing of parsers and models + +## File Descriptions + +### conftest.py +Pytest configuration and shared fixtures for the DeBank test suite. Provides: +- API client fixtures with different configurations (all tokens on/off) +- Parser fixtures for balance, portfolio, protocol, and usage parsing +- Mock response fixtures loaded from test data files +- Sample data fixtures for testing specific scenarios +- Protocol cache fixtures for testing caching behavior +- Chain and protocol model fixtures + +### test_debank.py +Core API client tests covering: +- URL building for different endpoints (balance, portfolio, protocols, chains) +- API key authentication in request headers +- Error response handling and logging +- Protocol fetching and caching integration +- Chain data retrieval +- Shared protocol cache across API instances +- Usage statistics retrieval + +### test_debank_balance_parser.py +Balance parser testing including: +- Token balance parsing from API responses +- Coin identification and mapping to internal models +- Protocol association with tokens +- Handling of zero balances and empty responses +- Chain-specific token mapping (ETH on different chains) +- Unknown chain handling +- Decimal precision and amount calculations +- Time parsing for last updated timestamps + +### test_debank_fetch.py +Integration tests for real API calls (marked with `@pytest.mark.integration`): +- Fetching actual balance data from DeBank API +- Retrieving portfolio positions for test addresses +- Fetching protocol and chain lists +- Saving fetched data for future test reference +- Usage statistics retrieval testing + +### test_debank_portfolio_parser.py +Portfolio/pool parsing tests covering: +- Complex DeFi position parsing (lending, borrowing, staking, liquidity) +- Asset type classification and mapping +- Position metadata extraction (health rates, lock times) +- Pool information parsing (IDs, names, tokens) +- Handling duplicate assets +- Unknown chain portfolio handling +- Vesting and time-locked position parsing +- Error handling for malformed data + +### test_debank_protocol_cache.py +Protocol cache functionality tests: +- Cache timeout behavior (3600 seconds) +- Cache update detection +- Protocol retrieval from cache +- Cache invalidation and refresh logic +- Efficient protocol data storage + +### test_debank_protocol_parser.py +Protocol parser testing for: +- Protocol metadata extraction (ID, name, chain) +- TVL (Total Value Locked) parsing +- Logo and site URL extraction +- Portfolio support flag parsing +- Protocol-to-blockchain mapping + +### test_debank_usage_parser.py +API usage statistics parser tests: +- Balance parsing (remaining API credits) +- Daily usage statistics extraction +- Historical usage data parsing +- Date formatting and timezone handling + +### test_pool.py +Pool model functionality tests: +- Pool item aggregation +- Balance item appending to pools +- Pool metadata preservation +- Protocol association with pools + +### data/ +Test data directory containing JSON fixtures for various API responses. See [data/STRUCTURE.md](data/STRUCTURE.md) for detailed information about each test data file and its purpose. + +## Key Testing Patterns + +### Fixture Usage +- Heavy use of pytest fixtures for reusable test data +- Fixtures cascade from conftest.py to provide consistent test environment +- Mock responses loaded from JSON files for predictable testing + +### Parser Testing Strategy +- Each parser (balance, portfolio, protocol, usage) has dedicated test file +- Tests cover both successful parsing and error conditions +- Edge cases like empty responses, unknown chains, and malformed data + +### Integration Points +- `requests_mock` for mocking HTTP requests in unit tests +- Real API calls in integration tests (requires API key) +- Protocol cache shared across test instances +- Parser chain: fetch → parse → model conversion + +## Important Conventions + +### Test Organization +- Unit tests run without external dependencies +- Integration tests marked with `@pytest.mark.integration` +- Test data stored as JSON files in data/ subdirectory +- Fixtures defined in conftest.py for reusability + +### Error Testing +- Comprehensive error response testing +- Logging verification with `caplog` fixture +- Unknown chain and protocol handling +- Malformed response handling + +### Data Validation +- Decimal precision testing for financial amounts +- Timezone-aware datetime parsing +- Address checksum validation +- Chain ID to blockchain mapping verification + +## Dependencies and Integration + +The test suite integrates with: +- DeBank Pro API (requires API key for integration tests) +- blockapi v2 models and coin definitions +- Protocol caching system for performance +- Multiple blockchain definitions and mappings +- Decimal and datetime libraries for precise financial data \ No newline at end of file diff --git a/blockapi/test/v2/api/debank/data/STRUCTURE.md b/blockapi/test/v2/api/debank/data/STRUCTURE.md new file mode 100644 index 00000000..c73a6714 --- /dev/null +++ b/blockapi/test/v2/api/debank/data/STRUCTURE.md @@ -0,0 +1,121 @@ +# DeBank API Test Data + +## Overview + +This directory contains JSON test data files used for testing the DeBank API integration within the blockapi library. DeBank is a Web3 portfolio tracker that provides comprehensive DeFi data across multiple blockchain networks. These test fixtures represent various API responses from DeBank's endpoints, including portfolio positions, token balances, and protocol interactions. + +## Purpose and Functionality + +The test data serves as mock responses for unit testing the DeBank API client implementation, ensuring proper parsing and handling of: +- DeFi protocol portfolio positions +- Token balance responses +- Error conditions and edge cases +- Complex nested data structures from various DeFi protocols + +## File Descriptions + +### aave_portfolio_response.json +Contains a sample response from the Aave V2 lending protocol showing: +- Staked AAVE positions with rewards +- Lending positions (supplied MKR tokens) +- Asset values, APY calculations, and health rates +- Demonstrates the structure for lending/borrowing protocol responses + +### balance_response.json +A comprehensive token balance response showing: +- Multiple token holdings across different chains (Ethereum, BSC, Polygon, Fantom, HECO) +- Token metadata including decimals, logos, prices, and amounts +- Both verified and unverified tokens +- Native chain tokens (ETH, BNB) and ERC-20/BEP-20 tokens + +### bio_pools.json +Portfolio data for the BIO protocol featuring: +- Multiple vesting positions with different indices +- Token amounts, claimable amounts, and vesting end dates +- Demonstrates handling of time-locked/vesting token positions +- Shows negative amounts in asset lists (representing locked tokens) + +### complex_portfolio_response.json +A large, multi-protocol portfolio response containing: +- Positions across multiple DeFi protocols (Trader Joe, Badger DAO, Bao Finance) +- Various position types: lending, staking, liquidity pools, rewards +- Complex nested structures with supply/borrow token lists +- Demonstrates real-world portfolio complexity + +### duplicates.json +Test data for handling duplicate entries, containing: +- Multiple protocol entries with similar structure +- Used to test deduplication logic and data aggregation +- Shows lending positions and locked token scenarios + +### esgmx_portfolio_response.json +GMX protocol positions on Arbitrum showing: +- Escrowed GMX (esGMX) staking and vesting +- Multiple reward tokens (GMX, esGMX, ETH) +- Complex staking mechanics with different pool IDs +- Demonstrates handling of escrowed/locked governance tokens + +### mist_response.json +A simple token balance response for MIST (Alchemist) token: +- Single token entry with full metadata +- Shows minimal response structure +- Useful for testing basic token parsing + +### position_index_portfolio_response.json +Uniswap V3 liquidity positions demonstrating: +- Multiple liquidity positions with position indices +- Token pair pools (PRIME/ETH) +- Supply and reward token lists +- Position-specific metadata and pool information + +### tokenset_portfolio_response.json +TokenSets investment positions showing: +- Leveraged token products (BTC2x-FLI, ETH2x-FLI) +- Negative USDC amounts (representing borrowed positions) +- Complex derivative products +- Investment strategy tokens + +### unauthorized.json +Error response for unauthorized API access: +- HTTP 401 status code +- Error headers including CloudFront and Istio metadata +- Standard error response structure +- Used for testing authentication error handling + +### unknown_chain_response.json +Test data with an invalid blockchain identifier: +- Contains "this-chain-will-never-exist" as chain value +- Tests error handling for unsupported chains +- Otherwise valid portfolio structure + +### usage_response.json +API usage statistics response showing: +- Current balance and daily usage stats +- Historical usage data over 30 days +- Remaining API calls per day +- Used for testing rate limit monitoring + +## Integration Points + +These test files integrate with: +- DeBank API client parsers that convert JSON responses to internal data models +- Error handling mechanisms for API failures +- Chain/protocol mapping logic +- Token price and metadata enrichment +- Portfolio aggregation and calculation engines + +## Testing Patterns + +The test data covers: +1. **Success Cases**: Valid responses with various data structures +2. **Error Cases**: Authentication failures, unknown chains +3. **Edge Cases**: Negative amounts, vesting positions, complex derivatives +4. **Data Variations**: Different protocols, chains, and position types + +## Important Conventions + +- All amounts are provided in both human-readable and raw formats +- Timestamps are Unix timestamps (seconds since epoch) +- Token addresses are checksummed Ethereum addresses +- Chain identifiers follow DeBank's naming convention (eth, bsc, matic, etc.) +- Prices are in USD with high precision decimals \ No newline at end of file diff --git a/blockapi/test/v2/api/nft/STRUCTURE.md b/blockapi/test/v2/api/nft/STRUCTURE.md new file mode 100644 index 00000000..5e28f06b --- /dev/null +++ b/blockapi/test/v2/api/nft/STRUCTURE.md @@ -0,0 +1,126 @@ +# NFT API Test Suite + +This directory contains comprehensive test suites for various NFT (Non-Fungible Token) marketplace APIs integrated into the blockapi v2 system. The tests verify the correct functioning of data fetching, parsing, and error handling for multiple NFT platforms across different blockchains. + +## Overview + +The NFT test suite ensures reliable integration with major NFT marketplaces including MagicEden, OpenSea, SimpleHash, and UniSat. Each test module focuses on a specific marketplace API and validates critical functionality such as: + +- Fetching NFT collections, individual tokens, and wallet holdings +- Parsing marketplace-specific data formats into standardized models +- Handling offers, listings, and collection statistics +- Managing API rate limits and retry conditions +- Processing different blockchain standards (ERC721, Ordinals, Runes) + +## File Structure + +### `__init__.py` +Empty initialization file marking this directory as a Python package. + +### `test_magic_eden.py` +Test suite for the MagicEden Solana API integration. + +**Key Features:** +- Tests NFT fetching for Solana wallets +- Validates collection statistics parsing +- Tests offer and listing data retrieval +- Handles MagicEden-specific retry conditions for service unavailability +- Uses fixture data from `data/magiceden/` directory + +**Test Coverage:** +- `test_parse_nfts`: Validates NFT token parsing from wallet responses +- `test_parse_collection`: Tests collection metadata and statistics +- `test_parse_offers`: Verifies NFT offer (bid) data parsing +- `test_parse_listings`: Tests NFT listing data parsing +- `test_retry_condition`: Ensures proper retry logic for API failures + +### `test_opensea.py` +Comprehensive test suite for OpenSea API integration supporting multiple EVM blockchains. + +**Key Features:** +- Supports Ethereum and other EVM-compatible chains (including BASE) +- Tests pagination with cursor-based navigation +- Validates locked listing filtering +- Handles API authentication and error responses +- Uses fixture data from `data/opensea/` directory + +**Test Coverage:** +- `test_fetch_ntfs`: Tests NFT fetching with pagination +- `test_parse_nfts`: Validates ERC721 NFT parsing +- `test_fetch_offers`: Tests collection offer fetching +- `test_parse_listings`: Validates listing data including locked listings +- `test_parse_collection`: Tests collection statistics across multiple time periods +- `test_supported_blockchains`: Verifies blockchain support + +### `test_simple_hash.py` +Test suite for SimpleHash API covering Bitcoin and Solana blockchains. + +**Key Features:** +- Tests Bitcoin Ordinals and Runes support +- Handles fungible NFT tokens (Runes) +- Tests listed NFTs with marketplace data +- Validates collection volume and floor price data +- Uses fixture data from `data/simplehash/` directory + +**Test Coverage:** +- `test_parse_nfts`: Tests Ordinals NFT parsing with market URLs +- `test_parse_listed_nfts`: Validates Solana NFTs with active listings +- `test_parse_fungible_nfts`: Tests Bitcoin Runes token parsing +- `test_parse_collection`: Validates collection data with floor prices and volumes +- `test_inscriptions_collection`: Special handling for inscription collections +- `test_parse_offers` & `test_parse_listings`: Full offer/listing lifecycle testing + +### `test_unisat.py` +Test suite specifically for UniSat API (Bitcoin NFT marketplace). + +**Key Features:** +- Focuses on Bitcoin NFT types (Ordinals collections) +- Tests icon URL handling (both relative codes and full URLs) +- Validates offer and listing parsing with BTC amounts +- Uses fixture data from `data/unisat/` directory + +**Test Coverage:** +- `test_fetch_collection_icon_code`: Tests automatic CDN URL prefixing for icon codes +- `test_fetch_collection_icon_full_url`: Validates handling of complete icon URLs +- `test_fetch_listings`: Tests listing data with proper BTC decimal conversion +- `test_fetch_offers`: Validates offer filtering by event status + +## Integration Points + +### Dependencies +- **pytest**: Test framework with custom fixtures +- **requests_mock**: HTTP request mocking for API responses +- **FakeSleepProvider**: Custom sleep provider for rate limit testing +- **blockapi.v2.models**: Standardized NFT data models +- **blockapi.v2.coins**: Cryptocurrency definitions (SOL, ETH, BTC, WETH) + +### Data Models Used +- `NftToken`: Individual NFT representation +- `NftCollection`: Collection metadata and statistics +- `NftOffer`: Buy offers and sell listings +- `AssetType`: Token availability status +- `Blockchain`: Chain identifiers +- `NftOfferDirection`: Offer vs Listing classification + +### Test Data Location +All test fixtures read JSON response data from: +- `data/magiceden/`: MagicEden API responses +- `data/opensea/`: OpenSea API responses +- `data/simplehash/`: SimpleHash API responses +- `data/unisat/`: UniSat API responses + +## Testing Patterns + +1. **Fixture-based Testing**: Each test uses pytest fixtures for API clients and response data +2. **Request Mocking**: All HTTP requests are mocked with pre-recorded responses +3. **Comprehensive Parsing**: Tests validate complete data transformation from raw API responses to domain models +4. **Error Handling**: Tests verify proper error propagation and retry logic +5. **Pagination Support**: Tests validate cursor-based pagination where applicable + +## Important Notes + +- All monetary values are converted to `Decimal` for precision +- Timestamps are parsed with timezone awareness (UTC) +- NFT identifiers follow platform-specific formats +- Tests ensure backward compatibility with API changes +- Rate limiting is simulated through FakeSleepProvider \ No newline at end of file diff --git a/blockapi/utils/STRUCTURE.md b/blockapi/utils/STRUCTURE.md new file mode 100644 index 00000000..cd3b5336 --- /dev/null +++ b/blockapi/utils/STRUCTURE.md @@ -0,0 +1,70 @@ +# blockapi/utils Directory Structure + +## Overview + +This directory contains utility functions that provide common functionality for the blockapi library. These utilities handle various aspects of blockchain-related data processing including address validation, datetime parsing, numerical conversions (especially for blockchain decimals/raw values), and HTTP request user agent management. + +## Files + +### `__init__.py` +- **Purpose**: Package initialization file (currently empty) +- **Functionality**: Makes the utils directory a Python package + +### `address.py` +- **Purpose**: Ethereum address validation and formatting utilities +- **Key Functions**: + - `make_checksum_address(address: str) -> Optional[str]`: Converts an Ethereum address to its checksum format using eth_utils + - Returns None if the address is invalid instead of raising an exception +- **Dependencies**: `eth_utils` +- **Usage**: Essential for validating and normalizing Ethereum addresses before blockchain interactions + +### `datetime.py` +- **Purpose**: DateTime parsing utilities for blockchain timestamps +- **Key Functions**: + - `parse_dt(dt: Union[str, int, float]) -> datetime`: Universal datetime parser that handles multiple input formats + - String timestamps (e.g., "1234567890") + - Integer/float timestamps + - ISO format strings and other date formats via dateutil +- **Dependencies**: `dateutil` +- **Usage**: Standardizes datetime handling across different blockchain APIs that return timestamps in various formats + +### `num.py` +- **Purpose**: Numerical conversion utilities optimized for blockchain decimal/raw value conversions +- **Key Functions**: + - `to_int(number: Union[int, str])`: Safe integer conversion + - `to_decimal(number: Union[int, float, str, Decimal]) -> Decimal`: Converts various number types to Decimal (handles float precision issues) + - `raw_to_decimals(raw: Union[int, str], decimals: Union[int, str]) -> Decimal`: Converts raw blockchain values (wei, gwei, satoshi) to decimal format + - `decimals_to_raw(amount: Union[int, str], decimals: Union[int, str]) -> Decimal`: Converts decimal values back to raw format + - `remove_exponent(d: Decimal) -> Decimal`: Removes scientific notation from Decimal values + - `safe_opt_decimal(obj: Optional[SupportsNumber]) -> Decimal`: Safely converts optional numbers to Decimal (None becomes 0) + - `safe_decimal(number: SupportsNumber) -> Decimal`: Duplicate of to_decimal function +- **Key Patterns**: Heavy use of Decimal for precision, essential for handling cryptocurrency values +- **Usage**: Critical for accurate cryptocurrency amount calculations and conversions between different decimal precisions + +### `user_agent.py` +- **Purpose**: HTTP user agent generation for API requests +- **Key Functions**: + - `get_random_user_agent() -> str`: Returns a random user agent string +- **Dependencies**: `fake_useragent` +- **Usage**: Helps avoid rate limiting and blocking when making requests to blockchain APIs and explorers + +## Architecture Patterns + +1. **Error Handling**: The utilities favor returning None or default values rather than raising exceptions (e.g., `make_checksum_address`) +2. **Type Safety**: Extensive use of type hints with Union types to handle multiple input formats +3. **Precision Focus**: Heavy emphasis on Decimal type for numerical operations to avoid floating-point precision issues critical in financial calculations +4. **Timezone Awareness**: All datetime operations use UTC timezone to maintain consistency across different blockchain networks + +## Integration Points + +- These utilities are likely used throughout the blockapi library for: + - Validating user-provided addresses before API calls + - Parsing timestamps from various blockchain explorers + - Converting between different token decimal formats (e.g., ETH has 18 decimals) + - Setting appropriate headers for HTTP requests to blockchain APIs + +## Important Conventions + +- All numerical operations involving cryptocurrency amounts use Decimal type +- Datetime operations always use UTC timezone +- Invalid inputs typically return None rather than raising exceptions (defensive programming) \ No newline at end of file diff --git a/blockapi/v2/STRUCTURE.md b/blockapi/v2/STRUCTURE.md new file mode 100644 index 00000000..676d20e3 --- /dev/null +++ b/blockapi/v2/STRUCTURE.md @@ -0,0 +1,204 @@ +# Blockapi v2 Directory Structure + +## Overview + +The v2 directory contains the next-generation blockchain API library for Crypkit, providing a unified interface for interacting with various blockchains, DeFi protocols, and NFT marketplaces. This version introduces a more modular architecture with standardized data models, improved error handling, and support for over 50 blockchain networks. + +## Architecture + +The v2 library follows several key architectural patterns: + +1. **Interface-Based Design**: Abstract interfaces (`IBalance`, `ITransactions`, `IPortfolio`, `INftProvider`) define standard contracts that implementations must follow +2. **Standardized Models**: Unified data models ensure consistent data format across all blockchain integrations +3. **Inheritance Hierarchy**: Base classes (`BlockchainApi`, `CustomizableBlockchainApi`) provide common functionality +4. **Rate Limiting**: Built-in rate limiting with configurable sleep providers +5. **Error Handling**: Consistent exception types and error propagation + +## Directory Structure + +``` +v2/ +├── __init__.py # Package initialization (empty) +├── base.py # Core base classes and interfaces +├── blockchain_mapping.py # Blockchain identifier mapping utilities +├── coin_mapping.py # Coin/token mapping configurations +├── coins.py # Pre-defined coin/token definitions +├── models.py # Data models and enumerations +└── api/ # API implementations directory + ├── STRUCTURE.md # Detailed API documentation + ├── covalenth/ # Multi-chain token aggregator + ├── nft/ # NFT marketplace integrations + ├── perpetual/ # Perpetual Protocol integration + └── synthetix/ # Synthetix protocol integration +``` + +## Core Files + +### `base.py` +The foundation of the v2 architecture, containing: + +**Key Classes:** +- `ISleepProvider` / `SleepProvider`: Rate limiting through configurable sleep mechanisms +- `CustomizableBlockchainApi`: Abstract base for APIs with customizable endpoints (proxy support, testnets, alternative RPC) +- `BlockchainApi`: Standard blockchain API base class +- `BalanceMixin`: Reusable implementation of balance fetching pattern + +**Core Interfaces:** +- `IBalance`: Contract for balance-fetching implementations +- `ITransactions`: Contract for transaction history retrieval +- `IPortfolio`: Contract for DeFi portfolio data +- `INftProvider` / `INftParser`: Contracts for NFT data fetching and parsing +- `IBlockchainFetcher` / `IBlockchainParser`: Generic fetch/parse pattern interfaces + +**Exception Types:** +- `ApiException`: General API errors +- `InvalidAddressException`: Malformed blockchain addresses + +**Features:** +- HTTP session management with automatic cleanup +- Retry logic for rate limits and connection errors +- Response time tracking from headers +- Configurable JSON parsing arguments +- Built-in error response handling + +### `blockchain_mapping.py` +Utilities for mapping blockchain identifiers across different systems: + +**Mapping Functions:** +- `get_blockchain_from_debank_chain()`: Maps DeBank chain IDs to internal Blockchain enum +- `get_blockchain_from_coingecko_chain()`: Maps CoinGecko platform IDs +- `get_blockchain_from_chain_id()`: Maps EVM chain IDs (e.g., 1 for Ethereum) +- `get_blockchain_from_rango_chain()`: Maps Rango exchange identifiers +- `get_blockchain_from_wormhole_chain()`: Maps Wormhole bridge identifiers + +**Key Mappings:** +- DeBank: 97 blockchain mappings (most comprehensive) +- CoinGecko: 40 platform mappings +- Chain ID: 74 EVM network mappings +- Supports automatic fallback to Blockchain enum values + +### `coin_mapping.py` +Token/coin mapping configurations for specific integrations: + +**Current Mappings:** +- `OPENSEA_COINS`: Maps OpenSea payment token symbols to Coin objects +- `OPENSEA_CONTRACTS`: Maps contract addresses to specific coins (WETH, ETH) + +**Supported OpenSea Tokens:** +- ETH, USDC, PRIME, DAI, MATIC + +### `coins.py` +Pre-defined coin/token definitions used throughout the library: + +**Native Coins Defined:** +- Ethereum: ETH, WETH +- Solana: SOL +- Terra: LUNA +- Polygon: MATIC +- Avalanche: AVAX +- Various other native tokens for 50+ blockchains + +**Coin Properties:** +- Symbol, name, decimals +- Blockchain association +- Contract address (for tokens) +- Metadata: logo URLs, CoinGecko IDs, websites +- Token standards supported + +### `models.py` +Comprehensive data models and enumerations: + +**Key Enumerations:** +- `Blockchain`: 200+ blockchain identifiers +- `AssetType`: Classification of assets (AVAILABLE, STAKED, REWARDS, DEBT, etc.) +- `CoingeckoId`: 1000+ CoinGecko cryptocurrency IDs +- `ProtocolType`: DeFi protocol classifications +- `Chain`: Simplified chain names for specific integrations + +**Core Data Models:** +- `Coin`: Token/coin definition with metadata +- `CoinInfo`: Extended coin metadata (logos, tags, links) +- `BalanceItem`: Standardized balance representation +- `TransactionItem`: Blockchain transaction data +- `Pool`: DeFi pool/position information +- `NftToken`, `NftCollection`, `NftOffer`: NFT-related models +- `FetchResult`, `ParseResult`: API operation results + +**Model Features:** +- Automatic decimal conversion +- DateTime parsing utilities +- JSON serialization support +- Optional field handling +- Nested data structure support + +## API Subdirectory + +The `api/` subdirectory contains all blockchain-specific implementations. See [api/STRUCTURE.md](api/STRUCTURE.md) for detailed information about: +- Native blockchain APIs (Bitcoin, Ethereum, Solana, etc.) +- Multi-chain aggregators (DeBank, Covalenth) +- DeFi protocol integrations (Synthetix, Perpetual) +- NFT marketplace APIs (OpenSea, Magic Eden, etc.) + +### Key Subdirectories: +- **`covalenth/`**: See [api/covalenth/STRUCTURE.md](api/covalenth/STRUCTURE.md) for multi-chain token balance aggregation +- **`nft/`**: See [api/nft/STRUCTURE.md](api/nft/STRUCTURE.md) for NFT marketplace integrations +- **`perpetual/`**: See [api/perpetual/STRUCTURE.md](api/perpetual/STRUCTURE.md) for Perpetual Protocol staking/vesting +- **`synthetix/`**: See [api/synthetix/STRUCTURE.md](api/synthetix/STRUCTURE.md) for Synthetix synthetic assets + +## Usage Patterns + +### Basic Balance Fetching +```python +from blockapi.v2.api import SolanaApi + +api = SolanaApi() +balances = api.get_balance("11111111111111111111111111111111") +``` + +### Custom RPC Endpoint +```python +from blockapi.v2.api import EthereumApi + +api = EthereumApi(base_url="https://mainnet.infura.io/v3/YOUR_KEY") +balances = api.get_balance("0x...") +``` + +### NFT Data Retrieval +```python +from blockapi.v2.api.nft import OpenSeaApi + +api = OpenSeaApi(api_key="...", blockchain=Blockchain.ETHEREUM) +result = api.fetch_nfts("0x...") +parsed = api.parse_nfts(result) +``` + +### DeFi Portfolio +```python +from blockapi.v2.api import DebankApi + +api = DebankApi(api_key="...", is_all=True) +portfolio = api.get_portfolio("0x...") +``` + +## Design Principles + +1. **Modularity**: Each blockchain/protocol has its own implementation file +2. **Consistency**: All APIs follow the same patterns and return standardized models +3. **Extensibility**: New blockchains can be added by implementing base interfaces +4. **Error Resilience**: Graceful degradation with detailed error information +5. **Performance**: Built-in caching, session reuse, and rate limit management + +## Dependencies + +- **Core**: requests, attrs, pydantic +- **Blockchain-Specific**: web3 (Ethereum), solders (Solana) +- **Utilities**: eth_utils, beautifulsoup4, marko +- **Internal**: blockapi.utils for number/datetime utilities + +## Important Notes + +1. Most APIs require API keys for production use +2. Rate limits vary by provider (configured in ApiOptions) +3. Address formats are blockchain-specific and validated +4. All monetary values use Decimal for precision +5. Timestamps are standardized to UTC datetime objects \ No newline at end of file diff --git a/blockapi/v2/api/STRUCTURE.md b/blockapi/v2/api/STRUCTURE.md new file mode 100644 index 00000000..d6b44e96 --- /dev/null +++ b/blockapi/v2/api/STRUCTURE.md @@ -0,0 +1,171 @@ +# Blockapi v2 API Directory + +## Overview + +This directory contains the blockchain API implementations for the blockapi v2 library. It provides unified interfaces for interacting with various blockchain networks, DeFi protocols, NFT marketplaces, and data aggregators. The implementations follow a consistent pattern of inheriting from base classes and implementing standardized interfaces for balance fetching, transaction querying, and specialized blockchain operations. + +## Architecture + +The API implementations follow these key patterns: +- **Inheritance**: All APIs extend from `BlockchainApi` base class and implement specific interfaces (`IBalance`, `ITransactions`, etc.) +- **Standardization**: Unified data models (`BalanceItem`, `TransactionItem`, `NftToken`, etc.) ensure consistent data format across all blockchains +- **Rate Limiting**: Built-in rate limiting to respect API provider limits +- **Error Handling**: Consistent error handling with custom exceptions for invalid addresses and API errors +- **Modularity**: Subdirectories group related implementations (NFT providers, DeFi protocols, multi-chain aggregators) + +## Files and Components + +### Core Module Files + +- **`__init__.py`**: Main module exports file that exposes all API implementations for easy importing. Includes APIs for Bitcoin, Ethereum, Solana, Cosmos ecosystem, and various aggregators. + +- **`web3_utils.py`**: Utility functions for Web3/Ethereum interactions + - `easy_call()`: Simplified smart contract function calling with error handling + - `get_eth_client()`: Web3 client initialization + - `map_struct()`: Maps raw contract results to structured dictionaries + - Address checksum utilities and decorators + +### Native Blockchain APIs + +- **`blockchainos.py`**: BOS (Blockchain Operating System) API implementation + - Fetches BOS token balances and transaction history + - Parses operations within transactions + - Uses REST API at `https://mainnet.blockchainos.org` + +- **`blockchair.py`**: Multi-blockchain explorer API (Bitcoin, Dogecoin, Litecoin) + - Implements `BlockchairApi` base class with specialized implementations + - Supports both regular addresses and extended public keys (xpub) + - Dashboard API for comprehensive account data + - Transaction fetching with UTXO model support + +- **`cosmos.py`**: Cosmos ecosystem API with support for multiple chains + - Base class `CosmosApiBase` with implementations for ATOM, OSMOSIS, DYDX, CELESTIA + - IBC token mapping and native token resolution + - Staking, unbonding, and rewards balance tracking + - Dynamic token data loading from GitHub repositories + +- **`ethplorer.py`**: Ethereum token balance API + - ERC20 token discovery and balance fetching + - Native ETH balance support + - Token metadata including logos, tags, and Coingecko IDs + +- **`optimistic_etherscan.py`**: Optimism L2 Etherscan API + - Simple ETH balance fetching for Optimism network + - Etherscan-compatible API interface + +- **`solana.py`**: Solana blockchain API with two implementations + - `SolanaApi`: Direct RPC implementation with SPL token support + - Metaplex metadata fetching for unknown tokens + - Staked SOL tracking including rent reserves + - Token list integration from multiple sources (Jupiter, Solana official) + - `SolscanApi`: Solscan explorer API for staking data + +- **`subscan.py`**: Polkadot/Kusama ecosystem API + - Abstract `SubscanApi` with implementations for DOT and KSM + - Complex balance types: available, staked, vesting, reserved + - Staking rewards and slashes tracking + - Search API with comprehensive account data + +- **`sui.py`**: Sui blockchain API via Blockberry + - Multi-token balance fetching + - Coin type formatting and metadata + +- **`terra.py`**: Terra blockchain with multiple data sources + - `TerraApi`: Combines FCD and Mantle APIs + - `TerraFcdApi`: Native and IBC token balances, staking data + - `TerraMantleApi`: CW20 token balances via GraphQL + - Dynamic token list loading from Terra assets + +- **`trezor.py`**: Trezor Blockbook API for Bitcoin and Litecoin + - UTXO-based balance and transaction fetching + - Support for both addresses and xpubs + - Transaction details with input/output analysis + +### Multi-Chain Aggregators + +- **`debank.py`**: DeBank multi-chain DeFi aggregator + - Comprehensive DeFi protocol integration + - Portfolio tracking with pools, farming, lending positions + - Protocol metadata caching system + - Complex asset type mapping (staking, vesting, debt, etc.) + - Supports 40+ blockchain networks + +- **`debank_maps.py`**: DeBank configuration and mappings + - Asset type conversions between DeBank and internal models + - Native coin mappings by blockchain and Coingecko ID + - Coingecko ID resolution for multi-chain tokens + +### Subdirectories + +- **`covalenth/`**: Covalenth (formerly Covalent) unified blockchain data API + - See [covalenth/STRUCTURE.md] for detailed information about multi-chain token balance aggregation + +- **`nft/`**: NFT marketplace and aggregator integrations + - See [nft/STRUCTURE.md] for detailed information about NFT APIs (OpenSea, Magic Eden, SimpleHash, Unisat) + +- **`perpetual/`**: Perpetual Protocol DeFi integration + - See [perpetual/STRUCTURE.md] for detailed information about PERP staking and vesting rewards + +- **`synthetix/`**: Synthetix synthetic assets protocol + - See [synthetix/STRUCTURE.md] for detailed information about SNX staking, debt, and rewards + +## Key Integration Patterns + +### Balance Fetching +All balance-fetching APIs implement a two-step process: +1. `fetch_balances()`: Retrieves raw data from the blockchain/API +2. `parse_balances()`: Converts raw data to standardized `BalanceItem` objects + +### Rate Limiting +Each API defines rate limits in `ApiOptions`: +- Default limits range from 0.05 to 20 requests per second +- Automatic throttling prevents API bans +- Some APIs support API keys for higher limits + +### Error Handling +- `InvalidAddressException`: Raised for malformed addresses +- `ApiException`: General API errors with descriptive messages +- Response validation with fallback handling + +### Token Resolution +Multiple strategies for token identification: +- On-chain metadata fetching (Solana Metaplex) +- External token lists (Terra, Solana) +- Contract address mapping (Ethereum tokens) +- IBC denom traces (Cosmos) + +## Common Dependencies + +- `web3`: Ethereum and EVM-compatible chain interactions +- `requests`: HTTP client for REST APIs +- `pydantic`: Data validation for complex response structures +- `eth_utils`: Ethereum address utilities +- `solders`: Solana-specific utilities +- Custom utilities from `blockapi.utils` and `blockapi.v2.base` + +## Usage Examples + +```python +# Native blockchain balance fetching +from blockapi.v2.api import SolanaApi +api = SolanaApi() +balances = api.get_balance("11111111111111111111111111111111") + +# DeFi portfolio aggregation +from blockapi.v2.api import DebankApi +api = DebankApi(api_key="...", is_all=True) +pools = api.get_portfolio("0x...") + +# NFT collection data +from blockapi.v2.api.nft import OpenSeaApi +api = OpenSeaApi(api_key="...") +nfts = api.fetch_nfts("0x...") +``` + +## Important Notes + +1. **API Keys**: Many providers require API keys for production use +2. **Rate Limits**: Respect provider limits to avoid bans +3. **Address Formats**: Each blockchain has specific address format requirements +4. **Token Standards**: Different chains use different token standards (ERC20, SPL, CW20, etc.) +5. **Balance Types**: Distinguish between available, staked, vesting, and locked balances \ No newline at end of file diff --git a/blockapi/v2/api/covalenth/STRUCTURE.md b/blockapi/v2/api/covalenth/STRUCTURE.md new file mode 100644 index 00000000..5f36b4fe --- /dev/null +++ b/blockapi/v2/api/covalenth/STRUCTURE.md @@ -0,0 +1,91 @@ +# Covalenth API Integration + +## Overview + +This directory contains the Covalenth API integration for the blockapi library. Covalenth (formerly Covalent) is a unified blockchain data API that provides access to balance and token information across multiple blockchain networks. The implementation follows a consistent pattern where each blockchain has its own module that inherits from a common base class. + +## Architecture + +The directory implements a clean inheritance pattern where: +- `base.py` contains the abstract base class with core functionality +- Each blockchain-specific file extends the base class with chain-specific configuration +- All implementations share the same API structure and authentication mechanism + +## Files and Purpose + +### Core Implementation + +- **`__init__.py`**: Empty initialization file for the Python package + +- **`base.py`**: Abstract base class that implements the core Covalenth API functionality + - Defines `CovalentApiBase` class that extends `BlockchainApi` and implements `IBalance` + - Implements HTTP Basic authentication using the API key + - Provides `get_balance()` method to fetch token balances for an address + - Handles response parsing and converts raw data to standardized `BalanceItem` objects + - Includes checksum address validation for EVM-compatible chains + - Filters out zero-balance tokens + - Supports ERC token standards detection + +### Blockchain-Specific Implementations + +Each blockchain implementation follows the same pattern, defining: +- `CHAIN_ID`: The unique chain identifier used by Covalenth +- `api_options`: Configuration object with blockchain type, base URL, and rate limits +- `coin`: The native coin/token for that blockchain + +#### EVM-Compatible Chains + +- **`ethereum.py`**: Ethereum mainnet (Chain ID: 1, Native coin: ETH) +- **`binance_smart_chain.py`**: BSC/BNB Chain (Chain ID: 56, Native coin: BNB) +- **`polygon.py`**: Polygon/Matic network (Chain ID: 137, Native coin: MATIC) +- **`avalanche.py`**: Avalanche C-Chain (Chain ID: 43114, Native coin: AVAX) +- **`arbitrum.py`**: Arbitrum One (Chain ID: 42161, Native coin: ETH) +- **`fantom.py`**: Fantom Opera (Chain ID: 250, Native coin: FTM) +- **`heco.py`**: Huobi ECO Chain (Chain ID: 128, Native coin: HT) +- **`moonbeam.py`**: Moonbeam on Polkadot (Chain ID: 1284, Native coin: GLMR) +- **`base.py`**: Base network (Chain ID: 8453, Native coin: ETH) + +#### Other Blockchain Networks + +- **`astar.py`**: Astar Network (Chain ID: 592, Native coin: ASTR) +- **`axie.py`**: Axie Infinity's Ronin chain (Chain ID: 2020, Native coin: RON) +- **`iotex.py`**: IoTeX network (Chain ID: 4689, Native coin: IOTX) +- **`klaytn.py`**: Klaytn network (Chain ID: 8217, Native coin: KLAY) +- **`palm.py`**: Palm network (Chain ID: 11297108109, Native coin: PALM) +- **`rsk.py`**: RSK (Rootstock) network (Chain ID: 30, Native coin: RBTC) + +## Key Features + +1. **Unified API Interface**: All blockchain implementations expose the same `get_balance()` method +2. **Automatic Token Detection**: Discovers and returns all tokens held by an address +3. **Native Coin Handling**: Properly distinguishes between native blockchain coins and tokens +4. **Rate Limiting**: Built-in rate limiting (0.2 requests per second) to respect API limits +5. **Error Handling**: Graceful error handling with logging for debugging +6. **Token Metadata**: Retrieves token names, symbols, decimals, logos, and supported standards + +## Integration Points + +- Inherits from `blockapi.v2.base.BlockchainApi` for common blockchain API functionality +- Uses standardized models from `blockapi.v2.models` (BalanceItem, Coin, CoinInfo) +- References native coins from `blockapi.v2.coins` module +- Implements the `IBalance` interface for balance-fetching capabilities + +## API Usage Pattern + +All implementations follow this pattern: +```python +api = SpecificBlockchainCovalentApi(api_key="your_key") +balances = api.get_balance(address="0x...") +``` + +The returned `balances` list contains `BalanceItem` objects with: +- Token balance (raw and formatted) +- Coin information (symbol, name, decimals, contract address) +- Metadata (logo URL, supported standards) +- Last transfer timestamp + +## Dependencies + +- `eth_utils`: For EVM address checksum validation +- `logging`: For debug and error logging +- Standard blockapi v2 modules for base classes and models \ No newline at end of file diff --git a/blockapi/v2/api/nft/STRUCTURE.md b/blockapi/v2/api/nft/STRUCTURE.md new file mode 100644 index 00000000..480e9080 --- /dev/null +++ b/blockapi/v2/api/nft/STRUCTURE.md @@ -0,0 +1,158 @@ +# NFT API Module Structure + +## Overview +This directory contains NFT (Non-Fungible Token) API implementations for various blockchain marketplaces and aggregators. It provides unified interfaces for fetching NFT data, collections, listings, and offers across multiple platforms including Magic Eden, OpenSea, SimpleHash, and Unisat. + +## Architecture +All NFT API implementations follow a consistent pattern: +- Inherit from `BlockchainApi` base class +- Implement `INftProvider` and `INftParser` interfaces +- Use standardized data models from `blockapi.v2.models` +- Handle rate limiting through sleep providers +- Return data in unified `FetchResult` and `ParseResult` formats + +## Files + +### `__init__.py` +**Purpose**: Package initialization and public API exports +- Exports all NFT API implementations for easy importing +- Provides centralized access to: + - `MagicEdenSolanaApi`: Solana NFT marketplace + - `OpenSeaApi`: Multi-chain NFT marketplace (Ethereum-based) + - `SimpleHashBitcoinApi`: Bitcoin ordinals/inscriptions + - `SimpleHashEthereumApi`: Ethereum and EVM-compatible chains + - `SimpleHashSolanaApi`: Solana NFTs via SimpleHash + - `UnisatApi`: Bitcoin ordinals marketplace + +### `magic_eden.py` +**Purpose**: Magic Eden marketplace API integration for Solana NFTs +- **Key Features**: + - Fetches NFTs by wallet address with pagination + - Retrieves collection statistics and metadata + - Handles listings and pool-based offers + - Implements retry logic for service unavailability +- **API Endpoints**: + - `wallets/{address}/tokens`: Get NFTs owned by address + - `collections/{slug}/stats`: Collection statistics + - `collections/{slug}/listings`: Active listings + - `mmm/pools`: AMM pool offers +- **Rate Limiting**: ~2 requests per second +- **Special Handling**: + - Calculates offer prices including fees (seller, LP, taker) + - Filters duplicate offers by UUID + - Handles service unavailable errors with 60-second retry + +### `opensea.py` +**Purpose**: OpenSea API integration for multi-chain NFT marketplaces +- **Supported Blockchains**: + - Ethereum, Arbitrum, Avalanche, Base, BSC, Klaytn, Optimism, Polygon, Zora +- **Key Features**: + - Paginated NFT fetching with cursor support + - Collection data with floor prices and statistics + - Offers and listings with Seaport protocol parsing + - Multi-chain contract resolution +- **API Endpoints**: + - `chain/{chain}/account/{address}/nfts`: NFTs by owner + - `offers/collection/{collection}/all`: Collection offers + - `listings/collection/{collection}/all`: Active listings + - `collections/{collection}`: Collection metadata + - `collections/{collection}/stats`: Trading statistics +- **Rate Limiting**: 4 requests per second +- **Special Handling**: + - Maps OpenSea-specific blockchain names + - Parses complex Seaport offer structures + - Handles multiple offer items and considerations + - Implements retry-after header support + +### `simple_hash.py` +**Purpose**: SimpleHash API integration - unified NFT data aggregator +- **Three Specialized Implementations**: + 1. **SimpleHashBitcoinApi**: Bitcoin ordinals and inscriptions + - Handles both NFTs and fungibles (Runes) + - Special collection mapping for inscriptions + - Default collection: "inscriptions" + 2. **SimpleHashSolanaApi**: Solana NFTs + - Combines NFT and listing data + - Wallet-specific listings support + 3. **SimpleHashEthereumApi**: Ethereum and EVM chains + - Supports 12+ EVM-compatible blockchains +- **Key Features**: + - Unified API across multiple blockchains + - Rich metadata including floor prices and volumes + - Bid/offer ordering and filtering + - Fungible token support (Runes on Bitcoin) +- **API Endpoints**: + - `nfts/owners`: NFTs by wallet + - `fungibles/balances`: Fungible tokens + - `nfts/collections/ids`: Collection details + - `nfts/bids/collection`: Collection offers + - `nfts/listings/collection`: Active listings +- **Rate Limiting**: 100 requests per second (free tier) +- **Special Handling**: + - Multi-cursor pagination for different data types + - Dynamic coin mapping from payment tokens + - Marketplace URL prioritization (Magic Eden > OpenSea) + +### `unisat.py` +**Purpose**: Unisat API integration for Bitcoin ordinals marketplace +- **Key Features**: + - Bitcoin inscription/ordinal fetching + - Collection statistics and metadata + - Listings and historical offer data + - BRC-20 and domain name support +- **API Endpoints**: + - `indexer/address/{address}/inscription-data`: Inscriptions by owner + - `market/collection/auction/list`: Active listings + - `market/collection/auction/actions`: Historical events + - `market/collection/auction/collection_statistic`: Stats + - `market/collection/auction/collection_summary`: Collection mapping +- **Rate Limiting**: 5 requests per second (free tier) +- **Special Handling**: + - Builds collection mapping from summary endpoint + - Handles "uncategorized-ordinals" as default collection + - Converts icon URLs to full paths + - Dummy result generation for server errors + - Timestamp conversion from milliseconds to ISO format + +## Common Patterns + +### Error Handling +- All APIs return `FetchResult` objects with error arrays +- Parse methods handle missing/malformed data gracefully +- Retry logic for rate limits and service unavailability + +### Data Normalization +- Unified `NftToken` model for all NFT data +- Standardized `NftCollection` with statistics +- Consistent `NftOffer` format for listings/offers +- Price conversions to decimal format + +### Rate Limiting +- Each API respects platform-specific rate limits +- Sleep providers handle request throttling +- Retry-after headers honored (OpenSea) + +### Pagination +- Cursor-based pagination where supported +- Offset/limit pagination fallback +- Maximum offset limits enforced (e.g., 15000 for Magic Eden) + +## Integration Points +- Uses coin mappings from `blockapi.v2.coins` +- Leverages base classes from `blockapi.v2.base` +- Returns standardized models from `blockapi.v2.models` +- Integrates with sleep providers for rate limiting + +## Usage Example +```python +# Initialize API +api = OpenSeaApi(api_key="...", blockchain=Blockchain.ETHEREUM) + +# Fetch NFTs +result = api.fetch_nfts("0x123...") +parsed = api.parse_nfts(result) + +# Get collection data +collection_result = api.fetch_collection("cryptopunks") +collection_data = api.parse_collection(collection_result) +``` \ No newline at end of file diff --git a/blockapi/v2/api/perpetual/STRUCTURE.md b/blockapi/v2/api/perpetual/STRUCTURE.md new file mode 100644 index 00000000..b4ce4747 --- /dev/null +++ b/blockapi/v2/api/perpetual/STRUCTURE.md @@ -0,0 +1,89 @@ +# Perpetual Protocol API Integration + +## Overview + +This directory contains the integration layer for Perpetual Protocol (PERP), a decentralized perpetual futures exchange. The implementation focuses on fetching and parsing staking rewards, vesting rewards, and claimable balances from the Perpetual Protocol's smart contracts and off-chain storage. + +## Files + +### `__init__.py` +Simple module initialization file that exports the main API class and utility functions: +- Exports `PerpetualApi` - the main API class for interacting with Perpetual Protocol +- Exports `perp_contract_address` - utility function to retrieve contract addresses + +### `perp_abi.py` +Contains the Ethereum smart contract ABI (Application Binary Interface) for the rewards contract: +- Defines `rewards_abi` - a comprehensive ABI for the MerkleRedeem contract +- Includes functions for claiming rewards, verifying claims, and managing merkle roots +- Key functions include: + - `claimWeek` / `claimWeeks` - for claiming rewards for specific periods + - `claimStatus` - check claim status for a liquidity provider + - `verifyClaim` - verify merkle proof for reward claims + - `seedAllocations` - admin function for setting up weekly allocations + +### `perpetual.py` +The main implementation file containing all business logic for interacting with Perpetual Protocol: + +#### Key Components: + +1. **Utility Functions**: + - `perp_contract_address()` - Retrieves contract addresses for PERP, sPERP, staking rewards, and vesting rewards + - `perp_contracts()` - Fetches contract metadata from Perpetual Protocol's metadata API + +2. **PerpOffChainStorage Class**: + - Handles interaction with AWS S3-hosted off-chain storage + - Methods: + - `get_epoch_snapshots()` - Fetches weekly snapshots for immediate or vesting rewards + - `get_rewards()` - Retrieves detailed reward data for a specific epoch + +3. **PerpProtocol Class**: + - Core protocol interaction logic + - Handles reward calculations and claim status checks + - Methods: + - `fetch()` - Main entry point returning all reward types + - `_fetch_staking_claimable_rewards()` - Gets immediately claimable staking rewards + - `_fetch_staking_vesting_rewards()` - Gets both claimable and locked vesting rewards + - `_fetch_non_claimed_snapshots()` - Filters out already claimed rewards + - `_get_total_reward()` - Aggregates rewards across multiple epochs + +4. **PerpetualApi Class**: + - Extends `CustomizableBlockchainApi` and `BalanceMixin` + - Implements the standard blockapi interface + - Methods: + - `fetch_balances()` - Fetches all reward balances for an address + - `parse_balances()` - Converts raw data to standardized BalanceItem objects + - `yield_balances()` - Generates balance items for claimable and vesting amounts + +## Architecture Patterns + +### Data Flow +1. API calls start with `PerpetualApi.fetch_balances()` +2. Creates a `PerpProtocol` instance to handle the actual data fetching +3. Retrieves snapshots from off-chain storage (AWS S3) +4. Checks on-chain claim status via Web3 smart contract calls +5. Calculates unclaimed rewards by cross-referencing snapshots with claim status +6. Returns standardized balance data + +### Key Dependencies +- **web3.py** - For Ethereum blockchain interactions +- **requests** - For HTTP calls to metadata and S3 endpoints +- **blockapi.v2.base** - Base classes for API implementation +- **blockapi.v2.coins** - Coin definitions (COIN_PERP) +- **blockapi.v2.models** - Data models for balances and results + +### Integration Points +- **Perpetual Protocol Metadata API**: `https://metadata.perp.exchange/production.json` +- **AWS S3 Storage**: `https://s3.amazonaws.com/staking.perp.fi/production` +- **Ethereum Smart Contracts**: + - Staking Rewards: `0xc2a9e84D77f4B534F049b593C282c5c91F24808A` + - Vesting Rewards: `0x49a4B8431Fc24BE4b22Fb07D1683E2c52bC56088` + +## Important Notes + +1. **Caching**: The implementation uses `@lru_cache` decorators to cache contract metadata and reward data, reducing API calls +2. **Reward Types**: Handles three types of rewards: + - Immediately claimable staking rewards + - Claimable vesting rewards (past redemption date) + - Locked vesting rewards (future redemption date) +3. **Merkle Proof System**: The protocol uses merkle trees for efficient reward distribution verification +4. **Time-based Logic**: Vesting rewards become claimable after their `redeemableUntil` timestamp passes \ No newline at end of file diff --git a/blockapi/v2/api/synthetix/STRUCTURE.md b/blockapi/v2/api/synthetix/STRUCTURE.md new file mode 100644 index 00000000..c03bf82b --- /dev/null +++ b/blockapi/v2/api/synthetix/STRUCTURE.md @@ -0,0 +1,108 @@ +# Synthetix API Integration + +## Overview + +This directory contains the blockchain API integration for Synthetix, a decentralized synthetic asset issuance protocol built on Ethereum and Optimism. The integration allows fetching balances, staking information, rewards, and collateralization data for Synthetix users across both Ethereum mainnet and Optimism L2. + +## Directory Structure + +``` +synthetix/ +├── __init__.py # Module exports and public API +├── synthetix.py # Core implementation of Synthetix API +└── synthetix_abi.py # Smart contract ABI definitions +``` + +## Files and Components + +### `__init__.py` +Exports the main API classes and utility functions for external use: +- `SynthetixApi` - Base abstract class for Synthetix integrations +- `SynthetixMainnetApi` - Ethereum mainnet implementation +- `SynthetixOptimismApi` - Optimism L2 implementation +- `snx_contract_address` - Contract address resolver for mainnet +- `snx_optimism_contract_address` - Contract address resolver for Optimism + +### `synthetix.py` +Core implementation containing: + +**Key Classes:** +- `SynthetixApi` - Abstract base class implementing `CustomizableBlockchainApi` and `IBalance` + - Handles SNX token balances, staking positions, debt, rewards, and vesting + - Supports both Ethereum mainnet and Optimism L2 networks +- `SynthetixMainnetApi` - Ethereum mainnet-specific implementation +- `SynthetixOptimismApi` - Optimism L2-specific implementation + +**Data Types:** +- `CollateralizationStats` - Collateralization ratio information +- `WeeklyReward` - Exchange and staking rewards structure +- `Staking` - Complete staking position including collateral, debt, rewards +- `Synth` - Synthetic asset metadata (symbol and contract address) + +**Key Functions:** +- `snx_contract_address()` - Dynamically resolves Synthetix contract names to addresses +- `snx_optimism_contract_address()` - Optimism-specific contract resolver using GitHub docs + +**Core Functionality:** +- Balance fetching for SNX and synthetic assets (sUSD, etc.) +- Staking information retrieval including: + - Transferable (unlocked) SNX + - Total debt owed in sUSD + - Staked/collateral amounts + - Vesting/escrowed SNX + - Weekly rewards (exchange fees and staking rewards) + - Liquidation rewards + - Collateralization ratios +- Support for multiple asset types: AVAILABLE, STAKED, DEBT, REWARDS, PRICED_VESTING, LIQUIDATION_REWARDS + +### `synthetix_abi.py` +Contains Ethereum smart contract ABI definitions for interacting with Synthetix contracts: +- `synthetix_abi` - Main Synthetix contract ABI +- `feepool_abi` - Fee pool contract for rewards distribution +- `exchangerates_abi` - Exchange rates oracle contract +- `system_settings_abi` - System configuration parameters +- `erc20_abi` - Standard ERC20 token interface +- `rewards_escrow_v2_abi` - Vesting/escrow contract for SNX rewards +- `liquidator_rewards_abi` - Liquidation rewards contract + +## Integration Architecture + +The Synthetix integration follows the blockapi v2 pattern: +1. Inherits from `CustomizableBlockchainApi` for standardized blockchain API structure +2. Implements `IBalance` interface for balance fetching capabilities +3. Uses Web3.py for direct smart contract interactions +4. Leverages caching (`@lru_cache`) for expensive contract calls +5. Supports multiple networks through polymorphic design + +## Key Dependencies + +- `web3` - Ethereum blockchain interaction +- `requests` - HTTP requests for contract address resolution +- `beautifulsoup4` - HTML parsing for Optimism contract addresses +- `marko` - Markdown parsing for documentation scraping +- Parent blockapi modules: + - `blockapi.v2.base` - Base classes and interfaces + - `blockapi.v2.api.web3_utils` - Web3 utility functions + - `blockapi.utils.num` - Decimal conversion utilities + +## Usage Pattern + +The API is designed to be instantiated with a network type and API URL: +```python +# Mainnet +api = SynthetixMainnetApi(api_url="https://mainnet.infura.io/v3/...") + +# Optimism +api = SynthetixOptimismApi(api_url="https://optimism-mainnet.infura.io/v3/...") + +# Fetch all balances +balances = api.get_balance("0x...address...") +``` + +## Important Implementation Details + +1. **Contract Address Resolution**: Uses dynamic resolution via synthetix.io redirect URLs for mainnet and GitHub markdown parsing for Optimism +2. **Balance Types**: Distinguishes between transferable, staked, vesting, debt, and reward balances +3. **Collateralization**: Calculates collateralization ratios and staking amounts based on system issuance ratios +4. **Decimal Handling**: All amounts use 18 decimal precision consistent with SNX token +5. **Error Handling**: Raises `ValueError` when contracts cannot be found \ No newline at end of file