Skip to content

Commit 1e8d6f6

Browse files
rotarurrotarur
committed
docs: create DEVELOPMENT.md and reorganize documentation
- Add comprehensive DEVELOPMENT.md with local development and testing guide - Move detailed development documentation from README.md to DEVELOPMENT.md - Add reference to DEVELOPMENT.md in main README.md - Include Dockerfile.dev usage, testing workflows, and troubleshooting - Maintain clean separation between user and developer documentation
1 parent 88085f7 commit 1e8d6f6

File tree

2 files changed

+282
-0
lines changed

2 files changed

+282
-0
lines changed

‎DEVELOPMENT.md‎

Lines changed: 276 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,276 @@
1+
# Development Guide
2+
3+
This document provides comprehensive guidance for local development, testing, and contributing to the Graph Protocol Indexer project.
4+
5+
## Local Development and Testing
6+
7+
The project includes a `Dockerfile.dev` for consistent local development and testing environments. This is particularly useful for testing performance improvements and ensuring compatibility across different systems.
8+
9+
### Prerequisites
10+
11+
- [Podman](https://podman.io/) or [Docker](https://docker.com/) installed
12+
- Git (for cloning the repository)
13+
- At least 4GB of available RAM
14+
15+
### Building the Development Image
16+
17+
```bash
18+
# Build the development image
19+
podman build -f Dockerfile.dev -t indexer-dev:latest .
20+
21+
# Or with Docker
22+
docker build -f Dockerfile.dev -t indexer-dev:latest .
23+
```
24+
25+
### Testing Performance Improvements Locally
26+
27+
1. **Mount your local project and run tests:**
28+
```bash
29+
# Test the complete build
30+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
31+
32+
# Test individual packages
33+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && yarn compile"
34+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && yarn compile"
35+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-cli && yarn compile"
36+
```
37+
38+
2. **Test the new CLI flag:**
39+
```bash
40+
# Verify the new flag is available
41+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help | grep -A 5 'indexer-min-stake-threshold'"
42+
```
43+
44+
3. **Run TypeScript type checking:**
45+
```bash
46+
# Check specific files for type errors
47+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && tsc --noEmit src/subgraphs.ts"
48+
```
49+
50+
### Interactive Development
51+
52+
```bash
53+
# Start an interactive shell in the container
54+
podman run --rm -it -v $(pwd):/opt/indexer indexer-dev:latest bash
55+
56+
# Inside the container, you can:
57+
cd /opt/indexer
58+
yarn install # Install dependencies
59+
yarn compile # Build all packages
60+
yarn test # Run tests
61+
```
62+
63+
### Environment Variables for Testing
64+
65+
The development image supports the same environment variables as the production build:
66+
67+
```bash
68+
# Test with custom batch sizes
69+
podman run --rm -v $(pwd):/opt/indexer -e INDEXER_DEPLOYMENT_BATCH_SIZE=1000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
70+
71+
# Test with custom stake thresholds
72+
podman run --rm -v $(pwd):/opt/indexer -e INDEXER_MIN_STAKE_THRESHOLD=5000000000000000000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
73+
```
74+
75+
### Troubleshooting
76+
77+
- **Build failures**: Ensure you have sufficient RAM (4GB+) and disk space
78+
- **Permission issues**: On some systems, you may need to use `sudo` with podman/docker commands
79+
- **Volume mount issues**: Ensure the current directory path is correct and accessible
80+
- **Dependency issues**: The image includes `yarn install` to ensure all dependencies are properly resolved
81+
82+
### Performance Testing
83+
84+
To test the performance improvements with large datasets:
85+
86+
```bash
87+
# Test compilation performance
88+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && time yarn compile"
89+
90+
# Test individual package compilation
91+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && time tsc --noEmit"
92+
```
93+
94+
## Project Structure
95+
96+
The project is organized as a monorepo using [Lerna](https://lerna.js.org/) and [Yarn workspaces](https://classic.yarnpkg.com/en/docs/workspaces/):
97+
98+
```
99+
packages/
100+
├── indexer-agent/ # Main indexer agent service
101+
├── indexer-cli/ # Command-line interface
102+
└── indexer-common/ # Shared utilities and types
103+
```
104+
105+
## Development Workflow
106+
107+
### 1. Setup Development Environment
108+
109+
```bash
110+
# Clone the repository
111+
git clone <repository-url>
112+
cd indexer
113+
114+
# Install dependencies
115+
yarn install
116+
117+
# Build the development image
118+
podman build -f Dockerfile.dev -t indexer-dev:latest .
119+
```
120+
121+
### 2. Make Changes
122+
123+
- Edit files in the appropriate package
124+
- Follow the existing code style and patterns
125+
- Add tests for new functionality
126+
127+
### 3. Test Your Changes
128+
129+
```bash
130+
# Test compilation
131+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
132+
133+
# Run tests
134+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test"
135+
136+
# Test specific functionality
137+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help"
138+
```
139+
140+
### 4. Commit and Push
141+
142+
```bash
143+
# Add your changes
144+
git add .
145+
146+
# Commit with a descriptive message
147+
git commit -m "feat: description of your changes"
148+
149+
# Push to your branch
150+
git push origin your-branch-name
151+
```
152+
153+
## Performance Optimization Features
154+
155+
The indexer agent includes several performance optimizations for handling large numbers of subgraph deployments:
156+
157+
### Batching and Filtering
158+
- **Deployment Batching**: Processes deployments in configurable batches (default: 500) to prevent event loop blocking
159+
- **Stake Threshold Filtering**: Automatically filters out deployments below a minimum stake/signal threshold to reduce processing overhead
160+
- **Rule Lookup Optimization**: Uses O(1) Map-based lookups instead of O(N) linear scans for indexing rules
161+
162+
### Configuration Options
163+
- `--indexer-min-stake-threshold`: Set minimum stake amount in wei (default: 1 GRT = 1000000000000000000 wei)
164+
- `INDEXER_DEPLOYMENT_BATCH_SIZE`: Environment variable for batch size (default: 500)
165+
166+
### Use Cases
167+
These optimizations are particularly beneficial when:
168+
- Processing 10,000+ subgraph deployments
169+
- Managing complex indexing rule sets
170+
- Running on resource-constrained environments
171+
- Requiring consistent response times during high-load periods
172+
173+
## Testing
174+
175+
### Running Tests Locally
176+
177+
To run the tests locally, you'll need:
178+
1. Docker installed and running
179+
2. Node.js and Yarn
180+
3. An Arbitrum Sepolia testnet RPC provider (e.g., Infura, Alchemy)
181+
4. An API key from The Graph Studio for querying subgraphs
182+
183+
#### Setup
184+
185+
1. Create a `.env` file in the root directory with your credentials. You can copy the example file as a template:
186+
```sh
187+
cp .env.example .env
188+
```
189+
190+
Then edit `.env` with your credentials:
191+
```plaintext
192+
# Your Arbitrum Sepolia testnet RPC endpoint
193+
INDEXER_TEST_JRPC_PROVIDER_URL=https://sepolia.infura.io/v3/your-project-id
194+
195+
# Your API key from The Graph Studio (https://thegraph.com/studio/)
196+
INDEXER_TEST_API_KEY=your-graph-api-key-here
197+
```
198+
199+
2. Run the tests:
200+
```sh
201+
bash scripts/run-tests.sh
202+
```
203+
204+
The script will:
205+
- Start a PostgreSQL container with the required test configuration
206+
- Load your credentials from the `.env` file
207+
- Run the test suite
208+
- Clean up the PostgreSQL container when done
209+
210+
### Using Docker for Testing
211+
212+
```bash
213+
# Run tests in the development container
214+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test"
215+
216+
# Run specific test suites
217+
podman run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && yarn test"
218+
```
219+
220+
## Contributing
221+
222+
### Code Style
223+
224+
- Follow the existing TypeScript patterns
225+
- Use meaningful variable and function names
226+
- Add JSDoc comments for public APIs
227+
- Ensure all tests pass before submitting
228+
229+
### Pull Request Process
230+
231+
1. Create a feature branch from `main`
232+
2. Make your changes following the development workflow
233+
3. Ensure all tests pass
234+
4. Update documentation if needed
235+
5. Submit a pull request with a clear description
236+
237+
### Performance Considerations
238+
239+
When making changes that affect performance:
240+
- Test with realistic data sizes (10K+ deployments)
241+
- Use the development container for consistent testing
242+
- Measure performance impact before and after changes
243+
- Consider the O(N×M) complexity implications
244+
245+
## Troubleshooting Common Issues
246+
247+
### Build Issues
248+
249+
**Problem**: `tsc: command not found`
250+
**Solution**: Ensure TypeScript is installed globally in the container or use `yarn tsc`
251+
252+
**Problem**: `lerna: command not found`
253+
**Solution**: Ensure Lerna is installed globally in the container
254+
255+
**Problem**: Dependency resolution errors
256+
**Solution**: Run `yarn install` in the container to ensure proper workspace linking
257+
258+
### Runtime Issues
259+
260+
**Problem**: Container can't mount volumes
261+
**Solution**: Check file permissions and ensure the path is accessible
262+
263+
**Problem**: Insufficient memory during build
264+
**Solution**: Increase container memory limits or use `--memory` flag
265+
266+
**Problem**: Port conflicts
267+
**Solution**: Use different ports or stop conflicting services
268+
269+
## Resources
270+
271+
- [The Graph Protocol Documentation](https://thegraph.com/docs/)
272+
- [Lerna Documentation](https://lerna.js.org/)
273+
- [Yarn Workspaces](https://classic.yarnpkg.com/en/docs/workspaces/)
274+
- [TypeScript Documentation](https://www.typescriptlang.org/)
275+
- [Docker Documentation](https://docs.docker.com/)
276+
- [Podman Documentation](https://podman.io/getting-started/)

‎README.md‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -278,6 +278,12 @@ These optimizations are particularly beneficial when:
278278
- Running on resource-constrained environments
279279
- Requiring consistent response times during high-load periods
280280
281+
## Development
282+
283+
For comprehensive development and testing guidance, see [DEVELOPMENT.md](./DEVELOPMENT.md).
284+
285+
The project includes a `Dockerfile.dev` for consistent local development and testing environments, particularly useful for testing performance improvements and ensuring compatibility across different systems.
286+
281287
## Terraform & Kubernetes
282288
283289
The [terraform/](./terraform/) and [k8s/](./k8s) directories provide a

0 commit comments

Comments
 (0)