Skip to content

Commit 4d84c09

Browse files
DaMandal0rianDaMandal0rian
committed
Merge branch 'rotarur/batching' into test/new-optimizations-merged
2 parents 27cb401 + 1257042 commit 4d84c09

File tree

8 files changed

+667
-52
lines changed

8 files changed

+667
-52
lines changed

‎DEVELOPMENT.md‎

Lines changed: 288 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,288 @@
1+
# Development Guide
2+
3+
This document provides comprehensive guidance for local development, testing, and contributing to the Graph Protocol Indexer project.
4+
5+
## Local Development and Testing
6+
7+
The project includes a `Dockerfile.dev` for consistent local development and testing environments. This is particularly useful for testing performance improvements and ensuring compatibility across different systems.
8+
9+
### Prerequisites
10+
11+
- [Docker](https://docker.com/) installed (or [Podman](https://podman.io/) as an alternative)
12+
- Git (for cloning the repository)
13+
- At least 4GB of available RAM
14+
15+
### Building the Development Image
16+
17+
```bash
18+
# Build the development image
19+
docker build -f Dockerfile.dev -t indexer-dev:latest .
20+
21+
# Note: You can also use Podman as a drop-in replacement for Docker
22+
# podman build -f Dockerfile.dev -t indexer-dev:latest .
23+
```
24+
25+
### Testing Performance Improvements Locally
26+
27+
**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`.
28+
29+
1. **Mount your local project and run tests:**
30+
```bash
31+
# Test the complete build
32+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
33+
34+
# Test individual packages
35+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && yarn compile"
36+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && yarn compile"
37+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-cli && yarn compile"
38+
```
39+
40+
2. **Test the new CLI flag:**
41+
```bash
42+
# Verify the new flag is available
43+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help | grep -A 5 'indexer-min-stake-threshold'"
44+
```
45+
46+
3. **Run TypeScript type checking:**
47+
```bash
48+
# Check specific files for type errors
49+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && tsc --noEmit src/subgraphs.ts"
50+
```
51+
52+
### Interactive Development
53+
54+
**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`.
55+
56+
```bash
57+
# Start an interactive shell in the container
58+
docker run --rm -it -v $(pwd):/opt/indexer indexer-dev:latest bash
59+
60+
# Inside the container, you can:
61+
cd /opt/indexer
62+
yarn install # Install dependencies
63+
yarn compile # Build all packages
64+
yarn test # Run tests
65+
```
66+
67+
### Environment Variables for Testing
68+
69+
**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`.
70+
71+
The development image supports the same environment variables as the production build:
72+
73+
```bash
74+
# Test with custom batch sizes
75+
docker run --rm -v $(pwd):/opt/indexer -e INDEXER_DEPLOYMENT_BATCH_SIZE=1000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
76+
77+
# Test with custom stake thresholds
78+
docker run --rm -v $(pwd):/opt/indexer -e INDEXER_MIN_STAKE_THRESHOLD=5000000000000000000 indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
79+
```
80+
81+
### Troubleshooting
82+
83+
- **Build failures**: Ensure you have sufficient RAM (4GB+) and disk space
84+
- **Permission issues**: On some systems, you may need to use `sudo` with docker commands
85+
- **Volume mount issues**: Ensure the current directory path is correct and accessible
86+
- **Dependency issues**: The image includes `yarn install` to ensure all dependencies are properly resolved
87+
88+
### Performance Testing
89+
90+
**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`.
91+
92+
To test the performance improvements with large datasets:
93+
94+
```bash
95+
# Test compilation performance
96+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && time yarn compile"
97+
98+
# Test individual package compilation
99+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-common && time tsc --noEmit"
100+
```
101+
102+
## Project Structure
103+
104+
The project is organized as a monorepo using [Lerna](https://lerna.js.org/) and [Yarn workspaces](https://classic.yarnpkg.com/en/docs/workspaces/):
105+
106+
```
107+
packages/
108+
├── indexer-agent/ # Main indexer agent service
109+
├── indexer-cli/ # Command-line interface
110+
└── indexer-common/ # Shared utilities and types
111+
```
112+
113+
## Development Workflow
114+
115+
### 1. Setup Development Environment
116+
117+
```bash
118+
# Clone the repository
119+
git clone <repository-url>
120+
cd indexer
121+
122+
# Install dependencies
123+
yarn install
124+
125+
# Build the development image
126+
docker build -f Dockerfile.dev -t indexer-dev:latest .
127+
```
128+
129+
### 2. Make Changes
130+
131+
- Edit files in the appropriate package
132+
- Follow the existing code style and patterns
133+
- Add tests for new functionality
134+
135+
### 3. Test Your Changes
136+
137+
```bash
138+
# Test compilation
139+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn compile"
140+
141+
# Run tests
142+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test"
143+
144+
# Test specific functionality
145+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && node bin/graph-indexer-agent start --help"
146+
```
147+
148+
### 4. Commit and Push
149+
150+
```bash
151+
# Add your changes
152+
git add .
153+
154+
# Commit with a descriptive message
155+
git commit -m "feat: description of your changes"
156+
157+
# Push to your branch
158+
git push origin your-branch-name
159+
```
160+
161+
## Performance Optimization Features
162+
163+
The indexer agent includes several performance optimizations for handling large numbers of subgraph deployments:
164+
165+
### Batching and Filtering
166+
- **Deployment Batching**: Processes deployments in configurable batches (default: 500) to prevent event loop blocking
167+
- **Stake Threshold Filtering**: Automatically filters out deployments below a minimum stake/signal threshold to reduce processing overhead
168+
- **Rule Lookup Optimization**: Uses O(1) Map-based lookups instead of O(N) linear scans for indexing rules
169+
170+
### Configuration Options
171+
- `--indexer-min-stake-threshold`: Set minimum stake amount in wei (default: 1 GRT = 1000000000000000000 wei)
172+
- `INDEXER_DEPLOYMENT_BATCH_SIZE`: Environment variable for batch size (default: 500)
173+
174+
### Use Cases
175+
These optimizations are particularly beneficial when:
176+
- Processing 10,000+ subgraph deployments
177+
- Managing complex indexing rule sets
178+
- Running on resource-constrained environments
179+
- Requiring consistent response times during high-load periods
180+
181+
## Testing
182+
183+
### Running Tests Locally
184+
185+
To run the tests locally, you'll need:
186+
1. Docker installed and running
187+
2. Node.js and Yarn
188+
3. An Arbitrum Sepolia testnet RPC provider (e.g., Infura, Alchemy)
189+
4. An API key from The Graph Studio for querying subgraphs
190+
191+
#### Setup
192+
193+
1. Create a `.env` file in the root directory with your credentials. You can copy the example file as a template:
194+
```sh
195+
cp .env.example .env
196+
```
197+
198+
Then edit `.env` with your credentials:
199+
```plaintext
200+
# Your Arbitrum Sepolia testnet RPC endpoint
201+
INDEXER_TEST_JRPC_PROVIDER_URL=https://sepolia.infura.io/v3/your-project-id
202+
203+
# Your API key from The Graph Studio (https://thegraph.com/studio/)
204+
INDEXER_TEST_API_KEY=your-graph-api-key-here
205+
```
206+
207+
2. Run the tests:
208+
```sh
209+
bash scripts/run-tests.sh
210+
```
211+
212+
The script will:
213+
- Start a PostgreSQL container with the required test configuration
214+
- Load your credentials from the `.env` file
215+
- Run the test suite
216+
- Clean up the PostgreSQL container when done
217+
218+
### Using Docker for Testing
219+
220+
**Note**: All `docker` commands in this section can be used with Podman by simply replacing `docker` with `podman`.
221+
222+
```bash
223+
# Run tests in the development container
224+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer && yarn test"
225+
226+
# Run specific test suites
227+
docker run --rm -v $(pwd):/opt/indexer indexer-dev:latest bash -c "cd /opt/indexer/packages/indexer-agent && yarn test"
228+
```
229+
230+
## Contributing
231+
232+
### Code Style
233+
234+
- Follow the existing TypeScript patterns
235+
- Use meaningful variable and function names
236+
- Add JSDoc comments for public APIs
237+
- Ensure all tests pass before submitting
238+
239+
### Pull Request Process
240+
241+
1. Create a feature branch from `main`
242+
2. Make your changes following the development workflow
243+
3. Ensure all tests pass
244+
4. Update documentation if needed
245+
5. Submit a pull request with a clear description
246+
247+
### Performance Considerations
248+
249+
When making changes that affect performance:
250+
- Test with realistic data sizes (10K+ deployments)
251+
- Use the development container for consistent testing
252+
- Measure performance impact before and after changes
253+
- Consider the O(N×M) complexity implications
254+
255+
## Troubleshooting Common Issues
256+
257+
### Build Issues
258+
259+
**Problem**: `tsc: command not found`
260+
**Solution**: Ensure TypeScript is installed globally in the container or use `yarn tsc`
261+
262+
**Problem**: `lerna: command not found`
263+
**Solution**: Ensure Lerna is installed globally in the container
264+
265+
**Problem**: Dependency resolution errors
266+
**Solution**: Run `yarn install` in the container to ensure proper workspace linking
267+
268+
### Runtime Issues
269+
270+
**Problem**: Container can't mount volumes
271+
**Solution**: Check file permissions and ensure the path is accessible
272+
273+
**Problem**: Insufficient memory during build
274+
**Solution**: Increase container memory limits or use `--memory` flag
275+
276+
**Problem**: Port conflicts
277+
**Solution**: Use different ports or stop conflicting services
278+
279+
**Note**: If you're using Podman instead of Docker, replace `docker` with `podman` in all commands. Podman is a drop-in replacement for Docker and supports the same command-line interface.
280+
281+
## Resources
282+
283+
- [The Graph Protocol Documentation](https://thegraph.com/docs/)
284+
- [Lerna Documentation](https://lerna.js.org/)
285+
- [Yarn Workspaces](https://classic.yarnpkg.com/en/docs/workspaces/)
286+
- [TypeScript Documentation](https://www.typescriptlang.org/)
287+
- [Docker Documentation](https://docs.docker.com/)
288+
- [Podman Documentation](https://podman.io/getting-started/) (alternative to Docker)

‎Dockerfile.dev‎

Lines changed: 44 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
########################################################################
2+
# Development build image for testing performance improvements
3+
########################################################################
4+
5+
FROM node:20.11-bookworm-slim
6+
7+
ENV NODE_ENV development
8+
ENV NODE_OPTIONS="--max-old-space-size=4096"
9+
10+
RUN apt-get update && apt-get install -y \
11+
python3 \
12+
build-essential \
13+
git \
14+
curl \
15+
&& rm -rf /var/lib/apt/lists/*
16+
17+
WORKDIR /opt/indexer
18+
19+
# Copy package files first for better Docker layer caching
20+
COPY package.json .
21+
COPY yarn.lock .
22+
COPY tsconfig.json .
23+
COPY lerna.json .
24+
25+
# Copy all packages
26+
COPY packages/ ./packages/
27+
28+
# Install all dependencies including dev dependencies
29+
RUN yarn --frozen-lockfile --non-interactive --production=false
30+
31+
# Install lerna and typescript globally for the build commands
32+
RUN npm install -g lerna typescript
33+
34+
# Install dependencies for all packages to ensure proper resolution
35+
RUN yarn install
36+
37+
# Build the packages
38+
RUN yarn compile || npx lerna run compile || echo "Build completed with possible warnings"
39+
40+
# Expose port for indexer management
41+
EXPOSE 8000
42+
43+
# Default command for development
44+
CMD ["bash"]

0 commit comments

Comments
 (0)