Implement scenario 07 combined (#210)

rid3thespiral · claude · web-flow · commit 4c9276a52e4d · 2025-11-25T11:09:08.000+01:00
Co-authored-by: Claude &lt;noreply@anthropic.com&gt;
diff --git a/.github/workflows/nightly_tests.yaml b/.github/workflows/nightly_tests.yaml
@@ -491,6 +491,14 @@ jobs:
         continue-on-error: false
         timeout-minutes: 2
 
+      - name: Run Chaos Test Scenario 07 (Combined Failures)
+        run: |
+          set -o pipefail
+          mkdir -p /tmp/teranode-test-results
+          go test -v ./test/chaos -run TestScenario07 2>&1 | tee /tmp/teranode-test-results/chaostest-scenario-07-results.txt
+        continue-on-error: false
+        timeout-minutes: 2
+
       - name: Upload chaostest results
         if: always()
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
diff --git a/test/chaos/README.md b/test/chaos/README.md
@@ -53,6 +53,9 @@ go test -v ./test/chaos/...
 
 # Scenario 6: Slow Close Connections (Slicer)
 ./test/chaos/run_scenario_06.sh
+
+# Scenario 7: Combined Failures (DB + Kafka)
+./test/chaos/run_scenario_07.sh
 ```
 
 The helper scripts will:
@@ -82,6 +85,9 @@ go test -v ./test/chaos -run TestScenario05
 
 # Scenario 6: Slow Close Connections (Slicer)
 go test -v ./test/chaos -run TestScenario06
+
+# Scenario 7: Combined Failures (DB + Kafka)
+go test -v ./test/chaos -run TestScenario07
 ```
 
 ### Run in Verbose Mode
@@ -409,6 +415,116 @@ go test -v ./test/chaos -run TestScenario06_KafkaSlowClose
 
 **Combined scenario duration:** ~55 seconds
 
+### Scenario 7: Combined Failures (3 variants)
+**File:** `scenario_07_combined_failures_test.go`
+
+Tests system behavior when multiple dependencies fail simultaneously or in sequence. This simulates realistic infrastructure-wide issues like datacenter problems, network partitions, or cascading failures.
+
+#### Variant A: Simultaneous Complete Failure
+**Test:** `TestScenario07_SimultaneousFailure`
+
+**What it tests:**
+- System behavior when both PostgreSQL AND Kafka fail at the same time
+- Failure detection when multiple dependencies down
+- Graceful degradation (errors, not crashes)
+- Simultaneous recovery of both services
+- Data consistency after dual failure
+
+**How to run:**
+```bash
+# Using helper script
+./test/chaos/run_scenario_07.sh
+
+# Using go test directly
+go test -v ./test/chaos -run TestScenario07_SimultaneousFailure
+```
+
+**Test phases:**
+1. Establish baseline with both services healthy
+2. Disable both PostgreSQL and Kafka simultaneously (complete failure)
+3. Test behavior during simultaneous outage
+4. Restore both services simultaneously
+5. Verify recovery and data consistency
+
+**Expected results:**
+- ✅ Baseline: Both services healthy and functional
+- ✅ Simultaneous failure: Both fail quickly and cleanly (no hangs)
+- ✅ During outage: Errors returned promptly (not timeouts or crashes)
+- ✅ Recovery: Both services restored successfully
+- ✅ Consistency: No data corruption from dual failure
+
+**Test duration:** ~10 seconds
+
+#### Variant B: Simultaneous Latency
+**Test:** `TestScenario07_SimultaneousLatency`
+
+**What it tests:**
+- System behavior when both PostgreSQL AND Kafka become slow simultaneously
+- Performance degradation when multiple dependencies affected
+- System remains functional despite infrastructure-wide slowdown
+- Recovery when latency removed from both
+
+**How to run:**
+```bash
+# Using helper script
+./test/chaos/run_scenario_07.sh
+
+# Using go test directly
+go test -v ./test/chaos -run TestScenario07_SimultaneousLatency
+```
+
+**Test phases:**
+1. Measure baseline performance (both services fast)
+2. Inject 500ms latency to both services simultaneously
+3. Test performance under simultaneous latency
+4. Remove latency and verify recovery
+
+**Expected results:**
+- ✅ Baseline: Fast operations on both services
+- ✅ With latency: Both services slower but still functional
+- ✅ Operations complete successfully despite 500ms delay
+- ✅ No cascading timeouts or failures
+- ✅ Recovery: Performance returns to baseline levels
+
+**Test duration:** ~15 seconds
+
+#### Variant C: Staggered Recovery
+**Test:** `TestScenario07_StaggeredRecovery`
+
+**What it tests:**
+- System behavior when services recover at different times
+- Partial functionality when one service up, one down
+- No cascading failures during staggered recovery
+- Data consistency with asynchronous recovery
+
+**How to run:**
+```bash
+# Using helper script
+./test/chaos/run_scenario_07.sh
+
+# Using go test directly
+go test -v ./test/chaos -run TestScenario07_StaggeredRecovery
+```
+
+**Test phases:**
+1. Disable both PostgreSQL and Kafka simultaneously
+2. Restore PostgreSQL first (Kafka still down)
+3. Verify PostgreSQL works while Kafka remains down
+4. Wait 3 seconds, then restore Kafka
+5. Verify both services healthy
+6. Confirm data consistency after staggered recovery
+
+**Expected results:**
+- ✅ Both services fail cleanly when disabled
+- ✅ PostgreSQL recovers independently while Kafka down
+- ✅ System handles partial recovery gracefully
+- ✅ Kafka recovers after delay with no issues
+- ✅ No data corruption from staggered recovery pattern
+
+**Test duration:** ~10 seconds
+
+**Combined scenario duration:** ~35 seconds
+
 ## Test Structure
 
 Each chaos test follows this pattern:
@@ -581,7 +697,8 @@ Chaos tests take longer than unit tests:
 - Scenario 4C (Load Under Failures): ~28 seconds (load testing under failures)
 - Scenario 5 (Bandwidth Constraints): ~4.4 seconds (database + Kafka bandwidth tests)
 - Scenario 6 (Slow Close Connections): ~55 seconds (slicer toxic tests)
-- Full suite: ~12-13 minutes (with all scenarios)
+- Scenario 7 (Combined Failures): ~35 seconds (simultaneous and staggered failures)
+- Full suite: ~13-14 minutes (with all scenarios)
 
 ## Troubleshooting
 
@@ -655,4 +772,6 @@ curl -X POST http://localhost:8474/reset
 - [x] Scenario 4C: Load Under Failures ✅ **Implemented**
 - [x] Scenario 5: Bandwidth Constraints ✅ **Implemented**
 - [x] Scenario 6: Slow Close Connections (Slicer toxic) ✅ **Implemented**
-- [ ] Scenario 7: Combined Failures (DB + Kafka simultaneously)
+- [x] Scenario 7: Combined Failures (DB + Kafka simultaneously) ✅ **Implemented**
+
+All planned chaos test scenarios have been implemented!
diff --git a/test/chaos/run_scenario_07.sh b/test/chaos/run_scenario_07.sh
@@ -0,0 +1,119 @@
+#!/bin/bash
+
+# Chaos Test Scenario 07: Combined Failures (DB + Kafka simultaneously)
+# This script runs the combined failures chaos test with proper setup validation
+
+set -e
+
+# Colors for output
+RED='\033[0;31m'
+GREEN='\033[0;32m'
+YELLOW='\033[1;33m'
+NC='\033[0m' # No Color
+
+# Configuration
+TOXIPROXY_POSTGRES_URL="http://localhost:8474"
+TOXIPROXY_KAFKA_URL="http://localhost:8475"
+POSTGRES_DIRECT_URL="localhost:5432"
+POSTGRES_TOXI_URL="localhost:15432"
+KAFKA_DIRECT_URL="localhost:9092"
+KAFKA_TOXI_URL="localhost:19092"
+COMPOSE_FILE="../compose/docker-compose-ss.yml"
+
+echo -e "${GREEN}[INFO]${NC} Checking if toxiproxy services are running..."
+
+# Check if toxiproxy-postgres is available
+if curl -s -f "${TOXIPROXY_POSTGRES_URL}/version" > /dev/null 2>&1; then
+    echo -e "${GREEN}[INFO]${NC} Toxiproxy for PostgreSQL is available at ${TOXIPROXY_POSTGRES_URL}"
+else
+    echo -e "${RED}[ERROR]${NC} Toxiproxy for PostgreSQL is not accessible at ${TOXIPROXY_POSTGRES_URL}"
+    echo -e "${GREEN}[INFO]${NC} Check with: docker ps | grep toxiproxy-postgres"
+    exit 1
+fi
+
+# Check if toxiproxy-kafka is available
+if curl -s -f "${TOXIPROXY_KAFKA_URL}/version" > /dev/null 2>&1; then
+    echo -e "${GREEN}[INFO]${NC} Toxiproxy for Kafka is available at ${TOXIPROXY_KAFKA_URL}"
+else
+    echo -e "${RED}[ERROR]${NC} Toxiproxy for Kafka is not accessible at ${TOXIPROXY_KAFKA_URL}"
+    echo -e "${GREEN}[INFO]${NC} Check with: docker ps | grep toxiproxy-kafka"
+    exit 1
+fi
+
+# Check if PostgreSQL is accessible
+echo -e "${GREEN}[INFO]${NC} Checking PostgreSQL..."
+if nc -z localhost 5432 2>/dev/null; then
+    echo -e "${GREEN}[INFO]${NC} PostgreSQL is accessible"
+else
+    echo -e "${RED}[ERROR]${NC} PostgreSQL is not accessible on ${POSTGRES_DIRECT_URL}"
+    echo -e "${GREEN}[INFO]${NC} Check with: docker ps | grep postgres"
+    exit 1
+fi
+
+# Check if PostgreSQL through toxiproxy is accessible
+echo -e "${GREEN}[INFO]${NC} Checking PostgreSQL through toxiproxy..."
+if nc -z localhost 15432 2>/dev/null; then
+    echo -e "${GREEN}[INFO]${NC} PostgreSQL through toxiproxy is accessible"
+else
+    echo -e "${RED}[ERROR]${NC} PostgreSQL through toxiproxy is not accessible on ${POSTGRES_TOXI_URL}"
+    exit 1
+fi
+
+# Check if Kafka is accessible
+echo -e "${GREEN}[INFO]${NC} Checking Kafka..."
+if nc -z localhost 9092 2>/dev/null; then
+    echo -e "${GREEN}[INFO]${NC} Kafka is accessible"
+else
+    echo -e "${RED}[ERROR]${NC} Kafka is not accessible on ${KAFKA_DIRECT_URL}"
+    echo -e "${GREEN}[INFO]${NC} Check with: docker ps | grep kafka"
+    exit 1
+fi
+
+# Check if Kafka through toxiproxy is accessible
+echo -e "${GREEN}[INFO]${NC} Checking Kafka through toxiproxy..."
+if nc -z localhost 19092 2>/dev/null; then
+    echo -e "${GREEN}[INFO]${NC} Kafka through toxiproxy is accessible"
+else
+    echo -e "${RED}[ERROR]${NC} Kafka through toxiproxy is not accessible on ${KAFKA_TOXI_URL}"
+    exit 1
+fi
+
+# Reset toxiproxy services to clean state
+echo -e "${GREEN}[INFO]${NC} Resetting toxiproxy services to clean state..."
+curl -s -X DELETE "${TOXIPROXY_POSTGRES_URL}/proxies/postgres/toxics" > /dev/null || true
+curl -s -X DELETE "${TOXIPROXY_KAFKA_URL}/proxies/kafka/toxics" > /dev/null || true
+curl -s -X POST "${TOXIPROXY_POSTGRES_URL}/proxies/postgres" -H "Content-Type: application/json" -d '{"enabled": true}' > /dev/null || true
+curl -s -X POST "${TOXIPROXY_KAFKA_URL}/proxies/kafka" -H "Content-Type: application/json" -d '{"enabled": true}' > /dev/null || true
+
+echo ""
+echo -e "${GREEN}[INFO]${NC} =================================================="
+echo -e "${GREEN}[INFO]${NC} Running Scenario 7: Combined Failures"
+echo -e "${GREEN}[INFO]${NC} =================================================="
+echo ""
+
+# Run the test
+go test -v ./test/chaos -run TestScenario07
+
+TEST_EXIT_CODE=$?
+
+echo ""
+if [ $TEST_EXIT_CODE -eq 0 ]; then
+    echo -e "${GREEN}[INFO]${NC} =================================================="
+    echo -e "${GREEN}[INFO]${NC} ✅ Test PASSED"
+    echo -e "${GREEN}[INFO]${NC} =================================================="
+else
+    echo -e "${RED}[ERROR]${NC} =================================================="
+    echo -e "${RED}[ERROR]${NC} ❌ Test FAILED"
+    echo -e "${RED}[ERROR]${NC} =================================================="
+fi
+echo ""
+
+# Cleanup: Reset toxiproxy to clean state
+echo -e "${GREEN}[INFO]${NC} Cleaning up toxiproxy state..."
+curl -s -X DELETE "${TOXIPROXY_POSTGRES_URL}/proxies/postgres/toxics" > /dev/null || true
+curl -s -X DELETE "${TOXIPROXY_KAFKA_URL}/proxies/kafka/toxics" > /dev/null || true
+curl -s -X POST "${TOXIPROXY_POSTGRES_URL}/proxies/postgres" -H "Content-Type: application/json" -d '{"enabled": true}' > /dev/null || true
+curl -s -X POST "${TOXIPROXY_KAFKA_URL}/proxies/kafka" -H "Content-Type: application/json" -d '{"enabled": true}' > /dev/null || true
+echo -e "${GREEN}[INFO]${NC} Cleanup complete"
+
+exit $TEST_EXIT_CODE
diff --git a/test/chaos/scenario_07_combined_failures_test.go b/test/chaos/scenario_07_combined_failures_test.go
