docs for foundation infra, updated infra api, addendum for ast layer docs

Author: NSHkr

docs/docs_module2_ast/ADDENDEUM_20250604.md

@@ -0,0 +1,161 @@
+Okay, this is an important clarification. The "Infrastructure Layer Design Document" (`10_gemini_plan.md`) describes new components that will reside **within the Foundation Layer (Layer 1)**, specifically under `lib/elixir_scope/foundation/infrastructure/`. It's not a separate numbered layer in the 9-layer model but rather an enhancement *to* Layer 1.
+
+The AST Layer (Layer 2) consumes services from Layer 1. Therefore, enhancements to Layer 1 (like the new Infrastructure components) will primarily benefit Layer 2 *indirectly* by making the Layer 1 services it depends on more resilient and performant. Layer 2 components will generally not call, for example, `ElixirScope.Foundation.Infrastructure.CircuitBreakerWrapper.execute/3` directly. Instead, a Layer 1 service like `ElixirScope.Foundation.Events` (which Layer 2 *does* use) would internally use these infrastructure wrappers to protect its own operations (e.g., writing to a database).
+
+However, there are specific points of integration and considerations for the AST Layer. Let's create a technical addendum to the AST Layer documentation reflecting these.
+
+---
+
+## Technical Addendum: AST Layer Integration with Enhanced Foundation Layer
+
+**Document Version:** 1.1 (AST Layer)
+**Date:** May 2025
+**Context:** This addendum updates the AST Layer (Layer 2) technical documentation to reflect the integration of and impact from the new Infrastructure sub-layer within the Foundation Layer (Layer 1), as detailed in `docs/FOUNDATION_OTP_IMPLEMENT_NOW/10_gemini_plan.md`.
+
+**Primary Impact Points:**
+1.  **Enhanced Reliability of Foundation Services:** AST Layer components can expect greater stability and resilience from the Foundation services they consume (e.g., `ElixirScope.Foundation.Config`, `ElixirScope.Foundation.Events`).
+2.  **Coordination with Foundation Infrastructure Services:** AST Layer's own management components (e.g., `AST.Repository.MemoryManager`) may need to coordinate or report to new Foundation-level infrastructure services (e.g., `Foundation.Infrastructure.MemoryManager`, `Foundation.Infrastructure.HealthAggregator`).
+3.  **Refined Error Handling Expectations:** AST Layer components should be prepared to handle potentially new error types surfaced by Foundation services that are now using infrastructure protections (e.g., an error indicating a circuit breaker is open in a Foundation service).
+4.  **Telemetry and Observability:** AST Layer components can contribute more detailed telemetry, knowing that the Foundation Layer has enhanced capabilities for its collection and analysis (via `Foundation.Infrastructure.PerformanceMonitor`).
+
+---
+
+### Revisions and Considerations for AST Layer Documentation:
+
+#### 1. Document: `AST_TECH_SPEC.md`
+
+*   **Section 1: Architecture Overview**
+    *   **Current Consideration:** General dependency on Foundation Layer.
+    *   **Impact of Enhanced L1:** The Foundation Layer is now internally more robust due to its own infrastructure components (circuit breakers, rate limiters, connection pools).
+    *   **Revised Consideration / Action Item:**
+        *   Update text to note that L1 services consumed by L2 (Config, Events, Utils) are now built on a more resilient internal infrastructure, leading to higher expected reliability for L2 interactions with L1.
+
+*   **Section 2.1: Repository System (`core.ex`, `enhanced.ex`)**
+    *   **Current Consideration:** `Repository.Core` is a GenServer managing ETS tables. `enhanced.ex` builds on this.
+    *   **Impact of Enhanced L1:**
+        *   If `Repository.Core` or `Enhanced` were to perform operations that could benefit from protection (e.g., complex state computations that could hang, or hypothetical writes to an external metadata store not covered by `EventStore`), they *could* internally use `ElixirScope.Foundation.Infrastructure.execute_protected/2`.
+        *   However, for standard ETS operations, this is likely overkill and adds unnecessary overhead. ETS is already highly performant and local.
+    *   **Revised Consideration / Action Item:**
+        *   Primarily, these components benefit from more reliable `ConfigServer` access if they fetch dynamic configurations.
+        *   No direct changes to their API for L1 infra, but internal implementation *could* adopt it for specific complex, state-altering `handle_call/cast` implementations if deemed necessary for self-protection, though this is less common for primarily ETS-bound GenServers.
+        *   Ensure `health_check` and `get_statistics` (from `MODULE_INTERFACES_DOCUMENTATION.md` for `Repository.Core`) are robust and detailed enough for consumption by `Foundation.Infrastructure.HealthAggregator` and `PerformanceMonitor`.
+
+*   **Section 2.1.3: Memory Manager (`memory_manager/`)**
+    *   **Current Consideration:** AST Layer has its own `MemoryManager` subsystem for its ETS tables.
+    *   **Impact of Enhanced L1:** Foundation Layer will have `ElixirScope.Foundation.Infrastructure.MemoryManager` for global/system-wide memory pressure.
+    *   **Revised Consideration / Action Item:**
+        *   The AST Layer's `MemoryManager.Monitor` should not only monitor AST-specific ETS tables but also consider subscribing to or querying the `Foundation.Infrastructure.MemoryManager` for global memory pressure signals.
+        *   `AST.Repository.MemoryManager.PressureHandler` should incorporate global pressure level information from the Foundation's `MemoryManager` into its decision-making for AST cache trimming, compression, etc.
+        *   A clear protocol for interaction/coordination between L1 and L2 memory managers needs to be defined (e.g., L2 MM reports its usage to L1 MM; L1 MM can signal L2 MM to take action).
+        *   Update `MODULE_INTERFACES_DOCUMENTATION.md` for `AST.MemoryManager.*` to reflect this potential coordination.
+
+*   **Section 2.4: Query System (`executor.ex`)**
+    *   **Current Consideration:** Executes queries against repository data (ETS).
+    *   **Impact of Enhanced L1:**
+        *   If query execution becomes a source of high load or involves potentially slow/complex computations (beyond simple ETS lookups), the `Query.Executor` could internally use `Foundation.Infrastructure.RateLimiter` to protect itself from too many concurrent complex queries, or `CircuitBreakerWrapper` if a query involves a risky computation.
+    *   **Revised Consideration / Action Item:**
+        *   For now, assume queries are primarily fast ETS reads. If future query types become computationally expensive or state-altering within the Executor GenServer, consider applying infrastructure protections internally to the `Executor`'s `handle_call` for those specific query types.
+        *   The `Query.Cache` can be an important part of this.
+
+*   **Section 6: Integration Points > Foundation Layer Integration**
+    *   **Current Consideration:** Lists `DataAccess`, `Utils`, `Config`.
+    *   **Impact of Enhanced L1:** New infrastructure components exist within L1.
+    *   **Revised Consideration / Action Item:**
+        *   Add a note: "The Foundation Layer now includes an `ElixirScope.Foundation.Infrastructure` sub-layer providing resilience patterns. While AST Layer components typically consume higher-level Foundation services (Config, Events), these services are now internally more robust. Direct use of `Foundation.Infrastructure` components by AST Layer is possible for specific advanced use cases but should be carefully evaluated for necessity."
+
+*   **Section 7: Implementation Guidelines > Error Handling & Monitoring**
+    *   **Current Consideration:** General guidelines.
+    *   **Impact of Enhanced L1:** L1 provides more structured error types from its infra and better monitoring.
+    *   **Revised Consideration / Action Item:**
+        *   AST Layer components should be prepared to handle specific `ElixirScope.Foundation.Types.Error` instances returned by L1 services that might indicate underlying infrastructure issues (e.g., circuit breaker open, rate limited).
+        *   AST Layer components should emit detailed telemetry that can be consumed by `Foundation.Infrastructure.PerformanceMonitor`. Standardize AST telemetry event names/payloads for this purpose.
+
+#### 2. Document: `MODULE_INTERFACES_DOCUMENTATION.md`
+
+*   **Section 1.1: Core Repository Interface (`ElixirScope.AST.Repository.Core`)**
+    *   **Current API:** `health_check(pid())`, `get_statistics(pid())`.
+    *   **Impact of Enhanced L1:** These will be consumed by `Foundation.Infrastructure.HealthAggregator` and `PerformanceMonitor`.
+    *   **Revised Consideration / Action Item:**
+        *   Ensure `health_check` returns a standardized format, e.g., `{:ok, %{status: :healthy | :degraded, details: map()}}` or `{:error, reason}`.
+        *   Ensure `get_statistics` provides metrics relevant for performance monitoring (e.g., ETS table sizes, hit/miss rates if applicable, queue lengths for its GenServer messages).
+
+*   **Section 8: Memory Management Interfaces (e.g., `ElixirScope.AST.Repository.MemoryManager.Monitor`)**
+    *   **Current API:** e.g., `get_memory_usage_report()`, `get_pressure_level()`.
+    *   **Impact of Enhanced L1:** Need to coordinate with `Foundation.Infrastructure.MemoryManager`.
+    *   **Revised Consideration / Action Item:**
+        *   Add functions or mechanisms for `AST.MemoryManager.Monitor` to report its specific memory usage (for AST data) to `Foundation.Infrastructure.MemoryManager`.
+        *   `AST.Repository.MemoryManager.PressureHandler` might subscribe to notifications from `Foundation.Infrastructure.MemoryManager` or periodically query its global pressure state to influence AST-specific cleanup actions.
+        *   The `handle_pressure_level/1` in `AST.PressureHandler` should take global context into account.
+
+#### 3. Document: `ETS_SCHEMA.md`
+
+*   **Section 8: Memory Management**
+    *   **Current Consideration:** Describes AST-specific LRU eviction and memory pressure response.
+    *   **Impact of Enhanced L1:** These local strategies now operate within a system that has a global `Foundation.Infrastructure.MemoryManager`.
+    *   **Revised Consideration / Action Item:**
+        *   The `ElixirScope.AST.MemoryPressure.handle_memory_pressure/1` logic should be updated to potentially be triggered or influenced by signals from `Foundation.Infrastructure.MemoryManager` in addition to its own local monitoring. For example, if Foundation L1 signals "critical pressure," AST's Memory Manager might trigger more aggressive cleanup than its local thresholds would indicate.
+
+#### 4. Document: `SUPERVISION_TREE.md`
+
+*   **Section 8: Error Recovery Patterns**
+    *   **Current Consideration:** Mentions an AST-local `ElixirScope.AST.CircuitBreaker`.
+    *   **Impact of Enhanced L1:** Foundation now provides `ElixirScope.Foundation.Infrastructure.CircuitBreakerWrapper`.
+    *   **Revised Consideration / Action Item:**
+        *   Clarify the role of any *AST-local* circuit breakers versus the *Foundation-level* ones.
+        *   AST-local CBs would be for protecting internal computational flows or non-critical internal tasks within Layer 2.
+        *   If an AST component needs to call an external service (which should be rare, usually L1 handles this), it *should* use `ElixirScope.Foundation.Infrastructure.CircuitBreakerWrapper.execute/3`.
+        *   More commonly, AST components benefit because the L1 services they call (e.g., `ConfigServer` hypothetically calling an external persistence layer for its config) are *already protected* by Foundation's CBs.
+        *   The `ElixirScope.AST.RecoveryPatterns.handle_repository_failure/1` might involve checking the health of underlying Foundation services before attempting complex recovery, as L1 issues could be the root cause.
+
+#### 5. Documents: `REQ-*.md` (e.g., `REQ-01-CORE-REPOSITORY.md`, `REQ-04-ADVANCED-FEATURES.md`)
+
+*   **General Impact:** Non-functional requirements related to reliability, performance, and monitoring can be met with higher confidence or to a greater degree due to the enhanced Foundation Layer.
+*   **`REQ-01-CORE-REPOSITORY.md`:**
+    *   **FR-1.4 (Lifecycle Management - Monitoring):** `Repository.Core` statistics collection can be more effectively utilized by `Foundation.Infrastructure.PerformanceMonitor`.
+*   **`REQ-04-ADVANCED-FEATURES.md`:**
+    *   **FR-4.2 (Memory Management System):** The description of this system *within AST* needs to clearly define its interaction and responsibility boundaries with `Foundation.Infrastructure.MemoryManager`. It should not duplicate efforts but rather manage AST-specific data structures based on its own heuristics *and* global signals from L1.
+    *   **NFR-4.5 (System Reliability - Fault tolerance):** The "graceful handling of memory exhaustion" will be a joint effort between L2's Memory Manager and L1's global Memory Manager.
+
+---
+
+### New Integration Patterns & API Considerations for AST Layer (Layer 2)
+
+1.  **Health Reporting to Foundation:**
+    *   Key AST GenServers (e.g., `Repository.Core`, `PatternMatcher.Core`, `Query.Executor`, `AST.MemoryManager.Monitor`, `FileWatcher.Core`) MUST implement a `health_check/0` function or respond to a `:health_check` call.
+    *   This health check should return `{:ok, details_map}` or `{:error, reason_map}`.
+    *   `Foundation.Infrastructure.HealthAggregator` will be configured to call these endpoints.
+    *   **Action Item:** Define a standardized health check response format/behaviour for ElixirScope services.
+
+2.  **Performance Telemetry for Foundation:**
+    *   AST components performing significant work (parsing, complex queries, pattern matching) SHOULD emit detailed telemetry events.
+    *   Event names should follow a convention allowing `Foundation.Infrastructure.PerformanceMonitor` to easily subscribe and aggregate them (e.g., `[:elixir_scope, :ast, :parser, :parse_file_duration]`, `[:elixir_scope, :ast, :query, :execute_duration]`).
+    *   Payloads should include relevant metadata (e.g., file size for parsing, query complexity for queries).
+    *   **Action Item:** Define standard AST-specific telemetry events and payloads.
+
+3.  **Memory Usage Reporting to Foundation:**
+    *   `ElixirScope.AST.Repository.MemoryManager.Monitor` SHOULD periodically report the memory usage specifically consumed by AST's ETS tables and caches to `Foundation.Infrastructure.MemoryManager`.
+    *   This allows the L1 `MemoryManager` to have a global view including L2's significant contribution.
+    *   **Action Item:** Define an API on `Foundation.Infrastructure.MemoryManager` for other layers/components to report their specialized memory consumption.
+
+4.  **Consuming Global Memory Pressure Signals:**
+    *   `ElixirScope.AST.Repository.MemoryManager.PressureHandler` SHOULD be able to subscribe to or poll `Foundation.Infrastructure.MemoryManager` for global memory pressure level updates.
+    *   When L1 signals high/critical pressure, L2's `PressureHandler` must trigger more aggressive local cleanup within AST's data stores, even if its own local thresholds haven't been met.
+    *   **Action Item:** Define the mechanism (callback, subscription, polling) for L2 MM to receive global pressure signals from L1 MM.
+
+5.  **Direct Use of Foundation Infrastructure (Limited Cases):**
+    *   While uncommon, if an AST component *itself* makes external calls or manages a pool of unique, long-lived, non-GenServer resources (highly unlikely for AST's domain), it *could* directly use `Foundation.Infrastructure.CircuitBreakerWrapper` or `ConnectionManager`.
+    *   Example: If `AST.Parsing.Parser` had to fetch some schema from a remote URL (not current design), that call would be wrapped.
+    *   **Action Item:** General guidance: AST components should rely on L1 services to handle external interactions. Direct use of L1 Infrastructure by L2 needs strong justification.
+
+---
+
+**Conclusion:**
+
+The enhancements to the Foundation Layer (Layer 1) with a dedicated Infrastructure sub-layer significantly bolster the overall robustness and manageability of the ElixirScope platform. For the AST Layer (Layer 2), this primarily translates to:
+
+*   **Increased reliability** of the L1 services it depends upon.
+*   New **coordination points** with L1's infrastructure services for global concerns like memory management and health aggregation.
+*   The **API contract for L2 consuming L1 services like `Config`, `Events`, `Telemetry` remains stable**.
+*   The **new `ElixirScope.Foundation.Infrastructure` API is primarily for L1 internal use** or very specific, advanced L2 scenarios.
+
+This addendum provides the necessary pointers to update AST Layer documentation and consider these integration points during its ongoing development and refinement. The AST Layer's own internal mechanisms for memory management, caching, and error handling remain important but will now operate within a more globally aware and resilient system.