update 0.1.1

Author: nshkrdotcom

README.md

@@ -57,15 +57,15 @@ mix pipeline.run evolved_pipelines/sentiment_analyzer_*.yaml
 defp deps do
   [
     # From Hex.pm (recommended)
-    {:pipeline_ex, "~> 0.1.0"}
+    {:pipeline_ex, "~> 0.1.1"}
 
     # Or from GitHub
-    # {:pipeline_ex, git: "https://github.com/nshkrdotcom/pipeline_ex.git", tag: "v0.1.0"}
+    # {:pipeline_ex, git: "https://github.com/nshkrdotcom/pipeline_ex.git", tag: "v0.1.1"}
   ]
 end
 ```
 
-> **⚠️ Breaking Change in v0.1.0:** Async streaming system removed. ClaudeCodeSDK already streams messages optimally - the custom buffering system added unnecessary complexity. See `docs/ASYNC_STREAMING_DEPRECATION.md` for details.
+> **⚠️ Breaking Change in v0.1.0** Async streaming system removed. ClaudeCodeSDK already streams messages optimally - the custom buffering system added unnecessary complexity. See `docs/ASYNC_STREAMING_DEPRECATION.md` for details.
 
 ### Simple API
 
@@ -761,4 +761,4 @@ TEST_MODE=live elixir run_example.exs  # All live
 
 ## License
 
-TODO: Add license information
+TODO: Add license information

docs/ASYNC_STREAMING_MIGRATION_GUIDE.md

@@ -0,0 +1,78 @@
+# Async Streaming Migration Guide
+
+This document helps teams migrate from the legacy async streaming hooks to the current streaming architecture introduced in `pipeline_ex` v0.3.
+
+## 1. Migration Checklist
+
+- Audit pipelines for `Pipeline.AsyncStreaming.Handler` usage.
+- Replace handler modules with the new `Pipeline.Streaming.*` namespace.
+- Confirm each LLM step sets `stream: true` in YAML or step configs.
+- Run `mix test --only streaming` to validate the new flow.
+
+## 2. Architectural Changes
+
+| Area | Legacy System | Current System | Migration Notes |
+| --- | --- | --- | --- |
+| Event shape | `{:chunk, binary}` | `{:delta, %{content: binary}}` | Update pattern matches to extract `%{content: chunk}`. |
+| Final callback | `{:complete, result}` | `{:done, response}` | Final responses now carry metadata such as tokens and finish reason. |
+| Handler state | Global Agent | Lightweight module functions | Move mutable state into per-run processes (e.g., Task or GenServer). |
+| Configuration | `async_stream: true` | `stream: true` | Remove the `async_` prefix everywhere. |
+
+## 3. Updating Elixir Steps
+
+```elixir
+# BEFORE
+Pipeline.AsyncStreaming.invoke!(
+  prompt: ctx.prompt,
+  handler: Pipeline.AsyncStreaming.ConsoleHandler
+)
+
+# AFTER
+Pipeline.LLM.invoke!(
+  ctx,
+  prompt: ctx.prompt,
+  stream: true,
+  on_event: &Pipeline.Streaming.ConsoleHandler.handle_event/1
+)
+```
+
+- Use `Pipeline.Streaming.Handler` behaviour for shared handlers.
+- Prefer function captures (`&Module.handle_event/1`) over anonymous functions so the handler remains testable.
+
+## 4. YAML Workflow Changes
+
+```yaml
+# BEFORE
+- id: draft_report
+  type: llm
+  provider: claude
+  async_stream: true
+  async_handler: Pipeline.AsyncStreaming.ConsoleHandler
+
+# AFTER
+- id: draft_report
+  type: llm
+  provider: claude
+  stream: true
+  stream_handler: Pipeline.Streaming.ConsoleHandler
+```
+
+Make sure the handler module is compiled and available when running `mix pipeline.run`.
+
+## 5. Testing Strategy
+
+- **Unit tests**: use the `Pipeline.TestMode.StreamingRecorder` helper to capture deltas without hitting real providers.
+- **Integration tests**: tag scenarios that depend on live streaming with `@tag :streaming` and execute via `mix test --only streaming`.
+- **Diagnostics**: set `PIPELINE_STREAM_DEBUG=1` to print raw events during runs.
+
+## 6. Common Issues
+
+- **Handler not invoked**: confirm the module implements `handle_event/1` and is referenced correctly in YAML.
+- **Duplicate output**: most often caused by reusing the same handler process; ensure each run starts a fresh handler.
+- **Fallback to sync execution**: some providers disable streaming for certain models; verify the chosen model supports it.
+
+## 7. Further Reading
+
+- `examples/STREAMING_GUIDE.md` for a quick-start implementation.
+- `docs/ASYNC_STREAMING_ASSESSMENT.md` for evaluation notes on the old system.
+- `docs/ASYNC_STREAMING_EVALUATION_REPORT.md` for design trade-offs and roadmap.

examples/STREAMING_GUIDE.md

@@ -0,0 +1,72 @@
+# Streaming Guide
+
+This guide walks through enabling streaming output in `pipeline_ex` pipelines.
+
+## When to Use Streaming
+
+- Long running prompts where you want partial progress updates.
+- Tooling scenarios that push incremental tokens to a UI.
+- Debugging runs that need to expose intermediate state as it is produced.
+
+If your pipeline already emits small responses, synchronous execution is fine. Streaming makes the biggest difference when a step may take tens of seconds or more.
+
+## Quick Start
+
+1. Add the `stream` flag to the step that invokes your LLM provider.
+2. Supply a streaming handler to capture events in real time.
+
+```elixir
+defmodule Pipeline.StreamingExample do
+  use Pipeline.Step
+
+  def run(ctx) do
+    Pipeline.LLM.invoke!(
+      ctx,
+      prompt: ctx.prompt,
+      stream: true,
+      on_event: &handle_event/1
+    )
+  end
+
+  defp handle_event({:delta, chunk}) do
+    IO.write(chunk.content)
+  end
+end
+```
+
+Store handlers under `lib/pipeline/streaming/` so other pipelines can reuse them.
+
+## YAML Configuration
+
+Streaming can be toggled directly in pipeline YAML workflows.
+
+```yaml
+steps:
+  - id: research_summary
+    type: llm
+    provider: claude
+    prompt: >
+      Summarize the latest findings on autonomous research pipelines.
+    stream: true
+    stream_handler: Pipeline.Streaming.ConsoleHandler
+```
+
+### Handler Requirements
+
+- Accept events shaped as `{:delta, chunk}` and `{:done, response}`.
+- Avoid blocking calls; use async tasks if you need to buffer or persist output.
+- Keep handlers idempotent—retries can trigger duplicate events.
+
+## Testing Streaming Pipelines
+
+- Run `mix test test/streaming` to execute streaming regression suites.
+- For ad-hoc verification, execute `mix pipeline.run pipelines/demo.yaml --stream`.
+- Capture output with `tee` if you need to diff streamed tokens: `mix pipeline.run ... | tee log.txt`.
+
+## Troubleshooting
+
+- **Nothing appears on screen**: confirm your handler writes to stdout and that the provider supports streaming for the selected model.
+- **Out-of-order chunks**: accumulate within a GenServer before printing to guarantee ordering.
+- **State bleed between runs**: reset any ETS tables or Agent state in your handler's `init/1`.
+
+For deeper design notes, continue with `docs/ASYNC_STREAMING_MIGRATION_GUIDE.md`.

mix.exs

@@ -1,7 +1,7 @@
 defmodule Pipeline.MixProject do
   use Mix.Project
 
-  @version "0.1.0"
+  @version "0.1.1"
   @source_url "https://github.com/nshkrdotcom/pipeline_ex"
 
   def project do
@@ -156,7 +156,9 @@ defmodule Pipeline.MixProject do
         "docs/specifications/data_processing_pipelines.md": [title: "Data Processing Pipelines"],
         "docs/specifications/model_development_pipelines.md": [
           title: "Model Development Pipelines"
-        ]
+        ],
+        "examples/STREAMING_GUIDE.md": [title: "Streaming Guide"],
+        "docs/ASYNC_STREAMING_MIGRATION_GUIDE.md": [title: "Async Streaming Migration"]
       ],
       groups_for_modules: [
         Core: [Pipeline, Pipeline.Config, Pipeline.Executor],