RFC: EmacsConf Talk Analysis System Requirements

[jwalsh]

Summary

We need a robust system for automatically generating technical summaries from EmacsConf talk transcripts, with quality validation and structured output.

Timeline of Development

gantt
    title Development Timeline
    dateFormat  HH:mm
    axisFormat %H:%M
    
    section Initial Attempt
    Basic script implementation      :20:00, 15m
    Initial failures with model      :20:15, 10m
    
    section Improvements
    Added structured terms           :20:25, 15m
    Implemented org tables          :20:40, 10m
    
    section Quality
    Added review system             :20:50, 15m
    Enhanced error handling         :21:05, 10m

Loading

Process Flow

sequenceDiagram
    participant User
    participant Script
    participant Phi3
    participant Llama3.2
    participant Filesystem

    User->>Script: Run with VTT files
    loop Each VTT File
        Script->>Filesystem: Read VTT
        Script->>Phi3: Generate Summary
        Phi3-->>Script: Summary Content
        Script->>Llama3.2: Review Summary
        Llama3.2-->>Script: Quality Assessment
        Script->>Filesystem: Write .org file
    end
    Script->>User: Report Results

Loading

Control Flow

flowchart TD
    A[Start] --> B{VTT Files Exist?}
    B -- Yes --> C[Process Next File]
    B -- No --> Z[End]
    
    C --> D[Extract Text]
    D --> E{Text Extracted?}
    E -- Yes --> F[Generate Summary]
    E -- No --> Y[Log Error]
    
    F --> G{Summary Generated?}
    G -- Yes --> H[Review Summary]
    G -- No --> I[Retry Logic]
    I --> F
    
    H --> J{Review Complete?}
    J -- Yes --> K[Write Output]
    J -- No --> L[Use Default Review]
    
    K --> M{More Files?}
    M -- Yes --> C
    M -- No --> Z
    
    Y --> M
    L --> K

Loading

Issues Encountered

1. Model Overload

   • Initial attempts failed due to context length
   • Solution: Added chunking and retry logic

2. Output Quality

   • Initial summaries lacked technical depth
   • Solution: Enhanced prompting and added review system

3. Formatting Consistency

   • Raw text output was hard to parse
   • Solution: Structured org-mode tables and properties

Current Requirements

Core Requirements

1. Input Processing

   • VTT file parsing
   • Text extraction
   • Audio duration extraction
   • Speaker identification

2. Summary Generation

   • Key points extraction
   • Technical term identification
   • Context preservation
   • Code snippet handling

3. Quality Control

   • Automated review
   • Manual review interface
   • Quality metrics tracking
   • Historical comparison

4. Output Format

   • Org-mode structure
   • Term tables
   • LaTeX export
   • HTML export

Optional Enhancements

1. Content Analysis

   • Topic clustering across talks
   • Technical term network analysis
   • Speaker expertise mapping

2. Search & Discovery

   • Full-text search interface
   • Technical term index
   • Cross-reference system

3. Integration

   • GitHub Actions workflow
   • Pre-commit hooks
   • CI/CD pipeline

4. User Interface

   • Web interface for review
   • CLI improvements
   • Progress visualization

Technical Implementation Options

Model Selection

1. Current: Phi3 + Llama3.2

   • Pros:
     • Local execution
     • No API costs
     • Good performance
   • Cons:
     • Resource intensive
     • Occasional timeout issues
     • Limited context window

2. Alternative: GPT-4 + Claude

   • Pros:
     • Larger context window
     • More consistent output
     • Better technical understanding
   • Cons:
     • API costs
     • External dependencies
     • Rate limiting

3. Hybrid Approach:

   • Use local models for initial processing
   • Fall back to API models for complex cases
   • Cache responses for efficiency

Architecture Options

1. Current Script-based:

flowchart LR
    A[VTT Files] --> B[Python Script]
    B --> C[Local Models]
    C --> D[Org Files]

Loading

2. Proposed Service-based:

flowchart LR
    A[VTT Files] --> B[API Server]
    B --> C[Model Pool]
    C --> D[Database]
    D --> E[Export Service]
    E --> F[Multiple Formats]

Loading

Next Steps

Immediate Priorities

1. Improve error handling and recovery
2. Add comprehensive logging
3. Implement quality metrics
4. Add cross-reference support

Long-term Goals

1. Build web interface
2. Create analysis dashboard
3. Implement search functionality
4. Develop plugin system

Questions for Stakeholders

1. What additional metadata would be valuable to extract?
2. Should we prioritize batch processing or interactive use?
3. What integration points are most important?
4. How should we handle manual corrections?

Open Issues

1. Model Reliability

   • Need better timeout handling
   • Consider caching mechanism
   • Implement fallback chain

2. Quality Metrics

   • Define objective measures
   • Set quality thresholds
   • Implement feedback loop

3. Resource Usage

   • Optimize memory usage
   • Consider distributed processing
   • Implement rate limiting

Appendix A: Example Configurations

models:
  primary:
    name: phi3
    timeout: 30
    retries: 3
  review:
    name: llama3.2
    timeout: 20
    retries: 2

output:
  format: org
  structure:
    - title
    - properties
    - key_points
    - technical_terms
    - review
    - meta

quality:
  minimum_terms: 3
  minimum_points: 5
  review_required: true

[jwalsh]

There should be a secondary reviewer noted for the quality of the generated summary:

#!/usr/bin/env python3

import os
from pathlib import Path
import click
import ollama
from typing import Optional, List, Tuple
from dataclasses import dataclass
import json
from webvtt import WebVTT
from datetime import datetime
from time import sleep

@dataclass
class TechnicalTerm:
    """A technical term with context."""
    name: str
    definition: str
    context: str = ""

@dataclass
class ReviewResult:
    """Summary review result"""
    quality: str  # high, medium, low
    reasoning: str
    suggestions: List[str]

@dataclass
class TalkSummary:
    """Schema for talk summaries."""
    title: str
    speaker: str
    key_points: list[str]
    technical_terms: list[TechnicalTerm]
    meta: dict
    review: Optional[ReviewResult] = None

    def to_org(self) -> str:
        terms_table = "| Term | Definition | Context |\n|-\n"
        for term in self.technical_terms:
            name = term.name.replace('|', '\\vert{}')
            definition = term.definition.replace('|', '\\vert{}')
            context = term.context.replace('|', '\\vert{}')
            terms_table += f"| {name} | {definition} | {context} |\n"

        review_section = ""
        if self.review:
            review_section = f"""
** Review
- Quality: {self.review.quality}
- Reasoning: {self.review.reasoning}
- Suggestions:
{chr(10).join(f'  - {s}' for s in self.review.suggestions)}"""

        return f"""* {self.title}
:PROPERTIES:
:SPEAKER: {self.speaker}
:END:

** Key Points
{chr(10).join(f'- {point}' for point in self.key_points)}

** Technical Terms
{terms_table}

** Meta
- Generated: {datetime.now().isoformat()}
- Model: {self.meta.get('model', 'unknown')}
{review_section}"""

def parse_vtt_content(vtt_path: Path) -> str:
    """Extract text content from VTT file."""
    try:
        vtt = WebVTT.read(str(vtt_path))
        text = " ".join(caption.text for caption in vtt)
        click.echo(f"\nExtracted {len(text)} characters from VTT")
        return text
    except Exception as e:
        click.echo(f"Error parsing {vtt_path}: {e}", err=True)
        return ""

def extract_talk_info(filename: str) -> tuple[str, str]:
    """Extract talk title and speaker from filename."""
    parts = filename.split('--')
    if len(parts) >= 3:
        title = parts[1].replace('-', ' ').title()
        speaker = parts[2].split('.')[0].replace('-', ' ').title()
        return title, speaker
    return "Unknown Title", "Unknown Speaker"

def get_summary_prompt(content: str, title: str, speaker: str) -> str:
    """Get prompt for summary generation."""
    return f"""As a technical documentation specialist, analyze this EmacsConf talk:
Title: {title}
Speaker: {speaker}

Extract key technical points and terms. For each technical term, provide:
1. Clear definition
2. How it was specifically used in this talk
3. Relationship to Emacs functionality

Structure your response exactly like this:
Key Points:
- [detailed technical point with context]
- [another point]

Technical Terms:
Term: [name of term]
Definition: [clear technical definition]
Context: [how it was used in the talk]

[repeat for each term]

Analyze this transcript:
{content[:4000]}"""

def parse_terms(text: str) -> List[TechnicalTerm]:
    """Parse technical terms from response text."""
    terms = []
    current_term = {}
    
    lines = text.split('\n')
    for line in lines:
        line = line.strip()
        if line.startswith('Term:'):
            if current_term:
                terms.append(TechnicalTerm(**current_term))
                current_term = {}
            current_term['name'] = line[5:].strip()
        elif line.startswith('Definition:'):
            current_term['definition'] = line[11:].strip()
        elif line.startswith('Context:'):
            current_term['context'] = line[8:].strip()
    
    if current_term:
        terms.append(TechnicalTerm(**current_term))
    
    return terms

def review_summary(summary: TalkSummary) -> ReviewResult:
    """Review summary using llama3.2."""
    try:
        review_prompt = f"""Review this EmacsConf talk summary for technical accuracy and completeness:

Title: {summary.title}
Speaker: {summary.speaker}

Key Points:
{chr(10).join(summary.key_points)}

Technical Terms:
{chr(10).join(f"- {t.name}: {t.definition}" for t in summary.technical_terms)}

Evaluate and rate:
1. Technical accuracy and specificity
2. Completeness of Emacs-specific details
3. Clarity of term definitions

Return exactly in this format:
Quality: [high/medium/low]
Reasoning: [brief explanation]
Suggestions:
- [concrete improvement 1]
- [concrete improvement 2]
"""
        response = ollama.chat(
            model="llama3.2",
            messages=[{"role": "user", "content": review_prompt}]
        )
        
        lines = response.message.content.split('\n')
        quality = "low"
        reasoning = ""
        suggestions = []
        
        for line in lines:
            if line.startswith('Quality:'):
                quality = line.split(':', 1)[1].strip().lower()
            elif line.startswith('Reasoning:'):
                reasoning = line.split(':', 1)[1].strip()
            elif line.strip().startswith('- '):
                suggestions.append(line.strip()[2:])
        
        return ReviewResult(quality, reasoning, suggestions)
    except Exception as e:
        click.echo(f"Review failed: {e}", err=True)
        return ReviewResult("low", "Review failed", ["Automatic review failed"])

def generate_summary_content(content: str, title: str, speaker: str) -> Optional[TalkSummary]:
    """Generate initial summary."""
    prompt = get_summary_prompt(content, title, speaker)
    
    response = ollama.chat(
        model="phi3",
        messages=[{"role": "user", "content": prompt}]
    )
    
    if not response.message.content:
        return None
        
    # Parse response sections
    lines = response.message.content.split('\n')
    key_points = []
    technical_content = []
    current_section = None
    
    for line in lines:
        if 'Key Points:' in line:
            current_section = 'points'
        elif 'Technical Terms:' in line:
            current_section = 'terms'
        elif line.strip().startswith('- ') and current_section == 'points':
            key_points.append(line.strip()[2:])
        elif current_section == 'terms':
            technical_content.append(line)
    
    technical_terms = parse_terms('\n'.join(technical_content))
    
    return TalkSummary(
        title=title,
        speaker=speaker,
        key_points=key_points,
        technical_terms=technical_terms,
        meta={"model": "phi3"}
    )

def generate_summary(content: str, title: str, speaker: str, max_retries: int = 3) -> Optional[TalkSummary]:
    """Generate and review summary with retries."""
    for attempt in range(max_retries):
        try:
            click.echo(f"\nGeneration attempt {attempt + 1}/{max_retries}")
            summary = generate_summary_content(content, title, speaker)
            if summary:
                click.echo("Reviewing summary...")
                summary.review = review_summary(summary)
                return summary
        except Exception as e:
            click.echo(f"Attempt {attempt + 1} failed: {e}", err=True)
            if attempt < max_retries - 1:
                click.echo("Waiting before retry...")
                sleep(2)
    return None

@click.command()
@click.argument('media_dir', type=click.Path(exists=True))
@click.option('--force', is_flag=True, help="Overwrite existing summaries")
@click.option('--output', default='summaries', help="Output directory for summaries")
@click.option('--verbose', is_flag=True, help="Show detailed processing information")
@click.option('--retries', default=3, help="Number of retries for failed generations")
def main(media_dir: str, force: bool, output: str, verbose: bool, retries: int):
    """Generate technical summaries for EmacsConf VTT files with quality review."""
    media_path = Path(media_dir)
    output_path = Path(output)
    output_path.mkdir(exist_ok=True)

    vtt_files = [f for f in media_path.glob('**/*.vtt') 
                 if '--main.vtt' in str(f) and '--chapters' not in str(f)]

    click.echo(f"Found {len(vtt_files)} VTT files to process")

    for vtt_file in vtt_files:
        summary_file = output_path / f"{vtt_file.stem}.org"
        
        if summary_file.exists() and not force:
            click.echo(f"Skipping (exists): {summary_file}")
            continue

        click.echo(f"\nProcessing: {vtt_file.name}")
        
        content = parse_vtt_content(vtt_file)
        if not content:
            continue

        title, speaker = extract_talk_info(vtt_file.name)
        if verbose:
            click.echo(f"Title: {title}")
            click.echo(f"Speaker: {speaker}")
        
        with click.progressbar(length=2, label='Generating summary') as bar:
            summary = generate_summary(content, title, speaker, retries)
            bar.update(1)
            
            if summary:
                if summary.review:
                    click.echo(f"\nQuality: {summary.review.quality}")
                summary_file.write_text(summary.to_org())
                click.echo(f"✓ Generated: {summary_file}")
            else:
                click.echo(f"✗ Failed: {vtt_file.name}")
            bar.update(1)

if __name__ == '__main__':
    main()