feat(skills): import tier 2 priority skills

components/skills/data-analysis-polars-dev/SKILL.md

@@ -0,0 +1,381 @@
+---
+name: polars
+description: "Fast DataFrame library (Apache Arrow). Select, filter, group_by, joins, lazy evaluation, CSV/Parquet I/O, expression API, for high-performance data analysis workflows."
+---
+
+# Polars
+
+## Overview
+
+Polars is a lightning-fast DataFrame library for Python and Rust built on Apache Arrow. Work with Polars' expression-based API, lazy evaluation framework, and high-performance data manipulation capabilities for efficient data processing, pandas migration, and data pipeline optimization.
+
+## Quick Start
+
+### Installation and Basic Usage
+
+Install Polars:
+```python
+uv pip install polars
+```
+
+Basic DataFrame creation and operations:
+```python
+import polars as pl
+
+# Create DataFrame
+df = pl.DataFrame({
+    "name": ["Alice", "Bob", "Charlie"],
+    "age": [25, 30, 35],
+    "city": ["NY", "LA", "SF"]
+})
+
+# Select columns
+df.select("name", "age")
+
+# Filter rows
+df.filter(pl.col("age") > 25)
+
+# Add computed columns
+df.with_columns(
+    age_plus_10=pl.col("age") + 10
+)
+```
+
+## Core Concepts
+
+### Expressions
+
+Expressions are the fundamental building blocks of Polars operations. They describe transformations on data and can be composed, reused, and optimized.
+
+**Key principles:**
+- Use `pl.col("column_name")` to reference columns
+- Chain methods to build complex transformations
+- Expressions are lazy and only execute within contexts (select, with_columns, filter, group_by)
+
+**Example:**
+```python
+# Expression-based computation
+df.select(
+    pl.col("name"),
+    (pl.col("age") * 12).alias("age_in_months")
+)
+```
+
+### Lazy vs Eager Evaluation
+
+**Eager (DataFrame):** Operations execute immediately
+```python
+df = pl.read_csv("file.csv")  # Reads immediately
+result = df.filter(pl.col("age") > 25)  # Executes immediately
+```
+
+**Lazy (LazyFrame):** Operations build a query plan, optimized before execution
+```python
+lf = pl.scan_csv("file.csv")  # Doesn't read yet
+result = lf.filter(pl.col("age") > 25).select("name", "age")
+df = result.collect()  # Now executes optimized query
+```
+
+**When to use lazy:**
+- Working with large datasets
+- Complex query pipelines
+- When only some columns/rows are needed
+- Performance is critical
+
+**Benefits of lazy evaluation:**
+- Automatic query optimization
+- Predicate pushdown
+- Projection pushdown
+- Parallel execution
+
+For detailed concepts, load `references/core_concepts.md`.
+
+## Common Operations
+
+### Select
+Select and manipulate columns:
+```python
+# Select specific columns
+df.select("name", "age")
+
+# Select with expressions
+df.select(
+    pl.col("name"),
+    (pl.col("age") * 2).alias("double_age")
+)
+
+# Select all columns matching a pattern
+df.select(pl.col("^.*_id$"))
+```
+
+### Filter
+Filter rows by conditions:
+```python
+# Single condition
+df.filter(pl.col("age") > 25)
+
+# Multiple conditions (cleaner than using &)
+df.filter(
+    pl.col("age") > 25,
+    pl.col("city") == "NY"
+)
+
+# Complex conditions
+df.filter(
+    (pl.col("age") > 25) | (pl.col("city") == "LA")
+)
+```
+
+### With Columns
+Add or modify columns while preserving existing ones:
+```python
+# Add new columns
+df.with_columns(
+    age_plus_10=pl.col("age") + 10,
+    name_upper=pl.col("name").str.to_uppercase()
+)
+
+# Parallel computation (all columns computed in parallel)
+df.with_columns(
+    pl.col("value") * 10,
+    pl.col("value") * 100,
+)
+```
+
+### Group By and Aggregations
+Group data and compute aggregations:
+```python
+# Basic grouping
+df.group_by("city").agg(
+    pl.col("age").mean().alias("avg_age"),
+    pl.len().alias("count")
+)
+
+# Multiple group keys
+df.group_by("city", "department").agg(
+    pl.col("salary").sum()
+)
+
+# Conditional aggregations
+df.group_by("city").agg(
+    (pl.col("age") > 30).sum().alias("over_30")
+)
+```
+
+For detailed operation patterns, load `references/operations.md`.
+
+## Aggregations and Window Functions
+
+### Aggregation Functions
+Common aggregations within `group_by` context:
+- `pl.len()` - count rows
+- `pl.col("x").sum()` - sum values
+- `pl.col("x").mean()` - average
+- `pl.col("x").min()` / `pl.col("x").max()` - extremes
+- `pl.first()` / `pl.last()` - first/last values
+
+### Window Functions with `over()`
+Apply aggregations while preserving row count:
+```python
+# Add group statistics to each row
+df.with_columns(
+    avg_age_by_city=pl.col("age").mean().over("city"),
+    rank_in_city=pl.col("salary").rank().over("city")
+)
+
+# Multiple grouping columns
+df.with_columns(
+    group_avg=pl.col("value").mean().over("category", "region")
+)
+```
+
+**Mapping strategies:**
+- `group_to_rows` (default): Preserves original row order
+- `explode`: Faster but groups rows together
+- `join`: Creates list columns
+
+## Data I/O
+
+### Supported Formats
+Polars supports reading and writing:
+- CSV, Parquet, JSON, Excel
+- Databases (via connectors)
+- Cloud storage (S3, Azure, GCS)
+- Google BigQuery
+- Multiple/partitioned files
+
+### Common I/O Operations
+
+**CSV:**
+```python
+# Eager
+df = pl.read_csv("file.csv")
+df.write_csv("output.csv")
+
+# Lazy (preferred for large files)
+lf = pl.scan_csv("file.csv")
+result = lf.filter(...).select(...).collect()
+```
+
+**Parquet (recommended for performance):**
+```python
+df = pl.read_parquet("file.parquet")
+df.write_parquet("output.parquet")
+```
+
+**JSON:**
+```python
+df = pl.read_json("file.json")
+df.write_json("output.json")
+```
+
+For comprehensive I/O documentation, load `references/io_guide.md`.
+
+## Transformations
+
+### Joins
+Combine DataFrames:
+```python
+# Inner join
+df1.join(df2, on="id", how="inner")
+
+# Left join
+df1.join(df2, on="id", how="left")
+
+# Join on different column names
+df1.join(df2, left_on="user_id", right_on="id")
+```
+
+### Concatenation
+Stack DataFrames:
+```python
+# Vertical (stack rows)
+pl.concat([df1, df2], how="vertical")
+
+# Horizontal (add columns)
+pl.concat([df1, df2], how="horizontal")
+
+# Diagonal (union with different schemas)
+pl.concat([df1, df2], how="diagonal")
+```
+
+### Pivot and Unpivot
+Reshape data:
+```python
+# Pivot (wide format)
+df.pivot(values="sales", index="date", columns="product")
+
+# Unpivot (long format)
+df.unpivot(index="id", on=["col1", "col2"])
+```
+
+For detailed transformation examples, load `references/transformations.md`.
+
+## Pandas Migration
+
+Polars offers significant performance improvements over pandas with a cleaner API. Key differences:
+
+### Conceptual Differences
+- **No index**: Polars uses integer positions only
+- **Strict typing**: No silent type conversions
+- **Lazy evaluation**: Available via LazyFrame
+- **Parallel by default**: Operations parallelized automatically
+
+### Common Operation Mappings
+
+| Operation | Pandas | Polars |
+|-----------|--------|--------|
+| Select column | `df["col"]` | `df.select("col")` |
+| Filter | `df[df["col"] > 10]` | `df.filter(pl.col("col") > 10)` |
+| Add column | `df.assign(x=...)` | `df.with_columns(x=...)` |
+| Group by | `df.groupby("col").agg(...)` | `df.group_by("col").agg(...)` |
+| Window | `df.groupby("col").transform(...)` | `df.with_columns(...).over("col")` |
+
+### Key Syntax Patterns
+
+**Pandas sequential (slow):**
+```python
+df.assign(
+    col_a=lambda df_: df_.value * 10,
+    col_b=lambda df_: df_.value * 100
+)
+```
+
+**Polars parallel (fast):**
+```python
+df.with_columns(
+    col_a=pl.col("value") * 10,
+    col_b=pl.col("value") * 100,
+)
+```
+
+For comprehensive migration guide, load `references/pandas_migration.md`.
+
+## Best Practices
+
+### Performance Optimization
+
+1. **Use lazy evaluation for large datasets:**
+   ```python
+   lf = pl.scan_csv("large.csv")  # Don't use read_csv
+   result = lf.filter(...).select(...).collect()
+   ```
+
+2. **Avoid Python functions in hot paths:**
+   - Stay within expression API for parallelization
+   - Use `.map_elements()` only when necessary
+   - Prefer native Polars operations
+
+3. **Use streaming for very large data:**
+   ```python
+   lf.collect(streaming=True)
+   ```
+
+4. **Select only needed columns early:**
+   ```python
+   # Good: Select columns early
+   lf.select("col1", "col2").filter(...)
+
+   # Bad: Filter on all columns first
+   lf.filter(...).select("col1", "col2")
+   ```
+
+5. **Use appropriate data types:**
+   - Categorical for low-cardinality strings
+   - Appropriate integer sizes (i32 vs i64)
+   - Date types for temporal data
+
+### Expression Patterns
+
+**Conditional operations:**
+```python
+pl.when(condition).then(value).otherwise(other_value)
+```
+
+**Column operations across multiple columns:**
+```python
+df.select(pl.col("^.*_value$") * 2)  # Regex pattern
+```
+
+**Null handling:**
+```python
+pl.col("x").fill_null(0)
+pl.col("x").is_null()
+pl.col("x").drop_nulls()
+```
+
+For additional best practices and patterns, load `references/best_practices.md`.
+
+## Resources
+
+This skill includes comprehensive reference documentation:
+
+### references/
+- `core_concepts.md` - Detailed explanations of expressions, lazy evaluation, and type system
+- `operations.md` - Comprehensive guide to all common operations with examples
+- `pandas_migration.md` - Complete migration guide from pandas to Polars
+- `io_guide.md` - Data I/O operations for all supported formats
+- `transformations.md` - Joins, concatenation, pivots, and reshaping operations
+- `best_practices.md` - Performance optimization tips and common patterns
+
+Load these references as needed when users require detailed information about specific topics.