feat(Story-3): Add IR Operations for Sigmoid Family (Group 2)

[ooples]

Story 3: IR Operations - Sigmoid Family

Added 10 IR operation classes for Sigmoid-family activations.

Activations Added:

• SwishOp (uses IEngine.Swish)
• SiLUOp (alias for Swish, uses IEngine.Swish)
• MishOp (uses IEngine.Mish)
• HardSigmoidOp (piecewise linear approximation)
• HardTanhOp (piecewise linear approximation)
• ScaledTanhOp (parameterized: a * tanh(b * x))
• SoftplusOp (ln(1 + exp(x)), numerically stable)
• SoftSignOp (x / (1 + |x|))
• BentIdentityOp ((sqrt(x² + 1) - 1) / 2 + x)
• IdentityOp (f(x) = x)

Implementation Details:

• All classes implement IROp interface
• Forward() uses IEngine.Swish and IEngine.Mish for GPU acceleration
• Backward() implements gradient computation for each activation
• Proper null checks for all parameters (no null-forgiving operators)
• Comprehensive XML documentation with mathematical formulas
• Numerical stability considerations (Softplus uses stable computation)

Pattern Followed:

• Validates engine and input parameters in constructor
• Type-safe tensor conversions with proper error messages
• Uses IEngine methods where available (Swish, Mish)
• ScaledTanhOp accepts parameters for flexibility

Build Status: ✅ Passed on all target frameworks (net462, net471, netstandard2.0)

Testing:

• Build succeeded with 0 warnings, 0 errors
• Ready for integration with DenseLayer JIT compilation

🤖 Generated with Claude Code

[coderabbitai]

Summary by CodeRabbit

• New Features
  • Added support for multiple activation functions including Swish, SiLU, Mish, HardSigmoid, HardTanh, ScaledTanh, Softplus, SoftSign, BentIdentity, and Identity operations for JIT-compiled neural networks.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

Walkthrough

Adds a new file containing an IROp interface and ten activation operation classes (Swish/SiLU, Mish, HardSigmoid, HardTanh, ScaledTanh, Softplus, SoftSign, BentIdentity, Identity) for JIT-compiled neural networks. Each class implements forward and backward passes using an IEngine abstraction, with input validation and both analytical and numerical derivative implementations.

Changes

Cohort / File(s)

Summary

New JIT Activation Operations src/JIT/ActivationOps.cs

Introduces IROp interface with generic Forward<T> and Backward<T> methods. Adds ten activation operation classes: SwishOp, SiLUOp (alias inheriting from SwishOp), MishOp, HardSigmoidOp, HardTanhOp, ScaledTanhOp (with configurable scales), SoftplusOp, SoftSignOp, BentIdentityOp, and IdentityOp. Includes internal NumOps<T> helper providing numeric constants and conversions. All classes require IEngine for tensor operations and validate inputs as Tensor<T> or Gradient objects.

Sequence Diagram(s)

sequenceDiagram
    participant Client
    participant ActivationOp as Activation Op
    participant IEngine
    participant Tensor

    Client->>ActivationOp: Forward(input)
    ActivationOp->>ActivationOp: Validate input type
    ActivationOp->>IEngine: Execute operation (e.g., Swish, Sigmoid)
    IEngine->>Tensor: Perform computation
    Tensor-->>IEngine: Result tensor
    IEngine-->>ActivationOp: Output tensor
    ActivationOp-->>Client: Return output

    Client->>ActivationOp: Backward(input, gradOutput)
    ActivationOp->>ActivationOp: Validate input & gradient types
    ActivationOp->>IEngine: Compute derivative
    IEngine->>Tensor: Derivative computation
    Tensor-->>IEngine: Gradient tensor
    IEngine-->>ActivationOp: Input gradient
    ActivationOp-->>Client: Return gradient

Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

• Mathematical correctness: Verify each activation function's forward formula and derivative calculations (both analytical and numerical implementations)
• Generic type constraints: Validate proper use of where T : struct constraints across all operations
• Input validation robustness: Ensure consistent error handling for invalid Tensor<T> and Gradient inputs across all classes
• Derivative implementations: Pay particular attention to MishOp's numerical gradient placeholder and confirm analytical derivatives for Swish, HardSigmoid, ScaledTanh, SoftSign, and BentIdentity
• IEngine abstraction usage: Verify that all engine method calls (Swish, Sigmoid, Tanh, etc.) are correctly employed

Poem

🐰 Activations now proliferate,
From Swish to Mish, they integrate!
With engines humming, tensors flow,
Forward, backward—watch them glow! ✨

Pre-merge checks and finishing touches

✅ Passed checks (3 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title accurately describes the main change: adding IR operations for the Sigmoid family of activation functions (10 new operation classes implementing IROp interface).
Description check	✅ Passed	The description is directly related to the changeset, detailing the 10 activation operations added, their implementations, and how they follow established patterns.
Docstring Coverage	✅ Passed	Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✨ Finishing touches

• 📝 Generate docstrings

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment
• Commit unit tests in branch feat/activation-ir-ops-group2

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 3

📜 Review details

Configuration used: CodeRabbit UI

Review profile: CHILL

Plan: Pro

📥 Commits

Reviewing files that changed from the base of the PR and between ef6df16 and 238ef51.

📒 Files selected for processing (1)

• src/JIT/ActivationOps.cs (1 hunks)

⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (1)

• GitHub Check: Build All Frameworks

[Copilot]

Pull request overview

This PR introduces IR (Intermediate Representation) operations for 10 Sigmoid-family activation functions to enable JIT compilation of neural network layers. The operations implement the IROp interface with Forward and Backward methods, leveraging GPU acceleration where available.

Key Changes:

• Defines IROp interface for JIT compilation operations
• Implements 10 activation function operation classes (Swish/SiLU, Mish, HardSigmoid, HardTanh, ScaledTanh, Softplus, SoftSign, BentIdentity, Identity)
• Adds NumOps helper class for generic numeric operations

Comments suppressed due to low confidence (3)

src/JIT/ActivationOps.cs:1

• Using numerical gradient approximation (finite differences) instead of analytical gradient is significantly slower and less accurate. The analytical formula is documented in the comments. Numerical differentiation requires an additional forward pass through Mish and introduces numerical errors.

using System;

src/JIT/ActivationOps.cs:338

• This assignment to three is useless, since its value is never read.

        var three = NumOps<T>.FromDouble(3.0);

src/JIT/ActivationOps.cs:339

• This assignment to negThree is useless, since its value is never read.

        var negThree = NumOps<T>.FromDouble(-3.0);

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.