[Opt] Manipulation of Q8_0 tensors with Tornado `ByteArray`s#79

Merged

mikepapadim merged 30 commits intobeehive-lab:mainfrom

orionpapadakis:opt/q8-load-bytearray

Dec 8, 2025

Commits on Dec 4, 2025

Refactor tensor loading and introduce support for Half-Float precision in TornadoVM acceleration.

mikepapadim
authored and
orionpapadakis
committed
Replace `loadTornadoTensorAsFP32` with `loadTornadoTensor` across model loaders for consistent tensor loading.

mikepapadim
authored and
orionpapadakis
committed
Add `modelType` to Configuration

mikepapadim
authored and
orionpapadakis
committed
Add `readModelType` integration for all model loaders
orionpapadakis
committed
Update `Q8_0` tensor creation to use `fromTornadoMemorySegment` method
orionpapadakis
committed
Add new `Q8_0TornadoTensor` constructor using `ByteArray` and `ByteArray.fromSegmentShallow`
orionpapadakis
committed
Add support for `Q8_0` weight type in `InferenceCore` embedding table copy
orionpapadakis
committed
Change `embeddingX` type from `HalfFloatArray` to `TornadoNativeArray`
orionpapadakis
committed
Add `type` field to `LlamaConfiguration` constructor
orionpapadakis
committed
Add FP16 and Q8_0 support to `Activation` layer initialization
orionpapadakis
committed
Add `convertQ8_0toFP32` kernel for dequantization in `TransformerComputeKernels`
orionpapadakis
committed
Add `matrixVectorGenericQ8Byte` and `matrixVectorRowMajorOptimizedQ8_0Byte` kernels for Q8_0 matrix-vector computations
orionpapadakis
committed
Update FFN and attention layers to use Q8_0 byte-based kernels for matrix-vector computations
orionpapadakis
committed
Update `LogitsQ8_0Layer` to use byte-based Q8_0 kernels
orionpapadakis
committed
Remove deprecated methods for Q8_0 tensor loading and conversion to FP32
orionpapadakis
committed
Add `matrixVectorGenericWithResidualQ8_0Byte` and `fusedFeedForwardWithSiLUAndGLUActivationQ8_0Byte` kernels for byte-based Q8_0 computations
orionpapadakis
committed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[Opt] Manipulation of Q8_0 tensors with Tornado `ByteArray`s#79

[Opt] Manipulation of Q8_0 tensors with Tornado `ByteArray`s#79
mikepapadim merged 30 commits intobeehive-lab:mainfrom
orionpapadakis:opt/q8-load-bytearray

Commits on Dec 4, 2025

Refactor tensor loading and introduce support for Half-Float precision in TornadoVM acceleration.

Replace `loadTornadoTensorAsFP32` with `loadTornadoTensor` across model loaders for consistent tensor loading.

Add `modelType` to Configuration

Add `readModelType` integration for all model loaders

Update `Q8_0` tensor creation to use `fromTornadoMemorySegment` method

Add new `Q8_0TornadoTensor` constructor using `ByteArray` and `ByteArray.fromSegmentShallow`

Add support for `Q8_0` weight type in `InferenceCore` embedding table copy

Change `embeddingX` type from `HalfFloatArray` to `TornadoNativeArray`

Add `type` field to `LlamaConfiguration` constructor

Add FP16 and Q8_0 support to `Activation` layer initialization

Add `convertQ8_0toFP32` kernel for dequantization in `TransformerComputeKernels`

Add `matrixVectorGenericQ8Byte` and `matrixVectorRowMajorOptimizedQ8_0Byte` kernels for Q8_0 matrix-vector computations

Update FFN and attention layers to use Q8_0 byte-based kernels for matrix-vector computations

Update `LogitsQ8_0Layer` to use byte-based Q8_0 kernels

Remove deprecated methods for Q8_0 tensor loading and conversion to FP32

Add `matrixVectorGenericWithResidualQ8_0Byte` and `fusedFeedForwardWithSiLUAndGLUActivationQ8_0Byte` kernels for byte-based Q8_0 computations

Commits on Dec 5, 2025

Replace `getHalf` with `getHalfFloat` for Q8_0 block scale loading in compute kernels

Add FP16 and Q8_0 activation initialization methods in `State` class

Use quantization-specific activation init in Llama models

Use quantization-specific activation init in Qwen3 models

Update Qwen3 FFN layers to use byte-based Q8_0 kernels

Use quantization-specific activation init in Qwen2 and Deepseek models

Update Qwen2 and Deepseek FFN layers to use byte-based Q8_0 kernels

Use quantization-specific activation init in Phi3 models

Update Phi3 FFN layers to use byte-based Q8_0 kernels

Commits on Dec 8, 2025

Rename `modelType` to `quantization` across configurations and update associated usages.

Cleanup unused memorySegment copy

Cleanup and document `Q8_0TornadoTensor`

Use Configuration.quantization() method in Activation

[CI] Update Tornado dependencies to version 2.0.1-dev to run CI

[Opt] Manipulation of Q8_0 tensors with Tornado `ByteArray`s#79mikepapadim merged 30 commits intobeehive-lab:mainbeehive-lab/GPULlama3.java:mainfrom orionpapadakis:opt/q8-load-bytearrayorionpapadakis/GPULlama3.java:opt/q8-load-bytearrayCopy head branch name to clipboard

Commits

Commits on Dec 4, 2025

Commits on Dec 5, 2025

Commits on Dec 8, 2025

[Opt] Manipulation of Q8_0 tensors with Tornado `ByteArray`s#79
mikepapadim merged 30 commits intobeehive-lab:mainfrom
orionpapadakis:opt/q8-load-bytearray