Can rknn quantization break something in the model that affects the correctness of the AUPIMO calculation?

[bananaspirit-nasonov]

Model: EfficientAD-S
Dataset: Custom

anomalib[vlm,clip] version: 2.1.0
rknn-toolkit2 version: 2.3.2
rknn-toolkit-lite2 version: 2.3.2

The metrics are calculated correctly with the onnx fp32 model
When trying to calculate metrics for the rknn fp16 model, the following error occurs:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 543, in <module>
    main()
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 539, in main
    detector.calculate_metrics()
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 491, in calculate_metrics
    metrics = self.metrics_calculator.compute()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 78, in compute
    "AUPIMO": float(self.__aupimo_metric.compute()),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/torchmetrics/metric.py", line 699, in wrapped_func
    value = _squeeze_if_scalar(compute(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/base.py", line 191, in compute
    return super().compute()  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/pimo/pimo.py", line 322, in compute
    thresholds, shared_fpr, per_image_tprs, _, aupimos, num_thresholds_auc = functional.aupimo_scores(
                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/pimo/functional.py", line 222, in aupimo_scores
    fpr_lower_bound_thresh_idx, _, fpr_lower_bound_defacto = thresh_at_shared_fpr_level(
                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/pimo/functional.py", line 382, in thresh_at_shared_fpr_level
    raise ValueError(msg)
ValueError: Invalid `fpr_level` because it's out of the range of `shared_fpr` = [1.100052234738372e-05, 1.0], and got 1e-05.

In this case, AUPRO and image_AUPR are calculated correctly.

When trying to calculate metrics for the rknn int8 model, the following error occurs:

2025-11-26 01:53:15,618 - W - The lower bound of the shared FPR integration range is not exactly achieved. Expected 1e-05 but got 1.7742777979651163e-06, which is not within rtol=0.01.
2025-11-26 01:53:15,618 - W - The upper bound of the shared FPR integration range is not exactly achieved. Expected 0.0001 but got 1.7742777979651163e-06, which is not within rtol=0.01.
Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 633, in <module>
    main()
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 629, in main
    detector.calculate_metrics()
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 542, in calculate_metrics
    metrics = self.metrics_calculator.compute()
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/efficient-ad/scripts/rknn_inference.py", line 72, in compute
    "AUPIMO": float(self.__aupimo_metric.compute()),
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/torchmetrics/metric.py", line 699, in wrapped_func
    value = _squeeze_if_scalar(compute(*args, **kwargs))
                               ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/base.py", line 191, in compute
    return super().compute()  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/pimo/pimo.py", line 312, in compute
    thresholds, shared_fpr, per_image_tprs, _, aupimos, num_thresholds_auc = functional.aupimo_scores(
                                                                             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/orangepi/obstacle-detector/anomalib/.venv/lib/python3.11/site-packages/anomalib/metrics/pimo/functional.py", line 267, in aupimo_scores
    raise RuntimeError(msg)
RuntimeError: The thresholds corresponding to the given `fpr_bounds` are not valid because they matched the same threshold or the are in the wrong order. FPR upper/lower = threshold lower/upper = 1 and 1.

At the same time, image_AUPR is correctly calculated, and AUPRO is almost 2 times less than expected.

At the same time, the predictions of the onnx_fp32 model completely coincide with the predictions of the original model, the predictions of the rknn_int8 model practically coincide with onnx_fp32, and the predictions of the rknn_fp16 model are worse than the predictions of rknn_int8 (which is paradoxical). At the same time, I am completely confident in the correctness of preprocessing, postprocessing, various transformations, etc., I have checked several times, including on the onnx_fp32 pipeline.

RKNN config for converting fp16 models:

rknn.config(
        target_platform=target_platform,
        optimization_level=3,
    )

RKNN config for converting int8 models:

rknn.config(
        mean_values=[[0, 0, 0]],
        std_values=[[255, 255, 255]],
        target_platform=target_platform,
        optimization_level=3,
    )

I have a hypothesis that some parameters are hardwired into the model that should not change, and then the AUPIMO calculation will be correct, and the quantization of fp16 and int8 simplifies these parameters, which leads to such errors.

P.S. I might have forgotten something when describing the problem, so if so, please indicate and I will provide everything.

[bananaspirit-nasonov]

I duplicated this question in #3165 to make it more likely to be noticed.

[bananaspirit-nasonov]

I figured out my problem, I hope it will be useful to someone.

If you look at the anomaly_map values obtained from the rknn int8 model, you can see that before the threshold > 0.5 occurs, the following values appear in the first row:

4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 4.990260004997253418e-01 5.010062456130981445e-01

Here you can see how the number 4.990260004997253418e-01 is repeated many times (including throughout the anomalous map). And in onnx fp32 there is no such repetition, there is a gradual increase and the range of values is wider:

4.994519650936126709e-01 4.994519650936126709e-01 4.994564652442932129e-01 4.994654357433319092e-01 4.994744062423706055e-01 4.994834065437316895e-01 4.994939267635345459e-01 4.995059669017791748e-01 4.995180368423461914e-01 4.995300769805908203e-01 4.995476901531219482e-01 4.995708763599395752e-01 4.995940625667572021e-01 4.996172189712524414e-01 4.996482431888580322e-01 4.996870756149291992e-01 4.997259080410003662e-01 4.997647702693939209e-01 4.998099207878112793e-01 4.998614490032196045e-01 4.999129474163055420e-01 4.999644458293914795e-01 5.000107288360595703e-01

So it turns out that the PIMO curve simply cannot be constructed, because at different thresholds we get only 3 values of TPR, 0, 1.7742777979651163e-06 and 1. Therefore, the lower and upper limits of integration collapse into one number - 1.7742777979651163e-06, and for calculating AUPIMO we need a continuous section, and not 1 point, because AUPIMO is the area under the PIMO curve.

Moreover, this behavior is absolutely normal for quantization, because during quantization the range of possible values is compressed. We need to fit the entire range of floating-point numbers from 0 to 1 into only 255 unique values, which means that when int8 numbers are dequantized, the following happens:

Original 1: 0.4994519 → quantization → 126 → dequantization → 0.500238
Original 2: 0.4994564 → quantization → 126 → dequantization → 0.500238
Original 3: 0.4994654 → quantization → 126 → dequantization → 0.500238

Now I see 2 ways out of this whole situation:

• Use Hybrid Quantization (leave outputs in fp16);
• Use a different metric to capture the difference in segmentation quality between the quantized and non-quantized models.

anomaly_map_rknn-int8_110-100.txt

anomaly_map_onnx_fp32_110-100.txt