Skip to content

When n_jobs > 128 the Data class crushes #145

@diegodoimo

Description

@diegodoimo

The Data class takes all the cores detected by "multiprocessing.cpu_count()" as a default number of jobs. When this number is larger than 128, the Class crushes due to an OPENBLAS memory issue:

OpenBLAS : Program is Terminated. Because you tried to allocate too many memory regions.
This library was built to support a maximum of 128 threads - either rebuild OpenBLAS
with a larger NUM_THREADS value or set the environment variable OPENBLAS_NUM_THREADS to
a sufficiently small number. This error typically occurs when the software that relies on
OpenBLAS calls BLAS functions from many threads in parallel or when your computer has more
cpu cores than what OpenBLAS was configured to handle.

I propose setting the default value to 1 or None, or the value set in the underlying libraries we use.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions