Skip to content

misleading performance comparison without hyperparameter tuning #10

@jannikmi

Description

@jannikmi

The detailed performance evaluation / comparison plots of the different model architectures in the readme make it seem like the results can be used to select the best model architecture.
Especially beginners in the field of data science might be fooled into selecting their preferred architecture based on these results, perhaps even in a completely different use case solely based on this information.

In the current version the training script does include only training with one selection of hyperparameters per architecture.

The performance especially of deep ML architectures are known to be heavily reliant on the selection of hyperparameters.
I want to point out, that without extensive(!) hyperparameter optimsation one should not draw any conclusions about the potential performance of an ML architecture. Hence the best performing model architecture cannot be selected by training with only one or few hyperparameter settings.

I suggest pointing this out in the readme and ideally referencing to hyperparameter tuning packages like https://optuna.org/ or similar.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions