Possible bug in ablation

[Michael190502]

Hi,

I think I found a possible bug in the ablation file.

Firstly I think after the following line

DeepCAVE/deepcave/evaluators/ablation.py

Line 128 in 58d6801

inc_cost, _ = self._model.predict(np.array([incumbent_encode]))

inc_cost = inc_cost[0] is missing (which is there for def_cost). This though only affects the comparison between these two values a few lines later I think.

Secondly the ablation path analysis did not work correctly for me when I had only categorical hyperparameters. I traced the problem to the ablation file where "encode_config" is used instead of "get_array". This resulted, in my case at least, in an encoding normalised to [0,1] instead of the number of the choice of the categorical hyperparameter. This resulted in the predicted performance of the default configuration being equal to the predicted performance of the incumbent.

It worked correctly when I replaced "encode_config" with "get_array".

Please let me know if this is something specific to my setup or if I did something wrong. I will append an image comparing the encoding of "get_array" to "encode_config" for clarity.

[mwever]

Hi @Michael190502 ,

thanks for reporting this issue, we will have a look at it!

[KrissiHub]

Hi @Michael190502 ,

I am currently looking into the issue, could you provide a minimal code example/snippet so I can recreate the problem?

[Michael190502]

Hi @KrissiHub,

as I dont have a prepared run at hand I utilized the example for ablation from examples/api/ablation_paths.py to create an example showing the difference. To demonstrate the difference I made the following simple changes:

• changed the loaded run in line 16 of the examples/api/ablation_paths.py file to "logs/DeepCAVE/digits_sklearn/run_1" to include a categorical variable
• changed the default value of the activation categorical variable in "logs/DeepCAVE/digits_sklearn/run_1/configspace.json" from logistic to tanh to avoid having the encoding be zero.
• Added print(default_encode[0], self.default_config.get_array()[0]) after line 123 in deepcave/evaluators/ablation.py to display the difference in encoding

I hope this suffices as an example even if it is not a code snippet.

Afterwards you could replace all encode_config with get_array to see the difference in the computed ablation paths. I included the two improvement plots before and after changing the encoding. (The first one is after)

[KrissiHub]

Thank you for your description of the problem. I tried to get the same results as you by applying the steps you provided to the api example. Unfortunately, the results I got were different from yours. Could you create a fork with your applied changes, so we can better see the differences?

[Michael190502]

Hi, I did the fork. https://github.com/Michael190502/DeepCAVE . I also added a minimal example script showing the difference between encode_config and get array. I also tested it again with a clean conda environment and got the same results. Please let me know if this was what you wanted as I have never forked a repository before.