Questions about using the model for denoising (reconstruction)

[mreyessierra]

Hello, I am interested in using your code to remove noise from a set of time series (reconstruction). I would like to confirm the following: the series are divided (in temporal order) into the first 80% for training and the remaining 20% for testing, correct? So, if I wish to remove noise from one (or several) complete time series, how should I input the data? Regarding this, I have another question: I see that the reconstruction example applies training and evaluation in the training, how would it be to train it and then only test it (I don't see an example where only the evaluation is applied)? One last question: does the denoising process take into account the interactions between the different series, or is it carried out independently for each one? Thank you very much in advance!

[JulesBelveze]

Hey @mreyessierra thanks for your interest. Let me try to answer your questions 🤓

1. Not sure to get what your problem is on that one. Once your model is trained you can use it in eval mode to reconstruct your time series. However, you won't be able to reconstruct for the first few timestamps as they are used as "history". Does that answer your question?
2. I haven't provided code to only perform inference. But if you only want to run the evaluation process you could use the function defined here.
3. Yeah it does leverage the global context! :)

Hope it helps!

[mreyessierra]

Thank you very much for the answers!
Regarding the evaluation, since the model fragments the time series (for training and evaluation), I'm not very clear on which parameter should I use in train_test_split so that the model applies the reconstruction process to the complete series.
On the other hand, regarding this condition: if cfg.general.do_eval and cfg.general.get("ckpt", False), how do I make sure to satisfy the part of cfg.general.get("ckpt", False)? Thank you again!"

[JulesBelveze]

Okay I think I got it. If you want to run the evaluation on the entire series (which is not advised as you would evaluate the model on windows it has been trained on), you would have to set the train_size parameter to 0 (dunno if scikit-learn will complain btw).

Regarding the second part of your question: to meet the condition you would have to either modify the yaml file to have a general.ckpt parameter. Or add a flag +=general.ckpt=PATH_TO_YOUR_CHECKPOINTS when running your file.

Let me know if this helps

[mreyessierra]

Thank you very much for responding, and I apologize for the delay in my reply. What I really wanted to do (and am trying to do) is use one dataset for training-evaluating and another just for evaluating, which is why I was wondering how to evaluate the model on the complete series. The train_test_split function indeed does not allow a zero value for train_size :) I've been working on this these days. Now, I have a question related to the results I've obtained. In the image, I send an example where the predicted series is noisier than the original, which was supposed to be cleaned... do you have any idea why this happens or in what cases this occurs? Thank you very much!

[vsimbalista]

Thank you very much for responding, and I apologize for the delay in my reply. What I really wanted to do (and am trying to do) is use one dataset for training-evaluating and another just for evaluating, which is why I was wondering how to evaluate the model on the complete series. The train_test_split function indeed does not allow a zero value for train_size :) I've been working on this these days. Now, I have a question related to the results I've obtained. In the image, I send an example where the predicted series is noisier than the original, which was supposed to be cleaned... do you have any idea why this happens or in what cases this occurs? Thank you very much!

Hey @mreyessierra,
I experienced this same 'scale' issue when applying this model for anomaly detection.
From my results, it looked like the predicted value was in a completely different scale than the target in the output.
As it turns out, there seems to be a minor bug causing this in the eval.py script:

Pay attention (pun intended) to what happens to the variables predictions and targets that are set on line 44 here:

    predictions, targets = torch.cat(predictions), torch.cat(targets)

    if config.general.do_eval:
        preds, targets = ts.invert_scale(predictions), ts.invert_scale(targets)

        plt.figure()
        plt.plot(preds, linewidth=.3)
        plt.plot(targets, linewidth=.3)
        plt.savefig("{}/preds.png".format(config.general.output_dir))
        plt.close()

        torch.save(targets, os.path.join(config.general.output_dir, "targets.pt"))
        torch.save(predictions, os.path.join(config.general.output_dir, "predictions.pt"))
        torch.save(attentions, os.path.join(config.general.output_dir, "attentions.pt"))

    results = get_eval_report(eval_loss / len(test_iter), predictions, targets)

First, preds and targets have their scales inverted, meaning they are back on the original input scale.
Then, we plot those curves together. So far, so good.

However, when saving that data in Pytorch Tensor format (.pt), we are saving targets (inverted) and predictions (not inverted!!!)
Furthermore, I got weird MSE values (in the hundreds) from the evaluation report in results as well.
That is also caused by the same bug: the arguments sent here are predictions (not inverted) and targets (inverted).

Anyway, I believe this is a slight oversight, as the var targets should not be overwritten inside the do_eval condition.
I actually figured this out because I set do_eval = False (didn't want the plots) and the issue solved itself.
I suspect you did the same I did, trying to read the .pt files to generate new plots.

I will work on a quick fix for this issue and submit a PR.
Let me know if there is anything else I can help with :)