Running and convergence issues with FH model

[nathanemac]

@MohamedLaghdafHABIBOULLAH, @dpo

When playing the demo, TR converges almost surely.
However, running R2N alone almost never works with the following error :

┌ Warning: Interrupted. Larger maxiters is needed. If you are using an integrator for non-stiff ODEs or an automatic switching algorithm (the default), you may want to consider using a method for stiff equations. See the solver pages for more details (e.g. https://docs.sciml.ai/DiffEqDocs/stable/solvers/ode_solve/#Stiff-Problems). └ @ SciMLBase ~/.julia/packages/SciMLBase/Pma4a/src/integrator_interface.jl:589
ERROR: DimensionMismatch: array could not be broadcast to match destination

To make R2N work on most runs we first need to run TR, then run R2N on the same problem (with no reset). iR2N with exact prox has the same behaviour and inexact prox often give $$\xi <<0 $$. When using TR before a run, both R2N and iR2N sometimes hit maxIter = 10 000.

R2 never works even with a prior run of TR.

Maybe we can play with how we define parameters infh_model here.

[MohamedLaghdafHABIBOULLAH]

Thanks @nathanemac for raising the issue, have you tried to change $$\sigma_{\min}$$ for both R2N and R2? because the gradient of the smooth term might not be continuous, so that they struggle to solve/evaluate it, as discussed here in this paper.

[nathanemac]

Indeed, some parameters have to be adapted. I tried several setups, and one that makes things work is the following :

1. regularization parameter $$\lambda = 10$$,
2. nice tolerences : $$\epsilon_a = 1e-2$$, $$\epsilon_r = 1e-4$$,
3. set a $$\sigma_{\min}$$ (1e-3 in my setup) and $$\sigma_0$$ (1e3 in my setup)

This way, R2, R2N, TR work with $$\ell_1$$ and give similar results. The solution to the problem is $$\bar{x} = (0.0, 0.2, 1.0, 0.0, 0.0)$$ and the methods cited above give $$x^* \approx (0.0, 0.3, 0.7, 0.0, 0.0)$$.

Eventually, iR2N gives better results than the ones above with the pseudo-ball indicator function (see here) instead of $$\ell_1$$ : $$x^* \approx \bar{x}$$.

[MohamedLaghdafHABIBOULLAH]

Thanks @nathanemac.
I believe a PR in RegularizedOptimization, specifically in the file demo-fh.jl
with the appropriate working parameters, and tests for R2 and R2N, would be quite useful.

What do you think @dpo ?