Use autodiff API based on ADTypes instead of symbols

[devmotion]

Fixes #1187

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 86.53%. Comparing base (de9250c) to head (adc9ff4).

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1195      +/-   ##
==========================================
- Coverage   87.48%   86.53%   -0.95%     
==========================================
  Files          45       45              
  Lines        3517     3514       -3     
==========================================
- Hits         3077     3041      -36     
- Misses        440      473      +33     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[devmotion]

One could consider switching to a different default AD backend (maybe ForwardDiff for univariate optimization and e.g. Mooncake - if at least all tests pass - for multivariate optimization problems? - but the choice for when to switch to which backend is likely also problem/dimension dependent, see also https://docs.sciml.ai/Optimization/stable/API/ad/#ad). But I think it would be better to make such more fundamental changes in a separate PR.

[pkofod]

I agree both to switch to reverse mode for multivariate and forward for univariate, but also that it's probably a separate PR

[github-actions]

Benchmark Results (Julia vlts)

Time benchmarks

	master	adc9ff4...	master / adc9ff4...
multivariate/solvers/first_order/AdaMax	0.544 ± 0.0091 ms	0.543 ± 0.0093 ms	1 ± 0.024
multivariate/solvers/first_order/Adam	0.544 ± 0.0091 ms	0.545 ± 0.009 ms	0.998 ± 0.023
multivariate/solvers/first_order/BFGS	0.263 ± 0.0082 ms	0.263 ± 0.0083 ms	1 ± 0.044
multivariate/solvers/first_order/ConjugateGradient	0.176 ± 0.0026 ms	0.175 ± 0.0028 ms	1.01 ± 0.022
multivariate/solvers/first_order/GradientDescent	1.56 ± 0.011 ms	1.55 ± 0.013 ms	1 ± 0.011
multivariate/solvers/first_order/LBFGS	0.234 ± 0.0076 ms	0.234 ± 0.0072 ms	0.999 ± 0.045
multivariate/solvers/first_order/MomentumGradientDescent	2.18 ± 0.014 ms	2.18 ± 0.016 ms	0.998 ± 0.0097
multivariate/solvers/first_order/NGMRES	0.434 ± 0.011 ms	0.431 ± 0.011 ms	1.01 ± 0.035
time_to_load	0.415 ± 0.0032 s	0.41 ± 0.00073 s	1.01 ± 0.0081

Memory benchmarks

	master	adc9ff4...	master / adc9ff4...
multivariate/solvers/first_order/AdaMax	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/Adam	0.34 k allocs: 7.16 kB	0.339 k allocs: 7.09 kB	1.01
multivariate/solvers/first_order/BFGS	0.36 k allocs: 15.5 kB	0.359 k allocs: 15.5 kB	1
multivariate/solvers/first_order/ConjugateGradient	0.362 k allocs: 14.2 kB	0.361 k allocs: 14.1 kB	1
multivariate/solvers/first_order/GradientDescent	2.09 k allocs: 0.0759 MB	2.08 k allocs: 0.0758 MB	1
multivariate/solvers/first_order/LBFGS	0.341 k allocs: 14.7 kB	0.34 k allocs: 14.7 kB	1
multivariate/solvers/first_order/MomentumGradientDescent	2.44 k allocs: 0.0815 MB	2.44 k allocs: 0.0815 MB	1
multivariate/solvers/first_order/NGMRES	1.56 k allocs: 0.117 MB	1.56 k allocs: 0.117 MB	1
time_to_load	0.153 k allocs: 14.5 kB	0.153 k allocs: 14.5 kB	1

A plot of the benchmark results has been uploaded as an artifact at .

[pkofod]

JET complains

[devmotion]

I checked it locally, and none of the things JET complains are introduced or changed by the PR. It seems the PR just helps JET to perform its type-signature-based analysis more thoroughly, and hence more problems are revealed 😄

I could fix a few of them in this PR but arguably it might be better to do that in a separate PR.