New DiffEntropyEst interface (#239)

* Change definition of `entropy(diff_ent_est, ...)`

* move Kraskov to new interface

* port kozacghenko to new interafce

* port Zhu to new intercace

* move ZhuSign to new interface

* move Lord to new interface

* move Gao and Goria to new interface

* move all order statistics estiamtors to new intercface

* [WIP] fixing tests

* add internal function for check with given entropy def

* port all estimator tests to new interface

* check if base is same in compatibility

* remove unecessary test

* discuss difference of estimator and definition

* add discrete entropy estimator type

* add it to normalizd/maximum as well

* completel;y remoe possibility of entropy(def, differential_estimator)

* rename to MLEntropy

* docstring of entropy should use estimator

* add discrete entropy estimator to docs

* remove testing of old interface

* correct Alizadeh error

* resolve all outstanding issues

* fix broken normalized entropy code

* done

* up verson

Author: Datseris

CHANGELOG.md

@@ -2,6 +2,11 @@
 
 Changelog is kept with respect to version 0.11 of Entropies.jl. From version v2.0 onwards, this package has been renamed to ComplexityMeasures.jl.
 
+## 2.3
+- Like differential entropies, discrete entropies now also have their own estimator type.
+- The approach of giving both an entropy definition, and an entropy estimator to `entropy` has been dropped. Now the entropy estimators know what definitions they are applied for. This change is a deprecation, i.e., backwards compatible.
+- Added `MLEntropy` discrete entropy estimator.
+
 ## 2.2
 
 - Corrected documentation for `SymbolicPermutation`, `SymbolicAmplitudeAwarePermutation`,

Project.toml

@@ -2,7 +2,7 @@ name = "ComplexityMeasures"
 uuid = "ab4b797d-85ee-42ba-b621-05d793b346a2"
 authors = "Kristian Agasøster Haaga <kahaaga@gmail.com>, George Datseries <datseris.george@gmail.com>"
 repo = "https://github.com/juliadynamics/ComplexityMeasures.jl.git"
-version = "2.2.0"
+version = "2.3.0"
 
 [deps]
 Combinatorics = "861a8166-3701-5b0c-9a16-15d98fcdc6aa"

docs/src/entropies.md

@@ -6,9 +6,10 @@ The entropies API is defined by
 
 - [`EntropyDefinition`](@ref)
 - [`entropy`](@ref)
+- [`DiscreteEntropyEstimator`](@ref)
 - [`DifferentialEntropyEstimator`](@ref)
 
-Please be sure you have read the [Terminology](@ref) section before going through the API here, to have a good idea of the different "flavors" of entropies and how they all come together over the common interface of the [`entropy`](@ref) function.
+Please be sure you have read the [Terminology](@ref terminology) section before going through the API here, to have a good idea of the different "flavors" of entropies and how they all come together over the common interface of the [`entropy`](@ref) function.
 
 ## Entropy definitions
 
@@ -30,13 +31,20 @@ entropy_maximum
 entropy_normalized
 ```
 
+### Discrete entropy estimators
+
+```@docs
+DiscreteEntropyEstimator
+MLEntropy
+```
+
 ## Differential entropy
 
 ```@docs
-entropy(::EntropyDefinition, ::DifferentialEntropyEstimator, ::Any)
+entropy(::DifferentialEntropyEstimator, ::Any)
 ```
 
-### Table of differential entropy estimators
+### [Table of differential entropy estimators](@id table_diff_ent_est)
 
 The following estimators are *differential* entropy estimators, and can also be used
 with [`entropy`](@ref).

docs/src/examples.md

@@ -95,7 +95,7 @@ end
 # Plot results
 # -------------
 fig = Figure(resolution = (700, 11 * 200))
-labels_knn = ["KozachenkoLeonenko", "Kraskov", "Zhu", "ZhuSingh", "Gao (not corrected)", 
+labels_knn = ["KozachenkoLeonenko", "Kraskov", "Zhu", "ZhuSingh", "Gao (not corrected)",
     "Gao (corrected)", "Goria", "Lord"]
 labels_os = ["Vasicek", "Ebrahimi", "AlizadehArghami", "Correa"]
 
@@ -354,7 +354,7 @@ When comparing different signals or signals that have different length, it is be
 
 ```@example MAIN
 using DynamicalSystemsBase
-N1, N2, a = 101, 100001, 10
+N1, N2, a = 101, 10001, 10
 
 for N in (N1, N2)
     local t = LinRange(0, 2*a*π, N)

docs/src/index.md

@@ -4,7 +4,7 @@
 ComplexityMeasures
 ```
 
-## Content and terminology
+## [Content and terminology](@id terminology)
 
 !!! note
     The documentation here follows (loosely) chapter 5 of
@@ -18,9 +18,8 @@ from input data.
 
 ### Probabilities
 
-Entropies and other complexity measures are typically computed based on *probability
-distributions* (or more precisely
-[*probability mass functions*](https://en.wikipedia.org/wiki/Probability_mass_function)),
+Entropies and other complexity measures are typically computed based on _probability
+distributions_,
 which we simply refer to as "probabilities".
 Probabilities can be obtained from input data in a plethora of different ways.
 The central API function that returns a probability distribution
@@ -36,14 +35,16 @@ even fundamentally, different quantities.
 In ComplexityMeasures.jl, we provide the generic
 function [`entropy`](@ref) that tries to both clarify disparate entropy concepts, while
 unifying them under a common interface that highlights the modular nature of the word
-"entropy". In summary, there are only two main types of entropy.
+"entropy".
 
-- *Discrete* entropies are functions of probabilities (specifically, probability mass functions). Computing a discrete entropy boils
+On the highest level, there are two main types of entropy.
+
+- *Discrete* entropies are functions of [probability mass functions](https://en.wikipedia.org/wiki/Probability_mass_function). Computing a discrete entropy boils
     down to two simple steps: first estimating a probability distribution, then plugging
-    the estimated probabilities into one of the so-called "generalized entropy" definitions.
+    the estimated probabilities into an estimator of a so-called "generalized entropy" definition.
     Internally, this is literally just a few lines of code where we first apply some
     [`ProbabilitiesEstimator`](@ref) to the input data, and feed the resulting
-    [`probabilities`](@ref) to [`entropy`](@ref) with some [`EntropyDefinition`](@ref).
+    [`probabilities`](@ref) to [`entropy`](@ref) with some [`DiscreteEntropyEstimator`](@ref).
 - *Differential/continuous* entropies are functions of
     [probability density functions](https://en.wikipedia.org/wiki/Probability_density_function),
     which are *integrals*. Computing differential entropies therefore rely on estimating
@@ -59,15 +60,15 @@ They are the good old discrete Shannon entropy ([`Shannon`](@ref)), but calculat
 *new probabilities estimators*.
 
 Even though the names of these methods (e.g. "wavelet entropy") sound like names for new
-entropies, they are *method* names. What these methods actually do is to devise novel
+entropies, what they actually do is to devise novel
 ways of calculating probabilities from data, and then plug those probabilities into formal
 discrete entropy formulas such as
 the Shannon entropy. These probabilities estimators are of course smartly created so that
 they elegantly highlight important complexity-related aspects of the data.
 
 Names for methods such as "permutation entropy" are commonplace, so in
 ComplexityMeasures.jl we provide convenience functions like [`entropy_permutation`](@ref).
-However, we emphasise that these functions really aren't anything more than
+However, we emphasize that these functions really aren't anything more than
 2-lines-of-code wrappers that call [`entropy`](@ref) with the appropriate
 [`ProbabilitiesEstimator`](@ref).
 

src/deprecations.jl

@@ -1,5 +1,14 @@
-@deprecate TimeScaleMODWT WaveletOverlap
+# from before https://github.com/JuliaDynamics/ComplexityMeasures.jl/pull/239
+function entropy(e::EntropyDefinition, est::DiffEntropyEst, x)
+    if e isa Shannon
+        return entropy(est, x)
+    else
+        error("only shannon entropy supports this deprecated interface")
+    end
+end
 
+# From before 2.0:
+@deprecate TimeScaleMODWT WaveletOverlap
 function probabilities(x::Vector_or_Dataset, ε::Union{Real, Vector{<:Real}})
     @warn """
     `probabilities(x::Vector_or_Dataset, ε::Real)`

src/encoding_implementations/rectangular_binning.jl

@@ -68,16 +68,17 @@ end
     RectangularBinEncoding(binning::RectangularBinning, x; n_eps = 2)
     RectangularBinEncoding(binning::FixedRectangularBinning; n_eps = 2)
 
-Struct used in [`outcomes`](@ref) to map points of `x` into their respective bins.
+An encoding scheme that [`encode`](@ref)s points `χ ∈ x` into their histogram bins.
 It finds the minima along each dimension, and computes appropriate
 edge lengths for each dimension of `x` given a rectangular binning.
 
 The second signature does not need `x` because (1) the binning is fixed, and the
 size of `x` doesn't matter, and (2) because the binning contains the dimensionality
 information as `ϵmin/max` is already an `NTuple`.
 
-Due to roundoff error when computing bin edges, a small tolerance `n_eps * eps()`
-is added to bin widths to ensure the correct number of bins is produced.
+Due to roundoff error when computing bin edges, the computed bin widths
+are increased to their `nextfloat` `n_eps` times
+to ensure the correct number of bins is produced.
 
 See also: [`RectangularBinning`](@ref), [`FixedRectangularBinning`](@ref).
 """

src/entropies_definitions/kaniadakis.jl

@@ -8,12 +8,13 @@ The Kaniadakis entropy (Tsallis, 2009)[^Tsallis2009], used with [`entropy`](@ref
 compute
 
 ```math
-H_K(p) = -\\sum_{i=1}^N p_i\\log_\\kappa^K(p_i),
+H_K(p) = -\\sum_{i=1}^N p_i f_\\kappa(p_i),
 ```
 ```math
-\\log_\\kappa = \\dfrac{x^\\kappa - x^{-\\kappa}}{2\\kappa},
+f_\\kappa (x) = \\dfrac{x^\\kappa - x^{-\\kappa}}{2\\kappa},
 ```
-where if ``\\kappa = 0``, regular logarithm to the given `base` is used, and `log(0) = 0`.
+where if ``\\kappa = 0``, regular logarithm to the given `base` is used, and
+0 probabilities are skipped.
 
 [^Tsallis2009]:
     Tsallis, C. (2009). Introduction to nonextensive statistical mechanics: approaching a

src/entropies_definitions/shannon.jl

@@ -11,6 +11,9 @@ H(p) = - \\sum_i p[i] \\log(p[i])
 ```
 with the ``\\log`` at the given `base`.
 
+The maximum value of the Shannon entropy is ``\\log_{base}(L)``, which is the entropy of the
+uniform distribution with ``L`` the [`total_outcomes`](@ref).
+
 [^Shannon1948]: C. E. Shannon, Bell Systems Technical Journal **27**, pp 379 (1948)
 """
 Base.@kwdef struct Shannon{B} <: EntropyDefinition
@@ -19,7 +22,7 @@ end
 
 function entropy(e::Shannon, probs::Probabilities)
     base = e.base
-    non0_probs = Iterators.filter(!iszero, probs.p)
+    non0_probs = Iterators.filter(!iszero, vec(probs))
     logf = log_with_base(base)
     return -sum(x*logf(x) for x in non0_probs)
 end

src/entropies_definitions/tsallis.jl

@@ -20,8 +20,8 @@ with `k` standing for the Boltzmann constant. It is defined as
 S_q(p) = \\frac{k}{q - 1}\\left(1 - \\sum_{i} p[i]^q\\right)
 ```
 
-If the probability estimator has known alphabet length ``L``, then the maximum
-value of the Tsallis entropy is ``k(L^{1 - q} - 1)/(1 - q)``.
+The maximum value of the Tsallis entropy is ````k(L^{1 - q} - 1)/(1 - q)``,
+with ``L`` the [`total_outcomes`](@ref).
 
 [^Tsallis1988]:
     Tsallis, C. (1988). Possible generalization of Boltzmann-Gibbs statistics.