cosmetic changes

Author: Julien Roussel

docs/analysis.rst

@@ -1,22 +1,22 @@
 
 Analysis
 ========
-The analysis section gives a better understanding of the holes in a dataset.
+This section gives a better understanding of the holes in a dataset.
 
 1. General approach
 -------------------
 
 As described in section :ref:`hole_generator`, there are 3 main types of missing data mechanism: MCAR, MAR and MNAR.
-The analysis brick provides tools to charaterize the type of holes.
+The analysis module provides tools to characterize the type of holes.
 
-The MNAR case is the trickiest, the user must first consider whether or not his missing data mechanism is MNAR. In the meantime, we make the assumption that the missing-data mechanism is ignorable (ie is not MNAR). If the MNAR missing data mechanism is suspected, please see this article :ref:`An approach to test for MNAR [1]<Noonan-article>`.
+The MNAR case is the trickiest, the user must first consider whether their missing data mechanism is MNAR. In the meantime, we make assume that the missing-data mechanism is ignorable (ie., it is not MNAR). If an MNAR mechanism is suspected, please see this article :ref:`An approach to test for MNAR [1]<Noonan-article>` for relevant actions.
 
 Then Qolmat proposes a test to determine whether the missing data mechanism is MCAR or MAR.
 
-2. How to use the results ?
----------------------------
+2. How to use the results
+-------------------------
 
-At the end of the MCAR test, it can then be assumed whether the missing data mechanism is MCAR or not. This could be used for several things :
+At the end of the MCAR test, it can then be assumed whether the missing data mechanism is MCAR or not. This serves three differents purposes:
 
 a. Diagnosis
 ^^^^^^^^^^^^
@@ -27,30 +27,30 @@ The test result can then be used for continuous data quality management.
 b. Estimation
 ^^^^^^^^^^^^^
 
-Some estimation methods are not suitable for the MAR case. For example, dropingn the nans introduces bias into the estimator, it is necessary to have validated that the missing-data mechanism is MCAR.
+Some estimation methods are not suitable for the MAR case. For example, dropping the nans introduces bias into the estimator, it is necessary to have validated that the missing-data mechanism is MCAR.
 
 c. Imputation
 ^^^^^^^^^^^^^
 
-Qolmat allows model selection imputation algorithms. For each of the K folds, Qolmat artificially masks a set of observed values using a default or user specified hole generator. It seems natural to create these masks according to the same missing-data mechanism as dtermined by the test. Here's the documentation on using Qolmat for imputation model selection. : `here <https://qolmat.readthedocs.io/en/latest/#:~:text=How%20does%20Qolmat%20work%20%3F>`_.
+Qolmat allows model selection imputation algorithms. For each of the K folds, Qolmat artificially masks a set of observed values using a default or user-specified hole generator. It seems natural to create these masks according to the same missing-data mechanism as determined by the test. Here is the documentation on using Qolmat for imputation `model selection <https://qolmat.readthedocs.io/en/latest/#:~:text=How%20does%20Qolmat%20work%20%3F>`_.
 
 3. The MCAR Tests
 -----------------
 
-There exist several statistical tests to determine if the missing data mechanism is MCAR or MAR. Most tests are based on the notion of missing pattern.
-A missing pattern, also called pattern, is the structure of observed and missing values in a dataset. For example, for a dataset with 2 columns, the possible patterns are : (0, 0), (1, 0), (0, 1), (1, 1). The value 1 indicates that the value in the column is missing.
+There are several statistical tests to determine if the missing data mechanism is MCAR or MAR. Most tests are based on the notion of missing pattern.
+A missing pattern, also called a pattern, is the structure of observed and missing values in a dataset. For example, for a dataset with two columns, the possible patterns are: (0, 0), (1, 0), (0, 1), (1, 1). The value 1 indicates that the value in the column is missing.
 
 The MCAR missing-data mechanism means that there is independence between the presence of holes and the observed values. In other words, the data distribution is the same for all patterns.
 
 a. Little's Test
 ^^^^^^^^^^^^^^^^
 
-The best-known MCAR test is the :ref:`Little [2]<Little-article>` test. Keep in mind that the Little's test is designed to test the homogeneity of means accross the missing patterns and won't be efficient to detect the heterogeneity of covariance accross missing patterns.
+The best-known MCAR test is the :ref:`Little [2]<Little-article>` test, and it has been implemented in :class:`LittleTest`. Keep in mind that the Little's test is designed to test the homogeneity of means across the missing patterns and won't be efficient to detect the heterogeneity of covariance accross missing patterns.
 
 b. PKLM Test
 ^^^^^^^^^^^^
 
-The :ref:`PKLM [2]<PKLM-article>` (Projected Kullback-Leibler MCAR) test compares the distributions of different missing patterns on random projections in the variable space of the data. This recent test applies to mixed-type data.
+The :ref:`PKLM [2]<PKLM-article>` (Projected Kullback-Leibler MCAR) test compares the distributions of different missing patterns on random projections in the variable space of the data. This recent test applies to mixed-type data. It is not implemented yet in Qolmat.
 
 References
 ----------
@@ -61,7 +61,7 @@ References
 
 .. _Little-article:
 
-[2] Little. `A Test of Missing Completely at Random for Multivariate Data with Missing Values. <https://www.tandfonline.com/doi/abs/10.1080/01621459.1988.10478722>`_ Journal of the American Statistical Association, Volume 83, 1988 - Issue 404.
+[2] Little, R. J. A. `A Test of Missing Completely at Random for Multivariate Data with Missing Values. <https://www.tandfonline.com/doi/abs/10.1080/01621459.1988.10478722>`_ Journal of the American Statistical Association, Volume 83, 1988 - Issue 404.
 
 .. _PKLM-article:
 

docs/index.rst

@@ -31,4 +31,4 @@
    :caption: ANALYSIS
 
    analysis
-   examples/tutorials/plot_tuto_mcar_test
+   examples/tutorials/plot_tuto_mcar

examples/tutorials/plot_tuto_mcar_test.py examples/tutorials/plot_tuto_mcar.pyexamples/tutorials/plot_tuto_mcar_test.py renamed to examples/tutorials/plot_tuto_mcar.py

@@ -5,6 +5,7 @@
 
 In this tutorial, we show how to test the MCAR case using the Little's test.
 """
+
 # %%
 # First import some libraries
 from matplotlib import pyplot as plt

qolmat/analysis/holes_characterization.py

@@ -8,7 +8,7 @@
 from qolmat.imputations.imputers import ImputerEM
 
 
-class MCARTest(ABC):
+class McarTest(ABC):
     """
     Astract class for MCAR tests.
     """
@@ -18,11 +18,11 @@ def test(self, df: pd.DataFrame) -> float:
         pass
 
 
-class LittleTest(MCARTest):
+class LittleTest(McarTest):
     """
-    This class implements the Little's test. The Little's test is designed to detect the
-    heterogeneity accross the missing patterns. The null hypothesis is "The missing data mechanism
-    is MCAR". Be aware that this test won't detect the heterogeneity of covariance.
+    This class implements the Little's test, which is designed to detect the heterogeneity accross
+    the missing patterns. The null hypothesis is "The missing data mechanism is MCAR". The
+    shortcoming of this test is that it won't detect the heterogeneity of covariance.
 
     References
     ----------
@@ -67,15 +67,16 @@ def test(self, df: pd.DataFrame) -> float:
             The p-value of the test.
         """
         imputer = self.imputer or ImputerEM(random_state=self.random_state)
-        fitted_imputer = imputer._fit_element(df)
+        imputer = imputer._fit_element(df)
 
         d0 = 0
         n_rows, n_cols = df.shape
         degree_f = -n_cols
-        ml_means = fitted_imputer.means
-        ml_cov = n_rows / (n_rows - 1) * fitted_imputer.cov
+        ml_means = imputer.means
+        ml_cov = n_rows / (n_rows - 1) * imputer.cov
 
         # Iterate over the patterns
+
         df_nan = df.notna()
         for tup_pattern, df_nan_pattern in df_nan.groupby(df_nan.columns.tolist()):
             n_rows_pattern, _ = df_nan_pattern.shape
@@ -89,4 +90,4 @@ def test(self, df: pd.DataFrame) -> float:
             d0 += n_rows_pattern * np.dot(np.dot(diff_means, inv_sigma_pattern), diff_means.T)
             degree_f += tup_pattern.count(True)
 
-        return 1 - chi2.cdf(d0, degree_f)
+        return 1 - float(chi2.cdf(d0, degree_f))

tests/analysis/test_holes_characterization.py

@@ -42,7 +42,7 @@ def mar_hc_df() -> pd.DataFrame:
     quantile_95 = norm.ppf(0.975)
     df = pd.DataFrame(matrix, columns=["Column_1", "Column_2"])
     df_nan = df.copy()
-    df_nan.loc[abs(df_nan["Column_1"]) > quantile_95, "Column_2"] = np.nan
+    df_nan.loc[df_nan["Column_1"].abs() > quantile_95, "Column_2"] = np.nan
 
     df_mask = df_nan.isna()
     return df.mask(df_mask)