Documentation for verifiable datasets and data sources (#1636)

teoparvanov · web-flow · commit 735e59302ffd · 2025-05-20T11:32:06.000+02:00
* Verifiable data set and data sources overview doc

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* README.md for the histology_s3 workspace template

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Adding copyright headers to the Mermaid files

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Mermaid syntax fixes

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Added styling to the class diagrams

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Moved the verifiable dataset page above the data splitters

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Cosmetic fix

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

* Addressing review comments

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;

---------

Signed-off-by: Teodor Parvanov &lt;teodor.parvanov@intel.com&gt;
diff --git a/docs/developer_guide/utilities.rst b/docs/developer_guide/utilities.rst
@@ -6,6 +6,9 @@ The following are utilities available in Open Federated Learning (OpenFL).
 :doc:`utilities/pki`
     Use the Public Key Infrastructure (PKI) solution workflows to certify the nodes in your federation.
     
+:doc:`utilities/verifiable_datasets`
+    Build and verify datasets composed of multiple data sources.
+
 :doc:`utilities/splitters_data`
     Split your data to run your federation from a single dataset.
 
@@ -17,5 +20,7 @@ The following are utilities available in Open Federated Learning (OpenFL).
    :hidden:
 
    utilities/pki
+   utilities/verifiable_datasets
    utilities/splitters_data
-   utilities/timeouts
+   utilities/timeouts
+   
diff --git a/docs/developer_guide/utilities/verifiable_datasets.rst b/docs/developer_guide/utilities/verifiable_datasets.rst
@@ -0,0 +1,36 @@
+.. # Copyright (C) 2025 Intel Corporation
+.. # SPDX-License-Identifier: Apache-2.0
+
+*************************************
+Verifiable Datasets and Data Sources
+*************************************
+
+.. _verifiable_datasets_overview:
+
+To accommodate for the proliferation of data sources and the need for trusted datasets, OpenFL provides a hierarchy of utility classes to build and verify datasets. 
+This includes an extensible class hierarchy that enables the creation of datasets from various data sources, such as local file system, object storage and others.
+
+The central abstraction is the :code:`VerifiableDatasetInfo` class that encapsulates the dataset's metadata and provides a method for verifying the integrity of the dataset.
+A dataset can be built from multiple data sources (not necessarily from the same type):
+
+.. mermaid:: ../../mermaid/verifiable_dataset_info.mmd
+    :caption: Verifiable Dataset with Multiple Data Sources
+    :align: center
+
+The :code:`VerifiableDatasetInfo` class can then be used to create higher-order dataset classes that enable iterating through multiple data sources, while verifying integrity if required.
+The :code:`root_hash` is used as a reference for integrity when loading items from the the data sources in the :code:`VerifiableDatasetInfo` object.
+
+OpenFL comes with a toolbox of dataset layout classes per ML framework. For PyTorch's :code:`torch.utils.data.Dataset` OpenFL curently provides: 
+- :code:`FolderDataset` - represents an iterable folder-layout dataset from a single data source, by implementing the :code:`__getitem__` method.
+- :code:`ImageFolder` - a specialization of the :code:`FolderDataset` that is able to load binary images from a foler-like structure
+- :code:`VerifiableMapStyleDataset` - a base class for map-style datasets that can be built from multiple data sources (as specified by a :code:`VerifiableDatasetInfo` object), including integrity checks.
+- :code:`VerifiableImageFolder` - a specialization of the :code:`VerifiableMapStyleDataset` encapsulating a collection of :code:`ImageFolder` datasets
+
+Note that the all those classes (directly or indirectly) extend :code:`torch.utils.data.DataLoader`, and are therefore compatible with all PyTorch utilities for pre-processing data sets.
+A similar class hierarchy can be created for other ML frameworks that offer dataset utilities, such as TensorFlow.
+
+.. mermaid:: ../../mermaid/verifiable_image_folder.mmd
+    :caption: Dataset hierarchy
+    :align: center
+
+A practical example for the :code:`VerifiableImageFolder` backed by a mix of :code:`LocalDataSource` and :code:`S3DataSource` objects is provided in the `s3_histology <https://github.com/securefederatedai/openfl/tree/develop/openfl-workspace/torch/histology_s3>`_ workspace template.
diff --git a/docs/mermaid/verifiable_dataset_info.mmd b/docs/mermaid/verifiable_dataset_info.mmd
@@ -0,0 +1,56 @@
+%% Copyright 2025 Intel Corporation
+%% SPDX-License-Identifier: Apache-2.0
+
+classDiagram
+    class VerifiableDatasetInfo {
+        +label: str
+        +data_sources: DataSource[] 
+        +metadata: dict[str, str] 
+        +root_hash: HASH 
+
+        +verify_dataset(root_hash: HASH)
+        +verify_single_file(file_path: str, file_hash: HASH)
+        +to_json() str
+        +from_json(json_str: str) VerifiableDatasetInfo
+    }
+
+    class DataSource {
+        <<abstract>>
+        +name: str 
+        +type: DataSourceType 
+        +compute_file_hash(path: str) str
+        +enumerate_files() Generator~str~
+        +read_blob(path: str) bytes
+        +from_dict(ds_dict: dict) DataSource
+        +is_valid_hash_function(func) bool
+        +to_dict() dict
+    }
+
+    class LocalDataSource {
+        +base_path: str
+        ...
+    }
+
+    class S3DataSource {
+        +uri: str 
+        +endpoint: str
+        ...
+    }
+
+    class AzureBlobDataSource {
+        +name: str 
+        +container_string: str 
+        +folder_prefix: str
+        ...
+    }
+
+    VerifiableDatasetInfo "1" o-- "*" DataSource
+    DataSource <|-- LocalDataSource
+    DataSource <|-- S3DataSource
+    DataSource <|-- AzureBlobDataSource
+
+    style VerifiableDatasetInfo fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style DataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style LocalDataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style S3DataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
+    style AzureBlobDataSource fill:#FFFFE0,stroke:#000,stroke-width:1px
diff --git a/docs/mermaid/verifiable_image_folder.mmd b/docs/mermaid/verifiable_image_folder.mmd
@@ -0,0 +1,50 @@
+%% Copyright 2025 Intel Corporation
+%% SPDX-License-Identifier: Apache-2.0
+
+classDiagram
+    class torch_utils_data_Dataset {
+        +__len__() int
+        +__getitem__(index: int) Any
+    }
+
+    class VerifiableDatasetInfo {
+        +verify_dataset(root_hash: HASH)
+        +verify_single_file(file_path: str, file_hash: HASH)
+        +from_json(json_str: str) VerifiableDatasetInfo
+    }
+
+    class VerifiableMapStyleDataset {
+        <<abstract>>
+        +__len__() int
+        +__getitem__(index: int) Any
+        +create_datasets() void*
+    }
+
+    class VerifiableImageFolder {
+        +__len__() int
+        +__getitem__(index: int) Any
+        +create_datasets() void
+    }
+
+    class FolderDataset {
+        <<abstract>>
+        +__len__() int
+        +__getitem__(index: int) Any
+        +load_file(file_path: str) void*
+    }
+
+    class ImageFolder {
+        +__len__() int
+        +__getitem__(index: int) Any
+        +load_file(file_path: str) void
+    }
+
+    torch_utils_data_Dataset <|.. VerifiableMapStyleDataset
+    torch_utils_data_Dataset <|.. FolderDataset
+    VerifiableMapStyleDataset o-- VerifiableDatasetInfo
+    VerifiableMapStyleDataset <|-- VerifiableImageFolder
+    VerifiableMapStyleDataset o-- FolderDataset
+    FolderDataset <|-- ImageFolder
+
+    style torch_utils_data_Dataset fill:#D3D3D3,stroke:#000,stroke-width:1px
+    style VerifiableDatasetInfo fill:#FFFFE0,stroke:#000,stroke-width:1px
diff --git a/openfl-workspace/torch/histology_s3/README.md b/openfl-workspace/torch/histology_s3/README.md
@@ -0,0 +1,31 @@
+## Overview
+This workspace template showcases the use of `VerifiableImageFolder` among a federation of two collaborators with a diverse mix of data sources (`LocalDataSource` and `S3DataSource`). The data source JSON descriptors are located under `./data/collaborator1` and `./data/collaborator2`. In a distributed setup, those would be located on separate machines at each collaborator's premises, to make the entire experiment executable locally.
+
+It is important to note that the integrity verification is done based on a hash of each dataset that is calculated _prior_ to the experiment via the `fx collaborator calchash` command. Doing so provides protection against various data integrity attacks that could occur between the dataset preparation and the actual federated learning process.
+
+## Steps to run the experiment
+1. Set up a MinIO server which hosts the S3 data sources from the JSON descriptors
+2. Configure the credentials
+    ```shell
+    export YOUR_ACCESS_KEY=<your_access_key>
+    export YOUR_SECRET_KEY=<your_secret_key>
+    ```
+
+    For example:
+    ```shell
+    export MINIO_ROOT_PASSWORD=minioadmin
+    export MINIO_ROOT_USER=minioadmin
+    ```
+3. Create a workspace folder from the `histology_s3` template
+    ```shell
+    fx workspace create --prefix ~/hist_s3 --template torch/histology_s3
+    cd ~/hist_s3/
+    pip install -r requirements.txt
+    ```
+4. Optional: Calculate the dataset hashes for both collaborators.
+The hash will be saved at `<data_path>/hash.txt`. Later on, when the Federated Learning process starts, the data loader will check for this file and verify the dataset's integrity against the hash stored inside:
+    ```shell
+    fx collaborator calchash --data_path plan/collaborator1
+    fx collaborator calchash --data_path plan/collaborator2
+    ```
+5. For the rest of the steps, follow the [quickstart guide](https://openfl.readthedocs.io/en/latest/tutorials/taskrunner.html)
