Skip to content

Commit e9f0918

Browse files
noopurintelnoopurintel
authored
E2E Task Runner automation for torch/histology_s3 and s3 datasource (#1640)
* Initial draft not tested Signed-off-by: noopur <noopur@intel.com> * dM Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Added to wf only for testing Signed-off-by: noopur <noopur@intel.com> * Added step to install minio and its client Signed-off-by: noopur <noopur@intel.com> * Final changes Signed-off-by: noopur <noopur@intel.com> * Extra comment for s3 wf Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Pass time_for_each_round Signed-off-by: noopur <noopur@intel.com> * New test and yaml for verifiable dataset Signed-off-by: noopur <noopur@intel.com> * Removed s3 test from basic e2e yamlo Signed-off-by: noopur <noopur@intel.com> * Lint and format check applied Signed-off-by: noopur <noopur@intel.com> * Docstring added Signed-off-by: noopur <noopur@intel.com> * Corrected logic to call prepare_data_for_s3 Signed-off-by: noopur <noopur@intel.com> * Added s3_bucket model Signed-off-by: noopur <noopur@intel.com> * Steps added in the test file Signed-off-by: noopur <noopur@intel.com> * Added no-op and s3 wf in publish dependency list Signed-off-by: noopur <noopur@intel.com> * Fixed the comment while checking calc_hash flag Signed-off-by: noopur <noopur@intel.com> --------- Signed-off-by: noopur <noopur@intel.com>
1 parent 7fcecd3 commit e9f0918

File tree

13 files changed

+1192
-15
lines changed

13 files changed

+1192
-15
lines changed

‎.github/actions/tr_post_test_run/action.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -24,7 +24,7 @@ runs:
2424
id: tar_files
2525
if: ${{ always() }}
2626
run: |
27-
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" --exclude="tensor.db" --exclude="workspace.tar" $HOME/results
27+
tar -cvf result.tar --exclude="cert" --exclude="data" --exclude="__pycache__" --exclude="tensor.db" --exclude="workspace.tar" --exclude="minio_data" $HOME/results
2828
# Model name might contain forward slashes, convert them to underscore.
2929
tmp=${{ env.MODEL_NAME }}
3030
echo "MODEL_NAME_MODIFIED=${tmp//\//_}" >> $GITHUB_ENV

‎.github/workflows/pq_pipeline.yml‎

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -158,6 +158,16 @@ jobs:
158158
with:
159159
commit_id: ${{ needs.set_commit_id_for_all_jobs.outputs.commit_id }}
160160

161+
tr_verifiable_dataset_e2e:
162+
if: |
163+
(github.event_name == 'schedule' && github.repository_owner == 'securefederatedai') ||
164+
(github.event_name == 'workflow_dispatch')
165+
name: TaskRunner Verifiable Dataset E2E
166+
needs: task_runner_e2e
167+
uses: ./.github/workflows/tr_verifiable_dataset_e2e.yml
168+
with:
169+
commit_id: ${{ needs.set_commit_id_for_all_jobs.outputs.commit_id }}
170+
161171
run_trivy:
162172
if: |
163173
(github.event_name == 'schedule' && github.repository_owner == 'securefederatedai') ||
@@ -198,6 +208,7 @@ jobs:
198208
wf_mnist_local_runtime,
199209
wf_watermark_e2e,
200210
wf_secagg_e2e,
211+
task_runner_connectivity_e2e,
201212
task_runner_e2e,
202213
task_runner_resiliency_e2e,
203214
task_runner_fedeval_e2e,
@@ -206,6 +217,7 @@ jobs:
206217
task_runner_dockerized_e2e,
207218
task_runner_secret_ssl_e2e,
208219
task_runner_flower_app_pytorch,
220+
tr_verifiable_dataset_e2e,
209221
run_trivy,
210222
run_bandit
211223
]

‎.github/workflows/tr_verifiable_dataset_e2e.yml‎

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
---
2+
# Task Runner Verifiable Dataset E2E
3+
4+
name: TR_Verifiable_Dataset_E2E # Please do not modify the name as it is used in the composite action
5+
6+
on:
7+
workflow_call:
8+
inputs:
9+
commit_id:
10+
required: false
11+
type: string
12+
workflow_dispatch:
13+
inputs:
14+
num_rounds:
15+
description: "Number of rounds to train"
16+
required: false
17+
default: "2"
18+
type: string
19+
num_collaborators:
20+
description: "Number of collaborators"
21+
required: false
22+
default: "2"
23+
type: string
24+
25+
permissions:
26+
contents: read
27+
28+
# Environment variables common for all the jobs
29+
# DO NOT use double quotes for the values of the environment variables
30+
env:
31+
NUM_ROUNDS: ${{ inputs.num_rounds || 2 }}
32+
NUM_COLLABORATORS: ${{ inputs.num_collaborators || 2 }}
33+
COMMIT_ID: ${{ inputs.commit_id || github.sha }} # use commit_id from the calling workflow
34+
35+
jobs:
36+
test_with_s3: # Run it only if the runner machine has enough memory and CPU
37+
name: With S3 (torch/histology_s3, 3.11)
38+
runs-on: ubuntu-22.04
39+
timeout-minutes: 120
40+
env:
41+
MODEL_NAME: "torch/histology_s3"
42+
PYTHON_VERSION: "3.11"
43+
44+
steps:
45+
- name: Checkout OpenFL repository
46+
id: checkout_openfl
47+
uses: actions/checkout@v4
48+
with:
49+
ref: ${{ env.COMMIT_ID }}
50+
51+
- name: Pre test run
52+
uses: ./.github/actions/tr_pre_test_run
53+
if: ${{ always() }}
54+
55+
- name: Install MinIO
56+
id: install_minio
57+
run: |
58+
wget https://dl.min.io/server/minio/release/linux-amd64/minio
59+
chmod +x minio
60+
sudo mv minio /usr/local/bin/
61+
62+
- name: Install MinIO Client
63+
id: install_minio_client
64+
run: |
65+
wget https://dl.min.io/client/mc/release/linux-amd64/mc
66+
chmod +x mc
67+
sudo mv mc /usr/local/bin/
68+
69+
- name: Run Task Runner E2E tests with S3
70+
id: run_tests
71+
run: |
72+
python -m pytest -s tests/end_to_end/test_suites/tr_verifiable_dataset_tests.py \
73+
-m task_runner_with_s3 --model_name ${{ env.MODEL_NAME }} \
74+
--num_rounds ${{ env.NUM_ROUNDS }} --num_collaborators ${{ env.NUM_COLLABORATORS }}
75+
echo "Task Runner E2E tests with S3 run completed"
76+
77+
- name: Post test run
78+
uses: ./.github/actions/tr_post_test_run
79+
if: ${{ always() }}
80+
with:
81+
test_type: "With_S3"

‎tests/end_to_end/models/collaborator.py‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -246,3 +246,26 @@ def ping_aggregator(self):
246246
log.error(f"{error_msg}: {e}")
247247
raise e
248248
return True
249+
250+
def calculate_hash(self):
251+
"""
252+
Calculate the hash of the data directory and store in hash.txt file
253+
Returns:
254+
bool: True if successful, else False
255+
"""
256+
try:
257+
log.info(f"Calculating hash for {self.collaborator_name}")
258+
cmd = f"fx collaborator calchash --data_path {self.data_directory_path}"
259+
error_msg = "Failed to calculate hash"
260+
return_code, output, error = fh.run_command(
261+
cmd,
262+
error_msg=error_msg,
263+
container_id=self.container_id,
264+
workspace_path=self.workspace_path,
265+
)
266+
fh.verify_cmd_output(output, return_code, error, error_msg, f"Calculated hash for {self.collaborator_name}")
267+
268+
except Exception as e:
269+
log.error(f"{error_msg}: {e}")
270+
raise e
271+
return True

0 commit comments

Comments
 (0)