[WIP - Don't review]Patch REST get_aggregated_tensor api for batched changes #1653
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
ishaileshpant
changed the title
ishaileshpant
requested review from
MasterSkepticista,
aayushgaintel,
gbikkiintel,
kminhta,
noopurintel,
pasokan-intel,
payalcha,
psfoley,
rahulga1,
rajithkrishnegowda,
srikanthenugul,
tanwarsh,
teoparvanov and
theakshaypant
as code owners
ishaileshpant
force-pushed
the
batched_fetch_restapi_fix
branch
2 times, most recently
from
c13a40b to
d6c93c4
Compare
* code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * remove workspace Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> * code changes Signed-off-by: yes <shailesh.tanwar@intel.com> --------- Signed-off-by: yes <shailesh.tanwar@intel.com>
…atedai#1500) - add new AggregatorClientInterface to allow switching b/w grpc and rest - endhance existing AggregatorGRPCClient to start using AggregatorClientInterface - added new transport package for rest with AggregatorRESTClient implementing AggregatorClientInterface - added streaming api support with custom content-type - added various connection flag for streaming request - send additional header key "Sender" for better request logging at server side - aligned Rest and gRPC client for most of the init params - added AggregatorRESTServer and necesary changes in aggregator cli and federated/plan get_server method - added transport_protocol settings in defaults/network.yaml, defaulted the same to 'grpc' - reduced cyclomatic complexity of Rest Server - fixed protobuf streaming issue for v1/task/results API - added more detailed logging for task progression and metadata for each api calls - pinned Flask version to latest stable 3.1.0 - addressing review comments - 13th-May - added ping api and `collaborato` constructor hint for `AggregatorClientInterface` - added send_message_to_server in client and AggregatorClientInterface, Rest Server is already at parity - changed base uri for REST server to 'experimental/v1', adjusted the client and tests accordingly - fixed issue related to mTLS in REST server/client - disabled TLS 1.2 in both server/client rebased 21st.May.1 Signed-off-by: Shailesh Pant <shailesh.pant@intel.com>
…ce (securefederatedai#1640) * Initial draft not tested Signed-off-by: noopur <noopur@intel.com> * dM Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Added to wf only for testing Signed-off-by: noopur <noopur@intel.com> * Added step to install minio and its client Signed-off-by: noopur <noopur@intel.com> * Final changes Signed-off-by: noopur <noopur@intel.com> * Extra comment for s3 wf Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Modified Signed-off-by: noopur <noopur@intel.com> * Pass time_for_each_round Signed-off-by: noopur <noopur@intel.com> * New test and yaml for verifiable dataset Signed-off-by: noopur <noopur@intel.com> * Removed s3 test from basic e2e yamlo Signed-off-by: noopur <noopur@intel.com> * Lint and format check applied Signed-off-by: noopur <noopur@intel.com> * Docstring added Signed-off-by: noopur <noopur@intel.com> * Corrected logic to call prepare_data_for_s3 Signed-off-by: noopur <noopur@intel.com> * Added s3_bucket model Signed-off-by: noopur <noopur@intel.com> * Steps added in the test file Signed-off-by: noopur <noopur@intel.com> * Added no-op and s3 wf in publish dependency list Signed-off-by: noopur <noopur@intel.com> * Fixed the comment while checking calc_hash flag Signed-off-by: noopur <noopur@intel.com> --------- Signed-off-by: noopur <noopur@intel.com>
…ecurefederatedai#1638) * Add TaskRunner Analytics E2E workflow and integrate into PQ pipeline Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * review comments Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * fix rest and add models Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * add rest test Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * add rest test Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * comment restapi Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Add Smokers health Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * remove logs Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Add review comments Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Review comments Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * restructure pq Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> --------- Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com>
…FlowerTaskRunner`/`FlowerDataLoader`, and `FlowerInteropServer`/`FlowerInteropClient` into core OpenFL (securefederatedai#1633) * integrate flower collaborator into core collaborator Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * change task name interop -> prepare_for_interop, change references to local_grpc_server to interop_server Signed-off-by: kta-intel <kevin.ta@intel.com> * add prepare_interop_server as callback to run on_experiment_begin Signed-off-by: kta-intel <kevin.ta@intel.com> * fix typo Signed-off-by: kta-intel <kevin.ta@intel.com> * change to on_round_begin() Signed-off-by: kta-intel <kevin.ta@intel.com> * return empty dict Signed-off-by: kta-intel <kevin.ta@intel.com> * set self.prepare_interop_server() to run on experiment begin Signed-off-by: kta-intel <kevin.ta@intel.com> * remove self-explanatory comment, added another comment for clarity Signed-off-by: kta-intel <kevin.ta@intel.com> * fix plan parsing and automatic assignment of grpc ports Signed-off-by: kta-intel <kevin.ta@intel.com> * move taskrunner and dataloader to core openfl.federated Signed-off-by: kta-intel <kevin.ta@intel.com> * move connector and interop components to core openfl Signed-off-by: kta-intel <kevin.ta@intel.com> * remove comments about patch Signed-off-by: kta-intel <kevin.ta@intel.com> * remove collaborator number setting, it is not being used Signed-off-by: kta-intel <kevin.ta@intel.com> * adjust handling of flower directory. enabling user configuration with default location and use openfl's directory traversal check Signed-off-by: kta-intel <kevin.ta@intel.com> * exit system if error Signed-off-by: kta-intel <kevin.ta@intel.com> * update REAMDE.md Signed-off-by: kta-intel <kevin.ta@intel.com> * update docstring Signed-off-by: kta-intel <kevin.ta@intel.com> * override self.port Signed-off-by: kta-intel <kevin.ta@intel.com> * remove tasks_connector.yaml Signed-off-by: kta-intel <kevin.ta@intel.com> * updates Signed-off-by: kta-intel <kevin.ta@intel.com> * pin flwr Signed-off-by: kta-intel <kevin.ta@intel.com> * remove time Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * formatting Signed-off-by: kta-intel <kevin.ta@intel.com> * copyright Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com> Co-authored-by: teoparvanov <teodor.parvanov@intel.com>
* Change some tests to run with restapi Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Add grpc and rest checks Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * small fix Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * small fix Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * add some fix Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * rebase fix Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * small fix Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * commen resiliency Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Change rest_api to rest protocol Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Change rest_api to rest protocol Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> * Add one test in analytics Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Add one docker test Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Add one windows test Signed-off-by: payalcha <payal.chaurasiya@intel.com> * small fix Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Small check Signed-off-by: payalcha <payal.chaurasiya@intel.com> * Small check Signed-off-by: payalcha <payal.chaurasiya@intel.com> --------- Signed-off-by: Chaurasiya, Payal <payal.chaurasiya@intel.com> Signed-off-by: payalcha <payal.chaurasiya@intel.com>
Bumps [flask](https://github.com/pallets/flask) from 3.1.0 to 3.1.1. - [Release notes](https://github.com/pallets/flask/releases) - [Changelog](https://github.com/pallets/flask/blob/main/CHANGES.rst) - [Commits](pallets/flask@3.1.0...3.1.1) --- updated-dependencies: - dependency-name: flask dependency-version: 3.1.1 dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Akshay Pant <16561942+theakshaypant@users.noreply.github.com> Co-authored-by: teoparvanov <teodor.parvanov@intel.com>
…ion (securefederatedai#1649) * add function to deserialize the flower message for further introspection Signed-off-by: kta-intel <kevin.ta@intel.com> * format Signed-off-by: kta-intel <kevin.ta@intel.com> --------- Signed-off-by: kta-intel <kevin.ta@intel.com>
… Framework (securefederatedai#1648) * E2E Framework: Folders cleanup and shuffling Signed-off-by: noopur <noopur@intel.com> * Lint check and fixes Signed-off-by: noopur <noopur@intel.com> * S3 - remove data from local oploaded to s3 or azure blob Signed-off-by: noopur <noopur@intel.com> * After final testing of all datasources Signed-off-by: noopur <noopur@intel.com> * Do not run resiliency in draft stage Signed-off-by: noopur <noopur@intel.com> * Final testing Signed-off-by: noopur <noopur@intel.com> --------- Signed-off-by: noopur <noopur@intel.com>
…ivate attributes (securefederatedai#1605) * Delayed aggregator initialization Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Added test case Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Added test case Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Documentation update Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Incorporated review comments Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Incorporated review comments. Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Incorporated review comments Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> * Incorporated review comments. Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com> --------- Signed-off-by: Sachin Gupta <sachin.gupta.dsp@gmail.com>
Signed-off-by: Shah, Karan <kbshah1998@outlook.com>
Signed-off-by: Shah, Karan <kbshah1998@outlook.com>
Signed-off-by: Shah, Karan <kbshah1998@outlook.com>
This reverts commit f6ac53e. Signed-off-by: Shah, Karan <karan.shah@intel.com> Signed-off-by: Shah, Karan <kbshah1998@outlook.com>
Signed-off-by: Shah, Karan <kbshah1998@outlook.com>
…efederatedai#1575 - rename the function to get_aggregated_tensors - adjust both grpc and rest client for name and signature change in api - for the protobuf changes of TensorSpeca and Batched response adjust both client and server in line with grpc - fix the test cases rebased 29th.May.1 Signed-off-by: Shailesh Pant <shailesh.pant@intel.com>
ishaileshpant
force-pushed
the
batched_fetch_restapi_fix
branch
from
d6c93c4 to
46d1e95
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
7 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
For the changes in get_aggregated_tensors api as per #1575 PR we need to adjust the REST api spec and the rest client for the changes
Type of Change (Mandatory)
Patch REST get_aggregated_tensor api for batched changes as per #1575
Description (Mandatory)
Current REST api implementation and the test cases needs to be adjusted for the api changes as per #1575