Skip to content

Conversation

@AlePouroullis
Copy link
Contributor

@AlePouroullis AlePouroullis commented Apr 24, 2025

Sync operation for the python SDK

Functionality

  • Implemented .sync() function on the extended Humanloop client
  • The sync() function internally called utils defined in src/humanloop/sync/sync_utils.py (added to .fernignore so that it doesn't get overriden by the auto-generated SDK)
  • Composed of existing client functionality, namely list_files() and serialize()
  • Supports prompt and agent files
  • Only allows for single version per file. Creates identical directory structure to that of the HL workspace (e.g. A/B/C where C is a prompt file will be stored in humanloop/A/B/C.prompt
  • Pulls default deployed files
  • No controls, like for environment, fetching from specific directory, etc. Only uses defaults: production files for the whole workspace

Tests

  • Wrote integration tests that use local environment (the client in the test uses localhost:80/v5 for the backend instead of prod)
  • Creates test prompt and agent files, and cleans up after
  • Checks that the path of the file on remote exists in the humanloop directory

Implementation details:

  • list_files() is called, and paginated to get all files
  • After a page is retrieved, each individual file is handed off to a thread pool where a thread processes a file, serializing it and writing it to disk

Caveats:

  • The serialize() function from the auto-generated SDK returns None, presumably because its response type is text/prompt on the API spec and fern doesn't know how to handle this. Consequently, it returns HttpResponse(response=_response, data=None) when it should do something like return HttpResponse(response=_response, data=_respnose.text) (which I did in this PR as a dirty workaround). I've flagged on the fern slack channel
  • Pagination is not done concurrently. Could be sped up by handing off fetching each page to a thread but difficult to coordinate
  • This operation could benefit quite a bit from async, but quite complicated.

@AlePouroullis AlePouroullis deleted the p_sync/sync branch April 28, 2025 16:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants