Skeleton for nightly run of Apps Codegen Evals #4103
Draft
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
- Change requires-python from >=3.11 to >=3.10 - Replace str | None union syntax with Optional[str] for 3.10 compat - Remove unused databricks-sdk and tqdm dependencies Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
- Remove bundle run dependency (databricks CLI not available in serverless) - Clone appdotbuild-agent repo and install klaudbiusz deps - Handle case of no apps gracefully - log sample metrics to MLflow - Job successfully validates infrastructure and logs to MLflow Note: Full eval requires Python 3.12+ or pre-populated apps Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
- Add apps_generation_job.job.yml with single-node Docker cluster
- Add generate_apps.py orchestrator using klaudbiusz framework
- Add init/setup_generation.sh to install Dagger and Python deps
- Update run_evals.py to read apps from UC Volume
- Add variables for CLI binary and generated apps volumes
Generation uses databricks experimental apps-mcp as the MCP server,
built from this repo for Linux x86_64.
Prerequisites:
- Create secret: databricks secrets put-secret apps-mcp-evals anthropic-api-key
- Upload CLI: GOOS=linux GOARCH=amd64 go build -o databricks-linux .
databricks fs cp databricks-linux /Volumes/main/evals/artifacts/
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Use main.default.apps_mcp_artifacts and main.default.apps_mcp_generated volumes which were created successfully. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
keugenek
temporarily deployed
to
test-trigger-is
GitHub Actions
Inactive
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes
This PR creates Lafeflow Job for long-running Apps Codegen Evals by cloning from this repo and running sample generation of Databricks apps using current cli mcp and running evals on them and publishing the result to mlflow.
Testing
evgenii.kniazev@FP424MF2FY cli % cd experimental/apps-mcp/evals
evgenii.kniazev@FP424MF2FY evals % databricks auth login
✔ Databricks profile name [DEFAULT]: █
Profile DEFAULT was successfully saved
evgenii.kniazev@FP424MF2FY evals % databricks bundle validate -t dev
Name: apps-mcp-evals
Target: dev
Workspace:..
Validation OK!
evgenii.kniazev@FP424MF2FY evals % databricks bundle deploy -t dev
Building apps_mcp_evals...
Uploading dist/apps_mcp_evals-0.1.0-py3-none-any.whl...
Uploading bundle files to /..
Deploying resources...
Updating deployment state...
Deployment complete!
evgenii.kniazev@FP424MF2FY evals % databricks bundle run -t dev apps_eval_job
Run URL: ...
2025-12-05 17:00:16 "[dev evgenii_kniazev] [dev] Apps-MCP Continuous Evals" RUNNING