Skip to content

Refactor apps logs log streaming #4086

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
pkosiec wants to merge 5 commits into
base: main
Choose a base branch
from
Open

Refactor apps logs log streaming #4086

pkosiec wants to merge 5 commits into from
+607 −349

Conversation

@pkosiec
Copy link
Member

@pkosiec pkosiec commented Dec 4, 2025

Changes

  • Show body details on dial error only if it's a JSON
  • Extract backoff, tail buffer, formatter and consume state for better readability

Why

To increase code readability for easier maintenance.

Tests

Unit tests are in place. I also did manual tests for the command usage as before.

Screenshots

Before

We printed HTML which weren't usefull really:
Screenshot 2025-12-04 at 12 17 05

After

We don't print HTML response. Only if JSON is returned then it would be printed.
image

@pkosiec pkosiec temporarily deployed to test-trigger-is December 4, 2025 13:12 — with GitHub Actions Inactive
@eng-dev-ecosystem-bot
Copy link
Collaborator

eng-dev-ecosystem-bot commented Dec 4, 2025 •
edited
Loading

Commit: 78e9334

Run: 19931035268

Env 🟨​KNOWN 🔄​flaky 💚​RECOVERED 🙈​SKIP ✅​pass 🙈​skip Time
🟨​ aws linux 7 2 374 637 22:11
🟨​ aws windows 7 2 2 374 635 18:30
🔄​ aws-ucws linux 3 4 2 517 522 27:39
💚​ aws-ucws windows 7 2 519 520 21:16
💚​ azure linux 1 4 375 635 19:09
💚​ azure windows 1 4 377 633 17:43
💚​ azure-ucws linux 1 4 514 520 28:09
💚​ azure-ucws windows 1 4 516 518 22:50
💚​ gcp linux 1 4 368 639 19:26
💚​ gcp windows 1 4 370 637 17:45
11 interesting tests: 7 KNOWN, 2 SKIP, 2 flaky
Test Name aws linux aws windows aws-ucws linux aws-ucws windows azure linux azure windows azure-ucws linux azure-ucws windows gcp linux gcp windows
🟨​ TestAccept 🟨​K 🟨​K 🔄​f 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
🙈​ TestAccept/bundle/resources/permissions 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions 🟨​K 🟨​K 🔄​f 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/with_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 🔄​f 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions 🟨​K 🟨​K 💚​R 💚​R 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=direct 🟨​K 🟨​K 💚​R 💚​R
🟨​ TestAccept/bundle/resources/permissions/jobs/destroy_without_mgmtperms/without_permissions/DATABRICKS_BUNDLE_ENGINE=terraform 🟨​K 🟨​K 💚​R 💚​R
🙈​ TestAccept/bundle/run/app-with-job 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S 🙈​S
🔄​ TestFilerWorkspaceNotebook ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
🔄​ TestFilerWorkspaceNotebook/sqlNb.sql ✅​p 🔄​f ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p ✅​p
Top 19 slowest tests (at least 2 minutes):
duration env testname
6:43 azure-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:24 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
6:05 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:54 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:45 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:34 gcp windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:34 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:29 aws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:24 aws-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
5:21 aws-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
5:09 gcp linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
4:21 azure linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:44 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=terraform
3:30 azure-ucws linux TestAccept/bundle/resources/synced_database_tables/basic
3:26 azure-ucws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
3:05 azure-ucws windows TestAccept/bundle/resources/synced_database_tables/basic
3:04 azure windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:11 aws linux TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct
2:02 azure-ucws windows TestAccept/bundle/resources/clusters/deploy/update-after-create/DATABRICKS_BUNDLE_ENGINE=direct

@pkosiec pkosiec force-pushed the pkosiec/apps-logs-improvements branch from e7b4ca6 to 78e9334 Compare December 4, 2025 13:41
@pkosiec pkosiec marked this pull request as ready for review December 4, 2025 13:41
@pkosiec pkosiec requested a review from a team as a code owner December 4, 2025 13:41
@pkosiec pkosiec temporarily deployed to test-trigger-is December 4, 2025 13:41 — with GitHub Actions Inactive
pietern
pietern approved these changes Dec 9, 2025
View reviewed changes
libs/apps/logstream/consume_state.go
"time"
)

// consumeState manages the tail buffer and flush timing during log consumption.
Copy link
Contributor

@pietern pietern Dec 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does it need to explicitly flush vs just write whenever?

Local I/O is cheap.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@pietern pietern pietern approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@pkosiec @eng-dev-ecosystem-bot @pietern