Quick-watch: Catch early bulk operation errors without --watch

[ericlee878]

Resolves: https://github.com/orgs/shop/projects/208/views/34?filterQuery=&pane=issue&itemId=140129246&issue=shop%7Cissues-api-foundations%7C1123

Inspired by: https://github.com/shop/issues-api-foundations/issues/1123#issuecomment-3603641001

Background

When users run bulk operations with --variables or --variable-file, invalid JSON or malformed inputs aren't validated upfront. Without the --watch flag, users would:

1. Submit a potentially invalid job
2. Walk away thinking it's running
3. Come back hours later to find errors in the result file

This is a frustrating experience, especially for long-running bulk operations.

We considered validating JSON on the client before sending to the API, but:

• There are too many ways for Core to reject input (incorrect query, incorrect variables, incorrect data format, etc.). There could also be errors we cannot catch from client side, making it possible for there to sometimes be errors caught on the client side and sometimes given in the output file (potentially causing confusion).
• It's impractical to catch every type of error at a validation stage.
• Full validation of large imports would hurt performance.

The solution: Quick-watch
Instead of duplicating validation logic, we leverage the fact that errors tend to surface in the first 2-3 seconds of a bulk operation. By polling briefly after creation, we can catch most "obvious issues" (like malformed JSON) and present them immediately—without requiring users to run a second command or wait for hours.

This approach:

• Catches early errors without duplicating Core's validation logic
• Doesn't significantly impact performance (only 3 seconds of polling)
• Works for the common case where validation errors appear quickly

Implementation Details

Detecting mutation-level errors in results
When a bulk operation completes (status: COMPLETED), we check the downloaded results for userErrors using a simple string check: results.includes('"userErrors":[{').

This catches cases where the operation "succeeds" but individual mutations failed (e.g., malformed JSON variables). Instead of showing a misleading "Bulk operation succeeded" message, we show:

• renderWarning with "Bulk operation completed with errors."
• If --output-file was provided: includes the file path so users know where to check

Why .includes() instead of ?.userErrors?.length: like in line 62 of execute-bulk-operation.ts?
The result file is a raw JSONL string, not a JavaScript object. Parsing every line would be expensive for large files — a simple string search is O(n) with no allocations. Checking for "userErrors":[{ only matches non-empty arrays ("userErrors":[] won't match).

Flag behavior
Quick watch only happens when the --watch flag isn't given. This decision is explained here.

Polling configuration

QUICK_WATCH_TIMEOUT_MS = 3000 - Total time to poll
QUICK_WATCH_POLL_INTERVAL_MS = 300 - Poll every 300ms (~10 polls max)

No null checks in quickWatchBulkOperation:
We use bulkOperation! (non-null assertion) because:

• executeBulkOperation already checks userErrors before calling quick-watch
• If bulkOperation is null, userErrors would exist and we'd return early

Tophatting

For catching errors early:

pnpm shopify:run app bulk execute \
  --path <PATH_TO_TEST_APP> \
  --output-file ./results.jsonl \
  -q 'mutation productUpdate($input: ProductInput!) { productUpdate(input: $input) { product { id tags } userErrors { field message } } }' \
  -v '{"input": invalid json}'

This is what my results.jsonl looked like:

{"data":{"productUpdate":{"userErrors":[{"message":"unexpected character: 'invalid' at line 1 column 11"}]}},"__lineNumber":0}

For super quickly finished bulk operations that successfully finish within the quick-watch:

pnpm shopify:run app bulk execute \
  --path <PATH_TO_TEST_APP>\
  -q '{ products(first: 1) { edges { node { id title } } } }'

For bulk operations that do not finish in quick-watch:

Create a variable file:

cat > /tmp/bulk-products.jsonl << 'EOF'
{"input":{"title":"Test Product 1","descriptionHtml":"<p>This is test product 1</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 2","descriptionHtml":"<p>This is test product 2</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 3","descriptionHtml":"<p>This is test product 3</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 4","descriptionHtml":"<p>This is test product 4</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 5","descriptionHtml":"<p>This is test product 5</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 6","descriptionHtml":"<p>This is test product 6</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 7","descriptionHtml":"<p>This is test product 7</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 8","descriptionHtml":"<p>This is test product 8</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 9","descriptionHtml":"<p>This is test product 9</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
{"input":{"title":"Test Product 10","descriptionHtml":"<p>This is test product 10</p>","productType":"Test Type","vendor":"Test Vendor","tags":["bulk","test"],"metafields":[{"namespace":"custom","key":"field1","value":"value1","type":"single_line_text_field"},{"namespace":"custom","key":"field2","value":"value2","type":"single_line_text_field"},{"namespace":"custom","key":"field3","value":"value3","type":"single_line_text_field"}]}}
EOF

Run the long bulk mutation:

pnpm shopify:run app bulk execute \
  --path ./coffee-and-more \
  --variable-file /tmp/bulk-products.jsonl \
  -q 'mutation productCreate($input: ProductInput!) {
    productCreate(input: $input) {
      product {
        id
        title
        metafields(first: 10) {
          edges {
            node {
              id
              namespace
              key
              value
            }
          }
        }
      }
      userErrors {
        field
        message
      }
    }
  }'

[ericlee878]

• Add user display during short poll #6700
• Quick-watch: Catch early bulk operation errors without --watch #6687 👈 (View in Graphite)
• main

This stack of pull requests is managed by Graphite. Learn more about stacking.

[github-actions]

We detected some changes at packages/*/src and there are no updates in the .changeset.
If the changes are user-facing, run pnpm changeset add to track your changes and include them in the next release CHANGELOG.

Caution

DO NOT create changesets for features which you do not wish to be included in the public changelog of the next CLI release.

[github-actions]

Coverage report

St.❔	Category	Percentage	Covered / Total
🟡	Statements	79.5% (+0.27% 🔼)	14167/17821
🟡	Branches	73.57% (+0.47% 🔼)	6955/9453
🟡	Functions	79.64% (+0.27% 🔼)	3634/4563
🟡	Lines	79.87% (+0.29% 🔼)	13393/16769

Show new covered files 🐣

St.❔	File	Statements	Branches	Functions	Lines
🟢	... / admin-as-app.ts	100%	100%	100%	100%
🟢	... / metafield_definitions.ts	100%	100%	100%	100%
🟢	... / metaobject_definitions.ts	100%	100%	100%	100%
🟢	... / bulk-operation-run-mutation.ts	100%	100%	100%	100%
🟢	... / bulk-operation-run-query.ts	100%	100%	100%	100%
🟢	... / get-bulk-operation-by-id.ts	100%	100%	100%	100%
🟢	... / list-bulk-operations.ts	100%	100%	100%	100%
🟢	... / staged-uploads-create.ts	100%	100%	100%	100%
🔴	... / execute.ts	0%	0%	0%	0%
🔴	... / status.ts	0%	0%	0%	0%
🔴	... / pull.ts	0%	100%	0%	0%
🟢	... / execute-operation.ts	92.86%	80%	100%	92.31%
🔴	... / pull.ts	0%	0%	0%	0%
🟢	... / bulk-operation-status.ts	96.61%	92.11%	100%	100%
🟢	... / download-bulk-operation-results.ts	100%	100%	100%	100%
🟢	... / execute-bulk-operation.ts	92.75%	87.23%	100%	93.94%
🟢	... / format-bulk-operation-status.ts	100%	100%	100%	100%
🟢	... / run-mutation.ts	100%	100%	100%	100%
🟢	... / run-query.ts	100%	100%	100%	100%
🟡	... / stage-file.ts	73.53%	62.5%	85.71%	72.73%
🟢	... / watch-bulk-operation.ts	100%	100%	100%	100%
🟢	... / declarative-definitions.ts	98.54%	93.18%	100%	98.51%
🟢	... / common.ts	96.15%	93.33%	100%	95.45%
🟢	... / execute-command-helpers.ts	100%	100%	100%	100%
🔴	... / promiseWithResolvers.ts	33.33%	50%	50%	33.33%

Show files with reduced coverage 🔻

St.❔	File	Statements	Branches	Functions	Lines
🔴	... / execute.ts	0%	0% (-100% 🔻)	0%	0%
🟢	... / extension-instance.ts	84.8% (+0.23% 🔼)	77.6% (-0.91% 🔻)	92.06% (+0.13% 🔼)	85.11% (+0.24% 🔼)
🟡	... / specification.ts	69.09%	75.61% (+2.44% 🔼)	76.47% (-1.31% 🔻)	68.75%
🟢	... / ui_extension.ts	85.38% (-9.44% 🔻)	72.34% (-8.91% 🔻)	84% (-16% 🔻)	88% (-8.46% 🔻)
🟢	... / developer-platform-client.ts	84.62% (-1.5% 🔻)	71.43% (+0.84% 🔼)	81.82% (+1.82% 🔼)	93.75% (+0.42% 🔼)
🟢	... / api.ts	87.07% (-0.43% 🔻)	76.71% (-0.1% 🔻)	100%	86.49% (-0.43% 🔻)
🟢	... / SingleTask.tsx	84.21% (-15.79% 🔻)	50% (-50% 🔻)	80% (-20% 🔻)	84.21% (-15.79% 🔻)
🔴	... / environment.ts	35% (-5% 🔻)	41.18%	40% (-10% 🔻)	36.84% (-5.26% 🔻)
🔴	... / ui.tsx	50.82% (-0.79% 🔻)	42.86% (-5.53% 🔻)	54.55% (+1.42% 🔼)	50% (-0.82% 🔻)
🟢	... / console.ts	81.82% (+15.15% 🔼)	75% (-25% 🔻)	100% (+33.33% 🔼)	81.82% (+15.15% 🔼)
🔴	... / dev.ts	14.29% (+0.95% 🔼)	3.13% (+0.18% 🔼)	50% (-7.14% 🔻)	14.29% (+0.95% 🔼)
🟢	... / init.ts	88% (-0.89% 🔻)	71.43% (+4.76% 🔼)	86.67% (+4.85% 🔼)	88% (-0.89% 🔻)
🟢	... / storefront-renderer.ts	90.2% (-0.54% 🔻)	78.95%	81.82% (-1.52% 🔻)	90.2% (-0.54% 🔻)
🟡	... / theme-polling.ts	67.12% (-0.93% 🔻)	68.75%	78.57%	66.67% (-0.98% 🔻)

Test suite run success

3544 tests passing in 1414 suites.

Report generated by 🧪jest coverage report action from 910f6b0

[jordanverasamy]

I wonder how much the extra polling (seems like right now we poll 10 times in a 3 second period) actually helps. Would be simple and nice to just e.g. wait 1 second and fetch once, instead of doing all of that work, and I bet it'd cover most of the same common cases anyways. What are your thoughts on the tradeoffs there?

[ericlee878]

Are there any impactful consequences with extra polling? Does it slow anything down or put too much pressure on the platform or something? Tbh, I don't think I know enough about the speed of the API, the user/agent's needs, or the impact of the extra polling to decide this. Happy to refactor either way though!

[jordanverasamy]

Are there any impactful consequences with extra polling? Does it slow anything down or put too much pressure on the platform or something? Tbh, I don't think I know enough about the speed of the API, the user/agent's needs, or the impact of the extra polling to decide this. Happy to refactor either way though!

Not really, it's not that many requests in the grand scheme of things. IMO the real cost there is code complexity and developer brainpower, not platform pressure. But that said I'm happy with keeping it and like where we landed, so don't worry :)

[gonzaloriestra]

Nice solution! It works well.

But I agree with @jordanverasamy that the current polling is maybe excessive, even if it won't cause problem. I think a 1s polling interval should be enough and consistent with #6695