Fix progress bar stuck during archive import

[GeigerJ2]

After removing filter_size batching in the QueryBuilder IN clause optimization in #6998, the _add_new_entities function's progress bar was broken.

Before PR #6998

for nrows, ufields_batch in batch_iter(ufields, filter_size):  # ~1000 IDs per batch
    rows = [transform(row) for row in QueryBuilder(...).dict()]  # ❌ List comprehension blocks
    bulk_insert(rows)
    progress.update(nrows)  # ✅ Updates every ~1000 entities

Progress updates:

• From filter_size batching only (every ~1000 entities)
• NOT from QueryBuilder streaming (blocked by list comprehension)
• Each batch still had a blocking collection phase

After PR #6998 (Broken)

rows = [transform(row) for row in QueryBuilder(...).dict()]  # ❌ Blocks for ALL
bulk_insert(rows)
progress.update(len(rows))  # ❌ Updates once at end

Progress updates: None until the very end

With this PR (restored AND improved)

for _, ufields_batch in batch_iter(ufields, 50_000):  # 50k IDs per batch
    query = QueryBuilder(...).append(..., filters={unique_field: {'in': ufields_batch}})
    for nrows, rows_batch in batch_iter(query.dict(), batch_size, transform):  # ✅ NEW!
        bulk_insert(rows_batch)
        progress.update(nrows)  # ✅ Updates every ~10k entities

Progress updates:

• Re-introduced query batching (50k instead of 1k—purely for UX, not due to SQL restrictions)
• NEW: Added QueryBuilder streaming-based updates (every ~10k entities—previously blocked by list comprehension)

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 79.58%. Comparing base (e79f0a4) to head (b4cfe89).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #7118      +/-   ##
==========================================
+ Coverage   79.58%   79.58%   +0.01%     
==========================================
  Files         566      566              
  Lines       43520    43523       +3     
==========================================
+ Hits        34631    34634       +3     
  Misses       8889     8889              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[GeigerJ2]

I considered also adding a report statement here:

Suggested change

	# collect the unique entities from the input backend to be added to the output backend
	ufields = []
	# Additional report statement, as this can take a while
	IMPORT_LOGGER.report(f'Finding unique new {etype.value}(s)')

As this can also take a while for a large number of nodes, however, it appears for all entities, so also Users and Computers, for which it completes quickly, so the report only adds noise. I decided to remove it for now.

[edan-bainglass]

Thanks @GeigerJ2. LGTM! Have you tested locally?