Progress not reporting correct number of rows

[mhkeller]

What happens?

I first reported this on the Node Neo client but was told it would be better to file here.

---

When logging the progress of an import query, the number of rows doesn't correspond to the rows in the file. If the rows remain the same but the file size changes, the total_rows_to_process changes, which makes me think this number is related to the bytes being processed, not rows. However, I can't figure out how this number relates to the file size. It seems to be 1/60th of the actual file size.

Is the key name here incorrect and it should be something like total_bytes_to_process?

To Reproduce

1. Clone the reproduction repo https://github.com/mhkeller/duckdb-import-issue
2. Install dependencies with npm i or npm i
3. Run the import with npm start or pnpm start

The file being imported is 100,000 rows but the logs come out as

{ percentage: 0, rows_processed: 0n, total_rows_to_process: 283628n }
{
  percentage: 21.97371101680997,
  rows_processed: 62323n,
  total_rows_to_process: 283628n
}
{
  percentage: 48.238472192600014,
  rows_processed: 136817n,
  total_rows_to_process: 283628n
}
{
  percentage: 75.35031028518631,
  rows_processed: 213714n,
  total_rows_to_process: 283628n
}
{
  percentage: 96.18581219394662,
  rows_processed: 272809n,
  total_rows_to_process: 283628n
}
{
  percentage: 99.99929485100202,
  rows_processed: 283626n,
  total_rows_to_process: 283628n
}
{
  percentage: 99.99929485100202,
  rows_processed: 283626n,
  total_rows_to_process: 283628n
}
{
  percentage: 99.99929485100202,
  rows_processed: 283626n,
  total_rows_to_process: 283628n
}

OS:

macOS 15.7.1 arm64

DuckDB Version:

"@duckdb/node-api": "1.4.1-r.4"

DuckDB Client:

NodeJS

Hardware:

No response

Full Name:

Michael Keller

Affiliation:

Self

Did you include all relevant configuration (e.g., CPU architecture, Linux distribution) to reproduce the issue?

• Yes, I have

Did you include all code required to reproduce the issue?

• Yes, I have

Did you include all relevant data sets for reproducing the issue?

Yes

[jraymakers]

Hey @szarnyasg, I directed the filer to the main DuckDB repo, because, although Node Neo was used in the example, the behavior of the progress reporting is in DuckDB core. The example could easily be reproduced with a different client, including C or C++.

[jraymakers]

To be clear, the behavior described is likely expected, but confusing. What would help is more documentation about the progress reporting system of DuckDB, which is not specific to Node Neo.

[mhkeller]

I don't know C so I can't create a reproduction but if you're able to do that @jraymakers, maybe that will have more success than my repro.

[jraymakers]

@szarnyasg I think this issue should be in the main duckdb repo.

The ask is to document the sometimes-confusing behavior of progress reporting. My understanding is that different operators report different things, and some don't report at all. So, how much information you get, and sometimes even what it means, depends on the details of the query, and even the details of the plan. But this could be inaccurate; I'm not knowledgeable enough about this behavior to document it sufficiently.

The repro above happens to be in Node Neo, but the behavior is in duckdb core.

[mhkeller]

My two cents: I'd really like to understand this better so I can build on top of it. I don't think @szarnyasg is going to accept this reproduction since 1) I flagged in my original issue that this was not node-neo specific and 2) @jraymakers raised it twice with no response.

So, in the interest of being pragmatic, I think we should close this issue and the two options for @jraymakers are:

1. Someone (not me, since I would do such a horrible job writing in a language I don't know that it would only add to the confusion) up an issue on the main repo with a C reproduction and clarification request.
2. Add something to the node-neo documentation that there is undefined behavior in this part of the API.