bug: Is stopOnAbortSignal as a LLamaChatPromptOption correct?

[rogerdcarvalho]

Issue description

When running llamaChatSession.prompt(), the option stopOnAbortSignal behaves odd

Expected Behavior

I can run the prompt() function on a LlamaChatSession, but when I add the option 'stopOnAbortSignal: true', it feels like the opposite happens. As if the label should have been continueOnAbortSignal?

Cause if I look at /src/evaluator/LlamaChatSession/LlamaChatSession.ts, I see you evaluate as

abortController.signal.addEventListener("abort", () => {
if (abortedOnFunctionCallError || !stopOnAbortSignal)
reject(abortController.signal.reason);

Wouldn't that mean you continue if stopOnAbortSignal is false?

Actual Behavior

When creating a LlamaChatSession and then aborting, an error isn't always thrown.

Steps to reproduce

Create a LlamaChatSession and run the prompt function, with either stopOnAbortSignal false or true and compare behavior.

My Environment

Dependency	Version
Operating System
CPU	Intel i9 / Apple M1
Node.js version	v22.13.1
Typescript version	x.y.zzz
node-llama-cpp version	3.14.2

npx --yes node-llama-cpp inspect gpu output:

Result of running `npx --yes node-llama-cpp inspect gpu`

OS: macOS 24.5.0 (arm64)
Node: 22.13.1 (arm64)
TypeScript: 5.9.2

node-llama-cpp: 3.14.2
Prebuilt binaries: b6845

Metal: available

Metal device: Apple M3 Max
Metal used VRAM: 0% (464KB/96GB)
Metal free VRAM: 99.99% (96GB/96GB)
Metal unified memory: 96GB (100%)

CPU model: Apple M3 Max
Math cores: 12
Used RAM: 98.4% (125.95GB/128GB)
Free RAM: 1.59% (2.05GB/128GB)
Used swap: 93.05% (18.61GB/20GB)
Max swap size: dynamic
mmap: supported

Additional Context

No response

Relevant Features Used

• Metal support
• CUDA support
• Vulkan support
• Grammar
• Function calling

Are you willing to resolve this issue by submitting a Pull Request?

Yes

[giladgd]

I'm not entirely sure what you mean the issue is. It could help if you can attach a reproduction code that demonstrates the specific issue you're facing.

By default, stopping the abort controller of the signal you pass to the .prompt(...) function causes the prompt to stop and to throw the error from the abort signal. It works this way since this is the expected standard behavior for JS APIs.

When you set stopOnAbortSignal to true, when the abort signal is stopped, if the response already started being generated then it just stops generating the response and returns the partial response that was generated.
Function calls that haven't returned a result yet will be ignored when they're fulfilled since the entire prompt is now cancelled (that's what the piece of code you linked does).

[rogerdcarvalho]

I guess I misunderstood how it should work. If I do:

try {
          const context = await this.llamaCppModel!.createContext({
            contextSize: {
              max: contextWindow,
            },
          });
          const session = new LlamaChatSession({
            contextSequence: context.getSequence(),
          });
          const abortController = new AbortController();
          await session.prompt(topicsPrompt, {
            stopOnAbortSignal: true,
            signal: abortController.signal,
            onTextChunk(chunk) {
              response += chunk;
              if (response.length >= maxLength) {
                abortController.abort();
              }
            },
          });
    }
catch (e) {
      console.log(e)
}
rest of code

It will not throw an error after response.length goes over maxLength, it will simply continue the code. If I however make stopOnAbortSignal: false, it will catch the error and console.log its output.

I basically assumed that stopOnAbortSignal as true would throw the error, while as false would continue the code without throwing an error. The way you explained it however aligns with what I'm seeing, it just read a little strange. But if this is by design I suppose this can be closed. Apologies!

[giladgd]

I agree that the name stopOnAbortSignal is a bit confusing, but that's the best I managed to come up with when designing this API with the constraint of aligning with JS standards for the default abort signal behavior.
If you have an idea of a better name that would be more intuitive for the behavior of the current option, please let me know!
I can use the opportunity of the upcoming v4 beta to rename the option to something better.

[rogerdcarvalho]

Sure makes sense! Perhaps something like 'partialResponseOnAbortSignal'? But no big deal really, could also just be solved in documentation. Thanks for explaining!