fix: detect actual image format in take_screenshot tool

[marcandreuf]

Description

Fixes #583

The take_screenshot tool was incorrectly declaring the MIME type based on the requested format parameter rather than
detecting the actual format of the image data returned by Puppeteer.

This caused a mismatch when Puppeteer returned a different format than requested, leading to Claude API validation
errors:
Image does not match the provided media type image/jpeg

Changes

Added

• src/utils/imageFormat.ts: Utility function to detect actual image format from binary data by inspecting magic numbers
  (PNG, JPEG, WebP)
• tests/utils/imageFormat.test.ts: Comprehensive unit tests for format detection (6 tests, all passing)

Modified

• src/tools/screenshot.ts: Updated to detect and use the actual image format instead of blindly trusting the request
  parameter

Testing

• ✅ All 6 new unit tests pass
• ✅ Manually tested with MCP Inspector against localhost:3000
• ✅ Verified PNG, JPEG, and WebP formats all work without MIME type errors
• ✅ Build succeeds with npm run build

Technical Details

The fix detects image formats by inspecting the first 12 bytes of the binary data:

• PNG: 89 50 4E 47 (‰PNG)
• JPEG: FF D8 FF
• WebP: 52 49 46 46 ... 57 45 42 50 (RIFF...WEBP)

This ensures the declared MIME type always matches the actual image data, preventing API validation errors.

Conventional Commits

This PR follows the conventional commits specification with type fix as it resolves a bug in the screenshot
functionality.

[google-cla]

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

[marcandreuf]

I sigend the google CLA, but it is not being picked up for this PR

[marcandreuf]

Not all fix yet. Let me try again.

[marcandreuf]

It looks like this issue in the Claude Code CLI might be the root cause. anthropics/claude-code#12015
Even though the fix in this branch ensures the MCP server does uses the correct mime type, the CC CLI is changing the mime type and the API rejects the image processing.
So, at this point the solution would be to add some configuration options to default the image mime type to jpeg. So, give the option to the user of the mcp server and thus we can align the image mime types. I am testing this solution and reopen the PR asap.

[marcandreuf]

Ok I think now it should be a valid fix to also work around the Claude Code CLI bug mentioned above. Now this mcp server defaults to use jpeg as this is the format passed by CC CLI to the CC API.
There is a new configuration option in this mcp server to change the default when needed.

[marcandreuf]

This is my test with my local build mcp server.

[OrKoN]

Thanks for the PR.

This caused a mismatch when Puppeteer returned a different format than requested, leading to Claude API validation
errors:

why was Puppeteer returning a different format than requested? Can we fix that on the Puppeteer side or mapping of the image types instead?

[marcandreuf]

Thanks for the PR.

This caused a mismatch when Puppeteer returned a different format than requested, leading to Claude API validation
errors:

why was Puppeteer returning a different format than requested? Can we fix that on the Puppeteer side or mapping of the image types instead?

Well, there is an open bug in Claude Code CLI, the link is above. However it looks like having a configuration option in this mcp we can detach this dependency of png only mime type. Also by aligning now to jpeg as default will make this mcp work again without extra configurations. We can change de default later on when the CC CLI bug is fix, if needed.

[OrKoN]

Sorry, I am still confused a little bit: in which situations does puppeteer provide an image that does not match the requested image format? It does appear that there is a bug in Claude Code CLI but is there any need to verify that Puppeteer's returned format is matching the requested format by looking at the image data? Does Claud Code CLI work if you explicitly pass the png format? does Puppeteer not return a png image if the png format is specified?

[marcandreuf]

Sorry, I am still confused a little bit: in which situations does puppeteer provide an image that does not match the requested image format? It does appear that there is a bug in Claude Code CLI but is there any need to verify that Puppeteer's returned format is matching the requested format by looking at the image data? Does Claud Code CLI work if you explicitly pass the png format? does Puppeteer not return a png image if the png format is specified?

Well, to check the actual mime type of the file is a good practice to make sure the this mcp is doing the right thing. The problem looks to be in the Claude Code CLI, which changes the mime type from png to jpeg, regardless of what mime type is used in the mcp. So if the mcp server takes a png image the CC CLI passes the image to the CC API as image/jpeg. And the CC API does not accept returns an error.

This is my understanding of the issue. I think that Puppeteer is the underlying dependency that CC CLI is using and I understand that this is the bug that is open in the CC CLI side.

From our side it looks better to have the flexibility to adapt the image format and thus we can align with the CC CLI.

[marcandreuf]

Do I need to open a new PR and remove the anthropic contributor? I guess that I can open a new branch and PR with only my user as contributor. Sorry, the anthropic contributor was added automatically by CC. Even though I think it is correct to add it as contributor it might block the CLA validation and the rest of the checks.

[OrKoN]

Is it possible to work around the issue by adding Pass jpeg for all take_screenshot() calls to the CLAUDE.md until the issue is fixed on their side? I think setting the default format might still be beneficial, but please remove the imageFormat.ts and detectImageFormat logic as it is redundant.

[marcandreuf]

OK yes I will amend the PR. Let me create a new PR with the right contributor and CLA approval so that all the required checks can run. I will close this PR and open a new one with the minimum changes for the fix.

[marcandreuf]

The new clean PR is #591