Streaming endpoints

[zamoshchin]

I'm getting a 502 (bad gateway) when calling my serverless function (API endpoint) that has a streaming response:

src/routes/api/soap/format.ts

...
export const POST: RequestHandler = async ({ request }) => {
  response = await openai.createChatCompletion({
      model: model,
      messages: [{ role: 'user', content: prompt }],
      stream: true,
    }, { responseType: 'stream' }
  );

  const customReadable = new ReadableStream({
    start(controller) {
      response.data.on('data', (data: Buffer) => {
        // ...etc
      });
    }
  });
  
  return new Response(customReadable, {
    headers: { 'Content-Type': 'text/html; charset=utf-8' },
  });
  ...

This works locally. Any idea what needs to be adjusted on CloudFront or Lambda to allow for streaming output?

[yarbsemaj]

Unfortunately, my adapter does not yet support request streaming, Amazon actually added support for request streaming for lambda a few months ago https://aws.amazon.com/blogs/compute/introducing-aws-lambda-response-streaming/. It’s not something that’s on my horizon to implement, but would appreciate a pull request if you would like to implement