Streaming encrypted images using Lambda Streaming Responses

Giving our users the ability to view online their backed up content has always been a feature that we wanted to provide. I am happy to announce that now you now can! The PhotoVaultOne service console now provides you with direct access to your stored photos, to view online in your browser, without need to download them first.

Our API is 100% built on AWS serverless services. For a long time, these services included no built-in mechanism for streaming large objects -- so how did we do this? AWS recently released Lambda streaming responses. With this capability, we can deliver your content directly to you, and also do it in a way that is highly performant.

TL;DR With streaming responses, we measured a 56% reduction in time-to-first-byte, and a 16% reduction in end-to-end latency for content delivery. Read more to find out how!

The following is technical information for the AWS developer community. If you are a user of PhotoVaultOne, this information may not be relevant for you.

Lambda Streaming Responses

The above linked AWS blog post does a good job at explaining what Lambda Streaming Responses are, and how to get started with them. I will not repeat all of that here, but simply state the two biggest benefits for PhotoVaultOne:

1. Increase of maximum payload size from 6MB to 20MB (soft limit)
2. Decrease of time-to-first-byte (TTFB)

The first one is easy to grasp. But how about the second? What real-life improvement can you expect for TTFB when using Lambda Streaming Responses?

To answer these questions, we ran a few simple measurements. The tests used the following setup:

Case 1: Buffered response (default)

Using buffered responses (the original and still current Lambda default) was tested with a simple NodeJS Lambda function:

const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = async (event) => {
  const params = {
      Bucket: "xxx",
      Key: "yyy"
  };
  const data = await s3.getObject(params).promise();

  return {
    statusCode: 200,
    headers: {
      'Content-Type': 'image/jpeg'
    },
    body: data.Body.toString('base64'),
    isBase64Encoded: true
  }
}

Using this ttfb tool, we measured the download performance against a Lambda function url endpoint associated with our function.

Case 2: Streaming response

A streaming compatible version of the Lambda function was created for this test case:

const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = awslambda.streamifyResponse(async (event, responseStream, context) => {
  const params = {
      Bucket: "xxx",
      Key: "yyy"
  };
  const requestStream = s3.getObject(params).createReadStream();

  const metadata = {
      statusCode: 200,
      headers: {
          "Content-Type": "image/jpeg",
      }
  };
  responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);
  await pipeline(requestStream, responseStream);
});

As can be seen by comparing the above two blocks of code, there is not a lot more to writing a streaming response than there is to a buffered one. As for results:

The results above are impressive: utilizing streaming responses reduced the time-to-first-byte by 56% and reduced the overall time to deliver the content by 16%.

Case 3: Streaming response with decryption

At PhotoVaultOne, we cannot simply stream your content from storage - it needs to be decrypted first. This is because our content is stored client-side encrypted for maximum privacy.

Building on top of the case 2, the following skeleton function represents how the image download function is implemented at PhotoVaultOne:

const util = require('util');
const stream = require('stream');
const pipeline = util.promisify(stream.pipeline);
const AWS = require('aws-sdk');
const s3 = new AWS.S3();

exports.handler = awslambda.streamifyResponse(async (event, responseStream, context) => {
  const mediaItemUuid = getMediaItemUuidFromHttpPath(event.requestContext.http.path);
  const [tenantId, fileKey] = verifyAndDecryptRequestToken(event.requestParams.token);

  const params = {
      Bucket: "user-content-bucket",
      Key: `${tenantId}/media/${mediaItemUuid}`,
  };
  const requestStream = s3.getObject(params).createReadStream();

  const decryptor = new PhotoVaultOneDecryptTransformer({
    key: fileKey,
    algo: "aes-256-ctr",
  });
    
  const metadata = {
      statusCode: 200,
      headers: {
          "Content-Type": "image/jpeg",
      }
  };
  responseStream = awslambda.HttpResponseStream.from(responseStream, metadata);
  await pipeline(requestStream, decryptor, responseStream);
});

The function structure and logic remains largely the same as for the case 2. Two key points to highlight:

• The client request contains an AES-256-GCM encrypted token, which contains everything the Lambda function needs to serve the file to the user. No DB lookups are needed.
• A custom NodeJS stream transformer is utilized for on-the-fly decryption of the AES-256-CTR encryption of the media file stored in S3.

So how does this perform compared to the previous case? How much overhead does the decryption add?

Turns out not much at all. The median time-to-first-byte is almost identical to that of the case 2.

This result further emphasizes the core architecture principal at PhotoVaultOne: privacy-first. There is no need to choose between high performance and privacy today - with the right choice of technologies, you can have both!

Conclusion

For the tested use-case, Lambda Streaming Responses provides a highly performant serverless implementation for streaming large files to end-users. Added to that, client-side encryption of the content at rest with a streaming encryption cipher such as AES-256-CTR adds virtually no noticable impact to the delivery of the content.

PhotoVaultOne creates a true backup of your precious digital memories. You can get started with a 3 month free trial today.

Blog resources

2023

• Streaming encrypted images using Lambda Streaming Responses
• Vault Mode now available
• When a copy is not a backup
• Protecting your personal data

2022

• Vault Mode for private encrypted backups
• Welcome to PhotoVaultOne