A company has 200 million existing objects in an S3 bucket and needs to invoke a Lambda function against every one of them to redact a specific field, without writing custom code to list and iterate over each object individually. Which S3 feature is designed for this kind of bulk action across a very large number of existing objects?
- S3 Event Notifications
- S3 Transfer Acceleration
- S3 Lifecycle expiration rules
- S3 Batch Operations, using a manifest of the objects and invoking the Lambda function as the batch action
Why D? And why not the others?
Correct answer: D. S3 Batch Operations, using a manifest of the objects and invoking the Lambda function as the batch action
S3 Batch Operations is designed exactly for this scenario: given a manifest listing the target objects, it can invoke a single action, including invoking a Lambda function, copying, or restoring, across millions or billions of existing objects with a single S3 API request or a few console clicks, handling the iteration, retries, and completion reporting internally. Event Notifications only fire in reaction to new activity on objects, such as a fresh upload, and have no mechanism to sweep across 200 million objects that already exist. Transfer Acceleration only speeds up network transfer of individual uploads and downloads over long distances and has nothing to do with running an operation across a bucket's existing contents. Lifecycle expiration rules only delete or transition objects based on age; they cannot invoke an arbitrary Lambda function to redact a field within each object.
Source: AWS S3 docs: Performing large-scale batch operations on Amazon S3 objects