An overview of the architecture for reading a file from the S3 Bucket is given in this document. When data is uploaded or modified in the Amazon S3 bucket, a Lambda function—also known as a ReadFile lambda function—is activated to publish the data to SNS and subsequently into SQS for additional processing. The lambda function will store the information in the database and publish alerts that will be subscribed to the SQS queue using an Amazon SNS topic. The message will be received by the SQS queue, and after further file processing, a seperate service will retrieve the message from the SQS queue.
The file is relocated to the processed folder in the same bucket after it has been read and the messages have been queued in SQS.
A seperate service will read and process the data from SQS. Unprocessed messages will be moved to another queue after being attempted x times.