Scan Amazon S3 buckets for content moderation using S3 Batch and Amazon Rekognition

Dealing with content in large scale is often challenging, costly, and a heavy lift operation. The volume of user-generated and third-party content has been increasing substantially in industries like social media, ecommerce, online advertising, and media sharing. Customers may want to review this content to ensure that it follows corporate governance and regulations. But they need a solution to handle scale and automation.

In this blog, I present a solution to scan existing videos in your Amazon Simple Storage Service (Amazon S3) buckets using S3 batch operations and Amazon Rekognition.

S3 batch operations is a managed solution for performing storage actions in large scale. S3 batch operations can perform actions on lists of Amazon S3 objects that you specify. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions, providing a fully managed, auditable, and serverless experience.

Amazon Rekognition enables you to analyze images and videos automatically on your applications and content. You provide an image or video to the Amazon Rekognition API, and the service can identify the objects, people, text, scenes, and activities, in addition to detecting any inappropriate content.

I show an example to demonstrate how to use S3 batch operations with Amazon Rekognition content moderation. Many companies rely entirely on human moderators to review third-party or user-generated content, while others primarily react to user complaints to take down offensive or inappropriate images, ads, or videos. However, human moderators alone cannot scale to meet these needs at sufficient quality or speed. That leads to a poor user experience, high costs to achieve scale, or even a loss of brand reputation. By using Amazon Rekognition, human moderators can review a much smaller set of content, typically 1% to 5% of the total volume, which has already been flagged by machine learning. This enables them to focus on more valuable activities and still achieve comprehensive moderation coverage at a fraction of their existing cost.

Solution Architecture

This solution uses S3 batch operations to invoke an AWS Lambda function that will scan videos in your S3 buckets using Amazon Rekognition. It will store the findings on an Amazon DynamoDB table and details on the activity will be logged on Amazon CloudWatch. In addition to that, it will send email alerts using Amazon Simple Notification Service (Amazon SNS) when Amazon Rekognition detects such content with high confidence levels.

The following diagram outlines the workflow:

1. S3 batch operations read the manifest file
2. S3 batch operations invoke a Lambda function
3. The Lambda function invokes Amazon Rekognition for content moderation on videos
4. Activity is logged on Amazon CloudWatch
5. Labels are stored in a DynamoDB table
6. Amazon SNS sends an email notification

Let’s look at each step in more detail:

1. S3 batch operations reads the manifest file

S3 batch operations can perform operations on large lists of Amazon S3 objects that you specify (manifest file). That allows to scale the solution as a single job can perform a specified operation on billions of objects containing exabytes of data. Amazon S3 tracks progress, sends notifications, and stores a detailed completion report of all actions. I use two buckets in the solution: one containing the videos and another containing the S3 inventory report (manifest file) and S3 batch operations logging. This allows you to isolate videos and processing information. The S3 batch job performs the operations on the objects listed in the manifest file, or you can use a CSV file with the list of videos.

2. S3 batch operations invokes a Lambda function

The Lambda function contains the Python code that will use Amazon Rekognition, store data on DynamoDB, and email notifications. AWS Lambda is a compute service that lets you run code without provisioning or managing servers. Lambda runs your code on a high availability compute infrastructure and performs all of the administration of the compute resources. It scales automatically and is cost efficient.

3. The Lambda function invokes Amazon Rekognition for content moderation on videos

This is the core machine learning activity in the solution. The S3 batch job provides the S3 key from each object in the manifest file to the Lambda function. The Lambda function calls the Amazon Rekognition APIs. The StartContentModeration operation starts asynchronous detection of inappropriate, unwanted, or offensive content in a stored video. The GetContentModeration operation gets the unsafe content analysis results.

4. Activity is logged on Amazon CloudWatch

AWS Lambda logs all requests handled by the function and stores logs generated by the code in Amazon CloudWatch Logs. Details from the runtime, labels identified, and workflow data are stored on CloudWatch.

5. Labels are stored in a DynamoDB table

Amazon DynamoDB is a key-value and document database that delivers single-digit millisecond performance at any scale. Details about the labels found on videos are durably stored in a DynamoDB table: Job ID, timestamps of label occurrence, confidence level, hierarchical taxonomy of labels, and video names are recorded in the table. That aggregates value since now you have a database with content moderation information from your videos. You can use the database to classify videos according to their content moderation.

6. Amazon SNS sends an email notification

The code checks the confidence level reported by Amazon Rekognition and sends email alerts when there is a high level of certainty (95% or more).

Label categories of inappropriate or offensive content

Amazon Rekognition uses a two-level hierarchical taxonomy to label categories of inappropriate or offensive content. For example, in the top-level category of tobacco you have a second-level category identifying tobacco products and smoking. More details in the following table.

Prerequisites

In order to implement this solution, you will need:

• an AWS account
• two S3 buckets
• a Lambda function
• a DynamoDB table
• an SNS topic

Implementation

There are four main steps to implement the solution:

1. Create the S3 buckets
2. Create a DynamoDB table
3. Create a Lambda function using the following Python code
4. Create and run the S3 batch job

Here are more details on each step:

1. Create two S3 buckets: one to store the videos (data bucket) and another to store the inventory reports and logs (work bucket). Use separate buckets to avoid scanning objects that are not videos and, consequently, wasting Lambda cycles. Set up S3 inventory reports from the bucket containing the videos (data bucket) into the work bucket. The inventory reports can take up to 48 hours to be created.
2. Create a DynamoDB table using the following Python code.

import boto3

# Create a table to store Rekognition labels
tablename = "rekognition-demo-table"

dynamodb = boto3.resource('dynamodb', region_name='us-east-2')
mytable = dynamodb.create_table(
TableName= tablename,
KeySchema=[
{
'KeyType': 'HASH',
'AttributeName': 'Id'
},
{
'KeyType': 'RANGE',
'AttributeName': 'Timestamp'
},
],
AttributeDefinitions=[
{
'AttributeName': 'Id',
'AttributeType': 'S'
},
{
'AttributeName': 'Timestamp',
'AttributeType': 'N'
},
],
ProvisionedThroughput={
'ReadCapacityUnits': 2,
'WriteCapacityUnits': 2
}
)
# Wait table creation
mytable.meta.client.get_waiter('table_exists').wait(TableName=tablename)
print('Table is ready for use')

3. Create a Lambda function to invoke the following Python code:

For videos lasting up to 10 minutes, increase the Lambda function timeout to 1 minute and allocated 128 MB of memory.

In the following code, replace the ARN (Amazon Resource Name) for the SNS topic of your choice in the TargetArn variable. That will allow you to receive the email alert.

import boto3
import time
import uuid
import datetime


def lambda_handler(event, context):
    s3Client = boto3.client('s3')
    rekognition = boto3.client('rekognition')
    client = boto3.client('sns')

    dynamodb = boto3.resource('dynamodb', region_name='us-east-2')
    table = dynamodb.Table('rekognition-demo-table')

    # Parse job parameters
    jobId = event['job']['id']
    print(f"JobID: {jobId}")
    invocationId = event['invocationId']
    invocationSchemaVersion = event['invocationSchemaVersion']

    # Process the task
    task = event['tasks'][0]
    print(f"Task: {task}")
    taskId = task['taskId']
    print(f"Task ID: {taskId}")
    s3Key = task['s3Key']
    print(f"S3 Key: {s3Key}")
    s3VersionId = task['s3VersionId']
    print(f"S3 Version ID: {s3VersionId}")
    s3BucketArn = task['s3BucketArn']
    print(f"S3Bucket ARN: {s3BucketArn}")
    s3Bucket = s3BucketArn.split(':')[-1]
    print('BatchProcessObject(' + s3Bucket + "/" + s3Key + ')')

    #Rekognition Content Moderation APIs
    startModerationLabelDetection = rekognition.start_content_moderation(
        Video={'S3Object': {'Bucket': s3Bucket, 'Name': s3Key, }}, )
    moderationJobId = startModerationLabelDetection['JobId']
    print("Job Id: {0}".format(moderationJobId))

    getContentModeration = rekognition.get_content_moderation(
        JobId=moderationJobId,
        SortBy='TIMESTAMP')

    while (getContentModeration['JobStatus'] == 'IN_PROGRESS'):
        time.sleep(5)
        print('.', end='')

        getContentModeration = rekognition.get_content_moderation(
            JobId=moderationJobId,
            SortBy='TIMESTAMP')

    print(getContentModeration['JobStatus'])
    print(getContentModeration)

    theObjects = {}

    strDetail = "Moderation labels in videon"
    strOverall = "Moderation labels in the overall video:n"

    # Potentially unsafe content detected in each frame
    for obj in getContentModeration['ModerationLabels']:
        ts = obj["Timestamp"]
        cconfidence = obj['ModerationLabel']["Confidence"]
        oname = obj['ModerationLabel']["Name"]
        strDetail = strDetail + "At {} ms: {} (Confidence: {})n".format(ts, oname, round(cconfidence, 2))
        if oname in theObjects:
            cojb = theObjects[oname]
            theObjects[oname] = {"Name": oname, "Count": 1 + cojb["Count"]}
        else:
            theObjects[oname] = {"Name": oname, "Count": 1}

    # Unique objects detected in video
    for theObject in theObjects:
        strOverall = strOverall + "Name: {}, Count: {}n".format(theObject, theObjects[theObject]["Count"])

    # Display results
    print(strOverall)

    mailer = ''
    timestamps = []
    for i in getContentModeration['ModerationLabels']:
        conf_level = i['ModerationLabel']['Confidence']
        name = i['ModerationLabel']["Name"]
        parent = i['ModerationLabel']["ParentName"]
        timestamp = i['Timestamp']
        if timestamp not in timestamps:
            if conf_level > 95:
                string = f'Amazon Rekognition detected "{parent}" at {timestamp}ms "{name}" on video "{s3Key} with a confidence of {round(conf_level, 1)}%";n'
                mailer += string
                timestamps.append(timestamp)

            table.put_item(
                Item={
                    "Id": str(uuid.uuid4()),
                    'Timestamp': timestamp,
                    'Confidence': str(round(conf_level, 1)),
                    'Name': name,
                    'ParentName': parent,
                    'Video': s3Key,
                    'Date': str(datetime.datetime.now())
                })

    if mailer != '':
        message = client.publish(TargetArn='arn:aws:sns:us-east-2:111122223333:reko-test', Message=mailer,
                                 Subject='Amazon Rekognition Video Detection')

    results = [{
        'taskId': taskId,
        'resultCode': 'Succeeded',
        'resultString': 'Succeeded'
    }]

    return {
        'invocationSchemaVersion': invocationSchemaVersion,
        'treatMissingKeysAs': 'PermanentFailure',
        'invocationId': invocationId,
        'results': results
    }

4. Create and launch the S3 batch job by using the following command:

aws s3control create-job 
--account-id  
--no-confirmation-required 
--operation '{"LambdaInvoke": { "FunctionArn": "arn:aws:lambda:Region:AccountID:function:LambdaFunctionName" } }' 
--manifest '{"Spec":{"Format":"S3BatchOperations_CSV_20180820","Fields":["Bucket","Key"]},"Location":{"ObjectArn":"arn:aws:s3:::ManifestLocal"}' 
--report '{"Bucket":"arn:aws:s3:::sample-bucket", "Format":"Report_CSV_20180820", "Enabled":true, "ReportScope":"AllTasks"}' 
--priority 10 
--role-arn arn:aws:iam::AccountID:role/BatchOpsRole 
--region Region 
--description "S3 Batch Lambda Job" 

Make sure to replace the appropriate fields in the preceding command. Replace your account ID, the Lambda function ARN, and the ARN of the role you will use to run this batch job. Also, replace the ARN for the manifest file, and the bucket for reporting. The manifest file is generated as part of the S3 inventory.

Results

After running the solution, you will find updates on Amazon CloudWatch, Amazon DynamoDB, and you will get email notifications if the confidence level is at least 95%.

You will get content moderation outputs for each of the videos listed in the S3 manifest file. This solution can take seconds to scan each video, despite of their size. You will be able to scan a large number of videos in a single S3 batch processing job. That allows scaling and automation.

The following is an extract of a CloudWatch Log for a sample video. You can see that for the video named Girlsfools_512kb.mp4, Amazon Rekognition, detected labels such as tobacco and nudity. It also provides Lambda runtime details such as billed duration and memory used.

The following is an extract of the DynamoDB table generated for this video. In this table we can see the timestamp details, the confidence level, and the hierarchical taxonomy findings.

The following is an extract of an email alert generated. In this example, it reported visually disturbing labels found in the video with their corresponding timestamps. It also reported a 96.5% confidence level on the findings.

Cleaning up

If you followed along and would like to remove resources used in this solution to avoid incurring any unwanted future charges, delete the following:

• the two S3 buckets created
• the S3 inventory configuration
• the S3 batch operations job
• the Lambda function
• the DynamoDB table
• the SNS topic

Conclusion

Amazon S3 batch operations can manage billions of objects at a time, saving you time and resources by simplifying storage management at massive scale. Amazon Rekognition is a service that lets you add powerful visual analysis to your applications. You can use Amazon S3 batch operations in addition to Amazon Rekognition to scan existing objects in your buckets.

In this blog, I describe a solution integrating Amazon S3 batch operations and Amazon Rekognition. I show an example that can be used to address content moderation across different domains: social media, broadcasting, law enforcement, public entities among others. In this example, we scan existing videos stored in an S3 bucket to detect explicit or suggestive adult content, violence, weapons, visually disturbing content, drugs, alcohol, tobacco, hate symbols, gambling, and rude gestures.

The process is logged on Amazon CloudWatch, the moderation content is durably stored in a DynamoDB database, and alert emails are sent when needed.

To learn more about Amazon S3 batch operations, you can look through the product documentation. You can also discover more uses and implementations using Amazon S3 batch operations on the AWS Storage Blog.

To learn more about Amazon Rekognition for image and video analysis, you can look through the product documentation. You can also discover more uses and implementations using Amazon Rekognition on the AWS Machine Learning Blog.

If you need any further assistance, contact your AWS account team or a trusted AWS Partner.

About the Author

Virgil Ennes is a Specialty Sr. Solutions Architect at AWS. Virgil enjoys helping customers take advantage of the agility, costs savings, innovation, and global reach that AWS provides. He is mostly focused on Storage, AI, Blockchain, Analytics, IoT, and Cloud Economics. In his spare time, Virgil enjoys spending time with his family and friends, and also watching his soccer team (GALO).

View Original Source (aws.amazon.com) Here.