Drive efficiencies with CI/CD best practices on Amazon Lex

Let’s say you have identified a use case in your organization that you would like to handle via a chatbot. You familiarized yourself with Amazon Lex, built a prototype, and did a few trial interactions with the bot. You liked the overall experience and now want to deploy the bot in your production environment, but aren’t sure about best practices for Amazon Lex. In this post, we review the best practices for developing and deploying Amazon Lex bots, enabling you to streamline the end-to-end bot lifecycle and optimize your operations.

We have covered the planning, design, and configuration phases in previous blog posts. We suggest reviewing these posts to help you build engaging conversations with your bot before you proceed. After you’ve initially configured the bot, you should test it internally and iterate on the bot definition. You’re now ready to deploy it in your production environment (such as a call center), where the bot will process live conversations. Once in production, you should monitor it continuously to make sure it’s meeting your desired business goals. This cycle repeats as you add new use cases and enhancements.

Let’s review the best practices for development, testing, deployment, and monitoring bots.

Development

Consider the following best practices when developing your bot:

Manage bot schema via code – The Amazon Lex console provides an easy-to-use interface as you design and configure the bot, but relies on manual actions to replicate the setup. We recommend converting the bot schema into code after finishing the design to simplify this step. You can use APIs or AWS CloudFormation (see Creating Amazon Lex V2 resources with AWS CloudFormation) to manage the bot programmatically.
Checkpoint bot schema with bot versioning – Checkpointing is a common approach often used to revert an application to a last-known stable state. Amazon Lex offers this functionality via bot versioning. We recommend using a new version at each milestone in your development process. This allows you to make incremental changes to your bot definition, with an easy way to revert them in case they don’t work as expected.
Identify data handling requirements and configure appropriate controls – Amazon Lex follows the AWS shared responsibility model, which includes guidelines for data protection to comply with industry regulations and with your company’s own data privacy standards. Additionally, Amazon Lex adheres to compliance programs such as SOC, PCI, and FedRAMP. Amazon Lex provides the ability to obfuscate slots that are considered sensitive. You should identify your data privacy requirements and configure the appropriate controls in your bot.

Testing

After you have a bot definition, you should test the bot to ensure that it works as intended and is configured correctly. For example, it should have permissions to trigger other services, such as AWS Lambda functions. In addition, you should also test the bot to confirm it’s able to interpret different types of user requests. Consider the following best practices for testing:

Identify test data – You should gather relevant test data to test bot performance. The test data should include a comprehensive representation of expected user conversations with the bot, especially for IVR use cases where the bot will need to understand voice inputs. The test data should cover different speaking styles and accents. Such test data can provide experience validation for your target customer base.
Identify user experience metrics – Defining the conversational experience can be hard. You have to anticipate and plan for all the different ways users might engage with the bot. How do you guide the caller without sounding too prescriptive? How do you recover if the caller provides incorrect or incomplete information? To manage the dialog through many different scenarios, you should set a clear goal that covers different speaking styles, acoustic conditions, and modality, and identify objective metrics that you can track. For example, an objective indicator would be “90% of conversations should have less than two re-prompts played to the user,” versus a subjective indicator such as “the majority of conversations should not ask users to repeat their input.”
Evaluate user experience along the way – In some cases, seemingly small changes can have a big impact on the user experience. For example, consider a situation where you inadvertently introduce a typo in the regular expression used for an account ID slot type, which leads to bot re-prompting the user to provide input again. You should evaluate the user experience, and invest in an automated testing to generate key metrics. You can refer to Evaluating an automatic speech recognition service and Testing accuracy and regression with Amazon Connect and Amazon Lex for examples of how to test and generate key metrics.

Deployment

Once you’re satisfied with the bot’s performance, you’ll want to deploy the bot to start serving your production traffic. As you iterate the bot over the course of its lifecycle, you repeat the deployments, making it a continuous process, so it’s critical to have a streamlined, automated deployment to reduce the chance of errors. Consider the following best practices for deployment:

Use a multi-account environment – You should follow the AWS recommended multi-account environment setup in your organization and use separate AWS accounts for your development stage and production stage. If you have a multi-Region presence, then you should also use a separate AWS account per Region for production. Using separate AWS accounts per stage offers you security, access, and billing boundaries for your AWS resources.
Automate promoting a bot from development through to production – When replicating the bot setup in your development stage to your production stage, you should use automated solutions and minimize manual touch points. You should use CloudFormation templates to create your bots. Alternatively, you can use Amazon Lex export and import APIs to provide an automated means to copy a bot schema across accounts.
Roll out changes in a phased manner – You should deploy changes to your production environment in a phased manner, so that changes are released to a subset of your production traffic before being released to all users. Such an approach gives you the chance to limit the blast radius in case there are any issues with the change. One way you can achieve this is by having a two-phased deployment approach: you create two aliases for a bot (for example, prod-05 and prod-95). You first associate the new bot version with one alias (prod-05 in this example). After you validate the key metrics meet the success criteria, you associate the second alias (prod-95) with the new bot version.

Note that you need to control the distribution of traffic on the client application used to integrate with Amazon Lex bots. For example, if you’re using Amazon Connect to integrate with your bots, you can use a Distribute by percentage contact block in conjunction with two or more Get customer input blocks.

It’s important to note that Amazon Lex provides a test alias out of the box. The test alias is meant to be used for ad hoc manual testing via the Amazon Lex console only, and is not meant to handle production-scale loads. We recommend using a dedicated alias for your production traffic.

Monitoring

Monitoring is important for maintaining reliability, availability, and an effective end-user experience. You should analyze your bot’s metrics and use the learnings as a feedback mechanism to improve the bot schema as well your development, testing, and deployment practices. Amazon Lex supports multiple mechanisms to monitor bots. Consider the following best practices for monitoring your Lex bots:

Monitor constantly and iterate – Amazon Lex integrates with Amazon CloudWatch to provide near-real-time metrics that can provide you with key insights into your users’ interactions with the bot. These insights can help you gain perspective on the end-user experience. To learn more about the different types of metrics that Amazon Lex emits, see Monitoring Amazon Lex V2 with Amazon CloudWatch. We recommend setting up thresholds to trigger alarms. Similarly, Amazon Lex gives you visibility into the raw input utterances from your users’ interactions with the bot. You should use utterance statistics or conversation logs to gain insights to identify communication patterns and make appropriate changes to your bot as necessary. To learn how to create a personalized analytics dashboard for your bots, refer to Monitor operational metrics for your Amazon Lex chatbot.

The best practices discussed in this post focus primarily on Amazon Lex-specific use cases. In addition to these, you should review and adhere to best practices when managing your cloud infrastructure in AWS. Make sure that your cloud infrastructure is secure and only accessible by authorized users. You should also review and adopt the appropriate AWS security best practices within your organization. Lastly, you should proactively review the AWS quotas for individual AWS services (including Amazon Lex quotas) and request appropriate changes if necessary.

Conclusion

You can use Amazon Lex to enable sophisticated natural language conversations and drive customer service efficiencies. In this post, we reviewed the best practices for the development, testing, deployment, and monitoring phases of a bot lifecycle. With these guidelines, you can improve the end-user experience and achieve better customer engagement. Start building your Amazon Lex conversational experience today!

About the Author

Swapandeep Singh is an engineer with the Amazon Lex team. He works on making interactions with bots smoother and more human-like. Outside of work, he likes to travel and learn about different cultures.