Continuously improve search application effectiveness with Amazon Kendra Analytics Dashboard

Unstructured data belonging to enterprises continues to grow, making it a challenge for customers and employees to get the information they need. Amazon Kendra is a highly accurate intelligent search service powered by machine learning (ML). It helps you easily find the content you’re looking for, even when it’s scattered across multiple locations and content repositories.

Amazon Kendra provides mechanisms such as relevance tuning, filtering, and submitting feedback for incremental learning to improve the effectiveness of the search solution based on specific use cases. As the data, users, and user expectations evolve, there is a need to continuously measure and recalibrate the search effectiveness, by adjusting the search configuration.

Amazon Kendra analytics provides a snapshot of how your users interact with your Amazon Kendra-powered search application in the form of key metrics. You can view the analytics data in a visual dashboard on the Amazon Kendra console or via Application Programming Interface (API) or using the AWS Command Line Interface (AWS CLI). These metrics enable administrators and content creators to better understand the ease of finding relevant information, the quality of the search results, gaps in the content, and the role of instant answers in providing answers to a user’s questions.

This post illustrates how you can dive deep into search trends and user behavior to identify insights and bring clarity to potential areas of improvement and the specific actions to take.

Overview of the Amazon Kendra analytics dashboard

Let’s start with reviewing the Search Analytics Dashboard of the Amazon Kendra index we use during this post. To view the Amazon Kendra analytics dashboard, open the Amazon Kendra management console, choose your index, and then choose Analytics in the navigation pane.

Just by looking at the top, there is a trend of an increasing number of queries, implying an increase in application adoption. There is little change since the last period in the clickthrough rate, zero click rate, zero search result rate, and the instant answer rate, signifying that the new queries and potentially new users’ usage pattern is consistent with that of the previous period.

Let’s look at the other macro trend charts available on the dashboard (see the following screenshots).

All the charts show a flat trend, meaning that the usage pattern is steady.

The clickthrough rate is at single digits with a slight downward trend. This either means that the users are finding the information through instant answers, FAQs, or document excerpts, or this could indicate that the results are totally uninteresting to the users.

The top zero click queries hover a little below 10%, which also means that the users are finding the information through instant answers, FAQs, or document excerpts, or the results are uninteresting to the users.

The instant answer rate is above 90%, which means that the overall quality of content is good and contains the information users are looking for.

The top zero result queries is lower than 5%, which is a good indicator that for the most part the users are finding the information they’re looking for.

Now let’s look at the drill-down charts starting with top queries, sorted high to low by Count.

The most important insight here is that the top queried items matter most to the users. The organization can use this information to potentially change their business priorities to focus more on these items of interest. It can also be an indicator to add more content on these topics.

When looking at the top queries sorted low to high on the Instant answer (%) column, we get the following results.

This provides insights into the items that the users are looking for but can’t find the answers. Depending on the query count, this may be a good indicator to add more content with specific information that answers the queries.

Now let’s look at the top clicked documents, sorted on the Count column from high to low.

These items indicate topics of interest to the users, not just for answers but also for detailed information. It could be an indicator to possibly add more content on these topics, but might be a business indicator to arrange training on these topics.

Let’s continue with the top zero click queries, sorted on 0 click count high to low.

This shows items of high interest that coincide with a high instant answer rate, implying that the users quickly find the answers through instance answers.

Now let’s look at the same chart sorted on Instant answer rate, low to high.

This indicates that there is lack of information on these topics that are of interest to users, and that the content owners need to add more content on these topics.

Now let’s look at the top zero result queries, sorted on the 0 result count column from high to low.

This is an indicator of a gap in content, because the users are looking for information that can’t be found. The content owners can fix this by adding content on these topics.

Using AWS CLI and API to get the Amazon Kendra analytics dashboard

So far we have used the visual dashboard in the Amazon Kendra management console to view all the available charts. The same dashboards are also available via API or the AWS CLI, which you can use to integrate this information in your applications as well as the tools of your choice for analytics and dashboards. You can use the following AWS CLI command to get the top queries this week based on their count:

aws kendra get-snapshots --index-id  --interval "THIS_WEEK" --metric-type "QUERIES_BY_COUNT"

The output looks similar to the following:

{
  "SnapShotTimeFilter": {
    "StartTime": "2021-11-14T08:00:00+00:00",
    "EndTime": "2021-11-20T07:00:00+00:00" 
  },
  "SnapshotsDataHeader": [ 
    "query_content", "count", "ctr", "zero_click_rate", "click_depth", "instant_answer",     "confidence"
    ],
  "SnapshotsData": [
    [
      "what is Kendra", 3216, 3.70, 96.30, 27.71, 97.01, HIGH,
      "NBA game schedule", 1632, 4.47, 95.53, 24.19, 95.47, MEDIUM,
      "Most popular search", 1603, 3.49, 96.51, 29.43, 94.14, MEDIUM,
      "New York City", 1551, 3.68, 96.32, 33.40, 94.58, MEDIUM,
      "how many weeks in a year", 1310, 2.21, 97.79, 42.10, 96.03, LOW,
      "what is my ip address", 859, 2.56, 97.44, 48.45, 96.97, MEDIUM,
      "how to draw", 857, 2.80, 97.20, 36.33, 96.38, HIGH,
      "what is love", 855, 2.46, 97.54, 27.33, 96.73, MEDIUM,
      "equal opportunity bill", 855, 5.26, 94.74, 23.62, 94.62, MEDIUM,
      "when are the nba playoffs", 836, 3.35, 96.65, 32.32, 92.34, LOW
    ]
  ],
  "NextToken":    "uVu4IDozCVdFz5klt0h9+YPTTNcCGGwGujsYChp1/vPp5nPdC+reHO8TRvg5ANhWQu10jvKltuM8KzUvYCvBGi7mWJdpOF7LFiBjFcIuY6cabYI9nb2b0u3AU3565RC9kCytG6RjeVcU/NjBAxLMyB96+WdEYv+jFCbejnM6YjWa0LRL+MmvlnXEkFMWvmgyrdF22JXWklTZc77NJILR+BTsCB5Xg34OJ4149968kDdb2CNhH4Bzk+qOGph+KoFDW/CpmQ=="
}

You can also get similar output using the following Python code:

import boto3
kendra = boto3.client('kendra')

index_ID = '${indexID}'
interval = 'THIS_WEEK'
metric_type = 'QUERIES_BY_COUNT'

snapshots_response = kendra.get_snapshots(
    IndexId = index_id,
    Interval = interval,
    MetricType = metric_type,
print("Top queries data: " + snapshots_response['snapshotsData'])

Conclusion

Growth of data and information along with evolving user needs make it imperative that the effectiveness of the search application also evolves. The metrics provided by the Amazon Kendra analytics empower you to dive deep into search trends and user behavior to identify insights. They help bring clarity to potential areas of improvement for Amazon Kendra-powered search applications. If you already implement an Amazon Kendra index-powered search solution, start looking at the Analytics Dashboard with the usage metrics for the last few weeks and get insights on how you can improve the search effectiveness. For new Amazon Kendra-powered search applications, the Analytics Dashboard is a great place to get immediate feedback with actionable insights on the search effectiveness. For a hands-on experience with Amazon Kendra, see the Kendra Essentials workshop. For a deeper dive into Amazon Kendra use cases, see the Amazon Kendra blog.

About the Author

Abhinav Jawadekar is a Senior Partner Solutions Architect at Amazon Web Services. Abhinav works with AWS Partners to help them in their cloud journey.