Introducing the AWS Panorama Device SDK: Scaling computer vision at the edge with AWS Panorama-enabled devices

Yesterday, at AWS re:Invent, we announced AWS Panorama, a new Appliance and Device SDK that allows organizations to bring computer vision to their on-premises cameras to make automated predictions with high accuracy and low latency. With AWS Panorama, companies can use compute power at the edge (without requiring video streamed to the cloud) to improve their operations by automating monitoring and visual inspection tasks like evaluating manufacturing quality, finding bottlenecks in industrial processes, and assessing worker safety within their facilities. The AWS Panorama Appliance is a hardware device that customers can install on their network to connect to existing cameras within your facility, to run computer vision models on multiple concurrent video streams.

This post covers how the AWS Panorama Device SDK helps device manufacturers build a broad portfolio of AWS Panorama-enabled devices. Scaling computer vision at the edge requires purpose-built devices that cater to specific customer needs without compromising security and performance. However, creating such a wide variety of computer vision edge devices is hard for device manufacturers because they need to do the following:

Integrate various standalone cloud services to create an end-to-end computer vision service that works with their edge device and provides a scalable ecosystem of applications for customers.
Invest in choosing and enabling silicon according to customers’ performance and cost requirements.

The AWS Panorama Device SDK addresses these challenges.

AWS Panorama Device SDK

The AWS Panorama Device SDK powers the AWS Panorama Appliance and allows device manufacturers to build AWS Panorama-enabled edge appliances and smart cameras. With the AWS Panorama Device SDK, device manufacturers can build edge computer vision devices for a wide array of use cases across industrial, manufacturing, worker safety, logistics, transportation, retail analytics, smart building, smart city, and other segments. In turn, customers have the flexibility to choose the AWS Panorama-enabled devices that meet their specific performance, design, and cost requirements.

The following diagram shows how the AWS Panorama Device SDK allows device manufacturers to build AWS Panorama-enabled edge appliances and smart cameras.

The AWS Panorama Device SDK includes the following:

Core controller – Manages AWS Panorama service orchestration between the cloud and edge device. The core controller provides integration to media processing and hardware accelerator on-device along with integration to AWS Panorama applications.
Silicon abstraction layer – Provides device manufacturers the ability to enable AWS Panorama across various silicon platforms and devices.

Edge devices integrated with the AWS Panorama Device SDK can offer all AWS Panorama service features, including camera stream onboarding and management, application management, application deployment, fleet management, and integration with event management business logic for real-time predictions, via the AWS Management Console. For example, a device integrated with the AWS Panorama Device SDK can automatically discover camera streams on the network, and organizations can review the discovered video feeds and name or group them on the console. Organizations can use the console to create applications by choosing a model and pairing it with business logic. After the application is deployed on the target device through the console, the AWS Panorama-enabled device runs the machine learning (ML) inference locally to enable high-accuracy and low-latency predictions.

To get device manufacturers started, the AWS Panorama Device SDK provides them with a device software stack for computer vision, sample code, APIs, and tools to enable and test their respective device for the AWS Panorama service. When ready, device manufacturers can work with the AWS Panorama team to finalize the integration of the AWS Panorama Device SDK and run certification tests ahead of their commercial launch.

Partnering with NVIDIA and Ambarella to enable the AWS Panorama Device SDK on their leading edge AI platforms

The AWS Panorama Device SDK will support the NVIDIA® Jetson product family and Ambarella CV 2x product line as the initial partners to build the ecosystem of hardware-accelerated edge AI/ML devices with AWS Panorama.

“Ambarella is in mass production today with CVflow AI vision processors for the enterprise, embedded, and automotive markets. We’re excited to partner with AWS to enable the AWS Panorama service on next-generation smart cameras and embedded systems for our customers. The ability to effortlessly deploy computer vision applications to Ambarella SoC-powered devices in a secure, optimized fashion is a powerful tool that makes it possible for our customers to rapidly bring the next generation of AI-enabled products to market.”

– Fermi Wang, CEO of Ambarella

“The world’s first computer created for AI, robotics, and edge computing, NVIDIA® Jetson AGX Xavier delivers massive computing performance to handle demanding vision and perception workloads at the edge. Our collaboration with AWS on the AWS Panorama Appliance powered by the NVIDIA Jetson platform accelerates time to market for enterprises and developers by providing a fully managed service to deploy computer vision from cloud to edge in an easily extensible and programmable manner.”

– Deepu Talla, Vice President and General Manager of Edge Computing at NVIDIA

Enabling edge appliances and smart cameras with the AWS Panorama Device SDK

Axis Communications, ADLINK Technology, Basler AG, Lenovo, STANLEY Security, and Vivotek will be using the AWS Panorama Device SDK to build AWS Panorama-enabled devices in 2021.

“We’re excited to collaborate to accelerate computer vision innovation with AWS Panorama and explore the advantages of the Axis Camera Application Platform (ACAP), our open application platform that offers users an expanded ecosystem, an accelerated development process, and ultimately more innovative, scalable, and reliable network solutions.”

– Johan Paulsson, CTO of Axis Communications AB

“The integration of AWS Panorama on ADLINK’s industrial vision systems makes for truly plug-and-play computer vision at the edge. In 2021, we will be making AWS Panorama-powered ADLINK NEON cameras powered by NVIDIA Jetson NX Xavier available to customers to drive high-quality computer vision powered outcomes much, much faster. This allows ADLINK to deliver AI/ML digital experiments and time to value for our customers more rapidly across logistics, manufacturing, energy, and utilities use cases.”

– Elizabeth Campbell, CEO of ADLINK USA

“Basler is looking forward to continuing our technology collaborations in machine learning with AWS in 2021. We will be expanding our solution portfolio to include AWS Panorama to allow customers to develop AI-based IoT applications on an optimized vision system from the edge to the cloud. We will be integrating AWS Panorama with our AI Vision Solution Kit, reducing the complexity and need for additional expertise in embedded hardware and software components, providing developers with a new and efficient approach to rapid prototyping, and enabling them to leverage the ecosystem of AWS Panorama computer vision application providers and systems integrators.”

– Arndt Bake, Chief Marketing Officer at Basler AG

“Going beyond traditional security applications, VIVOTEK developed its AI-driven video analytics from smart edge cameras to software applications. We are excited that we will be able to offer enterprises advanced flexibility and functionality through the seamless integration with AWS Panorama. What makes this joint force more powerful is the sufficient machine learning models that our solutions can benefit from AWS Cloud. We look forward to a long-term collaboration with AWS.”

– Joe Wu, Chief Technology Officer at VIVOTEK Inc.

Next steps

Join now to become a device partner and build edge computer vision devices with the AWS Panorama Device SDK.

About the Authors

As a Product Manager on the AWS AI Devices team, Shardul Brahmbhatt currently focuses on AWS Panorama. He is deeply passionate about building products that drive adoption of AI at the Edge.

Kamal Garg leads strategic hardware partnerships for AWS AI Devices. He is deeply passionate about incubating technology ecosystems that optimize the customer and developer experience . Over the last 5+ years, Kamal has developed strategic relationships with leading silicon and connected device manufacturers for next generation services like Alexa, A9 Visual Search, Prime Video, AWS IoT, Sagemaker Neo, Sagemaker Edge, and AWS Panorama.