Fully customizable action space now available on the AWS DeepRacer console

AWS DeepRacer is the fastest way to get rolling with machine learning (ML) through a global racing league, cloud-based 3D racing simulator, and fully autonomous 1/18th scale race car driven by reinforcement learning. Starting today, the model action space is fully customizable yet simplified with new dynamic graphics so developers have greater control and can easily unlock better model performance.

Advanced AWS DeepRacer developers need the ability to customize the action space to increase model performance. Until now, you were limited to 20 options for modifying steering angle and speed in your action space on the AWS DeepRacer console. To achieve greater customizability, developers directly edited the action space in the JSON file in Amazon Simple Storage Service (Amazon S3) after the model had been trained. With a fully customizable action space, you can now specify steering angle, speed, and add actions of your own to push your model performance to its limits!

New visualization tools help racers understand the changes to their action space, dynamically reflecting the exact set of actions defined. Additionally, until now, each change made to the action space required either a new vehicle or changes to an existing one, resulting in a garage full of similar-looking vehicles with different action spaces. Today, the sensor configuration and look of the vehicle are decoupled from the model creation flow so racers can use any of their appropriately configured vehicles for any of their models.

“This new feature is a great example of AWS taking community feedback on board and implementing the changes we’ve asked for,” says Darren Broderick (racing alias “DBro”).

Action space

In reinforcement learning, the set of all valid actions, or choices, available to an agent (in this case a vehicle) as it interacts with an environment (track) is called an action space.

A discrete action space represents all of an agent’s possible actions for each state in a finite set. For AWS DeepRacer, this means that for every incrementally different environmental situation, the agent’s neural network selects a speed and direction for the car based on input from its sensors. The choice is limited to a grouping of predefined steering angle and throttle value combinations.

A continuous action space allows the agent to select an action from a range of values for each state. Just as with a discrete action space, this means that for every incrementally different environmental situation, the agent’s neural network selects a speed and direction for the car based on input from sensors. However, in a continuous action space, racers define the minimum and maximum values for steering angle and speed to create a range of options the agent picks its action from.

For AWS DeepRacer, this means that for every incrementally different environmental (track) situation, the agent’s neural network selects a speed and direction for the car based on input from its cameras and (optional) LiDAR sensor. When using the continuous action space, this is a value between the minimum and maximum values set. For a discrete action space, you still start with the minimum and maximum values, but you adjust the size of the action space by altering the steering angle and speed granularity. Now you can further adjust the steering and speed values in the action space as well as add or remove actions from the list.

Create a new model

Let’s go through the new process of creating a new model.

On the AWS DeepRacer console, choose Your models.
Choose Create model.

This starts the multi-step flow to create a new model that racers are familiar with.

Give your model a name and description, and select the track to train on.

Select the training algorithm and hyperparameters.
Choose Next.

Next, we define the action space.

Select between either continuous or discrete action spaces (continuous action space is the default selection).

The updated dynamic sector graph provides better visibility of how changes in steering angle and speed settings impact the action space. In the following example screenshot, the steering angle is adjusted to the left 20 degrees and right -20 degrees, along with the speed having a minimum of 1 m/s and a maximum 3 m/s. The sector graph on the right instantly updates to reflect these changes.

When using the discrete action space, things look similar except for the radial polar graph on the right and the new option of Advanced configuration under Action list. You can make discrete action space changes directly on the radial polar graph and by updating the action list.

Toggle the Advanced configuration setting to fully define the discrete action space as desired.

Remove an action from the list by choosing X at the end of a row, or add an action by choosing Add an action to increase the size of the action space. In the following example, the action currently selected in the list (action 0) is also highlighted on the radial polar graph to make it easier to visualize how changes to the action list impact the action space. Alternatively, drag the arrow on the graph to change the steering angle and speed. Changes made on the graph update the action list and vice versa.

Choose Next.
Choose a vehicle.
Customize the reward function and begin the training process.

When model training is complete, clone the model to see what changes can be made to action space. The interface is similar to the advanced action space with the exception that actions can’t be added or removed from the action list. You can change the steering angles or speed using either the action list table or by dragging the arrows on the radial polar graph as required based on the model’s performance during training and evaluation.

All subsequent steps are unchanged.

Your garage

Now let’s look at the new garage experience.

On the AWS DeepRacer console, choose Your garage to set the sensor configuration and look of your agent. You can earn digital rewards to customize your car by taking part in the AWS DeepRacer League (my personal favorite is currently the Baja Truck).

Get rolling with AWS DeepRacer

It’s time to hit the track, train your models like a pro with customizable action spaces, and level up your race performance! Try it out and let us know how it goes in the comments!

About the Author

David Smith is a Sr. Solutions Architect for AWS DeepRacer. He is passionate about AWS DeepRacer, technology as an enabler and learning. Outside of work he’s into Formula 1, flying (and crashing) drones, 3D printing, running (Parkrun), tinkering with code and spending time with the family.