Google robots sorting waste[©Google Research 2023]

The study shows that reinforcement learning (RL) systems can enable robots to sort waste by interacting with their environment, receiving feedback through rewards or penalties, and optimizing their actions to maximize the overall reward. It allows artificial intelligence to resemble natural intelligence as closely as possible.

Through the application of RL, the mobile robots, with vision systems and an arm, were able to address real-world tasks in workplace environments, with a combination of offline and online data enabling them to adapt to the broad variability of real-world situations.

The study programmed the robots – supplied by Everyday Robots, a part of Google parent company Alphabet – to roam and search for ‘waste situations’ – bins for recyclables, compost, and trash. They were then tasked with sorting the items between the bins so that all recyclables (cans, bottles) were placed in the recyclable bin, the compostable items (cardboard containers, paper cups) were placed in the compost bin, and everything else was placed in the residual waste bin.

The robots were bootstrapped with a basic set of skills – the process through which a computer is loaded with a program using a much smaller initial program. The skills included four sets of experience:

1. A set of simple hand-designed policies that have a very low success rate, but serve to provide some initial experience;
2. A simulated training framework that uses sim-to-real transfer to provide some initial bin sorting strategies,
3. ‘Robot classrooms’ where the robots continually practice at a set of representative waste stations;
4. And the real deployment setting, where robots practice in real office buildings with real trash.

Discussing the motivation for the study, the research team said that as the real world is complex, diverse, and changes over time, RL-enabled robots struggle to adapt and therefore they are not yet commonly used in everyday settings.

The ‘robot classrooms’ provide a large portion of the robots’ experience. The team said that while real-world office buildings can provide the most representative experience, the throughput in terms of data collection is limited — some days there will be a lot of trash to sort, and some days not so much.

At the end of the two years, the team gathered 540,000 trials in the classrooms and 32,500 trials from deployment. It found that overall system performance improved as more data was collected. The final system was evaluated in the classrooms for controlled comparisons with scenarios based on what the robots saw during deployment.

Alongside the 84 per cent accuracy of the final system, real-world testing showed that the system could reduce contamination by between 40 per cent and 50 per cent by weight. This was determined based on statistics from three real-world deployments between 2021 and 2022.

The team notes that the final RL policies do not succeed every time, and larger and more powerful models will be needed to improve performance and extend them to a broader range of tasks. Other sources of experience, including from other tasks, other robots, and even online videos may serve to further supplement the bootstrapping experience.