Advances in Robots to See & Interpret within Warehouse Environments

Q&A with Ethan Lauer, Senior Software Engineer | Boston Dynamics

Tell us about yourself and your role with Boston Dynamics.

I’m a Robotics Engineer on the World Modeling team for the Stretch robot at Boston Dynamics. I completed my Masters in Robotics Engineering at Worcester Polytechnic Institute and spent a couple of years in classical industrial automation before joining Boston Dynamics as a Technical Support Engineer. After a few years of diagnosing and repairing customer robots and seeing anything and everything Stretch may encounter, I joined the World Modeling team to help make Stretch adapt to unique environments. This typically includes creating new software features to handle environmental changes, optimizing the existing perception systems, and testing the system’s robustness.

Describe some of the challenges mobile robots face in warehouse environments.

Warehouses are dynamic environments where the freight, trucks, and other surroundings are highly variable and often changing. Robots have to be trained on how to perceive and pick diverse freight, without colliding with a trailer wall or ceiling. The trailer opening might be higher or lower than the last one the robot operated in, or the opening might have a curtain hanging there that partially obscures the view. No two conveyors are alike either, varying in position, responsiveness, and appearance.

Robots also have to be built to withstand the extreme temperatures of warehouses. They can be freezing in the winter or sweltering in triple-digit temperatures amplified by the sun beating down on a metal container.

And at Boston Dynamics, safety is our top concern. Stretch is designed with redundant safety measures, including lidars in its base to detect surroundings and operating behind a virtual safety fence enabled by lidars installed at the base of the conveyor.

Finally, the way that freight is stacked inside a trailer presents challenges for robotics engineers to solve. This can include a large number of SKUs, cases tossed everywhere instead of being stacked in an orderly way, cases wedged in tight, or having cases side by side at very different depths.

How does Stretch’s 360-degree sensor provide a competitive advantage over traditional warehouse automation?

Traditional automation involved having one fixed camera pointed in one direction, all the time, or something like a six-sided scanner in a tunnel that is fixed in place. Stretch, by contrast, is driven into place and then set to unload boxes autonomously. This involves the robot working its way further into the container as it progresses, and the robot must constantly perceive its surroundings and its position relative to those walls, ceiling, conveyor, and boxes. The Stretch system includes a lidar-enabled safety zone without fencing where, if a person crosses, the robot ceases working in an instant so no one gets too close. This also reduces the infrastructure footprint at customer sites that is typically required for classic warehouse automation. Looking forward, we are developing Stretch to tackle other tasks in the warehouse including case picking. That will involve the robot roaming warehouse aisles to pick and build a palletized order, and 360° perception will be a must.

How does the robot’s sensing system differentiate between permanent infrastructure and temporary obstacles like fallen boxes?

Stretch runs various detectors for different types of objects and models permanent infrastructure differently from temporary obstacles. While Stretch is still constantly detecting its environment, permanent infrastructure is detected at a lower frequency compared to objects that are constantly moving. The detectors for temporary obstacles need to smoothly adapt to frequent perceived changes. Whereas when permanent infrastructure is detected to have moved drastically, Stretch is more wary of these unexpected changes and needs to run extra checks before proceeding.

In what ways does real-time environmental data improve the precision of Stretch’s maneuvers?

When it comes to trailer unloading, the environment is constantly changing because Stretch is moving all those boxes. So that real-time data equips Stretch with the information about where the remaining boxes are, which to pick next, how to approach and pick it, and where the conveyor is now so the robot can plan the placement.

If the robot is multipicking, or picking two, three, or four boxes at a time, that data helps the robot decide whether to multipick, which boxes to grasp, and how to approach those boxes and place them.

The robot also picks in different ways. Having that environmental data helps Stretch decide whether to pick from the front, the top, or the side of the box.

And, of course, this information helps the robot avoid colliding with the ceiling, walls, or conveyor.

What software hurdles exist in processing 360-degree sensor data in real time?

Computation is the largest hurdle. 360-degree sensor data that is granular enough to detect small obstacles in the environment is very large. For Stretch to have a real time reaction, that data has to be processed at a very high rate. This includes reading the raw data, processing it through the sensor drivers, and then analyzing it by the perception system. This all needs to be completed before Stretch can decide what to do with the detection results. Not only that, this processing is often done simultaneously with other decisions such as planning its arm trajectory. Optimizing data management and processing is just one step of getting over that hurdle.

How does Stretch’s level of environmental awareness facilitate safer and more efficient human-robot collaboration in the future?

A 3,000 pound moving robot with a powerful arm needs to be safe in the workplace, above all else. Stretch’s perception of everything around it at all times is key to that safety. On the efficiency side, Stretch is making decisions in real time on how best to pick a box, since saving mere seconds on a pick adds up in the long run when unloading thousands of boxes in a container. Its long battery life allows the robot to work continuously, without breaks, for more than one shift.

Stretch also offers real-time visible data on the progress of its unload workflow. After Stretch finishes unloading a truck, users can dive deeper into the data with our Orbit Performance Dashboards, which show metrics like the total number of cases unloaded or total multipicked. Orbit also captures images inside the trailer from Stretch at many points in the unload workflow, so users can look at the photos and analyze the exact trailer conditions and how they impacted the rate of the robot’s work.

Ethan Lauer is a Senior Software Engineer at Boston Dynamics where he helps robots understand the world around them. As part of the World Modeling team, Ethan develops perception systems for the Stretch robot, specializing in how the machine interprets unknown objects and adapts to new environments. His work enables Stretch’s 360-degree sensors to differentiate between environmental structures, temporary obstacles, and objects to manipulate. Ultimately, this allows Stretch to operate safely and seamlessly alongside humans in dynamic environments. Ethan brings a uniquely practical perspective to his software, having previously spent two years as a Technical Support Engineer at Boston Dynamics troubleshooting and repairing customer robots in the field. Before that, the Worcester Polytechnic Institute graduate spent two years working in industrial automation. Outside of the lab, you can usually find him running obstacle course races or playing rock music with his band.

The content & opinions in this article are the author’s and do not necessarily represent the views of RoboticsTomorrow

Featured Product

Harmonic Drive – Ultra Compact Products – Miniature Gearheads

Harmonic Drive is your one-stop headquarters for miniature speed reducer products designed for applications requiring smaller, space-saving components. These devices decrease a motor’s rotational speed while increasing output torque and generating zero backlash. Options include mini compact gearheads and miniature actuators. Our mini gearheads, also referred to as miniature gearboxes, are the ideal choice in confined areas where maximum precision is essential. Key features include a compact four-point contact ball bearing mounted to the main shaft, outline dimensions up to 13 mm and precision-matched motors and encoders. Our compact gearboxes come in five sizes, four ratios and flange or shaft outputs.

Advances in Robots to See & Interpret within Warehouse Environments