Source: Fraunhofer IPA/Foto: Rainer Bez
Automation of automation and bin picking
An example of pioneering technology is a separation process using robots known as bin picking. Its development exemplifies the path from research result to a purchasable product of Liebherr-Verzahntechnik GmbH. The technology is still being further developed today under the motto “Automation for Automation”.
A robot moves its arm over a lattice box containing the bulk material, pauses for barely a few seconds, reaches in, lifts out a part, puts it down, and immediately dips back again into the mass of different workpieces, separating cast, stamped or forged parts from the box and placing them in the correct position and facing the right way on a conveyor belt for further processing. This continues until the box is empty. Decades of research into bin picking systems has gone into this application, which now seems so easy and effortless.
Fraunhofer IPA has been developing technologies for this purpose for many years. In 2007, for example, the institute applied for a patent for its object recognition method. Just a year later, Fraunhofer IPA presented an application at Automatica that had already been implemented. While bin picking cells are now common exhibits and are also finding their way into production, the IPA robot cell was still a rarity then. At this time, the bin picking research team also first came into contact with the company Liebherr-Verzahntechnik GmbH.
When Werner Kraus, now Head of the Department of Robots and Assistive Systems, first encountered bin picking in 2009, the economic situation in Germany and the world looked bleak. The financial crisis had made a deep mark and the ifo business climate index had reached a low point. At that time, bin picking technologies, especially the efficient object recognition processes, were still in their infancy. Existing solutions such as vibrating bowl feeders were not flexible enough.
It meant that an employee was often responsible for separating tasks. This took place directly on the production line, and was a monotonous, dirty and physically demanding task because of the heavy weights of the workpieces and the non-ergonomic posture involved. A typical “3D task” can be neatly summarised by the three Ds: dull, dirty and dangerous. This burden for the employees and a regular amortisation of less than two years across a three-shift operation is all the motivation that both employees and management need to implement robot-based bin-picking solutions.
“The years of cooperation between Liebherr and Fraunhofer IPA have paid off – and that excites me”
Intensive collaboration leads to first industrial implementation
»Weil es eine wirtschaftlich schwierige Zeit war, plante Liebherr, sich strategisch neu auszurichten«, erklärt Kraus. »Sie wünschten, nicht mehr ›nur‹ Maschinenhersteller zu sein, sondern wollten sich zum Gesamtlösungsanbieter wandeln, nicht mehr ›nur‹ Verzahnmaschinen verkaufen, sondern eine gesamte Fertigungszelle.« Darin sollte ein Roboter die Bauteile der Maschine anreichen und sie nach der Bearbeitung dem nächsten Schritt im Prozess zuführen. »Die jahrelange Zusammenarbeit zwischen Liebherr und IPA hat sich ausgezahlt, und das begeistert mich«, blickt Kraus zurück. Gemeinsam gingen das IPA und Liebherr durch schwierigere und erfolgreichere Zeiten und erarbeiteten Meilensteine: 2010 gab es den ersten »Proof of Concept« einer Roboterzelle, die die Bauteile handhaben konnte. Ein Jahr später folgte die erste Realisierung für semichaotisch, 2013 für vollständig chaotisch gelagerte Bauteile.
Scaling challenge
At the time, everyone involved was euphoric, thinking that the technology transfer to industry had succeeded. The “handling” award initially confirmed this. However, it is not enough to rest on your laurels once the task at hand is completed. Kraus explains: “When it came to scaling the solution we had developed, we still had to overcome a few hurdles. We wanted to use the bin picking system in another project, where it would place connecting rods on the workpiece transporter for subsequent machining. But the speed was not right. We kept at it, fought for acceptance of the cell and ultimately succeeded”. The IPA researchers were able to use the knowledge gained for the Liebherr application and improve it further. They also developed a graphic user interface to make the software more user-friendly.
Best of Industry Award
In 2017, Liebherr won the “Best of Industry” award for its overall solution based on Fraunhofer IPA’s bin picking technology. Last year, the technology in the Fraunhofer IPA product portfolio was also made available to other robot integrators in the form of the “LHRobotics.Vision” technology package. Both Fraunhofer IPA and Liebherr benefit equally from this knowledge transfer. “Developing customer-specific technologies, constantly improving and following up, systematically overcoming hurdles and working together to reach the goal”. This is how Kraus formulates his understanding of innovation.
Bin picking application falls short of expectations
Liebherr took the right step in times of crisis and set the course for the future. For some years, the IPA has also been working on the bin picking technology of the future: greater autonomy, improved cycle times, higher degree of robustness achieved by machine learning methods. Critically speaking, the spread of bin picking application is still far below expectations. Every year, more than 200,000 new robots are installed worldwide for workpiece handling. Of these, only a fraction in the per thousand range perform bin picking tasks. Conversely, the majority of robots grip blindly or at most, use 2D image processing at for semi-sorted deliveries such as depalletizing.
At the same time, there are numerous production stages that would be suitable for the application of bin picking, not to mention the potential applications in logistics. The user problem of the “3D tasks”, which has existed for decades, still looms very large and has not yet been resolved, despite good economic efficiency.
One reason for this is that cells with the bin picking facility are the first link in an interconnected production or assembly line. The line balancing of an interlinked line such as this is based on the fact that each station delivers a guaranteed output. There are two inherent uncertainties in a “typical” bin picking system: “At present, there is still no guarantee that a robot can remove all the parts from the bin. This means that the last remaining parts must then be separated by hand”, Kraus explains. In addition, the cycle time increases significantly as the bin empties. “The fluctuations in the cycle time can be compensated either via worst-case design or buffers. The entire line could possibly be adapted for high cycle times or bin picking could start earlier in order to gain a head start and to prevent downtimes”, Kraus adds. “These uncertainties currently prevent the widespread use of bin picking in practice”.
Vision Zero: complete bin emptying with constant cycle time
Fraunhofer IPA is tackling these central problems with its “Vision Zero”, which aims to completely empty a bin at the same cycle time. There are several reasons why this vision is not yet a reality: “It can be due to the image processing, unsuitable sensors or grippers, combined with objects that are difficult to grasp or other customer-specific challenges”, explains Kraus. “Overall, we have identified eight typical end-user problems that limit the use of bin picking solutions. We are developing solution approaches for these and in some cases, resorting to new technologies such as machine learning”.
Practical experience shows, for example, that the degree of emptying and the cycle times currently depends greatly on the expertise of the operator. IPA researchers are therefore working on algorithms for the self-configuration of a bin picking system, which will automatically set the parameters at the level that might be expected of an expert. Among other aspects, this concerns the hand-eye-calibration, defining algorithm parameters and determining gripping points.
Key component: the gripper
The gripper design is the basis for complete bin emptying. On the one hand, it determines the number of potential workpiece gripping points and therefore the flexibility of the application as well; on the other hand, its overall configuration contributes to the success – or otherwise – of the process, for example, through susceptibility to collisions. In practice, this important element is often selected according to ‘gut feeling’ and the performance only becomes apparent when the robot system is commissioned. “In order to arrive at a well-founded selection, we use a simulated environment to verify the gripper design”, Kraus emphasizes. In this environment, 3D point clouds of virtually filled boxes are generated and used to statistically evaluate aspects such as accessibility of the gripper to the workpiece avoiding collisions. Without hardware investment and tests on real production lines, which would have to come to a standstill for this, a variety of gripper designs can be examined and their performance demonstrated. In consultation with the IPA, Liebherr is further developing a simulation tool in line with industry requirements to enable simpler and more productive use.
New level achieved by machine learning
Another reason why cycle times increase when bins are almost empty or the robot cannot empty the bins completely is the challenge of object recognition to correctly identify the parts at the bottom of the bin. Kraus describes the difficulties with sheet metal parts, for example: “The parts are shiny, thin and difficult to recognize because they merge into bottom of the bin in the 3D point cloud”. To overcome this challenge and empty the bin, the researchers use a deep neural network for segmentation. To illustrate this, the neural network isolates the areas of the point cloud containing the workpieces, i.e. segments them, and passes these segments on for further image processing. Pre-processing ensures that even the last remaining parts are detected quickly and reliably.
Automatisierung der Automatisierung
Fraunhofer IPA puts this latest developments under the slogan “Automation of Automation”. This is intended to convey that in future the planning process for automation solutions should run on a more autonomous basis, for example, or that production means such as robots should be automatically reconfigured when a new product variant is being manufactured. The qualities of the IPA solutions for bin picking described above demonstrate how this kind of “Automation of Automation” can be made possible.
Businesses to test bin pickingsoftware
At the moment, interested companies can apply to become “first adopters” for these technologies that are developed as part of the initiative. “In this way, companies can be the first parties to test new technologies and help us to adapt the bin picking software in addition to six other industrial and service robot technologies in the field of cognitive robotics to market requirements and interests with their feedback”, Kraus explains.
Further information
- Deep Picking – KI-basierte Kommissionierung von Objekten
- Projektwebsite Deep Picking
- Produktblatt Greifplanung
Contact
Dr. Werner Kraus
Head of the Robots and Assistive Systems Department
Phone: +49 711 970-1049