A sneak peek into the factory of the future has a sweet spot—so to speak. Several conveyor belts transport chocolate bars that are part of a demonstrator machine that shows how artificial intelligence can be used for motion control. What remains to be done in a real factory is to automatically pack the chocolate bars. In this Intelligent Infeed Demonstrator machine from Siemens Digital Industries, the chocolate bars must be placed in evenly spaced slots on the outfeed belt.

“The bars are placed on the inlet belt at random intervals,” says Martin Bischoff, expert in virtual mechatronics at technology, the research division at Siemens. “The system controller achieves this by altering the speeds of the conveyor belts. A line of three conveyor belts can be accelerated or slowed down to ensure the chocolate is positioned correctly on the outlet belt. The development of an optimized control algorithm for this application is a tricky programming task—if you don’t believe it: just try it yourself. Via Reinforcement Learning, we have trained an artificial intelligence controller to realize this task.”

Reinforcement learning, according to Siemens Digital Industries, is an artificial intelligence method that works in much the same way as most people learn to ride a bicycle—by trial and error, without any knowledge of the basic physics: The novice cyclist experiences whether his or her own technique is good directly during the riding tests and thus gradually becomes better and better.

“This is exactly how reinforcement learning works,” explains Michel Tokic, a fellow expert at technology and a lecturer in applied reinforcement learning at Munich’s Ludwig Maximilian University. “The AI is given a target specification, such as ‘the candy bars may only be placed in the target fields, and the system should work as quickly as possible in the process.’ The AI then makes—initially completely random—control attempts on the simulation model and receives feedback, triggered by light barrier signals, on how good each attempt was. With this feedback, a goal-directed control algorithm emerges after many automated training cycles.”

Errors in a plant control system can have expensive or dangerous consequences. For this reason, controls are developed and tested on digital twins of the plants without risk as standard (Siemens Virtual Commissioning). The digital twin of the plant can also be used to train the AI.

“After about 72 hours of training with the digital twin (on a standard computer; about 24 hours on computer clusters in the cloud), the AI is ready to control the real machine. That’s definitely much faster than humans developing these control algorithms,” Bischoff says.

Using reinforcement learning, the AI has developed a solution strategy in which all the chocolate bars on the front conveyor belts are transported as quickly as possible and the exact speed is only controlled on the last conveyor belt.

This is interestingly quite different from that of a conventional control system.

The researchers led by Bischoff were able to make their approach even more practical by compressing and compiling the trained control models in such a way that they run cycle-synchronously on the Siemens Simatic controllers in real time.

There is great potential in the methodology of letting AI learn complex control tasks independently on the digital twin.

“Under the name AI Motion Trainer, this method is now helping several co-creation partners to develop application-specific optimized controls in a much shorter time. Production machines are now no longer limited to tasks for which a PLC control program has already been developed but can realize all tasks that can be learned by AI,” says Thomas Menzel, digital machines and innovation, production machines at Siemens.

sw.siemens.com