A Commentary on Advancing Neuromorphic Embodied Intelligence: “Neurorobotic Reinforcement Learning for Domains with Parametrical Uncertainty”

Description

This paper explores the integration of neuromorphic hardware into robotic simulation frameworks for advancing ways to learn embodied behaviors by interaction. Embodiment seems to play an important role on giving rise to human-like knowledge representations and decision-making strategies. Therefore, facilitating the exploration of ways to connect efficient brain-inspired computation with embodied physics simulators for learning can pave the road towards more human-like intelligence and cognition. Neurorobotic simulations have multiple features that allow faster, safer, and more efficient learning, but such knowledge acquired via interactions with the simulated environment cannot always be directly applied to the real world. Therefore, exploring strategies for extending the behaviors learned in simulation to real domains also has a crucial role.

More specifically, the study states a use-case on robot control through reinforcement learning. The study focuses on implementing spiking reinforcement learning for a robotic arm engaged in a force-torque feedback-driven "peg-in-hole" task. This paper explores the integration of neuromorphic hardware, specifically Intel's Loihi chip, into the Neurorobotics Platform (NRP) simulation framework for advancing robot control through Reinforcement Learning (RL). The focus on a force-guided object insertion task aims to demonstrate the versatility of the proposed approach.

Neuromorphic devices follow an emergent brain-inspired sensing- and computing paradigm. This technology, known for its energy efficiency and adaptability, holds promise for real-time decision-making in edge robotics control tasks. However, optimal training of spiking neural networks for neuromorphic hardware remains an open challenge. The paper addresses this problem by maximizing the entropy and expected reward offline with surrogate gradients in a set of randomized domains with a spiking version of the soft actor critic algorithm. While the study is limited to model-free actor-critic architectures, it provides a foundation for exploring alternative training algorithms and optimization method.

The results highlight the success of the proposed approach in achieving a stable behavior with a 100% success rate in the robotic task. The discussion addresses limitations, suggesting potential enhancements, such as real-time learning during episodes and support for multiple parallel backends.

The study presents a comprehensive framework within the NRP for training neurorobots robustly to variable parameters and employing neuromorphic hardware for inference. Addressing the existence of known parametric uncertainties by randomizing the learning environment effectively reduces the sim-to-real gap during the transition from simulation to real-world applications. The paper presents an approach to randomize the training environments according to variability in the target domain, by actively sampling environments with different variability according to performance metrics.

In conclusion, this work contributes to the burgeoning field of neuromorphic reinforcement learning by successfully integrating the Loihi chip in the NRP and addressing key challenges in neurorobotic control tasks. The findings pave the way for further exploration of advanced learning algorithms and optimization techniques, emphasizing the pivotal role of the NRP and simulation with integrated neuromorphic computing in shaping the future of neurorobotics research. Furthermore, exploring more complex algorithms, knowledge structures, and models with higher expressive capacity might be a way to give rise to features observed in higher cognition with knowledge grounded to the real world; such cognitive features could be easily deployed to edge applications by means of neuromorphic hardware.