BehaviorKit: A Multi-modal Real-Time Behavior Analysis Library for Robots

We present BehaviorKit, an open-source plug-and-play module that enables the addition of real-time behavior analysis using deep learning to any robot with internet access. The library bundles together gaze tracking, real-time transcription, textual sentiment analysis, facial emotion (valence and arousal) estimation, and face and pose landmark localization. The library is designed to be run on a GPU-powered computer, either on the robot or externally, to process the incoming visual and auditory inputs that the robot receives. The library connects via WebSocket to the robot, which receives the processed outputs from all of the models. The WebSocket client can easily connect to a ROS-powered robot, or a custom client can be written to adapt to any robot; alternatively, the library can also be run between two laptop computers. A small dataset is provided to test the framework. By packaging together and optimizing many commonly used models, we hope to enable easier access to high-performing behavior models for the HRI community. The code is available at https://github.com/IUB-RHouse/BehaviorKit.