OpenSense

OpenSense is a platform for real-time multimodal acquisition and recognition of social signals. OpenSense enables precisely synchronized and coordinated acquisition and processing of human behavioral signals. Powered by the Microsoft’s Platform for Situated Intelligence, OpenSense supports a range of sensor devices and machine learning tools and encourages developers to add new components to the system through straightforward mechanisms for component integration. This platform also offers an intuitive graphical user interface to build application pipelines from existing components. OpenSense is freely available for academic research.

Components

OpenSense currently includes the following components.

• Facial expression analysis with OpenFace
• Body tracking with OpenPose and Azure Kinect
• Acoustic analysis with OpenSMILE
• Speech recognition with Azure and Google Cloud
• Data acqusition from BIOPAC (requires additional license) and Shimmer EDA+
• Head gesture recognition (native; requires OpenFace)

Access

OpenSense is available on GitHub and is free to use for academic research.

Windows binary files are available here.

You can cite the following paper, if you use our software in your research.

@inproceedings{StefanovICMI2020,
  author = {Stefanov, Kalin and Huang, Baiyu and Li, Zongjian and Soleymani, Mohammad},
  title = {OpenSense: A Platform for Multimodal Data Acquisition and Behavior Perception},
  booktitle = {Proceedings of the 2020 International Conference on Multimodal Interaction},
  publisher = {Association for Computing Machinery},
  pages = {660–664},
  numpages = {5},
  year = {2020}
}

Support

OpenSense development has been supported by multiple grant by the US Army Research Office.