The new hybrid device combines decoupled sensors with a flexible wireless powering and transmitting module for emotion recognition, according to a research team headed by Penn State scientists.
The wearable patch can simultaneously and accurately track multiple emotional signals. Image credit: Yangbo Yuan / Penn State.
“This is a new and improved way to understand our emotions by looking at multiple body signals at once,” said Penn State Dr. Huanyu ‘Larry’ Cheng.
“Relying only on facial expressions to understand emotions can be misleading.”
“People often don’t visibly show how they truly feel, so that’s why we’re combining facial expression analysis with other important physiological signals, which will ultimately lead to better mental health monitoring and support.”
The sticker-like patch tracks a range of physiological responses, such as skin temperature, humidity, heart rate and blood oxygen levels, that are associated with emotional states.
Most importantly, the device’s sensors are designed to work independently, minimizing any interference between the different measurements.
The device combines its analysis of the physiological signals with facial expression data to better distinguish between genuine emotions and acted ones.
It then wirelessly transmits the real-time measured data to mobile devices and the cloud, where clinicians could potentially use it to better assess patients virtually.
The device does not record personal information, only signals, meaning personal privacy is protected through the device’s design.
“This technology has the potential to help people who are struggling with their mental health, but maybe aren’t being fully honest with others or even themselves about how much they are struggling,” said Yangbo Yuan, a doctoral student at Penn State.
“The collected data could also help bridge cultural or social gaps, which can manifest as a person appearing more stoic or expressive to their health care providers.”
“By keeping track of these signals, it could be possible to detect problems like anxiety or depression earlier in its progression.”
The researchers built the stretchy device by folding together thin layers of flexible metals like platinum and gold and cutting them into wave-like shapes that maintain sensitivity even when pulled or twisted.
They also used layers of materials that change the flow of electrical current with temperature and built in hollow tubes made of carbon atoms, which absorb water and track humidity levels.
The multiple sensors were designed and placed on the device in such a way that they would not interfere with each other’s measurements.
For example, the scientists put a rigid layer under the temperature and humidity sensors to protect them from the stretching that the facial expression sensors would experience.
They also used a waterproof layer to protect the temperature and strain sensors from humidity.
“We’ve engineered this device to measure these different signals independently, without them interfering with each other, providing a much clearer and more accurate picture of what’s happening beneath the surface,” said Dr. Libo Gao, a researcher at Xiamen University.
The authors then trained an artificial intelligence (AI) model to read and understand signs of performed and real human emotion.
They recruited eight people, a common sample size for pilot studies, to perform six common facial expressions: happiness, surprise, fear, sadness, anger and disgust.
The participants displayed each emotional expression 100 times while the device tracked their movement.
The researchers then fed this data to an AI model, training it to correlate specific facial movements with different emotions.
They then recruited an additional three participants to further evaluate the model’s abilities.
It classified performed facial expressions with 96.28% accuracy.
When it came to tracking real emotions, the scientists tested how well the device tracked the psychological responses of the same participants as they watched video clips designed to elicit emotions.
The device correctly identified emotions with 88.83% accuracy, with the sensors confirming that the psychological responses were consistent with known links between emotions and psychological reactions, such as increases in skin temperature and heart rate during surprise and anger.
“The ability to wirelessly transmit the data means that health care professionals could potentially monitor individuals remotely and provide timely emotional support through telemedicine,” Dr. Cheng said.
“This sensor can serve a vital function in bridging gaps in access to care.”
“Given the rising stress levels in modern society, the ability to monitor emotions can provide early indicators of debilitating conditions and allow for proactive support.”
The new device is described in a paper published in the journal Nano Letters.
_____
Yangbo Yuan et al. 2025. Stretchable, Rechargeable, Multimodal Hybrid Electronics for Decoupled Sensing toward Emotion Detection. Nano Lett 25 (13): 5220-5230; doi: 10.1021/acs.nanolett.4c06392
English