Technology and methods

The BamLiD multi-sensory technology and its methodology at a glance

Extracting interpretable information from diverse data is a fundamental task in the Living Lab. What kind of behaviour is a person currently performing, what behavioural intention is being realised, and what emotional state is the person in? To ascertain this, hybrid artificial intelligence approaches combining machine learning and knowledge-based inference are used.

It is essential that multimodal derivation is possible while the person is moving freely in the BamLiD so that everyday activities and actions can be carried out unhindered. This multimodal derivation takes place via recordings:

A total of 12 cameras can cover every corner of the Living Lab.

1. Facial expressions

BamLiD is equipped with 12 (mobile) cameras so that lateral and frontal images of the face are assured at all times.
The cameras record synced videos (VideoSyncPro by Mangold International).
Facial reactions are manually analysed offline using the Facial Action Coding System.
Automatic facial expression recognition systems (Face Reader, Pain Face Reader, Open Face) are also used to analyse facial expression reactions.
Parameters: e.g. type of action unit (AU) shown, AU frequency, AU intensity, AU on and offset, AU duration

Additional microphones provide optimal sound quality for phonetic analyses.

2. Voice

In BamLiD, 5 gooseneck microphones ensure optimal recording quality from any position in the room.
Audio is recorded in sync with the video recordings (VideoSyncPro by Mangold International).
The phonetic analyses of the audio tracks are carried out offline with the ProsodyPro program.
Parameters: e.g. average fundamental frequency of phonation, frequency variation of phonation, average intensity and roughness

3. Gait analyses

BamLiD is fitted with an intelligent sensor floor (Smart Floor) (Fraunhofer IGD, Darmstadt). Sensors installed in the floor enable the location of a person to be determined with a high degree of accuracy based on passive electric field measurements.
Gait analyses are performed offline.
Parameters: e.g. speed, number of steps, step length, variance of speed/step length, asymmetry between left and right

The Smart Floor tracks movements which can be reproduced on a monitor.

The sensors installed in the floor in the Living Lab are not visible to persons in the lab.

4. Mobile biosignal recorder

BamLiD has a variety of mobile sensors (BioNomadix Wireless Wearable Physiology by BIOPAC, Biosignal Recorder NeXus-10 MKII by Hasomed/MindMedia) for recording blood pressure, heart rate, skin conductivity and actigraphy (motor activation).
These analyses are also performed offline.
Parameters: e.g. heart rate, heart rate variability, sympathetic arousal, agitation/restlessness

Attaching electrodes to the index and middle fingers to measure skin conductivity.

Biosignal recorder for capturing a wide range of physiological parameters.

In the control room, video and sound are recorded and prepared for further data processing.

5. Monitoring, trigger and storage technology in the control room

All data streams converge in the control room.
Trigger circuits ensure synchronous evaluation of the multi-sensor data.
The multi-sensor data is collected in the control room and camera settings are configured.

There are 42 daylight luminaires in the laboratory that allow the brightness and colour temperature to be adjusted to any level.

6. Lighting system

BamLiD is fitted with 42 luminaires that can be controlled individually.
This allows targeted control of light brightness and colour temperature (up to 3,000 lux).

7. Controlling ambient conditions

Temperature: BamLiD is air-conditioned so that temperatures can be monitored and set to the desired level.
Noise/noise level: BamLiD is equipped with loudspeakers so that any background noise can be played.