Affective Movement Recognition and Generation

We build models to teach machines to recognize and synthesize affect expression through human movement. In particular, we investigate question such as:

  • Can people perceive valence and arousal in motion capture without facial expressions?
    • Do people agree on what they see?
  • Can we train machine learning models using human ratings to classify movements of different affects?
  • Can we use deep learning for modelling and controlling the generation of motion capture-based movement information? In particular:
    • Can we integrate the control of multiple movement dimensions (function, planning, expression)?
    • Can the model generalize the movements to generate unseen combinations and qualities?
    • Can we use the model in interactive scenarios in which the responsiveness of the control is important?

Members:Omid AlemiWilliam Li, Philippe Pasquier

Data Collection

Since we want to be able to control movement across the three dimensions of function, planning, and expression, we created our own data set and captured the movements of three professional actors and dancers (two female, one male).

The performers carry out 9 different types of movements:

  • Walking in a figure eight pattern
  • Hugging the other performer
  • Standing idle
  • Free improvisation
  • Sitting down
  • Pointing while sitting
  • Walking with sharp turns
  • Improvisation with the other performer
  • Lying down.

Each movement type is performed with multiple expression, corresponding to the 9 different possible valence and arousal combinations as shown in Affect Representation. Each valence and arousal combination is repeated 4 times to capture the existing motor variabilities.

circumplex
The motion capture data can be found at:  http://moda.movingstories.ca/projects/29-affective-motion-graph

Affect Recognition

Valence 8-walk ratings                                                Arousal 8-walk ratings

Valence 8-walk ratings    Arousal 8-walk ratings

Valence sharp walk ratings                                      Arousal sharp walk ratings

Valence sharp walk ratings  Arousal sharp walk ratings

 

Inter-rater agreement, measured with Cronbach’s α:

cronbach

 

Categorical approach:

Support Vector Machines: 43% accuracy

k-Nearest Neighbour: 54% accuracy

Hidden Markov Models: 73% accuracy

 

Continuous approach:

Stepwise linear regression, measured with coefficient of determination (R2):

0.925 for valence, 0.985 for arousal

 

Affect-Expressive Movement Generation

We are developing a multidimensional agent movement controller, modelling the function, execution, affect-expression, personal style, and planning dimensions of movement. The system is designed and built iteratively in three stages.

AffectiveMGraph

In the first iteration, we only model the expressive dimension.

AffectGen1

In the second iteration, we add the planning dimension, as well as the choice of personal movement signature.

In the third iteration, we add the functional dimension, which allows us to transition from one action to another.

Papers:

  • O. Alemi, W. Li and P. Pasquier, “Affect-expressive movement generation with factored conditional Restricted Boltzmann Machines,” Affective Computing and Intelligent Interaction (ACII), 2015 International Conference on, Xi’an, 2015, pp. 442-448.
  • Li, W., and Pasquier, P.,  “Automatic Affect Classification of Human Motion Capture Sequences in the Valence-Arousal Model,” To Appear in Proceedings of the 3rd International Symposium On Movement & Computing, 2016, Thessaloniki, Greece.