Tracking gestures of hands and fingers: A tool for pianists

Tracking gestures of hands and fingers:
A tool for pianists’ identification
March 26th, 2014
Paper submitted for MUMT 621
Information Music Retrieval
Professor Ichiro Fujinaga
Catherine Massie-Laberge
Student number: 260320177
Tracking gestures of hands and fingers: A tool for pianists’ identification
1. Introduction
Famous pianists, such as Vladimir Horowitz or Arthur Rubinstein, mark us deeply with
brilliant performances. However, the reasons why it is possible to distinguish performers’
personal style and expression are little understood (Dalla Bella and Palmer 2006). Research have
not yet pinpoint precisely which features in performance are the most distinguishing ones.
However, it was shown that movement dynamics in music performance, such as pianists’
fingers movement, can contain sufficient evidence to identify performers, and reveal essential
information about note preparation, accent, expression, and personality (MacRitchie and Bailey
2013). Before discussing data acquisition processes (i.e., motion capture set up, marker
placement) and the gesture recognition and classification methods, essential for the study on
pianists’ finger and hand gestures recognition (Sotirios and Georgios 2008), this summary will
first examine how pianists segment music in their head; the influence of finger movements on
the resulting sound; and the cognitive and biological constraints.
2. Ancillary gestures
Wanderley (1999) found that each time a player reproduced a musical sequence, the same
ancillary gestures (i.e., gestures that are not essential for sound production, but may have a
sound-facilitating function) were found in their performance. This means that once a solid
mental representation of a piece is accomplished, similar execution time profiles between
performances of the same piece in different instrumentalists will be found. Without asserting
that pianists’ finger movements are considered as ancillary gestures, pianists’ finger motion
measurements were still shown to influence the characteristics of the resulting sound, such as
intensity of tones and timing (Dalla Bella & Palmer 2011; Goebl & Palmer 2008), suggesting
that fingers’ movements typify pianists’ originality and have an impact on their control of force
and tempo.
3. Chunking and segmenting
“[…] Perceived [music related] action and sound are broken down into a series of chunks in
people’s minds when they perceive or imagine music” (Godøy et al 2010, p. 690). The term
chunking refers to the synthesis and transformation of smaller units that leads to unification
(Caramiaux et al 2012). When we listen to music, we tend to perceive units, such as phrases,
motives, measures, reminiscent of how we perceive these units when hearing speech (Godøy et
al 2010). Gestures that create sequence of elements can be influenced by the gestures that
generate surrounding elements (Loehr and Palmer 2007). For instance, pianists’ finger
movements rapidly change in velocity and acceleration one to three events before a keystroke.
This suggests that such movements reveal planning of upcoming events. There exist different
motion segmentation methods, such as Pincipal Component Analysis (PCA) and Gaussian
Mixture Model (GMM) that are based on the information available in the motion sequence
Caramiaux et al 2012). It is also possible to use velocity properties of the joint angles to perform
segmentation. The cognitive constraint referring to chunking has been studied in various motor
tasks in which participants realize movements under speeded response conditions. Loehr and
Palmer (2007) showed that chunking affected the timing and motion of pianists’ tapping, which
required temporal precision.
2 Tracking gestures of hands and fingers: A tool for pianists’ identification
4. Cognitive and biological constraints
Cognitive constraints affect the fluidity of performance, and individual differences in shortterm memory restrict how many and which pitch events can be anticipated prior to the actual
production of note. Anticipatory movements, which are measured by the divergence of finger
movement trajectories, may occur as much as 500ms in advance of the last note shared by the
melody. Moreover, biomechanical factors, such as the degree of independence between fingers,
restrain the range of possible movements. According to these constraints, it is propose that to
achieve spatial and temporal accuracy, pianists might use diverse movement strategies that in
turn yield different sound outcomes. Dalla Bella and Palmer (2011) investigated the uniqueness
of a particular pianist’s style and the relationships with idiosyncratic kinematic properties of
finger movement.
4.1 Control of the finger movement
The anatomical and physiological foundations of a finger-stroke required for an efficient
pianists’ finger technique, or the patterns of finger movement they employed to generate sound,
are described as very complex (Goebl and Palmer 2009; Watson 2009). Each joint angle
between adjacent segments of the fingers, the hand, and the forearm, contributes to an optimal
configuration of an independent finger technique (see figure 1). The fingertips are flexed by the
deep flexor muscles in the forearm; the middle phalanx is flexed by the superficial flexor
muscles. The proximal phalanx of the fingers is moved by the intrinsic muscles of the hand,
which correspond to lumbrical and interosseus muscles. The interosseus muscles move the
fingers sideways and the lumbrical muscles flex the metacarpal-phalangeal joint (MCP).
Figure 1. Control of the finger movements and the role of the different structures (Watson
2009).
3 Tracking gestures of hands and fingers: A tool for pianists’ identification
5. Data acquisition
If one desires to capture accurately these intricate and fine movements produced the by
movements of the fingers, appropriate systems need to be employed. Image subtraction
combines with motion capture techniques have revealed that motion cues such as quantity
(amount of detected motion), velocity, fluidity (staccato/legato articulation), and amplitude of
movements played a primordial role in conveying and distinguishing between different affects
(Camurri et al., 2004; Wanderley et al., 2005). With three-dimensional optical motion capture
systems, and passive markers, pianists’ finger and hand movements can be accurately captured.
In previous studies, a Vicon motion capture system was used, with six infrared cameras that
tracked the movements of 4-mm reflective markers at a sampling rate of 250 frames/s (see
figure 2) (Goebl and Palmer 2013; Palmer and Dalla Bella 2004, 2006, 2011). Fifteen markers
were placed on the keys to monitored the motion of the keys. Twenty-five markers were placed
on pianists’ hand, finger joints, and wrist to track the movements (see figure 3). Kinematic
landmarks were extracted from each finger movement toward the keys in the vertical dimension
(Goebl and Palmer 2013). The maximum finger height (mxH, refers to the beginning of the
finger movement) and the key-bottom landmark (KB, refers to the moment when the finger is
stopped by the keybed) were computed.
Figure 2. Motion capture setup and
marker placement on the keyboard. Fifteen
markers were placed on the piano keyboard
(Goebl and Palmer 2013).
Figure 3. Hand markers and joint angles.
Twenty-five markers were positioned on the
hand, fingers, and forearm (Goebl and
Palmer 2013).
5.1 Joint analysis
Joint angles were calculated for all adjacent phalanges of the fingers, the wrist, and the
forearm. The wrist angle was computed (between forearm and the metacarpals), as well as the
wrist rotation (the degree of supination and pronation in relation to the horizontal plane). For
4 Tracking gestures of hands and fingers: A tool for pianists’ identification
each finger, three joint angles were calculated: metacarpophalangeal joint (MCP), proximal interphalangeal joint (PIP), and distal inter-phalangeal joint (DIP).
5.2 Timing analysis
In Dalla Bella and Palmer’s (2011) study, each interonset interval (IOI) was computed from
the keyboard MIDI data. The first and last tones in the performances were ignored from MIDI
and movement analyses because they did not have comparable beginnings and endings.
5.3 Movement analysis
Movement data in the vertical plane from the fingertip markers are analyzed as regard to the
piano key movements to assess how loud keys are struck. The discrete data obtained from finger
positions were converted into continuous functions with Functional Data Analysis (FDA).
Movement velocity and acceleration are measured as continuous functions starting from discrete
data. Position values are smoothed with order-6 splines as basis functions. Faster performances
are associated with greater finger heights above the keys, which is also related to larger key
velocity. Keypresses were aligned for all performances in terms of the key bottom positions. The
region between two successive vertical lines is called an event region. The finger movement
amplitude refers to the difference between the maximum finger height before keypress and the
minimum finger height at keypress. Anticipation time before keypress refers to the difference
between the time of the maximum finger height within the two event regions right before the
keypress (see figure 4).
Figure 4. Movement of the thumb during one pianist’s performance at slow tempo and the event
regions (Dalla Bella 2011).
6. Is it possible to classify pianists based on finger trajectories?
To verify whether the movement trajectories for attacks, keypresses, and at-rest positions
contained sufficient information to discriminate between performers on the basis of individual
5 Tracking gestures of hands and fingers: A tool for pianists’ identification
keystrokes, Dalla Bella and Palmer (2006, 2011) tested a neural network classifier on different
time segments of the finger trajectories. The part of the velocity-acceleration trajectories for each
finger that was significantly different across pianists was identified by functional ANOVA. The
significant portion of the velocity-acceleration curves was entered in a Principal Components
Analysis to decrease the amount of information. Five components were used to train a twohidden-layer network. Pianists’ classification was performed separately for each finger. The
results were then combined to obtain the final classification rate. It was trained on 450 samples.
The pianists were accurately classified in 87% of attack trajectories and in 84% of keypress
trajectories. Certain pianists were identified correctly more often than others. Experience and
training can be important factors in correctly identifying pianists. High-skilled pianists also use
more consistent finger movements.
7. How can we define a musical signature?
Pianists’ movement kinematics can be considered as an indicator of personal identify (Dalla
Bella and Palmer 2011; Goebl and Palmer 2009; Palmer and Dalla Bella 2004). Finger
movements’ observations of piano performance have shown that it is possible to discriminate
between different performers. Moreover, studies on pianists’ touch at various tempi have
revealed that the motion of the fingers effectuated toward the key throughout note production
enable performers to control timing accuracy (Goebl and Palmer 2009a,b) and maintain a high
level of independence between each finger. The variability of finger movements in the three
motion planes shown that pianists executed movements with greater amplitude at faster tempi,
before striking the keys. These results indicate that increased movement amplitudes at fast tempi
may help with speed or accuracy tradeoffs, and provide more tactile and haptic feedback. In
other words, the larger the change in acceleration at finger-key contact, which coincides with
greater tactile (sensation of pressure, local features such as curvature, orientation, texture,
thermal properties, softness, vibration, etc.) and kinesthetic feedback (awareness of one's body
state, including position, velocity and forces supplied by the muscles through a variety of
receptors located in the skin, joints, muscles, and tendons), the more precise the subsequent
temporal interval will be.
7.1 Influence of experience and training
It was shown that fast pianist produced faster and more accurately timed sequences than slow
pianists, who deviated from the original tempo as it became faster (Goeble and Palmer 2013).
More importantly, more experienced pianists executed the keystroke movements from the MCP
joint, whereas the other finger joints (PIP and DIP) extended only slightly. The less experienced
pianists extended the finger joins (PIP and DIP) significantly during a keystroke and had to
compensate through exaggerated MCP flexion. Although it is not clear how neuro-physiological
and biological constraints may affect individual production of notes, simple mechanicalprinciples of movements of the fingers, hands, and forearms, executed in an efficient manner,
contribute to individual motor optimization processes.
8. Applications
Piano pedagogy: Performing music requires from the student a great comprehension about
their body and the movements they make in order to obtain a good sonority. Moreover,
interactions between the performer and his audience are important in the production of
6 Tracking gestures of hands and fingers: A tool for pianists’ identification
different types of gestures, used specifically to entertain the audience or communicate an
emotion. Body movements need to be employed in an optimal level, for instance, too many
movements may arouse an overly exaggerated performance while too timid gestures can make
the musical performance looks constrained. The style of the piece can play an important role in
the production of movements; for example, grand and elegant gestures are appropriate for the
musical language of the nineteenth century. Being able to track x and y positional information,
related to time and space of the hands along with measurements of z (depth), can help
understand pedagogical issues on hand movement, as well as identifying ancillary gesture and
note onsets. Through the analysis of the distance between joint markers, one can assess how
fingers’ position and curvature influence the resulting sound.
Finally, Goebl and Palmer (2009) found that pianists’ finger motion dynamics change while
performing melodies at various tempi. Individual differences across participants revealed that
dissociating different finger motion properties from performance tempo may be important in
pieces played fast. It is interesting to note that pedagogical techniques foster a principle of
economy of finger movement on various instruments (Haken et al 1985; Russianoff 1982;
Taylor 1983). Pianists are advised to keep their fingers as close to the keys as possible during fast
passages. The results of Dalla Bella and Palmer’s (2011) study refute pedagogical
recommendations for music performance practice.
Bibliography
Caramiaux, B., M. M. Wanderley, and F. Bevilacqua. 2012. Segmenting and parsing
instrumentalist’s gestures. Journal of New Music Research 41 (1): 13-29.
This article explores a segmentation model applied to musician ancillary gestures. They used HMM as a model
for shape modeling and segmentation. The model enables continuous signals to be segmented and indexes
simultaneously. A dictionary is first needed to describe input gestures as a sequence of basic shapes.
Dalla Bella, S., and C. Palmer. 2006. Personal identifiers in musicians' finger movement
dynamics. Journal of Cognitive Neuroscience 18, Supplement, G84.
Similar to Dalla Bella and Palmer’s (2011) study, this research evaluates whether pianists’ finger motion are
characterized by dynamic landmarks (fingers’ velocity and acceleration) that could identify pianists and fingers.
Dalla Bella, S., and C. Palmer. 2011. Rate effects on timing, key velocity, and finger kinematics
in piano performance. PLoS ONE, 6, e20518. doi: 10.1371/journal.pone.0020518.
This paper investigates the possibility of movement kinematics to identify different pianists’ style. They specifically
looked at fingers’ amplitude, velocity, and acceleration, as well as temporal variability at faster tempi. They used a
neural network classifier to assess whether finger velocity and accelerations as fingers approached keys were
sufficient to discriminate between pianists.
Davidson, J.W. 1995. What does the visual information contained in music performances offer
the observer? Some preliminary thoughts. In Music and the Mind Machine: Psychophysiology and
Psychopathology of the Sense of Music, ed. R. Steinberg, 105-14. Heidelberg: Springer.
7 Tracking gestures of hands and fingers: A tool for pianists’ identification
This research suggests a qualitative observation of gestures in music performance, and ask questions, such as which
body movements contribute to musical expression and to the audience’s perception.
Goebl, W., and C. Palmer. 2008. Tactile feedback and timing accuracy in piano performance.
Experimental Brain Research 186: 471-479.
Here the pianists’ finger motion trajectories are examined and recorded with a motion capture system. It was
shown that it is possible to discriminate between pianists according to their experience based on temporal accuracy
of finger movements.
Goebl, W., and C. Palmer. 2009. Finger motion in piano performance: Touch and tempo.
Proceedings of the International Symposium on Performance Science. Auckland, New Zealand. 65-70.
With a 3-D motion capture system, this study examines movement characteristics of pianists’ fingers while
performing melodic extracts at different tempi. Individual differences reveal that various finger movement properties
may be truly important for fast piano playing.
Goebl, W., and C. Palmer. 2013. Temporal control and hand movement efficiency in skilled
music performance. PLoS ONE 8(1): e50901. doi:10.1371/journal.pone.0050901.
This paper discusses the timing and force control of finger movements in expert pianists, as well as the potential
individual differences that can emerge from various performances. The method is clearly exposed with graphs to
support the procedures and material used.
Godoy, R., A. Jensenius, and K. Nymoen. 2010. Chunking in music by coarticulation. Acta
Acustica united with Acustica 96 (4): 690–700.
This paper discusses the characteristics of chunking and coarticulation, as well as their role in the production and
perception of music. The authors claim the coarticulation (such as in speech) is an essential element of music.
Loehr, J. D., and C. Palmer. 2007. Cognitive and biomechanical influences in pianists’ finger
tapping. Experimental Brain Research 178: 518-28.
This paper concentrates on cognitive and biological constraint that may affects timing and motion trajectories. The
results of this study provided important information regarding anticipatory motion in finger tapping. It comes to
the conclusion that movement in advance of a finger’s tap may be influenced by finger coupling and anticipatory
goals.
MacRitchie, J., and N. J. Bailey. 2013. Efficient tracking of pianists’ finger movements. Journal of
New Music Research 24 (1): 79-95.
This paper explores various methods to efficiently capture pianists’ finger movements, and discuss the motion
capture technologies.
8 Tracking gestures of hands and fingers: A tool for pianists’ identification
Palmer, C., and S. Dalla Bella. 2004. Movement amplitude and tempo change in piano
performance. Journal of the Acoustical Society of America 115: 2590.
This paper discusses the contribution of movement amplitude changes in rate tempo investigated with motion
capture.
Palmer, C., E. Koopmans, J. D. Loehr, and C. Carter. 2009. Movement-related feedback and
temporal accuracy in clarinet performance. Music Perception 26: 439-49.
This study is similar to pianists’ assessment of sensory information given by the instrument’s keys and how it
contributes to temporal accuracy of performers, except that it was conducted near clarinetists. The numerous graphs
and figures contribute to the understanding of the paper. The findings suggest that peak accelerations happen as
the finger released the key, thus providing another example of tactile information.
Sotirios, M., and P. Georgios. 2008. Computer vision methods for pianist’s fingers information
retrieval. Proceedings of the 10th International Conference on Information Integration and Web-based
Applications & Services. Linz, Austria. 604-8.
This article suggest a low-cost video camera-based method for retrieving musical information from pianists’ finger
movements, which is based on signal processing of video recording of the right hand of the pianist. It does not
discuss motion capture techniques.
Wanderley, M. M. 1999. Non-obvious Performer Gestures in Instrumental Music. In Lecture
Notes on Artificial Intelligence, ed. A. Braffort, 37-48. Heidelberg: Springer-Verlag.
This paper first discusses the different significations of the term gesture in various contexts. It then suggests a study
conducted on clarinetists to assess the influence of non-obvious gestures (gestures that are not essential in the
production of sound) on performances.
Wanderley, M. M., B. Vines, N. Middleton, C. McKay, and W. Hatch. 2005. The musical
significance of clarinetists' ancillary gestures: An exploration of the field. Journal of New Music
Research 34 (1): 97-113.
This research suggests an analysis of clarinetists’ ancillary gestures performing classic concert solo repertoire. They
concentrated on the timing of the different performance conditions (i.e., immobile, standard, and exaggerated). An
interesting point was that they gathered clarinetists’ perception of how they experienced body movements and how
they related their perception of the utility of ancillary movements to musically relevant events.
Watson, A. H. D. 2009. The biology of musical performance and performance-related injury. Plymouth, UK:
Scarecrow Press.
This book considers the considerable demand made on music performers’ mind and body. It discusses the issues
that singers and instrumentalists encounter during their musical training, and provides interesting schemas, figures,
and animations to visualize what a healthy position looks like.
9

Download Report