Vote Summary: Gesture application range

I’m so grateful for those who submitted your assignment and those who attended our gesture workshop.
Now, we have a preliminary picture to guide the selection of gestures for in-car interaction.
Great Job!!

Gesture interaction_summary


Brainstorming Session Summary

Scenario 1. Rear-seat HMI interaction for kids

Children are required to be seated in the rear-passenger row with restraint systems.  Comforting children and providing entertainment to them not only makes them happy, but also helps their parents focus on driving.

Design Question: How can we use gesture in rear-seat HMI design to meet the needs of children between 4 and 12 years old?

Group 1




Long point


Point vs. Grasp


Quad Point

Sound up/down

Swipe through cards

Group 2 

  • Thumb in the mouth – signs of tiredness (typical child gesture)
    • soothing music to help or
    • Keep awake if you don’t want them to sleep
  • Games using hand shapes – animals
    • Tennis game L-R or F-B
  • Pointing
    • activation of entertainment
    • assistive for parent to help see what they are looking at trying to see etc
  • Taking Pictures
    • Enjoyment game
  • Conflict Resolution / Management – Arguing siblings, or ones being naughty!
  • Crossing legs/fidgeting
    • Gestures or Signs of the need to go to the toilet
  • Gesture to indicate hunger / thirst
  • Child entertainments
    • Spoken/vocal
    • The question why? Always give an answer back!
  • Reach, unlock gesture for seat belt – for smaller children who can’t reach the buckle – only when stopped
  • Sickness – wretching gesture.
  • Suspicious behavior cam – driver alert, are they doing something they shouldn’t

Just the system for the rear!

Group 3

Scenario 1- gesture for kid

Group 4: [Lost in transportation]

Scenario 2. Gestures assist driver in motion

The one-way commute time for an average American is one hour.  We spend a lot of time on the road. Sometimes, nothing can keep us away from multi-tasking. We may answer phone calls, change music, look up routes having the least traffic, etc.  Voice, touch- screen and remote controllers may not be enough to keep us safe while getting everything done efficiently.

Design Question: How can we use gesture to help drivers who are in motion?

Group 5:  Enhancing Social Interactions through Gestures

Use-case 1

Apologizing to car behind when after unintentional poor driving

Car X:

  • Locating vehicle (Car Y) to communicate with = Finger point to near view mirror
  • Visual feedback correct car located
  • Speak word “Sorry”

Car Y:

  • Feedback that X vehicle wants to communicate
  • Gesture Accept/reject=wave/palm up or speak “yes/no”

(message spoken – possibly in driver’s voice)

Vote! Gesture application range

Remember to share your thought about the visual & manual demand for different types of gestures. Since a type of gestures may have wide range of applications, giving us an example will help understand what’s your understanding of a certain gesture type.

You can either scan your paper survey  and send them to:

Or fill a online form (see link below)

Thank you for your attention. Look forward to hearing from you.

Gesture interaction_Def Sheet_ overview

Research Needs & Knowledge Gaps

Research Needs

Big Questions

  • Which are real reasons for gestures in automotive? Where does it make sense?
  • How can universal gestures be established & whose job is that?
  • What are uses of gestures during periods of automated/non-automated driving?
  • Better theory of gestures – Unified

Design considerations

  • Ho to Multimodal combination of gestures
  • How to provide results feedback?
  • How to properly navigating user to sensitive area for air gestures
  • How many gestures are possible / feasible?
  • How to selection of meaningful gesture for certain situations

Behavior observation

  • Which gestures are done unintentional during driving?
  • What are methods for discovering gestures?
  • What are the emotional aspects of gesture interaction?

Knowledge Gaps

Performance consideration

  • Experimental evidence for performance
  • What domains are gestures actually optimal for?
  • Where are gestures really useful?

Gesture discovery

  • How do users know a gesture is there?
  • How can we communicate what a gesture is to a user?

Technology consideration

  • How “complete” should a gesture be? -> trade-off between completeness & recognition
  • Capability of gesture trackers – limitations
  • Can gesture inputs be accurate enough to allow efficient and non-distracting inputs?


  • Gesture navigation guidelines.
Design challenges

Workshop presentations

Gesture in Car – an Overview [by Dr. Yu Zhang]  
Link address: Gesture in car-an overview

Gesture in Automotive Application [by Dr. Linda Angell]
Link address: Gesture in Automotive Applications

Apply Popular Usability Heuristics to Gesture Interaction in the Vehicle [by Thomas M. Gable, Keenan R. May, & Bruce N. Walker]
Link address: Gesture Design Heuristics

The Steering Wheel as a Touch Interface [by Dr. Steffen Werner]
Link address: Thumb Gesture on Steering Wheel

Design Exercise
Link address: Design exercises

Evaluation Exercise.
link address:Evaluation exercises
Check list address: Evaluation checklist

Funny Gesture Video

Presenter Information

Applying Popular Usability Heuristics to Gesture Interaction in the Vehicle

Thomas M Gable
Department of Psychology, Georgia Institute of Technology

Thomas is a doctoral student in engineering psychology at Georgia Tech. Thom graduated with a BA in Psychology from The College of Wooster, and worked in the Sonification Lab as a research associate before starting the PhD program. His interests lie in the area surrounding interaction with complicated systems while multitasking, attempting to increase performance of these tasks through the application of new technologies and novel approaches to information display.

Keenan R May
Department of Psychology, Georgia Institute of Technology

Keenan is a doctoral student in engineering psychology at Georgia Tech. He became interested in the way humans interact with technology while majoring in cognitive science at Rice University, and completed a MS in Human-Computer Interaction at Georgia Tech in 2014. His research interests center on the use of nontraditional interfaces in the vehicle

Applying Popular Usability Heuristics to Gesture Interaction in the Vehicle.Gable, May, and Walker_permission formated

The Steering Wheel as a Touch Interface: Using Thumb-Based Gesture Interfaces as Control Inputs While Driving

Steffen Werner, Ph.D. [Associate Professor]
Department of Psychology – Human Factors Program, Interdisciplinary Neuroscience Program, University of Idaho

Steffen Werner leads the Human Cognition and Usability Lab at the University of Idaho. He teaches courses in HCI, Cognitive Neuro-science, Cognitive Psychology, and UX seminars at Idaho’s Human Factors graduate program. He received his Ph.D. at the University of Göttingen (Germany) in 1994 followed by a postdoc at MIT. His applied research projects center on visual and spatial displays, graphical passwords, driver distraction, and more recently, gesture interfaces in cars.

The Steering Wheel as a Touch Interface_swerner

Preliminary Time Line




Round Table / Participants introduction


Session 1 – Gesture interaction: the big picture


Group Exercise – Generating & analyzing gestures


Session 2 – Current applications of gesture interaction and trends


Session 3 – Heuristics for applying gesture in the Automotive HMI




Session 4– The steering wheel as touch interface: Using Thumb-based gesture interface as control inputs while driving


Session 5 – Creative design exercise


Session 6 – Design evaluation exercise


Workshop wrap-up