Google’s ‘Project Soli’ Radar Hand Tracking and How VR and AR Might Use it

See discussion in:


Brainstorming Session Summary

Scenario 1. Rear-seat HMI interaction for kids

Children are required to be seated in the rear-passenger row with restraint systems.  Comforting children and providing entertainment to them not only makes them happy, but also helps their parents focus on driving.

Design Question: How can we use gesture in rear-seat HMI design to meet the needs of children between 4 and 12 years old?

Group 1




Long point


Point vs. Grasp


Quad Point

Sound up/down

Swipe through cards

Group 2 

  • Thumb in the mouth – signs of tiredness (typical child gesture)
    • soothing music to help or
    • Keep awake if you don’t want them to sleep
  • Games using hand shapes – animals
    • Tennis game L-R or F-B
  • Pointing
    • activation of entertainment
    • assistive for parent to help see what they are looking at trying to see etc
  • Taking Pictures
    • Enjoyment game
  • Conflict Resolution / Management – Arguing siblings, or ones being naughty!
  • Crossing legs/fidgeting
    • Gestures or Signs of the need to go to the toilet
  • Gesture to indicate hunger / thirst
  • Child entertainments
    • Spoken/vocal
    • The question why? Always give an answer back!
  • Reach, unlock gesture for seat belt – for smaller children who can’t reach the buckle – only when stopped
  • Sickness – wretching gesture.
  • Suspicious behavior cam – driver alert, are they doing something they shouldn’t

Just the system for the rear!

Group 3

Scenario 1- gesture for kid

Group 4: [Lost in transportation]

Scenario 2. Gestures assist driver in motion

The one-way commute time for an average American is one hour.  We spend a lot of time on the road. Sometimes, nothing can keep us away from multi-tasking. We may answer phone calls, change music, look up routes having the least traffic, etc.  Voice, touch- screen and remote controllers may not be enough to keep us safe while getting everything done efficiently.

Design Question: How can we use gesture to help drivers who are in motion?

Group 5:  Enhancing Social Interactions through Gestures

Use-case 1

Apologizing to car behind when after unintentional poor driving

Car X:

  • Locating vehicle (Car Y) to communicate with = Finger point to near view mirror
  • Visual feedback correct car located
  • Speak word “Sorry”

Car Y:

  • Feedback that X vehicle wants to communicate
  • Gesture Accept/reject=wave/palm up or speak “yes/no”

(message spoken – possibly in driver’s voice)

Vote! Gesture application range

Remember to share your thought about the visual & manual demand for different types of gestures. Since a type of gestures may have wide range of applications, giving us an example will help understand what’s your understanding of a certain gesture type.

You can either scan your paper survey  and send them to:

Or fill a online form (see link below)

Thank you for your attention. Look forward to hearing from you.

Gesture interaction_Def Sheet_ overview

Research Needs & Knowledge Gaps

Research Needs

Big Questions

  • Which are real reasons for gestures in automotive? Where does it make sense?
  • How can universal gestures be established & whose job is that?
  • What are uses of gestures during periods of automated/non-automated driving?
  • Better theory of gestures – Unified

Design considerations

  • Ho to Multimodal combination of gestures
  • How to provide results feedback?
  • How to properly navigating user to sensitive area for air gestures
  • How many gestures are possible / feasible?
  • How to selection of meaningful gesture for certain situations

Behavior observation

  • Which gestures are done unintentional during driving?
  • What are methods for discovering gestures?
  • What are the emotional aspects of gesture interaction?

Knowledge Gaps

Performance consideration

  • Experimental evidence for performance
  • What domains are gestures actually optimal for?
  • Where are gestures really useful?

Gesture discovery

  • How do users know a gesture is there?
  • How can we communicate what a gesture is to a user?

Technology consideration

  • How “complete” should a gesture be? -> trade-off between completeness & recognition
  • Capability of gesture trackers – limitations
  • Can gesture inputs be accurate enough to allow efficient and non-distracting inputs?


  • Gesture navigation guidelines.
Design challenges

Workshop presentations

Gesture in Car – an Overview [by Dr. Yu Zhang]  
Link address: Gesture in car-an overview

Gesture in Automotive Application [by Dr. Linda Angell]
Link address: Gesture in Automotive Applications

Apply Popular Usability Heuristics to Gesture Interaction in the Vehicle [by Thomas M. Gable, Keenan R. May, & Bruce N. Walker]
Link address: Gesture Design Heuristics

The Steering Wheel as a Touch Interface [by Dr. Steffen Werner]
Link address: Thumb Gesture on Steering Wheel

Design Exercise
Link address: Design exercises

Evaluation Exercise.
link address:Evaluation exercises
Check list address: Evaluation checklist

Funny Gesture Video