Gamification of Wearable Data Collection A Tool for both Friend and Foe

work involving gamifiying mobile apps for data collection for ML-based case study that expose privacy risks

Introduction

User authenticating on a device using pattern lock
Question 1:
Are the wrist movement dynamics similar when data is collected from pattern execution at the authentication screen and a gamified version of the pattern entry mechanism?
Question 2:
Can user's APL patterns be decoded by an adversarial gaming app through underlying sensor data?
*UA (User Authentication), GR (Gesture Recognition), ST (Sleep Tracking), HAR (Human Activity Recognition)
Gamification of cybersecurity training and experiment data collection
  • De Nadai et al. used Foursquare API to get information about user behavior across Italian cities.
  • Dergous et al. proposed gamifying data collection experiments using in-game powerups.
  • Cechanowicz et al. designed three gamified versions of market research survey.
Mining user inputs using sensor data
Mining user inputs using smartphone sensor data
  • Xu et al. used sensor data to learn motion change patterns of tap events for inference of keys on a number pad and PINs.
Mining user inputs using smartwatch sensor data
  • Lu et al. inferred PINs and Android Pattern Locks (APL) drawn on the smartwatch.

Sensing + Hardware Setups

A. Android Pattern Lock Mimicing App

23 pattern locks entered in lock screen on our Android App.
IMU sensing on smartwatch:
Dominant Right hand setup with smartwatch logging IMU sensor information.
Gyroscope: angular shifts in xy, yz, zy planes
Accelerometer: linear shifts in x, y, z direction
Timestap: ts in milliseconds Pattern label: a, b, c--w
Timestamps log on smartphone:
Time stamps of start and end of patterns entered. Start of gesture: ts in milliseconds End of gesture: ts in milliseconds

B. Gamified App Mimicing APL entries

A Google Play App: Free Flow The game asks users to connect similar color dots on a grid without colliding with other circle's path with patterns that form very similar to APL lock screen.
A completed APL like gesture on the game.
Our App: allows intersection with a moving ball that disappears as it traces a pattern. User follows the pattern as it disappears to complete a level.
Our game ends at each APL entry.
Our Game in Action: A unity3D game designed for each 23 pattern locks. The pattern is not shown in the game, it is presented for illustration purposes.

Data collection & Experiment Design

  • Number of participants (N): 24
  • Sessions/participants: 2 sessions/a day apart
  • Task constraints:
    • user entered 23 patterns explicitly in app P.
    • played game containing 23 patterns in game G.
Scenarios tested using data from app A and game G.
Scenarios tested using data from app A and game G.
Scenarios tested using data from app A and game G.
Scenarios tested using data from app A and game G.

Signal processing & feature logic

Segmentation of Raw Signals
  • Both accelerometer and gyroscope data from a smartwatch are used.
  • For each drawn pattern, a time-series segment is extracted.
  • Each time series is preprocessed to remove noise and outliers.
Feature Extraction
  • Feature vectors are computed for each time series.
  • Examples include Fast Fourier Transform (FFT), derivatives, and related features.
Training and Testing Scenarios
  • Scenarios: I, II, III
Classifiers
  • Support Vector Machine (SVM)
    • Kernel: Linear
    • Penalty parameter: C = 2
    • Probability estimates enabled
  • Logistic Regression (LR)
    • Solver: libfgs
    • Loss function: Multinomial
    • Penalty parameter: C = 2
Feature Type Features
Type A (time–freq domain) mean, standard deviation, mean absolute deviation, minimum, maximum, energy, interquartile range, entropy
Type B (frequency-domain) spectral maximum index, spectral mean frequency, spectral skewness, spectral kurtosis
Type C (inter-axis: xy, yz, xz) correlation
Type D (all-axis: x, y, z) signal magnitude area

Results

Average accuracy from SVM classifier's cumulative top 3 guesses.
  • Our study’s results are x% better than random guesser.
  • For 23 patterns, the random first guess probability is 4.34%.
  • From our study, we pick the lowest accuracy from all three scenarios to make the comparison.

Pattern Lock Inference Accuracy

Observation
  • Scenario I performs 10-15% better than scenarios II and III.
  • Scenario III performs better than scenario II only for LR classifier.
Implications:
  • Game designed to collect data is feasible as shown by Scenario II and III.
  • From LR, 76% of patterns are predicted within three guesses, whereas only 62% if using target’s data.

User-level and Pattern-Level Performance

  • Scenario I performs far better than scenarios II and III for most users.
  • 80% of users had an accuracy of over 40% for the scenarios II and III.
  • Scenario I performs far better than scenarios II and III for most users.
  • 80% of users had an accuracy of over 40% for the scenarios II and III.

Pattern Analysis

Patterns similar to one another harder to distinguish examples (r) and (s).
Pattern (f) frequently misclassified as (l), (n), (p), (w).
Pattern (r) and (u) although similar, never misclassified.

Conclusion and Future Work

  • Gamification of data collection for malicious intent matches real world data collection scenarios.
  • Application of gamification for the purpose of malicious intent in data collection is plausible.
Future Work:
  • Study of gamification in another data collection setting.
  • Study of another potential attack with gamification.