A Proposal of an Internal-state Inference System Based on Multi-modal Sensory Fusion Method 1K. Hirota, 1N. Iwamatsu, and 1,2Y. Takama Tokyo Institute of Technology PREST, Japan Science and Technology Corporation (JST) 1 Striking-based Internal-state Diagnosis • Palpation in hospitals, judgment of fruit’s matureness, inspection of tunnel walls, etc. • High dependence on the experts’ skill • Difficulty in transferring implicit knowledge from experts to novices • Proposal: Design of automatic internal-state inference system with low cost 2 Target Objects EP: Empty Plastic HP: Half Plastic FP: Full Plastic ES: Empty Steel HS: Half Steel FS: Full Steel Steel can Plastic can • 2 types of materials (plastic & steel) • 3 types of water amount (empty, half, full) 3 Proposed System Configuration Auditory sensor part (Microphone with amp) Inference part Feature extraction Target object (Unknown internal state) Reexamination? Tactile sensor part (Hammer attached with Tactile Sensor) Inference based on multi-modal fusion 4 Developed System Hammer part Target object 5 Used Sensors • Auditory sensor part – Condenser microphone (Max 120dB) – Amplifier (40dB) • Tactile sensor part – Accelerometer: MA304Aa (Microstone Inc.) – Range: ±4G – Sensitivity: 50±10%(mV/G) – Frequency:0.8-1KHz 6 Sound wave Feature Extraction • Ampmax: maximum amplitude • T: time interval from 1st to 2nd stroke • T1: time interval from Ampmax to 1/2Ampmax (last time within T) • T2: time interval from Ampmax to 1/4Ampmax (last time within T) Tactile info. 7 Inference of material & amount of water • Material identification – Based on the difference of decay time between steel and plastic cans – Target is plastic if the following holds, otherwise steel • Discriminating amount of water – (A: identified material): system’s output 8 Experimental Results Target’s state EP EP 17 HP 2 FP 0 ES 0 HS 0 FS 0 HP 2 17 2 0 0 0 System’s output FP ES 1 0 1 0 18 0 0 19 0 2 0 0 HS 0 0 0 1 17 0 FS 0 0 0 0 1 20 • Accuracy: 80% (total) , 100% (material only) 9 Inference Failure Cases • E.g. hammer may slip on the target’s surface • One of input variables has apparently different values from average • No significant difference are found between membership values : Error detection • Reexamination is easy because of low-cost and non-destructive diagnosis 10