Development of a game software using virtual 3D sound spatialization for the visually impaired Department of Advanced Course of Electrical and Information System, Nagano National College of Technology Matsushima Fumiaki 1/24 About virtual 3D sound 5.1ch 2/24 About virtual 3D sound from back left spatialization spatialize virtual 3D sound 3/24 About virtual 3D sound Changing original characteristics left ear original characteristics right ear Head-Related Transfer Function 4/24 Back ground An app using virtual 3D sound An app using sound A game for the visually impaired 5/24 The number of game for the visually impaired In Vector(a software download site in Japan) The Number of Registered Software 115,000 Search results Keywords: “ゲーム 視覚障害” (Japanese words for "game visually impaired“) 4 6/24 Current game for the visually impaired • Only sound • Almost solo • Nothing can play over network • Using pan (output sound from right or left speaker) to indicate direction Example : direction of ball → not intuitive 7/24 Purpose A game for the visually impaired Virtual 3D sound Communication 8/24 Summary HRTF DS3D Development 9/24 Result DS3D HRTF Using HRTF 10/24 Method HRTF Convolving the HRTF database released by Nagoya University 11/24 HRTF database 44.1kHz 512 points 11.7ms 1.52m 5° 360° 12/24 The developed game Catch! 13/24 Developing environment Language Library Microsoft Visual C# Microsoft Managed DirectX - loading WAV file, etc Skype4COM (Skype API) - voice chat, communication 14/24 How to play (mouse) Catch! 15/24 How to play (keyboard) forward W left A S D right back 16/24 How to play (keyboard) forward W left A S D right back 17/24 Flowchart Loading sound and the HRTF data Calculation of distance and angle Convolving the HRTF Output virtual 3D sound 18/24 Calculation of distance and angle θ l l θ 19/24 Convolving the HRTF FFT original HRTF(θ) IFFT 3D sound 20/24 Communication between players dog position, catching call requirement of catching call Host Client Client Client 21/24 Test-playing At Nagano school for the visually impaired 22/24 Feedback • To use vibration • Virtual 3D sound has a potential to be applied to other games – Submarine game – Fight in the dark 23/24 A game for the visually impaired Communication Virtual 3D sound The new game for the visually impaired 24/24 25/24 26/24 Convolving the HRTF Source Source1 HRTF(θ) 3D Sound1 Source2 HRTF(θ) 3D Sound2 3D Sound 27/24 Convolving the HRTF Source Calculation Source1 3D Sound1 → Output Calculation Source2 3D Sound2 → Output 28/24 Convolving f∗g t = f τ g t − τ dτ • Output characteristic sound to convolve transfer function Example : Get sound reverberation like generated in a large room when convolve the room transfer function 29/24 Convolving theorem F f ∗ g = F(f) ∙ F(g) • Computation order Convolving O(n2 ) FFT O(n log n) 30/24 Skype, Skype API Skype A Internet phone service using P2P Skype API The Skype API enables third-party applications to communicate with Skype. 31/24 Skype, Skype API Skype Call with multiple people Get through NAT, Firewall Communication with voice chat Using as infrastructure 32/24 Communication using Skype Client Host Skype Skype Internet Game Game 33/24 Method DS3D Microsoft DirectSound3D Using position, direction, speed, time difference between right and left ears, etc 34/24 Experimental method • 8 directions • Ask where sounds source • 10 times each methods • 27 sighted people (20’s) • 2 visually impaired 4 times 35/24 Result(sighted) 36/24 Result(visually impaired) 37/24 Result(HRTF) forward 80% forward left 60% forward right 40% 20% left 0% right back left right forward left back back right back 38/24 Result(HRTF) forward 80% forward left 60% forward right 40% 20% left 0% right back left forward right forward left back left back right back right back 39/24 Result(DS3D) forward 50% forward left 40% forward right 30% 20% 10% left 0% right back left right forward left back back right back 40/24 Result(DS3D) forward 40% forward left 30% forward right 20% 10% left 0% right back left forward right forward left back left back right back right back 41/24
© Copyright 2024 ExpyDoc
