NEXUS UI simplified expressive mobile development Experimental Music & Digital Media @ Louisiana State University Ben Taylor • Dr. Jesse Allison • Yemin Oh • William Conlin • Daniel Holmes ! What is Nexus UI? • Nexus is a project focused on Distributed Performance Systems via the web. • Nexus Project began focused on the server tools to enable distribution of UI - Rails, PHP, Node.js • Nexus UI is a framework that solves the issue of creating browser based interfaces. The need arose for simplified development of standard and experimental UI objects Vox Instrument ! Enabling musical expressivity Vox Instrument ! Enabling musical expressivity Objectives • cross-device compatibility for apps • enabling unconventional UIs using browser technologies • no preference for platform • extensibility • focus on making touch display UIs easy to create Objectives • cross-device compatibility for apps • enabling unconventional UIs using browser technologies • no preference for platform • extensibility • focus on making touch display UIs easy to create Interfaces Standard graphical user interfaces for electronic music Dial Slider Button Toggle Multislider Keyboard Position Interfaces Non-standard interfaces made easy by HTML5 Tilt Metroball Typewriter (computer keyboard) Pixels Mouse Joints Colors Matrix METROBALL Now with added Tilt! METROBALL Now with added Tilt! Accessing Interfaces • Link to nexusUI.js library <script src=“nexusUI.js”></script> • Add HTML5 canvas with valid nx attribute <canvas nx=“multitouch”></canvas> <canvas nx=“tilt”></canvas> Interface Example <!doctype html> <html> <head> <script src="jquery.js"></script> <script src="nexusUI.js"></script> </head> <body> <canvas nx="tilt"></canvas> <canvas nx="position"></canvas> <canvas nx="joints"></canvas> </body> </html> Synthesis done separately in Max (or whatever). The web page is simply the UI. Interface Example <!doctype html> <html> <head> <script src="jquery.js"></script> <script src="nexusUI.js"></script> </head> <body> <canvas nx="tilt"></canvas> <canvas nx="position"></canvas> <canvas nx="joints"></canvas> </body> </html> Synthesis done separately in Max (or whatever). The web page is simply the UI. Interface Example <!doctype html> <html> <head> <script src="jquery.js"></script> <script src="nexusUI.js"></script> </head> <body> <canvas nx="tilt"></canvas> <canvas nx="position"></canvas> <canvas nx="joints"></canvas> </body> </html> Enabling: • Continuous control • Gestural control • Automation / animation • Multi-point interfaces Synthesis done separately in Max (or whatever). The web page is simply the UI. Objectives • cross-device compatibility for apps • enabling unconventional UIs using browser technologies • no preference for platform • extensibility • focus on making touch display UIs easy to create Communication and Servers No preference for audio platform or server configuration JavaScript • Web Audio Servers (send via UDP to Max, SuperCollider, ChucK) • Node.js • PHP/Ajax • Rails iOS • libPD Android - yes, but not incorporated yet… Communication and Servers No preference for audio platform or server configuration JavaScript • Web Audio nx.sendsTo(“js”) Servers ! ! ! (send via UDP to Max, SuperCollider, ChucK) iOS • libPD Android - yes, but not incorporated yet… nx.sendsTo(“node”) nx.sendsTo(“ajax”) • Node.js • PHP/Ajax • Rails ! ! nx.sendsTo(“ios”) Open Sound Control <canvas nx=“tilt”></canvas> /tilt1/x 0.587 /tilt1/y 0.394 /tilt1/z 0.008 <canvas nx=“dial” id=“volume”></canvas> /volume 0.587 nexusosc.com/ nexusosc.com/api/ Objectives • cross-device compatibility for apps • enabling unconventional UIs using browser technologies • no preference for platform • extensibility • focus on making touch display UIs easy to create Core Nexus is a platform > Template for building new interface objects • • • • • Touch-compatible interaction event handling Transmit functions Helper functions Animation pulse Integration with all UI functionality and UI builders nexusui.js holds the nx object which manages all of the UI objects on page. Built in Individual User interface files are bundled into nexusui.js though you can make and import your own. moonball iOS app built on NexusUI platform ! • • • Touch Interaction Animation Communication with libPD TRAVERSAL Web App with node server, gibberish, max, and midi controlled Organ • Touch interactions • Animation • 2 way communication with Node server & other participants Performed at SEAMUS National Conference 2014 Objectives • cross-device compatibility for apps • enabling unconventional UIs using browser technologies • no preference for platform • extensibility • focus on making touch display UIs easy to create Accessibility Several server paradigms: PHP, Rails, Node.js ! Tutorials on how to set up a local server ! Folder of examples for integration with •Max •Web Audio (in gibberish) ! UI Builders so you don’t have to code UI Builders NexusDrop Drag-and-drop interface editor NexusUp Max bpatcher for duplicating patches in NexusUI Drop Drop Drop NexusDrop creates reusable and editable UIs ! ! • Save to UI Database • Save as HTML • View UIs by others Up Max bpatcher constructs NexusUI HTML from .maxpat JSON Up Up EMDM ACADEMY Developing Web instruments using NexusUI and Gibberish with Middle and High School kids emdmacademy.music.lsu.edu Instruments can be found on the EMDM Academy website linked above. Objectives • cross-device compatibility for apps • no preference for audio platform • extensibility • unconventional UIs using browser technologies • focus on making mobile phone orchestras easy to create Objectives • cross-device compatibility for apps • no preference for audio platform • extensibility • unconventional UIs using browser technologies • focus on making mobile phone orchestras easy to create • make it awesome! Mobile Phone Orchestra Visualizer connecting all interfaces with node.js VISUALIZATION Node.js based visualization of performers for the Louisiana Mobile App Orchestra (LMAO) VISUALIZATION Node.js based visualization of performers for the Louisiana Mobile App Orchestra (LMAO) LDMC.js object used to localize sounds in the Louisiana Digital Media Center 92 speaker Meyer Constellation system Tactile Overlays An offshoot of this has been exporting UI images as templates for laser cutting various materials to create Vox TouchOSC interface turned into NexusUI interface NxPetals Example of an iOS app with nexusUI and libPd Humming Mississippi http://hmiss.in Collaboration through node server, Gibberish audio in each individual browser, wood river-routed planks in real life. Humming Mississippi http://hmiss.in Collaboration through node server, Gibberish audio in each individual browser, wood river-routed planks in real life. Thanks! Want to see more? Demo Thursday! nexus.cct.lsu.edu ! nexusosc.com ! github.com/lsu-emdm/nexusUI Ben Taylor - [email protected] Dr. Jesse Allison - [email protected] William Conlin - @walkersaxplayer - [email protected] Experimental Music & Digital Media @ Louisiana State University Ben Taylor • Dr. Jesse Allison • Yemin Oh • William Conlin • Daniel Holmes !
© Copyright 2024 ExpyDoc
