Tackling The Challenges of Big Data Introduction to Big Data Samuel Madden Professor and Director of Big Data at CSAIL Massachusetts Institute of Technology BIG Data Transport • Medicine • Social Media • Education • Industry Government • Retail • Science • Personal Tackling the Challenges of Big Data © 2014 Massachusetts Institute of Technology Example 1: Medical Costs MGH Cancer Center ! “Super-Database”! Largest cancer database in the world (173,301 patients)! Based on national tumor registry! Cross linked with death registry! Includes billing, reports, labs, imagery, genome SNPs! ! ! Question: ! What are the factors driving costs for lung cancer patients?! ! Source: Dr. James Michaelson, PhD, MGH, Harvard Medical School! Tackling the Challenges of Big Data © 2014 Massachusetts Institute of Technology 1 Example 2: Shopycat Recommend Things My Friends Like That Are Compatible With My Interests Tackling the Challenges of Big Data © 2014 Massachusetts Institute of Technology Example 2: Flight Prediction EV EV EV K5 K5 NGF NGF WIG How to Predict Flight Delays? 5647PIT 7/5/13 2:10 PM 7/5/13 2:21 Historical Information 5648ABE 7/5/13 12:10 PM 7/5/13 12:00 5651RIC 7/5/13 3:40 PM 7/5/13 3:56 Weather 96PDT 7/5/13 8:15 PM 7/5/13 11:35 96PDX 7/5/13 6:25 PM 7/5/13 6:26 Availability of Planes 4BFM 7/5/13 3:10 PM 7/5/13 3:18 64ISW 7/5/13 4:10 PM 7/5/13 4:16 Other Concurrent Flights 8416PWM 7/5/13 12:01 PM 7/5/13 12:37 Radar Tackling the Challenges of Big Data PM PM PM PM PM PM PM PM © 2014 Massachusetts Institute of Technology Example: Flight Inconsistency! Source: Luna Dong! FlightView! FlightAware! Orbitz! 6:15 PM! 6:22 PM! 6:15 PM! 9:40 PM! 8:33 PM! Tackling the Challenges of Big Data 9:54 PM! © 2014 Massachusetts Institute of Technology 2 Example 2: Flight Prediction! EV EV EV K5 K5 NGF NGF WIG How to Predict Flight Delays? 5647PIT 7/5/13 2:10 PM 7/5/13 2:21 Historical Information 5648ABE 7/5/13 12:10 PM 7/5/13 12:00 5651RIC 7/5/13 3:40 PM 7/5/13 3:56 Weather 96PDT 7/5/13 8:15 PM 7/5/13 11:35 96PDX 7/5/13 6:25 PM 7/5/13 6:26 Departure Time 4BFM 7/5/13 3:10 PM 7/5/13 3:18 64ISW 7/5/13 4:10 PM 7/5/13 4:16 Concurrent Flights 8416PWM Other 7/5/13 12:01 PM 7/5/13 12:37 Radar Tackling the Challenges of Big Data PM PM PM PM PM PM PM PM © 2014 Massachusetts Institute of Technology When Do You Have a Big Data Problem?! • Too many bytes (Volume)! • Too high a rate (Velocity)! • Too many sources (Variety)! • Non-scalable analysis! – Human-intensive! – Computer Intensive! Tackling the Challenges of Big Data Flight Data! Mass General Hospital Data! © 2014 Massachusetts Institute of Technology Sources of Big Data learning science transportation retail Big Data finance social entertainment government health/medical Tackling the Challenges of Big Data © 2014 Massachusetts Institute of Technology 3 What is this course going to cover? Taught by experts in the field! ! • Case Studies! • Ingesting and Integrating Data! • Storage and Compute Platforms! • Presentation and Visualization! • Algorithms & Analytics! ! Tackling the Challenges of Big Data Regina Barzilay! John Guttag! Piotr Indyk! Tommi Jaakkola! David Karger! Security ! Andrew Lo! & Privacy! Sam Madden! Ron Rivest! Ronitt Rubinfeld! Daniela Rus! Michael Stonebraker! Matei Zaharia! Nickolai Zeldovich! ! © 2014 Massachusetts Institute of Technology Tackling The Challenges of Big Data Introduction to Big Data THANK YOU 4
© Copyright 2024 ExpyDoc
