Data - RETHINK big

RETHINK big Project
Consuelo GONZALO MARTÍN
UNIVERSIDAD POLITÉCNICA DE MADRID
24 March 2015
Vivir en un mar de Datos 2015:
Big Data una mirada Global
Fundación Telefónica
www.rethinkbig-project.eu
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619788.
MIDAS expertise
• 20+ years in Data Value
Chain: collection, analysis,
knowledge extraction
• Multiple-source data integration
and analysis
• Data Mining on text, image and
structured data
• Data Mining on streaming data
• High Performance Data
Analysis
• Numerical and agent-based
simulations
• Large scale Heuristic
optimization
• Complex data interaction and
visualization
2
Funding: FP7, private (industry)
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
MIDAS technology
• Medical information
Systems:
• Prediction of patient
recovery
• Early detection of mental
decay
• Pharma applications
• Drug NSLC effectiveness
• Pharmaeconomics
• Mining the IoT:
• Context aware
recommender based on
social network analysis
• Mining portable and
wearable monitoring
3
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
3
24/03/2015
MIDAS Project
• Design of an analytical platform to "monetize”
electronic medical data
EMH
Images
Genomics
Other data
(demographic,
geographic…)
4
Emergencies:
• Re-admissions
• Prioritization of Rx
Nuclear Medicine:
• Automatic
identification of
tumors
Basic research:
• MEG data (Alzheimer)
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Rethink big Project Overview
INDUSTRY-DRIVEN
The Project:
Coordination and Support Action (CSA), 2-year.
Coordinated by BSC,
Start: 1 Mar 2014
The Mission:
To deliver a strategic roadmap for how European technology
advancements in hardware, networking and algorithms can be
exploited for Big Data analytics, in the next 10 years.
5
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Rethink big Project Overview
The Partners
6
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Motivation
Big Data a fast growing market
with impact on diverse sectors
Big Data market is growing six times
faster than the overall ICT market
(source IDC)
Big Data is becoming a key economic
asset:
“Big Data is the new oil”
(EU – N. Kroes)
World Wide Big Data
Market Forecast
40,0
30,0
20,0
10,0
0,0
7
EUR Billion
Sectors/Domains
Big Data Value
Public administration
EUR 150 billion to EUR 300 billion in new
value (Considering EU 23 larger governments)
Healthcare & Social
Care
EUR 90 billion considering only the reduction
of national healthcare expenditure in the EU
Utilities
Reduce CO2 emissions by more than 2
gigatonnes, equivalent to EUR 79 billion
Transport and
Logistics
USD 500 billion in value worldwide in the form
of time and fuel savings, or 380 megatonnes of
CO2 emissions saved
Retail & Trade
60% potential increase in retailers’ operating
margins possible with Big Data
Geospatial
USD 800 billion in revenue to service
providers and value to consumer and business
end users
Applications &
Services
USD 51 billion worldwide directly associated
to Big Data market (Services and applications)
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Motivation
Ensure Europe’s leading role in the datadriven world
addressing competitiveness, innovation, and
society
covering the all aspects of Big Data Value
Skills
Social
Legal
Data
Technical
Business
Application
8
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Big Data Definition
9
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Challenges
http://www.ibmbigdatahub.com/sites/default/files/infographic_file/4-Vs-of-big-data.jpg
10
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Challenges
Work with different requirements
Velocity
Volume
Variety
Real Time
Sensors
Power consumption…
11
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Challenges
Work with different areas
Software Tools
Systems
Network
Hardware
Applications and end users
12
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Application Challenges
Science and Engineering Applications
Life Sciences
Future Internet and Social Networking
Business, Finance, Information Marketplaces
13
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Key: Hardware/Software Holistic Design
Hardware needs to be software-aware
Software needs to be hardware-aware
14
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
What happens if HW does not consider SW
• Many (supposedly great) changes in
HW architecture do not survive
Cell processor (Playstation 3 processor)
Master-Slave processor model programmed using
DMAs
-> Extremely difficult for programmers
Itanium processor (VLIW)
Very Long instruction word explicitly harnesses
instruction level parallelism through Compiler
-> Compilers could not extract required parallelism
15
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
What happens if SW does not consider HW
Terasort contest: sorting 100TB data
Number 1: Vanilla Hadoop
2100 nodes, 12 cores per node, 64 Gb per node
24.000 cores
134 Tb memory
Vanilla
Hadoop
Time: 4300
segs is easy to program, but
Cost57X
in Amazon:
$ 8.800100X more memory,
needs
more cores,
Number
2: only
Tritonsort
and
gets 2X performance
52 nodes, 8 cores per node, 24 Gb
416 cores
1,2 Tb memory
Time: 8300 secs and 6400 secs
Cost in Amazon: $ 294 and 226
16
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Enabling Technologies
Conventional / Unconventional HW and
processing technology
Distributed Architectures, Devices and
Sensors, Memory and Storage
Networks
Frameworks, SW Models, Algorithms,
Data Stuctures and Visualization
17
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Rethink big Methodology
Identification of European Big Data Competencies
Review& First Working Group Meeting
Refine a group of technology and bussines experts
Technical and bussines oriented surveys
Interactive Working Group Workshop
SWOT Elaboration
18
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
First Working Group Meeting: 18,19 Sep 2014
Objectives: Identify challenges across European Big Data sectors,
Develop a shared language, Engage key strategists
Attendees: 70 Experts from 49 Organizations, 38 External
Project /
Programme
2
Research
Institution
6
19
SME
16
Large Company
12
Users
Academic
13
THALES, AIRBUS,
Boehringer-Ingleheim,
AGT International,
Capgemini, Cloud&Heat,
The Unbelievable Machine Company,
NextWorks
Providers
INDUSTRY PARTICIPANTS INCLUDED:
ARM, THALES,
Alcatel Lucent Bell Labs,
Telefonica, T-Systems, Bull,
TT Tech, Lacie (Seagate),
Kalray, Okkam
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
20
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Initial Expert List
Interest in Participation per Area of Expertise
35
30
25
20
15
Implicit NO
Explicit NO
10
YES
5
0
21
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
First Working Group Meeting: 9,10 Dec. 2014
Objectives: to synthesize findings so far and analyzing the
hardware and networking situation for Big Data in Europe
Attendees: Around 30 partners and external experts participated
from seven European countries, representing both those
researching and producing the Big Data infrastructure and those
who rely on it for their research or business objectives.
Conclusions:
While Europe may not be less competitive in software and co-design, it holds a
leading position in hardware areas such as embedded systems and device
design.
Software areas such as algorithms and data analytics, domain-specific expertise
were also perceived strengths.
Opportunities identified include distributed computing, leveraging datasets and
real-time analytics.
Europe benefits from strong political leadership in this field and the funding to
facilitate scaling, although securing cooperation between its vast patchwork of
SMEs may prove challenging.
Complex bureaucracy and legal frameworks in Europe mean that other regions
may move faster to capitalize on such openings.
22
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
RETHINK big Project
Big Data Value cPPP
www.rethinkbig-project.eu
This project has received funding from the European Union’s Seventh Framework Programme for research, technological development and demonstration under grant agreement no 619788.
BDV cPPP Content: Multidisciplinary approach
24
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015
Thank you
www.RETHINKbig-project.eu
25
Vivir en un mar de datos: Big Data, una mirada global. Fundación Telefónica
24/03/2015

Download Report