Green Cube GSI FAIR Tier 0 Thorsten Kollegger FAIR & GSI 30. March 2015 AIME Big Data - Budapest Green Cube New data center at GSI, mainly for FAIR Tier0 6 floors, ~25x35 sqm footprint T. Kollegger | AIME Big Data, Budapest | 30. March 2015 2 GSI & FAIR GSI – Helmholtz Center for Heavy Ion Research FAIR – Facility for Antiproton and Ion Research Located in Darmstadt, Germany ~1.300 employees and ~3.000 worldwide users T. Kollegger | AIME Big Data, Budapest | 30. March 2015 3 FAIR Currently under construction Fully operational in the next decade, ~1B EUR investment European project of 8 member states + ~50 international partners T. Kollegger | AIME Big Data, Budapest | 30. March 2015 4 FAIR Science time Atomic, applied and plasma physics - APPA Astrophysics and nuclear structure - NUSTAR Hadron structure - PANDA QCD phase transition - CBM temperature big bang T. Kollegger | 21.11.2014 FAIR Computing The FAIR experiments will • produce tens of PetaBytes of raw data per year, with up to 1 TByte/s from the experiments to the Tier0, which need to be safely and permanently stored • analyze these datasets over several years, requiring also extensive simulations and theory calculations T. Kollegger | AIME Big Data, Budapest | 30. March 2015 6 FAIR Computing FAIR Tier 0 @ full FAIR operation • 300.000 CPU-cores • 35 PByte/year Online-Storage (HDD) • 30 Pbyte/year Permanent-Storage (Tape) Comparable resources in FAIR partner countries T0@FAIR/GSI Key Points • Few big, highly efficient data centers • Further partner connected via regional federations T. Kollegger | AIME Big Data, Budapest | 30. March 2015 T0 Teralink Federation Partners (~T2) Co-T0(s) in FAIR Partner Country(-ies) T1’s or Regional Federations in FAIR Partner Countries (T1+T2) 7 FAIR Tier 0 @ GSI FAIR Tier 0 is the core of the FAIR Computing Model – Experiment Online Farms connected via ~4000 fibres Co-T0(s) in FAIR Partner to Experiments Country(-ies) – Experiment Storage Systems – Experiment Processing Farm – Integration of HPC Centers of surrounding partners (Teralink) – Distribution of experimental data to FAIR Partners; several 100 Gb/s links T. Kollegger | AIME Big Data, Budapest | 30. March 2015 T0@FAIR/GSI T0 Teralink Federa on Partners T1’s or Regional Federa ons in FAIR Partner Countries (T1+T2) 8 FAIR Data Center A common data center for FAIR (Green Cube) Key Facts 6 floors, 4.645 sqm Space for 768 19” racks (2,2m) 4 MW cooling (baseline) Max cooling power 12 MW Fully redundant (N+1) PUE <1.07 T. Kollegger | AIME Big Data, Budapest | 30. March 2015 9 An Alarming Truth World wide air-traffic causes around 2 % of the global CO2 emission – just as much as commercial IT! (ca. 30 GW) [Gartner] Volker Lindenstruth (www.compeng.de) May 22, 2012— Copyright ©, Goethe Uni, Alle Rechte vorbehalten 10 Data Center State of the Art Electrical Power UPS 10% Data Center Cooling 12% Back Cooling 25% Light, etc. 3% IT Equipment 50% PUE = Total Facility Power IT Equipment Power Power Usage Effectiveness PUE = 2 (typical PUE = 1,6 to 3,5) Note: 12C/kWh 1W = 1,05€/a Source: EYP Mission Critical Facilities Inc., New York Volker Lindenstruth (www.compeng.de) May 22, 2012— Copyright ©, Goethe Uni, Alle Rechte vorbehalten 11 Innovative Cooling System Architecture evaporation Cooling Overhead 5% of computer power cooling tower data center building air air server room cooling tower fixtures makekupwater draw-off equipment container M M Secondary cooling circuit Works with every commercial computer secondary pump primary heat exchanger pump Volker Lindenstruth (www.compeng.de) 30. März 2015— Copyright ©, Goethe Uni, alle Rechte vorbehalten Primary Cooling circuit Wet bulb temperature 2010 Volker Lindenstruth (www.compeng.de) 30. März 2015— Copyright ©, Goethe Uni, alle Rechte vorbehalten 13 Heiko Volker Lindenstruth (www.compeng.de) 30. März 2015— Copyright ©, Goethe Uni, alle Rechte vorbehalten Ebermann 14 Prizes • Green-IT Award Bundesregierung „Visionäre Gesamtkonzepte“ • Deutscher Rechenzentrumpreis 2012 – energy efficiency • Deutscher Rechenzentrumspreis 2013 – Visionary Datacenter Architecture • Nominated for Deutscher Rechenzentrumspreis 2014 – energy efficency • DataCenterDynamics EMEA Award 2013 – Data Center Blueprint • BroadGroup EMEA Awards Special Commendation – Energy Efficiency • „Land der Ideen“2012 for LOEWE-CSC • Green Cube Project of the Month, BMBF • 5 Nominations with 4 ranks at Data Center Dynamics EMEA Awards – 2011, 2012, 2013 • 2nd rank at Deutsche Internet Awards 2012 Volker Lindenstruth (www.compeng.de) 30. März 2015— Copyright ©, Goethe Uni, alle Rechte vorbehalten 15 The FAIR Data Center (Green Cube) • • • • Space for 768 19” racks (2,2m) 4 MW cooling (baseline) Max cooling power 12 MW Fully redundant (N+1) • • • • Can be used for any commercial IT No battery backup required PUE <1.07 Baseline building cost 12 M€ Volker Lindenstruth (www.compeng.de) 30. März 2015— Copyright ©, Goethe Uni, alle Rechte vorbehalten 16 Green Cube Construction started in Dec 2014 On schedule for start of operation in Q4/2015 T. Kollegger | AIME Big Data, Budapest | 30. March 2015 17 Green Cube T. Kollegger | AIME Big Data, Budapest | 30. March 2015 18 Green Cube Prototype Two floor prototype in “Testinghalle” @ GSI Home of current GSI Computing Systems T. Kollegger | AIME Big Data, Budapest | 30. March 2015 19 GSI Scientific Computing Systems Compute clusters Prometheus ~10.000 cores Kronus ~4.000 cores + HT L-CSC ~3.000 cores + 4 GPUs/node (AMD S9150/S10000) Storage systems Hera ~7 PByte netto (Lustre) Nyx ~5 PByte netto (Lustre) x2 by end of year Used for FAIR&GSI Experiments, ALICE Tier2 & National Analysis Facility, Theory T. Kollegger | AIME Big Data, Budapest | 30. March 2015 20 GSI Scientific Computing Systems Compute clusters Prometheus ~10.000 cores Kronus ~4.000 cores + HT L-CSC ~3.000 cores + 4 GPUs/node (AMD S9150/S10000) Storage systems #1 in Hera ~7 PByte netto (Lustre) Green 500 list Nyx ~5 PByte netto (Lustre) (Nov 2014) 5,27 GFLOPS/W x2 by end of year Used for FAIR&GSI Experiments, ALICE Tier2 & National Analysis Facility, Theory T. Kollegger | AIME Big Data, Budapest | 30. March 2015 21 TeraLink Tight integration of LOEWE-CSC Computing Center in Frankfurt (~30km away; 84km in fibres) • Router cluster (LNET) for the TeraLink 12 x 10 Gbit Ethernet => lustre IB loadbalancing T. Kollegger | AIME Big Data, Budapest | 30. March 2015 22 TeraLink lustre clients at Gauss Supercomputer Frankfurt Hera lustre cluster at GSI Reading: 13.5 GB/s Writing: 14GB/s Offloading of jobs to Frankfurt, opportunistic resource usage Additional storage system for selected users in Frankfurt T. Kollegger | AIME Big Data, Budapest | 30. March 2015 23 Software Stack FairRoot Common Software Framework for Simulation, Reconstruction and Analysis for the FAIR Experiments https://github.com/FairRootGroup/FairRoot T. Kollegger | AIME Big Data, Budapest | 30. March 2015 24 FairRoot and ALFA CbmRoot R3BRoot SofiaRoot MPDRoot PandaRoot AsyEosRoot FopiRoot EICRoot T. Kollegger | AIME Big Data, Budapest | 30. March 2015 Runtime DB VGM Magnetic Field Event Generator Geant3 Geant4 Module Detector CMake ZeroMQ MC Application Fair DB BOOST ROOT Fair MQ Libraries and Tools Protocol Buffers FairRoot Testing Building configuraion DDS ???? ALFA Genat4_VMC AliRoot6 (O2) 25 Summary Green Cube Data Center at GSI under contruction PUE < 1.07, low Cap. and Op. Exp Supplemented by Software Development and System Design Program for FAIR Computing T. Kollegger | AIME Big Data, Budapest | 30. March 2015 26
© Copyright 2024 ExpyDoc
