The OmegaCAM-VST and MUSE-VLT Survey Federations Gijs A. Verdoes Kleijn OmegaCEN, Kapteyn Astronomical Institute Target, University of Groningen The OmegaCAM-VST and MUSE-VLT Survey Federations Gijs A. Verdoes Kleijn OmegaCEN, Kapteyn Astronomical Institute Target, University of Groningen Collaborations • • • • • • OmegaCAM consortium KiDS survey team Astro-WISE consortium OmegaCEN center MUSE consortium MUSE-WISE team PI Konrad Kuijken PI Konrad Kuijken PI Edwin Valentijn, head E. Valentijn PI Roland Bacon lead J. Brinchmann KiDS / Astro-WISE production team • Kor Begeman, Danny Boxhoorn, Ewout Helmich, Nancy Irisarri, Gert Sikkema, Jelte de Jong, John McFarland, Nicola Napolitano, Lino Grado, Emmanuele Puddu, Fedor Getman, Mario Radovich, Massimo D’allora, Maurizio Paolillo, Francesco LaBarbera, Zhuouy Huang, MUSE-WISE team • Jarle Brinchmann, Willem-Jan Vriend, Thomas Martinsson, Genevieve Soucail, Nicolas Bouche, Thierry Contini, Patrick Maeght, Marie Larrieu, Cynda Othmani, Ole Streicher, Harry Enke, Kristing Riebe, Adrian Partl, Peter Weilbacher, Jim Pizagno OmegaCEN team • Edwin Valentijn, Andrey Belikov, Hugo Buddelmeijer, Parisa Noorishad, Willem-Jan Vriend, Ewout Helmich, Gert Sikkema, Danny Boxhoorn, Kor Begeman, John McFarland, Gijs Verdoes Kleijn KiDS survey team Douglas Applegate, Kor Begeman, Andrey Belikov, Chris Blake, Jeffrey Bout, Danny Boxhoorn, Massimo Brescia, Margot Brouwer, Hugo Buddelmeijer, Axel Buddendiek, Marcello Cacciato, Massimo Capaccioli, Stefano Cavuoti, Ami Choi, Oliver-Mark Cordes, Giovanni Covone, "Massimo DallOra", Jelte de Jong, Alastair Edge, Thomas Erben, Ian Fenech Conti, Jeroen Franse, Fedor Getman, Aniello Grado, Joachim Harnois-Deraps, Ewout Helmich, Ricardo Herbonnet, Catherine Heymans, Hendrik Hildebrandt, Henk Hoekstra, Zhuoyi Huang, Nancy Irisarri, Benjamin Joachimi, Fabian Köhlinger, Thomas Kitching, Leon Koopmans, Konrad Kuijken, Francesco Labarbera, Pedro Lacerda, Giuseppe Longo, John McFarland, Lance Miller, Reiko Nakajima, Nicola Napolitano, Maurizio Paolillo, John Peacock, Carlo Enrico Petrillo, Berenice Pila-Diez, Emanuella Puddu, Mario Radovich, Agatino Rifatto, Nivya Roy, Peter Schneider, Tim Schrabback, Elisabetta Semboloni, Cristobal Sifon, Gert Sikkema, Patrick Simon, William Sutherland, Crescenzo Tortora, Alexandru Tudorica, Edwin Valentijn, Remco van der Burg, Edo van Uitert, Ludovic van Waerbeke, Malin Velander, Gijs Verdoes Kleijn, Massimo Viola, Willem-Jan Vriend, Kristian Zarb Adami KiDS survey team (~75 persons) LVANWAERBEKE, JHARNOISDERAPS WFAU/ROE CHEYMANS ,LMILLER, JPEACOCK,BJOACHIMI TSHANKS MVELANDER CASU Potsdam Bonn HILDEBRANDT, OCORDES, TERBEN, TSCHRABBAC K,KRAHE, EVANUITERT,RNAKAJIMA, MIRWIN SWARREN, WSUTHERLAND PSCHNEIDER, AJAKOBS, DKLAES ATUDORICA, PPLEGATE, ACHOI TKITCHING BVENEMANS PLACERDA YMELLIER,PHUDELOT, RVANDERBURG Munich JSNIGULA, RBENDER, RSAGLIA, SSEITZ Zürich Lyon Padua MRADOVICH Toulouse FGETMAN, AGRADO, MCAPACCIOLI, Naples , NNAPOLITANO MPAOLILLO, EPUDDU, ARIFATTO, GCOVONE, MDALLORA,ZHUANG, CAVUOTI KADAMI SMIESKE, GBECCARI CBLAKE,AROBOTHAM “Glass, steel AND BITS” Remember: WFAU/ROE LSST : “large software project with a mirror” Gaia: ~250p?, ~6 datacenters, $DFS ~ 1.3$INST (ex. Telescope?) Euclid: ~250p, 9 datacenters, $DFS ~ 1.25$INST (ex. telescope) Groningen PTF: ~7p, 1 datacenter, $DFS ~ $INST (ex. telescope) CASU Potsdam Munich Zürich Lyon Toulouse Naples Chile “Glass, steel AND BITS” WFAU/ROE Groningen CASU Potsdam Leiden Nijmegen Bonn Munich Zürich Lyon Padua Toulouse Naples -WISE Chile KiDS Survey Team LVANWAERBEKE, JHARNOISDERAPS WFAU/ROE CHEYMANS ,LMILLER, JPEACOCK,BJOACHIMI Science member WISE member Science/WISE member TSHANKS MVELANDER CASU Potsdam Bonn HILDEBRANDT, OCORDES, TERBEN, TSCHRABBAC K,KRAHE, EVANUITERT,RNAKAJIMA, MIRWIN SWARREN, WSUTHERLAND PSCHNEIDER, AJAKOBS, DKLAES ATUDORICA, PPLEGATE, ACHOI TKITCHING BVENEMANS PLACERDA YMELLIER,PHUDELOT, RVANDERBURG Munich JSNIGULA, RBENDER, RSAGLIA, SSEITZ Zürich Lyon Padua MRADOVICH Toulouse FGETMAN, AGRADO, MCAPACCIOLI, Naples , NNAPOLITANO MPAOLILLO, EPUDDU, ARIFATTO, GCOVONE, MDALLORA,ZHUANG, CAVUOTI KADAMI SMIESKE, GBECCARI CBLAKE,AROBOTHAM Survey “data-onion” www.astro-wise.org • One system, one approach • orchestration & monitoring, archiving, production, publishing, quality control, science analysis RAW raw Image+catalogs Morphs, lightcurves Users Users Users Survey team • Federated DBase, storage, compute •Survey tasks captured in data model with lineage •Web + Python user interface • 260 users (astro,prod,dev), • ~dozen projects • 80B objects, 2Pb storage, >>45M files “Survey task = query” • Who is my team? • Are we on track? – www.astro-wise.org Survey project progress monitoring Retrieval / OTF processing www.astro-wise.org Calibrate the instrument not the data www.astro-wise.org Improved calibration for smaller GTO surveys Quality Control Quality: flags & timestamps • • • • Target.verify() #automated inspection Target.quality_flags #set by system Target.inspect() # User inspection Target.is_valid=value #set by user 0,1,2 = bad,OK,Qualified - ready for delivery • Calib.timestamp_start,end # validity time range Software (r)evolutions • Updating / expanding methods + attributes • Updating / expanding data model raw RAW Users Users Users Image+catalogs Morphs, lightcurves Users -WISE • WISE recycling: datamodel-centric approach with full lineage. Federation of RDBMS, storage and compute resources. Python/SQL/C(++) • From DRS (Potsdam) to datamodel (WISE) • Quality Control as layer on-top (Toulouse) • Two science harvest machines!! -WISE 5e8 objects at 0.65” 5e6 spectra with an 8m! on Orion Recycling WISE • MUSE: GTO has started • LOFAR: Long-term archive in operation • Euclid: EXT data with Euclid-D (Mission Archive with ESAC) • MICADO: Data Flow System (E-ELT) • Target = Big Data R&D, and visualization for opt/radio astronomy and AI and medicine References / links • Astro-WISE topical issue, Exp.Astro,2013 (19 articles) – – – – • • • • • • Astro-WISE datacentric information system, Begeman et al, 2013 Monitoring OmegaCAM, Verdoes Kleijn et al,m 2013 Data Zoo in Astro-WISE, Verdoes Kleijn et al., 2013 The Kilo-Degree Survey, de Jonh et al., 2013 OmegaCAM, Kuijken et al, 2011 MUSE commissioning, Bacon et al., 2014 Astro-WISE OmegaCEN MUSE consortium KiDS survey
© Copyright 2024 ExpyDoc
