An Application for the Data Grid : A Prototype of the Japanese Virtual Observatory System Masatoshi Ohishi National Astronomical Observatory of Japan and Sokendai [email protected] 2015/9/30 GGF7 in Tokyo Collaborators NAOJ Y. Mizumoto N. Yasuda Y. Shirasaki M. Tanaka Ochanomizu U. Y. Masunaga 2015/9/30 Fujitsu Ltd. K. Monzen K. Kawarai Y. Ishihara H. Yanaka Y. Yamaguchi M. Ishida N. Saito GGF7 in Tokyo Supported by Research Grant from the MEXT （文部科学省 科研費 特定領域研究 「情報学」 松岡・下條組） 2015/9/30 Research Grant from the Sokendai （総合研究大学院大 学 企画研究） GGF7 in Tokyo Orion nebula In the IR 2015/9/30 GGF7 in Tokyo More than 5,000 Galaxies 2015/9/30 GGF7 in Tokyo SUBARU telescope in Hawaii (optical & IR) 2015/9/30 GGF7 in Tokyo 45m Radio Telescope at Nobeyama 2015/9/30 GGF7 in Tokyo Time and Spectral Dimensions The Multiwavelength Crab Nebulae Crab star 1053 AD X-ray, optical, infrared, and radio views of the nearby Crab Nebula, which is now in a state of chaotic expansion after a supernova explosion first sighted in 1054 A.D. by Chinese Astronomers. 2015/9/30 GGF7 in Tokyo Slide courtesy of Robert Brunner @ CalTech. Unknowns of Astronomy Physics just after the Big Bang ? How Galaxies were formed ? Averaged density of the Universe etc. → statistical analyses for homogeneous survey data are crucial 2015/9/30 GGF7 in Tokyo Data Productivity Nobeyama Radio ：~1TB/yr Subaru@Hawaii ：~20TB/yr ALMA（planned） ：~PB/yr Flood of excellent data (survey data) Digitized & Archived 2015/9/30 GGF7 in Tokyo HPC + High Speed NETWORK +GRID Technology → it is possible to connect and seamlessly use remote DBs and computing facilities for data analyses → Virtual Observatory 2015/9/30 GGF7 in Tokyo Telescope is behind me ! 2015/9/30 GGF7 in Tokyo 観 測 アイデア 実望遠鏡での データ 検 解 アイデア 析 研究発表 観測 2015/9/30 索 データ 解 仮想望遠鏡での 観測 GGF7 in Tokyo 析 JVO (will) consists of Seamless Access to Federated DBs Various Data Analysis tools Advanced Analysis such as Data Mining Federation with Real Telescopes → crucial to control data quality in VO 2015/9/30 GGF7 in Tokyo JVO prototype : Ver. 1 Establish seamless access to federated DBs by using the Grid technology Define JVO Query Language, and implement its parser Test environment is in a closed subnet 2015/9/30 GGF7 in Tokyo Query request Web Broser JVO Query Language 利用者独自 サービス JVO Browser Catalog Query Service Commands callable from the GT2 Parse JVO QL into each query sequence JVO Portal ASTRONOMER JVO Viewer Catalog DB Data Archive Service skycat JVO Contoroller Globus Toolokit V2 Data Management DB UDDI Server （FreeSoft) Globus Toolkit Service Registry Security Service Resource Management Data Management GridFTP each query Resolve servers to request queries 2015/9/30 Data GGF7 in Tokyo Data Analysis Services JVO Query Language(1) Catalog Query create view myEROtable as select s.Bmag, s.Rmag, t.Hmag, t.Kmag, ..., sr.BOX(POINT(s.ra,s.dec),w,h) as Rimage, tk.BOX(POINT(s.ra,s.dec),w,h) as Kimage, ... from SUBARU s, 2MASS t, ..., SUBARU.R sr, 2MASS.K tk, ... where XMATCH(s,b,...) < 3 arcsec and (s.Rmag-t.Kmag) > 6 mag and BOX(POINT(ra0,dec0), w0, h0) 2015/9/30 GGF7 in Tokyo and ... Create view with the user specified name in JVO system. Select attributes from each catalog server. Column names can be expressed in UCD. Select cutout images from each image data server. Image area can be specified by BOX or CIRCLE operand. Select the catalog server. Select the image data server. Cross-match distributed catalogs. Query condition based on distributed catalog. Specify search area with the same syntax as cutout image specification. Partition to small segment JVO Query Language(2) Image Query AREA table s t a1 a2 a3 select s.a, t.a, ... from SUBARU.R s, 2MASS.K t, cutout request ... OVERLAP where (s.AREA() OVERLAP t.AREA()) as a s.AREA() SUBARU.R s 2015/9/30 GGF7 in Tokyo t.AREA() 2MASS.K t Redhat Linux JVO Prototype System Architecture GSDL UDDI Maintenance Tool JVO Client Apache Tomcat Netscape Communicator JVO Service (X match) Maintenance Tool UDDI4J soapuddi Apache ＳＯＡＰ ① Applet DownLoad Globus Toolkit (Server) Java2SE 1.4 Java2SE 1.4 Hypertext contents Redhat Linux JVO Service Gb Ether Solaris8 (Possible at Linux) JVO Controller UDDI Registory / Servlet Engine 2Mass DataBase JVO Service (select) • register • update • delete JDBC SQL library Java2 Plugin JVO Client (applet) ③ Find Service ② Request Observation JVO Controller DB Service Java2SE 1.4 PostgreSQL (DBMS) UDDI4J • • • • Image Data SUBARU SupCam z-band UDDI Data get & parse GSDL pre condition check execute Service post condition check ⑤ execute Service ⑤’ execute Service Internet ③ get GSDL Check status ④’ Get GSDL by GridFTP ⑥’ spawn Service GridFTP SQL library JVO Service (select) SQL library JVO Service (image) Image Data 2015/9/30 GSDL ⑥ spawn Service JVO Service (count) Free Software In House Software Commercial Software Function JVO Service (image) JVO Service (X match) Globus Toolkit (Server) Globus Toolkit GSDL Globus Toolkit (Server) JVO Service SUBARU DataBase SUBARU DataBase JVO Service ⑦ issue SQL GSDL (copy) ④ Get GSDL by GridFTP AVO 2Mass DB Redhat Linux Globus Toolkit (Client) NVO PostgreSQL JVO Server Monitoring Globus Toolkit DB Service Image Data (copy) JVO Service (X match) Image Data (copy) The following free software is included by globus toolkit. • OpenSSL (Secure Socket Library) • OpenLDAP (LDAP Server) • wu-ftpd (ftp server) GGF7 in Tokyo JVO Service (select) SQL library JVO Service (count) SQL library ⑦’ issue SQL Copy imate data by GridFTP Copy imate data by GridFTP Image Data (copy) Image Data DB Service ORACLE SUBARU SupCam i-band How it works ? Read JVO QL Create “Work Flow”, and execute each step When necessary, access to the registry to expand the step of the work flow Thus the steps increases dynamically 2015/9/30 GGF7 in Tokyo 2015/9/30 GGF7 in Tokyo 2015/9/30 GGF7 in Tokyo Query Files can be stored in a local PC (WS) 2015/9/30 GGF7 in Tokyo Monitor Window: Refreshed automatically in every 15 seconds 2015/9/30 GGF7 in Tokyo Sample Outputs 2015/9/30 GGF7 in Tokyo 2015/9/30 GGF7 in Tokyo FAQs on our Prototype Q) Contents of environment variables are not passed to the executing machines. A) Prepare a shell script to get environment variables and another script to execute the command. Then GridFTP these shell scripts , and execute the shells in the remote machine. Q) It is impossible to prepare all users’ account to all relevant machines. A) Map all accounts on a portal to a single representative one, such as globus. 2015/9/30 GGF7 in Tokyo FAQs Q) GT2 has too large overhead before starting actual jobs !! They are several seconds. A) For batch jobs, such as database accesses, the overhead could be negligible. However users can never wait for several second for interactive data analyses. Request) Overheads must be < 0.1 seconds excluding the network delays. 2015/9/30 GGF7 in Tokyo Future Prototype version 2 (in progress) implement data analysis tools Interoperation with VOs in the world We have joined the International Virtual Observatory Alliance, and are contributing standardizing protocols for interoperability. 2015/9/30 GGF7 in Tokyo VOs in the World NVO (USA), AVO (EU),AstroGrid (UK) Germany, Canada, Russia, Australia India, China, JAPAN Protocols to interconnect VOs → International Virtual Observatory Alliance (IVOA) 2015/9/30 GGF7 in Tokyo www.ivoa.net 2015/9/30 GGF7 in Tokyo MVC 天文台殿 天体カタログDB MVC サービス サービス 天体カタログ サービス MVC VO用 ブラウザ JVO ポータル Web サーバ セキュリティ管理 VO用 Viewer ワーク フロー サービス サービス 利用者独自 サービス セキュリティ管理 資源管理 データ管理 仮想観測実行 サービス 仮想観測 指示 研究者 天体カタログ 天文台殿 DA/DB サービス サービス DB/DA サービス 資源管理 サービス レジストリA データ管理 セキュリティ管理 資源管理 DB Data データ管理 GRID環境 A サービス レジストリC Web サーバ サービス レジストリB 資源管理 セキュリティ管理 VO用 ブラウザ セキュリティ管理 研究者 VO用 Viewer 2015/9/30 GRID環境 C サービス サービス サービス セキュリティ管理 VO ポータル サービス サービス サービス 資源管理 他システム データ管理 処理 データ管理 資源管理 セキュリティ管理 データ管理 資源管理 データ管理 サービス 他システム サービス サービス サービス pipeline 他システム サービス サービス 処理 処理 他システム 処理 GGF7 in Tokyo GRID環境 B ALMA in Chile (sub-mm wave) 2015/9/30 GGF7 in Tokyo JVO Home Page http://jvo.nao.ac.jp/ 2015/9/30 GGF7 in Tokyo
© Copyright 2024 ExpyDoc
