DDN & iRODS iRODS User Group Meeting Cambridge, MA June 18, 2014 David Martin WOS Product Line Manager Storage should improve collaboration … Not make it harder ▶ Distributed, not centralized ▶ Minutes to install, not hours ▶ Milliseconds to retrieve data, not seconds ▶ Replication built in, not added on ▶ Instantaneous recovery from disk failure, not days ▶ Built in data integrity, not silent data corruption ©2013 DataDirect Networks. All Rights Reserved. ddn.com DDN is the World’s Largest Privately Held Data Storage Company Key Statistics Industry Validation Overview: World’s Largest Privately-Held Storage Co. Providing the data backbone for the world’s most extreme, data-intensive environments – enabling organizations across multiple industries to maximize the value of their information globally. Fast500 Technology Company • Established: 1998 • Headquarters: Chatsworth, California USA • Over 1,000 Customers in 5 Continents Key Industries: • High Performance Computing & Life Science • Cloud & Web Content • Rich Media • Intelligence/Federal Inc. 500|5000 High-Growth Company Best HPC Storage Product Best Practice for Digital Media World Class Set of Global Customers Go To Market: • Global Partners, VARs, Resellers ©2013 DataDirect Networks. All Rights Reserved. ddn.com Big Data & Cloud Infrastructure DDN Announced & Pending Product Portfolio Infinite Memory Engine™ Distributed File System Buffer Cache* [Demo] Cloud Tiering Petascale Lustre® Storage Analytics Reference Architectures Enterprise Scale-Out File Storage Cloud Foundation S3 Big Data Platform Management EXAScaler™ GRIDScaler™ 10Ks of Clients 1TB/s+, HSM Linux HPC Clients NFS & CIFS ~10K Clients 1TB/s+, HSM Linux/Windows HPC Clients NFS & CIFS WOS® 3.0 32 Trillion Unique Objects Geo-Replicated Cloud Storage 256 Million Objects/Second Self-Healing Cloud Parallel Boolean Search Storage Fusion Architecture™ Core Storage Platforms DirectMon SFA12KX SFA7700 48 GB/s 1.7M IOPS 1,680 Drives, Embedded Option 12.5 GB/s 450K IOPS 60 Drives in 4U; 228 Drives in 12U Flexible Drive Configuration SATA SAS SSD SFX Automated Flash Caching Adaptive cache with application & filesystem support for prestaging, alignment & by-pass WOS7000 60 Drives in 4U Self-Contained Servers * Technology Demonstration 4 ©2013 DataDirect Networks. All Rights Reserved. ddn.com High Performance Data Center With iRODS tie-in Users DATA CENTER IME High speed processing GRIDScaler Scalable storage engine WOS Live Archive Storage Public Cloud WOS Bridge Tape Archive 5 ©2013 DataDirect Networks. All Rights Reserved. ddn.com IRODS Data Grid & WOS Made for Big Data • Hyperscale • Accessible • Distributed • Collaborative • Secure • Flexible DDN products assure data longevity with low latency over geographies and iRODS adds the important policy management & accessibility layers ©2013 DataDirect Networks. All Rights Reserved. ddn.com WOS Key Feature Breakdown Federated, Global Object Storage Namespace • Up to 256 nodes per cluster • Across 64 geographies, • Replicate data with smart policies for performance and/or storage efficiency on a per-object basis Flexible Data Protection Supports any combination of replication, erasure coding & Distributed Data Protection DeClustered Data Protection • No hard tie between physical disks & data. • Failed drives are recovered through dispersed data placement • Rebuilds only data, not drives Latency-Aware Access Manager WOS intelligently makes decisions on the best geographies to get from based upon location access load and latency REPLICATION Self-Healing Architecture • Protection from node, Disk & Data Center failure • Completely automated Fully-Integrated Object Storage Appliance • 60 Drives in 4U • WOS7000 nodes support innode failover for ideal HA Object Disk Architecture • NOFS-no Linux file I/Os, no fragmentation, • Fully contiguous object read and write operations for maximum disk efficiency ©2013 DataDirect Networks. All Rights Reserved. User Defined Metadata and Metadata Search • Applications can assign their own metadata via object storage API • Supports batch search of user metadata ddn.com Why DDN WOS & iRODS are perfect complements WOS extends iRODS by… ▶ WOS makes iRODS more performant & scalable • Single iCAT DB entry references multiple WOS copies • iRODS retrieves lowest latency WOS object instance ▶ WOS enables iRODS users to implement better QOS • High speed access to distributed project data • Long tail data can be stored on deep archive WOS Nodes ▶ WOS dramatically improves iRODS storage efficiency & availability • Self healing – at local node SAS bus speeds which increases data durability • Highly available – no interruption if disk, node, or site failure • WOS implements highly efficient Global Object Assure • Lowest TCO in the industry, <1 FTE for multi petabyte multi-site deployments iRODS extends WOS by… • Integration with heterogeneous storage environments (multi-vendor disk & tape) • Additional clients and access methods • Richer storage rules and migration policies 8 ©2013 DataDirect Networks. All Rights Reserved. ddn.com iRODS Standard Operation iRODS Operation • iRules determine file store locations • iCat maintains file location metadata • All retrieves get primary file location info from iCAT (all retrieves use primary) Store & Retrieve Operation • iRule policy determines where file is stored iRODS Zones iRule iRODS Server .. Replicas: BOS, SF iRODS Server RAID BOS iRODS Server .. • FileA stored in BOS is replicated to SF automatically by iRODS • iCat maintains file location metadata iRODS Server ICAT DB Paris • File Access always reads primary copy • FileA read in SF or Paris will go to BOS server (primary) iRODS Server .. iRODS Server JBOD ICAT DB SF ©2013 DataDirect Networks. All Rights Reserved. ddn.com WOS + iRODS Integration WOS Location Intelligence WOS as a iRODS Cloud resource iRODS Zones • WOS Cluster is defined as IRODS “Cloud Resource” • Multiple WOS replicas stored once in iCAT • iRODS utilizes WOS data location intelligence iRODS Server Site 1 iRule .. Replicas Cloud iRODS Server RAID File store / retrieve operation ICAT DB BOS • iRule policy specifies cloud storage method • FileA stored in SF writes to closest node of WOS Cluster • WOS replicates file to BOS per WOS storage policy • Single entry in iCAT DB iRODS Server • FileA read in BOS will retrieve from BOS WOS node • FileA read from Paris will retrieve from closest location (normally Boston) iRODS Server iRODS Server ICAT DB WOS Cluster • File read always reads from closest WOS location .. Paris .. iRODS Server JBOD ICAT DB SF iRODS optimizations for WOS is available today ©2013 DataDirect Networks. All Rights Reserved. ddn.com Power of WOS Location intelligence Minimize Latency & Enable True Collaboration Data Locality & Collaboration 1. LA site user edits video “A”, which replicates to 1 Mexico City & Tokyo based on policy 2 2. MP Gateway immediately synchronizes iCat metadata DB with Tokyo user 3. Tokyo user requests video “A” for processing, Tokyo 3 WOS-Lib selects Tokyo site (lowest latency) & Latency Map retrieves for the user 80 4. The Tokyo user extracts frames from the video & 4 ms writes to WOS (new object), which replicates to 10 30 Mexico City & LA & synchronizes the iCAT DB’s ms ms San Los Francisco Angles Mexico City New York Tokyo London Tokyo Real Time Editing App IRODS Server Los Angeles Latency Map WOS-Lib Los Angles User 80 ms 10 ms 40 ms 1 2 4 4 33 10.8.24.101 10.8.26.101 10.8.25.101 . . . . . . 10.8.24.105 B A 10.8.25.105 A A Zone Los Angeles B Zone Mexico City B 10.8.26.105 A Zone Tokyo Cluster “Acme WOS 1” iRODS Server A Real Time Editing App Tokyo User Best viewed in presentation mode ©2013 DataDirect Networks. All Rights Reserved. WOS-Lib ddn.com WOS Replication Replication delivers both multi-site collaboration + data & disaster protection ▶ Select either Asynchronous or Synchronous replication per policy • Definable on a per-object basis ▶ ▶ Synchronous Replication replicates to remote sites to insure data protection before acknowledging a successful “write” to the application Asynchronous replication to remote WOS nodes increases performance • Two copies of data are always written to “local” (lowest latency) node before “PUT” call returns OID to minimize/eliminate risk of data loss • Replication occurs in background, once replication successfully completes, extra local copy is removed • Especially useful for big files and big data sets ©2013 DataDirect Networks. All Rights Reserved. ddn.com WOS Local & Replicated Object Assure WOS Replicated OA (ROA) WOS Local OA (LOA) Client App `` Asynchronous Replication WOS-Lib Data Center 1 “PUT” “GET” Data Center All shards Failure stored locally 1 229 8 Writes Reads @ LAN line speed 1 2 3 4 P1 5 6 7 8 P2 1 Data Center 2 Splits files into 10 shards 1122229910 8 … 1 12 22910 910 WAN speed ONLY on read failure Application Servers Local Object Assure: • Erasure-code based declustered data protection • 8+2 (Data + Parity) for single copy protection • Each WOS node can withstand up to two concurrent drive failures without loss of data or availability • Local Data rebuilds ©2013 DataDirect Networks. All Rights Reserved. Replicated Object Assure • • • • DR for OA storage methods Sync & Async replication supported Optimized for large objects Two OA replicas provide equal protection as 3 standard replicas NDA Confidential Information ddn.com WOS Global Object Assure & Replicated Object Assure WOS Global Object Assure Data Center 2 Data Center 1 WOS OA Replicated Data Center 3 Data Center 1 Data Center All shards Failure stored locally Stores single data copy across 3 WOS zones 1 2 3 15 1617 18 Dual encodes files & spreads across 3 data centers Application Servers Asynchronous Replication 1 229 8 1122229910 8 Writes Reads @ LAN line speed Splits files into 10Shards across 2 data centers … 1 12 22910 910 Application Servers The Net-Net - Ultimate customer flexibility Global Object Assure • A minimum of 3 data centers must be deployed to provide disaster protection • High storage efficiency • Data rebuilds from drive failures occur @ SAS Bus speeds 14 ©2013 DataDirect Networks. All Rights Reserved. Data Center 2 WAN speed ONLY on failure Best viewed in presentation Mode Fastest LAN Path = Performance (unless DR scenario) • Two data centers required for full DR • Store/retrieve data @ full LAN line speeds • Data replicates to remote asynchronously @ WAN Speeds • Only retrieve data over slower WAN in DR scenarios ddn.com DDN-IRODS Summary Data Center View Applications and Users IME Bridge IME Bridge PFS Performance Optimized Tier PFS Namespace Federation ES/GS Bridge Geo-location & Collaboration (Active Archive) WOS TSM /ACSLS TSM / HPSS Data Center 1 15 Bridge WOS Capacity Optimized (Deep Archive) ©2013 DataDirect Networks. All Rights Reserved. ES/GS Additional Data Centers Data Center 2 ddn.com Building a DDN & iRODS storage Grid Replication & Multi-site Collaboration Karlsruhe, DE Seattle Austin Tx (UT) ©2013 DataDirect Networks. All Rights Reserved. UNC (RENCI) ddn.com Summary WOS & iRODS WOS – iRODS Integration" User • Petabyte scalability: Scale out by simply adding storage modules" Can Search, Access, Add and Manage Data & Metadata iRODS Rule Engine iRODS Data System Track policies • Unrivaled Simplicity: Management simplicity translates directly to lower cost of ownership" iRODS Metadata Catalog Track data WOS Object Supervisor Policy-‐driven Replication • Rapid Rebuilds: Fully recover from lost drives in moments" • Replication Ready: Ingest & distribute data globally" WOS Object Store WOS Data Locality Mgr • Self-Healing: Zero intervention required for failures, automatically recovers from lost drives" Global Data Management • Disaster Recoverable: For uninterrupted transactions no matter what type of disaster occurs" • File Layout: Capacity and performance optimized" • Object Metadata: User-defined metadata makes files smarter" ©2013 DataDirect Networks. All Rights Reserved. ddn.com
© Copyright 2024 ExpyDoc
