[Cinder] Support LVM on a shared LU July/2/2014 Mitsuhiro Tanino <[email protected]> 0 1-1. Big Picture Control Node Cinder node API Compute Nodes Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler Proposed LVM volume Driver qemu-kvm qemu-kvm Attach/Detach via qemu-kvm Attach/Detach via qemu-kvm Volume operations /dev/sdx (LU1) VG1 LV1 LV2 LV3 LV4 FC Storage Storage Controller FC Storage LU1 VG1 LV1 LV2 LV3 LV4 /dev/sdx (LU1) VG1 LV1 /dev/sdx (LU1) LV2 LV3 LV4 VG1 LV1 LV2 LV3 LV4 I/Os from VMs on compute nodes • Volume operations are implemented by using LVM, which is light weight. •I/Os are issued to a storage controller, which has enough scalability. 1 1-2. Preparation by administrator Control Node Cinder node API Compute Nodes Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler Proposed LVM volume Driver /dev/sdx (LU1) qemu-kvm /dev/sdx (LU1) qemu-kvm /dev/sdx (LU1) FC Storage Storage Controller FC Storage LU1 (1) Set up FC connectivity between storage, control and compute nodes. (2) Create LU1 on FC storage. (3) Attach the LU1 to control and compute nodes as a shared LU outside of OpenStack. After (3), LU1 appears in all nodes as a /dev/sdx. 2 1-2. Preparation by administrator Control Node Compute Nodes Cinder node API Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler /dev/sdx (LU1) /dev/sdx (LU1) VG1 VG1 qemu-kvm qemu-kvm Proposed LVM volume Driver /dev/sdx (LU1) VG1 FC Storage Storage Controller FC Storage LU1 VG1 (4) Create VG1 on top of LU1 at control node. (5) Scan LVM metadata on LU1 on each node. As a result, VG1 appears in all nodes. (6) Edit cinder.conf and use VG1 as a “volume_group” for proposed LVM driver. 3 1-3. Work flow of volume creation and attachment Control Node Compute Nodes Cinder node API Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler qemu-kvm qemu-kvm Proposed LVM volume Driver Create/Delete Extend/etc /dev/sdx (LU1) VG1 LV1 /dev/sdx (LU1) VG1 /dev/sdx (LU1) VG1 FC Storage Storage Controller FC Storage (1) Create LV1 on the top of VG1 at control node. LU1 VG1 4 1-3. Work flow of volume creation and attachment Control Node Cinder node API Compute Nodes Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler Proposed LVM volume Driver qemu-kvm qemu-kvm Create/Delete Extend/etc /dev/sdx (LU1) /dev/sdx (LU1) VG1 LV1 VG1 LV1 /dev/sdx (LU1) VG1 FC Storage Storage Controller FC Storage LU1 VG1 LV1 (2) LV1 is written to the shared LU. (3) Before volume attachment, scan LVM metadata on LU1 at compute1. As a result, LV1 appears in compute1. 5 1-3. Work flow of volume creation and attachment Control Node Cinder node API Compute Nodes Nova Compute1 Proposed fix Nova Compute2 Proposed fix VM1 VM4 VM2 VM3 VM5 VM6 Scheduler Proposed LVM volume Driver /dev/sdx (LU1) VG1 LV1 VG1 LV1 Attach/Detach via qemu-kvm Attach/Detach via qemu-kvm Create/Delete Extend/etc /dev/sdx (LU1) qemu-kvm qemu-kvm /dev/sdx (LU1) VG1 FC Storage Storage Controller FC Storage LU1 VG1 LV1 (4) Activate LV1 at compute1. => After activation, access device path /dev/VG1/LV1 is created at compute1. (5) Attache LV1 to VM1. => Libvirt and qemu can handle device path and attach a volume using this. 6 2. Target of Proposed LVM volume driver There are many kinds of Cinder plugins available. DFS Softwarebased File system based Block based Cinder Plugins GlusterFS, GPFS, RBD(ceph) NFS LVMiSCSI Proposed LVM volume Driver Fibre Channel Hardwarebased iSCSI NFS Vendor specific drivers, such as, Dell, EMC, HP, Hitachi and etc … 7 3. Benefit of Shared LVM driver Reduce hardware based storage workload by offloading the workload to software based volume operation. In OpenStack cloud environment, workloads of storages have been increasing than usual and it is difficult to manage the workloads because every user have a permission to execute storage operations via cinder. In order to use expensive storage more efficiently, I think it is better to reduce hardware based storage workload by offloading the workload to software based volume operation on a case by case basis. Using these two drivers to one enterprise storage, cloud provider can provide volumes both way as the situation demands. For example, As for "Standard" type storage, use proposed software based SharedLVM driver. As for "High performance" type storage, use hardware based cinder driver. (Ex. Higher charge than "Standard" volume) 8 3. Benefit of Shared LVM driver Enable cinder to any kinds of shared storage volumes without specific cinder storage vendor driver. User can apply this driver to a storage which does not have specific cinder driver or does not have features such as efficient snapshot, backup. Provide quicker volume creation and snapshot creation without storage workloads. LVM has lightweight volume creation. And also lightweight snapshot using copy on write. Better I/O performance using direct volume access via Fibre channel. This driver can issue I/O without scsi target, therefore performance is better compared to LVMiSCSI driver. I/O bandwidth and latency are almost same as raw FC volume.(See test result at P13 and P14) 9 4. Comparison of Proposed LVM volume driver Proposed LVM Proposed LVM volume Driver FC (Vendor Driver) Implementation LV (managed by LVM) of volume LV LU (managed by LVM) (managed by storage) Volume Operation By software By software By hardware (LVM) (LVM) (Storage) LVMiSCSI Work Load Supported Storage Server side Any storage Any storage (Storage independent) (Storage independent) Volume Access Via software Path iSCSI target Performance HA Server side Direct from fibre channel Better I/O performance Less volume operations to storage. Storage side Specific storage (Requires specific plugin) Direct from fibre channel Better I/O performance Better support coverage. Better I/O performance. Active/Passive：〇 Active/Passive： 〇 Active/Passive：〇 Active/Active ： ー Active/Active ： ー Active/Active ：〇 A/A is not supported. 10 4. Comparison of Proposed LVM volume driver LVMiSCSI Admin work - Initial setup Proposed LVM Proposed LVM volume Driver FC (Vendor Driver) 5 steps 5 steps 4 steps (a) Setup network connectivity for all nodes. (b) OS and OpenStack setup (c) Create a storage volume and attach it to cinder node. (d) Create volume group on top of storage volume (e) Configure cinder.conf (a) Setup network connectivity for all nodes. (b) OS and OpenStack setup (c) Create a storage volume and attach it to cinder node. (d) Create volume group on top of storage volume (e) Configure cinder.conf (a) Set up FC connectivity between storage, cinder node and compute nodes. (b) OS and OpenStack setup (c) Create storage pool in the storage. (d) Configure cinder.conf Volume group creation step is required. 11 4. Comparison of Proposed LVM volume driver LVMiSCSI Admin work add new server Proposed LVM Proposed LVM volume Driver FC (Vendor Driver) 2 steps 3 steps 2 steps (a) Setup network connectivity for new node. (b) OS and OpenStack setup (a) Set up FC connectivity between storage and new compute node (b) Set up an access control to a Shared LU from new node. (c) OS and OpenStack setup (a) Set up FC connectivity between storage and new compute node. (b) OS and OpenStack setup Access control(FCZone or Shared LU) is required 12 5. Comparison of I/O performance - Following results are I/O performance of issuing direct I/O from single instance to single volume. Bandwidth of I/O [O_DIRECT] Unit: [KB/s] I/O size LVMiSCSI SharedLVM Raw FC volume Read 1M 49788.0 85146.0 88203.0 512K 62382.0 160517.0 157810.0 4K 4026.4 8630.8 8865.2 1M 76691.0 141315.0 144173.0 512K 59200.0 142408.0 144006.0 4K 3870.1 7738.9 7867.1 Randread 1M 16152.0 17665.0 17105.0 13677.0 20397.0 19971.0 4K 417.3 480.6 476.8 Randwrite 1M 15606.0 17067.0 16526.0 13666.0 20381.0 19955.0 417.1 4803.3 476.5 Write 512K 512K 4K I/O performance of SharedLVM and Raw FC volume are almost same. *Larger is good. 13 5. Comparison of I/O performance Average latency of I/O [O_DIRECT] Unit: [msec] I/O size LVMiSCSI SharedLVM Raw FC volume Read 20.56 12.02 11.60 512K 8.20 3.18 3.24 4K 0.99 0.46 0.45 1M 13.35 7.24 7.10 512K 8.64 3.59 3.55 4K 1.03 0.51 0.50 Randread 1M 50.47 52.31 54.22 28.75 21.58 22.13 4K 8.38 7.76 7.84 Randwrite 1M 13.37 5.84 5.83 512K 8.68 3.51 3.50 4K 1.42 0.75 0.73 Write 1M 512K Average latency of SharedLVM and Raw FC volume are almost same. *Smaller is good. 14
© Copyright 2024 ExpyDoc
