October, 2013 IBM announced new midrange system Storwize V5000. After the new software version came, the system has begun to support more useful options and configurations. In this article we will make a brief overview of the system and talk a little about some of its features.
We’ll talk a little later about apparatus features of V5000, and now let’s look at the main characteristics of the system.
Basis things (Type – Model: 2072-24С):
- Each controller has 4-core 64-bit Intel CPU, based on Ivy Bridge. Thus, one pair of controllers has 8 cores.
- The amount of memory on the pair of controllers is 16 GB, i.e. 8 GB per controller.
- An access to hosts may be organized in the following external interfaces to block access: FC – 8Gbps, iSCSI – 1Gbps and 10Gbps, FCoE – 10Gbps, SAS – 6 Gbps.
- The standard PCIE V3 is used for Backend IO connection. The throughput of this standard allows to use SAS Backend at the speed up to 6Gbps.
- The maximum number of disk drives per pair of controllers is 480.
- A form factor of the shelve with a pair of controllers is 2U in the 19-inch rack.
- Up to 24 disk drives size 2.5″ can be installed in the shelf with a pair of controllers (2078-24С).
- Up to 12 disk drives size 3.5″ can be installed in the shelf with a pair of controllers (2078-12С).
- Up to 24 disk drives size 2.5″ can be installed in the disk shelf (2078-24E).
- Up to 12 disk drives size 3.5″ can be installed in the disk shelf (2078-12E).
- The Flash disk drives 2.5” sizes 200, 400, 800 GB with 12Gbps SAS interface are supported.
- The SAS disk drives 2.5” sizes 146, 300, 600 GB 15K rpm with 6 Gbps SAS interface are supported (600 GB – 12 Gbps SAS).
- The SAS disk drives 2.5” sizes 600, 900, 1200, 1800 GB 10K rpm with 6Gbps SAS interface are supported (1800 GB – 12 Gbps SAS).
- The SAS disk drives 3.5” sizes 300, 600 GB 15K rpm with 6Gbps SAS interface are supported (600 GB – 12 Gbps SAS).
- The SAS disk drives 3.5” sizes 900, 1200, 1800 GB 10K rpm with 6Gbps SAS interface are supported (1800 GB – 12 Gbps SAS).
- The NL-SAS disk drives 2.5” 7.2K rpm size 1 TB with 6Gbps SAS interface are supported.
- The NL-SAS disk drives 3.5” 7.2K rpm size 2, 3, 4, 6 TB with 6Gbps SAS interface are supported (6 TB – 12 Gbps SAS).
- 12 Gbps SAS ports of new drives work at 6 Gbps as SAS ports on the controllers.
- The different disk drive types with the same form factor can be supported in one shelf.
- One pair of controllers can support up to 19 disk shelves.
- Thin Provisioning technology.
- Hybrid pools with automatic data migration between storage tires (Easy Tier V3).
- External block storage virtualization.
- Local replication (FlashCopy).
- Remote replication synchronous and asynchronous (Remote Mirror).
- 12 Gbps SAS drives are supported with software version 7.4 or above.
Software based on SAN Volume Controller (SVC) code is used in Storwize V5000. The scheme of organization RAID groups and Pools was borrowed from High end systems DS8xxx series. Base platform – Linux.
These features appeared in 7.1:
- Support 10 Gbps iSCSI/FCoE cards for host attachment.
These features appeared in 7.2:
- Data replication to a remote system (Remote mirror) over IP.
- Data migration between hosts which are connected by SAS.
These features appeared in 7.3:
- The memory architecture on the controllers was changed.
- The maximum number of disk shelves per controller pair is 9. The quantity of disks per controller pair is 240.
- Support of 3 tiers in hybrid Pools (EasyTier V3).
Version 7.4 adds
- Support for new SAS drives 12Gbps (NL-SAS 6TB 7K 3.5″ SAS 1,8TB 10K 3.5″ SAS 1,8TB 10K 2.5″)
- The maximum number of disk shelves per controller pair is 19. The quantity of disks per controller pair is 480.
The form factor of V5000 controllers doesn’t differ from that of V3700. Each controller has one PCIE slot that can be used for various cards.
Embedded ports on the controller
- Up to 2 ports Ethernet 1Gbps, (1 port is also used for service needs, initial install and etc). These 2 ports can be used to block access iSCSI 1Gbps protocol.
- Up to 2 USB ports (used for configuration purposes).
- Up to 4 SAS ports on the controller. The first 2 ports are used to connect hosts; the 3d and 4th ports are used for connecting disk shelves. One connector consists of 4 bidirectional serial lanes, each one operates at the speed up to 6 Gbps (max theoretical bandwidth).
Connections of disk shelves
As it was mentioned before, the 3d and 4th SAS ports of the controllers are used for Backend connectivity. A typical connection scheme is shown in the figure below. Note, that 19 disk shelves (19 ESM modules) are connected by chains, so that 9 shelves (9 ESM modules) are connected by chain to the 4th SAS port and other 10 shelves are connected by chain to the 3d SAS port. Thus, we see a slight imbalance in the number of disk shelves between the 3d and 4th SAS ports of the controllers.
PCIE cards for controllers
At the moment, you can install different cards into PCIE slot, these are:
- 4 ports FC 8Gbps card. It comes with Shot Wave SFP 8Gbps transceivers. There is a possibility to use Long Wave SFP 8Gbps transceivers.
- 2 ports iSCSI, FCoE 10Gbps card. It comes with SFP+ transceivers.
The cards with SAS 6Gbps and iSCSI, FCoE 1Gbps are not available. Thus, V5000 has for 2 cards less than V3700.
The table contains possible ports configurations:
|embedded ports||additional ports|
|SAS 6Gbps||iSCSI 1Gbps||FC 8Gbps||iSCSI / FCoE 10Gbps|
Among the new things we want to note that it is now possible to make the replication via Ethernet, using the integrated ports 1Gbps and 10Gbps ports on the cards.
Host connection thru FCoE 10 Gbps is supported only when the network infrastructure is used, i.e. direct connection to the host is not supported.
Besides, there is a recommendation to separate FCoE and iSCSI protocols between different ports.
Key components of the solution
In general, it is worth noting that the system V5000 is not much different from V3700 regarding the use of SAS chips and PCIE switches.
The fact that both systems use the same SAS chips, which are responsible for Backend and SAS Frontend. And the apparatus configuration is changed by software. For example, only one of 4 SAS controller ports is used to connect the disk shelves in V3700, and 2 SAS ports are used to connect the disk shelves in V5000. Therefore V5000 supports 2 times more disk shelves and disks. More powerful processors are used to provide required performance of controllers.
The main difference of V5000 from V3700 is the use of more powerful processors. This allows to have more drives in the system and extend the virtualization of external storage.
The key element of the controller is the use of 4-core CPU Intel Хeon E3-1265L V2, based on Ivy Bridge architecture. Cores frequency is 2,5 GHz. An embedded video core. Support PCE V3. It supports memory with correcting errors (ECC). The processor together with IO chipset (PCH) forms a powerful two-chip platform with broad functionality, low cost and compact arrangement of elements.
The processor supports a sufficiently large number of PCIE ports configurations. We assume that the configuration “4″ is used (2×8, 1×4). In this case, the configuration consists of two ports by 8 lanes each and one port of four lanes. Possible configurations of ports are shown below.
Each controller has a SAS chip PMC-Sierra Inc. PM8018 Tachyon SPCv 16x6G. This chip has a large number of SAS ports. It supports PCIE V3. 8 lanes are used for connection to PCIE fabric. The same SAS chip is used to connect disk shelves (Backend) and hosts (Frontend).
Not less than 2 SAS ports are used to connect the drives in the controllers’ shelf (2 controllers). We assume that the disks in the controllers shelf are connected to SAS chips more than 2 SAS ports, because SAS chips have a large number of ports. This gives increased bandwidth for this disks shelf. Hence follows the recommendation to install high-performance Flash drives in the controller shelf.
The same FC HBA is used to connect hosts by FC, as in the first generation of V7000. This is 4 ports HBA PM8032 Tachyon QE8 designed by PMC-Sierra, Inc., it supports PCIE v2. 8 lanes are used for connection to PCIE fabric.
You can see information about PCI device ids for V5000 here.
Low cost 8 ports PCIE switches PEX8609 PLX technology are used for connection PCIE devices. If you look closely at this information, you can see that low cost 8 ports PCIE V2 switches PEX8609 PLX Technology are used for some connections of PCIE devices.
We assume that one of these switches is used to exchange data between controllers. In this case, the port x4 PCIE V3 on the processor is connected to the switch PCIE V2. Therefore, the bandwidth between the controllers must not be greater than the rate of 4 lanes PCIE V2 (2 GBps teoretical and 1,6GBps more real). This indicates that the writing speed does not exceed these values, because it is necessary that the data are copied to another controller.
The rest 2 switches PEX8609 PCIE V2 should be used to connect FC HBA PM8032 via x8 PCIE V3 port to CPU.
Cache memory architecture and hybrid pools EasyTier V3
The advent of the 7.3 version has changed the cache memory architecture and has provided an ability to use hybrid pools with 3 tiers. You can find more detailed information about these matters in the article «IBM Storwize V7000 Gen2».
We finish our review with a small comparative table:
|Max drives per controller pair (IO Group)||480|
|Memory per controller, GB||8|
|Memory per controller pair (IO Group), GB||16|
|CPU type||Xeon E3|
|CPU per controller||1|
|CPU core q-ty||4|
|CPU core clock, GHz||2,5|
|Max FC 8Gbps ports per controller (FE)||4|
|Max FC 8Gbps ports per controller pair (IO Group) (FE)||8|
|Max iSCSI 10Gbps ports per controller (FE)||2|
|Max iSCSI 10Gbps ports per controller pair (IO Group) (FE)||4|
|Max SAS 6Gbps ports per controller (FE)||2|
|Max SAS 6Gbps ports per controller pair (IO Group)||4|
|Max Disks per RAID Group||16|
|Max Disks per Pool||128|