This dispatch covers Microsoft Exchange Solution Review Program (ESRP)[1] performance results for the over 5000 mailboxes results category.  Prior reports discussed the over 1000 to 5000 mailboxes and the under 1000 mailboxes result categories[2].

ESRP was never intended to compare subsystem performance but rather as a proof of concept for Microsoft and storage vendors to depict a configuration supporting a given workload.  Hence, any comparisons necessarily come with some caveats and may not be real.  Nonetheless, SCI feels comparisons can well serve both the vendor and end-user storage community and thus, worth noting.

Latest ESRP V2.0 results

The first chart refers to aggregate average database backup throughput across all storage groups.  This value correlates moderately to aggregate database transfers per second.  (See figure 1).

(SCIESRP091030-001) (c) 2009 Silverton Consulting, All Rights Reserved

As can be seen from Figure 1, backup activity is dominated by HDS and HP (an HDS OEM).  Partly HDS’s commanding result is due to the overall number of drives and mailboxes being serviced.  However, 3PAR’s 96Kmbx and Dell’s Equal Logic iSCSI result for 90Kmbx results only came in at number five and seven respectively.  Also, the number nine result for Pillar Data’s Axiom 600 supports only 34K mailboxes and stands almost as well as the other results supporting many more mailboxes.  A couple of caveats worth noting here database backup performance can be impacted by

• Number of disk drives in a configuration
• How message store databases are split across those spindles
• Subsystem RAID level

(SCIESRP091030-002) 2009 (c) Silverton Consulting, All Rights Reserved

This chart shows ESRP reported database latencies for read, write and log write operations.  SCI and others feel the read latency metric best shows what an end-user experience would be from a subsystem configuration.  The list is sorted by read latency.  New results from Pillar Data with a read latency of 3ms and Sun’s J4400 at 8ms showed up very well here.  A couple of considerations to note:

• While read latency is unaffected by replication mode, write and log write latency can be seriously impacted by how the Exchange database is replicated.  For example if one examines the EMC SRDF/S in the number two position, its write latency is pretty high.  However if one considers that SRDF/S was active this means the data has to be written to the secondary subsystem in parallel to being written to the primary subsystem and as such its write latency does not look that bad.
• There are a couple of ways to impact or game this value. One easy way is to reduce the overall load on the storage.  As ESRP reports are intended to show a viable performing solution to handle a simulated user workload we assume that these products are all optimizing cost and performance, so believe this is not an issue here.
• For an ESRP benchmark to be accepted, read latency must be under 20 msecs.  Some vendors may try to push read latency out closer to 20msecs in order to support more mailboxes with less hardware.  As such, those vendors may not show up well on this chart.

(SCIESRP091030-003) 2009 (c) Silverton Consulting, All Rights Reserved

The new result for Pillar Data broke into the top 10 log playback time.  The top 4 systems LSI 399x, Pillar Data Axiom 600, Lefthand SANIQ, and IBM DS4800 would all be considered mid range storage subsystems although HP’s Lefthand was supporting a heavy workload at 50K mailboxes and was configured accordingly.   The range for the top ten subsystems is fairly large, over 2X from lowest to highest.  It’s unclear how one succeeds in this metric other than having fast disk and low latency database operations.  Similar to the backup discussions above, playback performance can be impacted by:

• Number of disk drives in a solution,
• How message store databases are split across those spindles,
• Subsystem RAID level,
• Replication type

(SCIESRP091030-004) (c) 2009 Silverton Consulting, All Rights Reserved

Finally, we now turn to normalized database transfer results and the Top 3 normalized ESRP/Jetstress results belong to HDS AMS2100, HDS AMS2300 and IBM XIV, all midrange systems. A few considerations are warranted on normalized results:

• Normalized results do not always scale well.  Although four of these results were for 20,000 mailboxes or over, (HDS USP-V, Sun StorageTex 6540, 3PAR Inserv T800, and EMC Symmetrix at 60KMbx) the top result from HDS supported only 5400 mailboxes and may not scale much beyond that quantity of mailboxes.
• One surprise here is the close running of everyone behind the top five results and may be an artifact of the ESRP benchmark striving to generate equivalent workloads per user mailbox.   But the workload simulated for these results varied considerably (0.3 to 1.0 iops/sec/mbx).

Conclusions

From our perspective, ESRP results in this over 5K mailbox tier are getting more competitive.  There were a number of new ESRP results in this category over the last 9 months, and at least 3 over the last quarter.  Seeing HDS, EMC and IBM highend systems, all running the same performance tests is a good indicator of their willingness to show their products in the best light as well as high customer interest in Exchange solutions.  Most likely, these vendors do not see their individual results as entirely comparable and arguably they may have a point, but we differ with them on this assessment.  Moreover, seeing DAS (HP MSA), SATA (IBM XIV), SAS (Sun, HDS AMS2100, & HP MSA), iSCSI (EMC, HP, NetApp, & Dell) and FC storage subsystem results compete in this top-tier at least on various metrics indicates to us that all storage interfaces can be competitive in mission critical applications.

ESRP/Jetstress results are inherently difficult to compare.  Nonetheless we believe Exchange results provide a unique real world benchmark and deserve some comparison so that the public can make properly informed storage purchases.  Our next ESRP/Jetstress report will return to the 1K to 5K mailbox tier.  We continue to welcome any feedback on how to do better.

This performance dispatch was sent out to our newsletter subscribers in October of 2009.  If you would like to receive this information via email please consider signing up for our free monthly newsletter (see subscription request, above right) or subscribe by email and we will send our current issue along with download instructions for this and other reports.  Also, if you need an even more in-depth analysis of SAN storage system features and performance please take the time to examine our SAN Storage Briefing available for purchase from our website.

A PDF version of this can be found at

Silverton Consulting, Inc. is a Storage, Strategy & Systems consulting services company, based in the USA offering products and services to the data storage community

[1] ESRP results from http://technet.microsoft.com/en-us/exchange/bb412164.aspx, as of 27 January 2009

[2] All of prior SCI ESRP Dispatches can be found at ofhttp://www.silvertonconsulting.com/page2/page2d/storage_int_dispatch.html