3PAR StoreServ for work in a corporate mail environment
Microsoft Exchange 2010 is the most common and popular corporate messaging product in the Windows environment. Quite often, our customers have a question in the selection of the necessary configurations for mail servers and data storage systems. In this article I will try to tell about the features that have appeared in the product Microsoft Exchange 2010, what requirements must be considered when calculating the specifications, and what solutions HP has.
The Exchange 2010 product has several features related to replication mechanics and fault tolerance.
Exchange Server is tightly integrated with Active Directory: most user data is stored in Active Directory (link between user accounts and mailboxes, contact lists). Only mailboxes themselves are stored separately from Active Directory. Thanks to the Active Directory replication mechanism, when multiple Microsoft Exchange Servers are used, the data on all servers remains current.
To implement fault tolerance in Exchange 2010, a single DAG technology is offered, the former SCR, LCR, CCR technologies and the single storage cluster have been removed. DAG (Database Availability Group) - technology of asynchronous database replication between nodes with protection against data loss. DAG is not a pure cluster technology, since no virtual shared node. In this regard, the client connection point was moved from the Mailbox role to the Client Access role. In a DAG, only the base is clustered, which can move between servers included in the DAG. But the Windows Clustering service is still used to determine the quorum. DAG can work only on block devices (local disks and SAN) and cannot work on NAS network drives.
Based on the features of the Exchange 2010 architecture, as well as the advent of more efficient hardware (this applies primarily to processors), the approach to architecture sizing has changed.
')
Microsoft publishes on its website a series of documents that describe the role of each component of the solution. Understanding the many articles is quite a non-trivial task. The following are theoretical sizingings.
For a productive environment, you need a processor that works with the 64-bit version of Exchange 2010 on Windows Server 2008 or 2008 R2. Microsoft recommends using Intel processors that support Intel Extended Memory 64 Technology or AMD processors that support AMD64.
The use of the Hyperthreading option requires careful planning, and the effect of deployment in the Exchange environment is usually not significant.
By default, the Hyperthreading option should be disabled. Use is only possible in the form of a temporary measure to improve processor performance until the additional hardware is commissioned.
Below is a table with the minimum and recommended maximum settings for Exchange 2010.
The recommended Microsoft configuration is 8 processor cores, for an environment where several servers and several thousand mailboxes are supposed to be used. A server with a large number of cores can be effectively used when the Hub Transport role is configured to work with anti-virus and anti-spam software. The load factor on the processors depends on such indicators as the average message size, the number of forwarded messages, the number of involved transport agents, the configuration of the antivirus and other applications.
In Microsoft Exchange 2010, most client functions were moved from the Mailbox Server role to the Client Access Server role. In the Exchange 2010 environment, messages are processed on the Client Access server if they are sent by a non-MAPI client (for example, POP3 or IMAP4). In addition, Microsoft Office Web App is rendered by the Client Access server role, and not by the Microsoft Exchange Information Store service, as in previous versions of Exchange.
These architectural features allow the Client Access server to seriously unload the Mailbox server and effectively use 8 processor cores. For organizations with a small number of mailboxes or with a low volume of traffic from non-MAPI clients, it is recommended to use 2 cores.
Microsoft recommends the configuration of a Unified Messaging Server with 8 processor cores. Such a number of cores is necessary for the operation of voice mail, for example, the conversion of the .wav format to Windows Media Audio (WMA). A server with 2 cores can be used for organizations with a small number of mailboxes or in an environment with a small activity of the Unified Server Messaging role.
Microsoft recommends configuring this role based on the number of mailboxes and user profiles. 4 processor cores give a good balance between performance and cost, so many cores can handle up to several thousand mailboxes. Resource assessment requires an understanding of the average user profile. This image can be obtained using transport performance counters. The use of third-party utilities is also allowed.
The choice of RAM is determined by the following values
Exchange 2010 on the Windows Server 2010 operating system efficiently works with RAM up to 64 GB per server.
The table below shows the minimum and maximum values ​​for each role.
These roles do not require a large amount of RAM, at least 4GB is enough to process all requests, the optimal value is 1GB per core, but not less than 4 GB in total.
The consumption of RAM in this role is almost linearly dependent on the number of client connections and the number of transactions. Based on Microsoft recommendations, the role configuration from 2GB per core is balanced.
This role is quite complicated in the calculations, because The optimal parameters of RAM depend on the number of deployed roles, the number of mailboxes, the average user profile and the number of active databases.
Memory calculation of this role is critical for reducing the I / O load on the server disk subsystem.
Exchange 2010 uses a mechanism to reduce the load on the disk subsystem of the server or data storage, based on the use of the Extensible Storage Engine (ESE) cache. The more cache available to the Exchange 2010 server, the lower the I / O load on the disk subsystem.
Cache performance in Microsoft Exchange 2010 has been enhanced during several technical changes. One of the most significant changes is the increase in the depth target check log. This log is used to ensure that the logs and database cache are written to the database within reasonable time limits. The size of the log depends on the number of copies and the type of database:
Using the Extensible Storage Engine, the I / O load for a database with two or more copies can be up to 40% lower than for a single database configuration. In situations with high load on the disk subsystem, the database stores change files in memory for a long time, this behavior reduces the number of sequential write operations (repeated write) and creates a merging effect (coalescing), the so-called hybrid storage model. This allows you to quickly restore the database in case of malfunction. At the same time, inactive copies of databases also use memory to store a log (5MB), this allows you to quickly translate an inactive database into active mode.
The cache size, and, accordingly, the amount of physical memory directly depends on the number of databases and can be calculated from the table:
In previous versions of Echange, the key metric was the volume of I / O operations, now the current metrics are the volume of the database cache and the number of messages sent by the user per day.
The following table allows you to determine IOPS, based on the volume of the box 2GB and an average letter size of 75KB.
In Exchange 2010, the same server can handle both active copies of databases and inactive ones in a configuration with a fault-tolerant solution. Processor requirements for processing inactive copies should also be considered at the design stage. Inactive database copies use processor resources to check replicated logs and process the database index. Each mailbox of an inactive database requires 15% of the CPU resources of the mailbox resources of the active database.
Each Exchange 2010 database can create up to 16 copies. Each copy of the database consumes 10% of the CPU from the main database.
Understanding the number of copies and the number of inactive databases is necessary in order to be able to create several models of the behavior of the Exchange 2010 environment depending on external conditions (for example, on the number of failed servers or platforms). Typical models for the Exchange 2010 environment are the model with 100% use of all mailboxes and the behavior model for the worst case scenario.
The planning steps are as follows:
The megacycles parameter is determined by the processor's core performance in MHz, for example, 2 x 4 core Intel Xeon x5470 3.33-gigahertz (GHz) is 3333 megacycles per core.
The choice of storage system depends on many factors, but is primarily determined by the number of mailboxes and the requirement for expansion.
For smaller organizations with 1,000 mailboxes, HP and Microsoft recommend the use of an entry-level P2000G3 iSCSI storage or ProLiant disk subsystem .
For large installations, high-level storage is recommended, supporting a large number of disks. One of these arrays is 3PAR StoreServ, announced on December 4th, 2012. This array has already been tested in the Exchange 2010 Solution Reviewed Program (ESRP) - Storage v3.0.
As you know, 3PAR arrays are intended for use under virtualization tasks in an environment with unpredictable loads. Another possible destination is corporate mail.
In conjunction with Microsoft, a 3PAR StoreServ array of mail for 60,000 users was tested. The solution used HP Proliant Bl460c Gen8 blade servers, C7000 blade baskets, Brocade optical switches.
Test software: Microsoft Windows Server 2008 R2 and Microsoft Exchange 2010.
The document provides information about testing an array in an environment of 60,000 users.
High performance is organized through the use of wide-striping technology (using all drives to build fault-tolerant RAID) on a 3PAR array using the Database Availability Group (DAG), where each copy of DAG is served by 18 servers (9 active servers are served by 60,000 users in 2 DAGs). High data availability can be ensured by organizing data replication between two 3PAR arrays. The test solution used one array and 9 servers.
In the 3PAR disk array, all used disks are automatically divided into blocks of 1 GB (chunklet) in size, blocks with the same types of disks and one RAID level form a logical disk. Common Provisioning Group (CPG) - a set of policies responsible for the formation of RAID groups. CPG is also responsible for generating virtual volumes (LUNs) that are presented to the host.
In the described solution, 3PAR is configured with one CPG for all hosts in both DAGs containing active databases. In the event of a disk failure, the array will continue servicing all hosts; RAID rebuild processes will use spare chunklets on all remaining disks.
Figure 1 shows the process of forming logical drives in the 3PAR array.
In the test environment, the configuration contains 18 servers connected to a 3PAR array with 216 2TB NL SAS. Virtual volumes are presented to hosts. From these virtual volumes, each server was provided with 12 volumes for databases / logs and one for recovery / maintenance.
In this solution, virtual volumes were created on the 3PAR array, which made it easy to increase the size of the volume as the databases grow. Active and inactive copies of the databases are placed on different arrays to ensure fault tolerance of the solution.
All database volumes, maintenance and recovery volumes were created using RAID protection. The actual size of databases and volumes for each specific customer depends on a large number of factors, including the Service Level Agreement (SLA), backup methodology, virtual volume management factors and the desired size of databases on disk.
Figure 2 shows the active data center with the layout of volumes and servers.
Figure 3 shows a complete solution diagram with 18 servers, each DAG has 9 servers. In the event of a loss of communication or in the event of a power outage at a remote site, the 9 remaining servers are able to serve all the databases.
The scheme of the presented databases for each host:
In the event of failure of the primary site - the remaining servers will be able to continue processing requests:
Consider a sample calculation scenario for the solution described.
Number of mailboxes: 60000
User profile: 100 emails per day (0.14 IOPS per mailbox user with a margin of 20%).
Load profile: 0.12 x 60000 = 8400 IOPS per active mailbox.
Mailbox size: 2GB.
Storage Architecture: RAID 10.
DAGs: 2.
Servers: 18 (9 servers for each DAG).
Calculate the number of mailboxes per server: 60000/18 = ~ 3300.
Number of mailboxes per database: 833.
The size of the virtual volume on the storage system for each database (LUN) Database + Log = 1740.88GB.
The storage system provides a raw capacity of at least 418,608 GB.
Storage Connection Type: Fiber Channel SAN.
Connection speed: 12x 8Gb = 96Gb / s
Switches: 2x Brocade 8/24
Northern platform: 18 servers 2 x 8 core Intel Xeon E5-2650 2.0 GHz, 64 GB RAM.
Since DAG should work in case of failure of 9 servers, then we start the calculation with 18 nodes. Assuming that all databases are equally distributed across 18 nodes, we get 3,300 mailboxes per server. We calculate the mailbox rate in case of failure of 9 nodes (60,000 / 9) = ~ 6700 mailboxes per server.
We calculate the number of megacycles per mailboxes: 6700 x 2 (according to the megacycle table per mailbox) = 13,400 megacycles. Multiply this value by 10% for each copy. In this example, for each active copy there are 2 inactive ones, therefore, the number of megacycles increases by 20%: 13400 x 1.2 = 17700 megacycles.
Multiply the number of inactive mailboxes (when the server processes the maximum number of mailboxes) by the megacycles index by the inactive mailbox from the table (3300 x 0.3 megacycles = 990 megacycles).
Add up megacycles values ​​for active mailboxes with values ​​for inactive mailboxes: 17700 + 990 = 18690 megacycles. We translate these values ​​to the platform.
Two eight-core Intel Xeon E5-2650 2.0 GHz processors correspond to a value of 16x 2000 = 32,000 megacycles. We consider the peak server utilization: 18690/32000 = 60%. This value shows that at the time of the peak of requests from all users, in case of failure of half of the physical servers, we get a load of 60% on the processor, this figure is considered acceptable. It is recommended to calculate the parameters so that the CPU utilization factor in a peak situation does not exceed 70%.
Calculate the required number of cores for active mailboxes.
For a fault tolerant solution:
Required number of cores = (required CPU performance for active mailboxes) / (megacycles per core) x (number of remaining servers) x (number of DAGs).
For a solution without fault tolerance:
Required number of cores = (required CPU performance for active mailboxes) / (megacycles per core) x (number of servers in the data center)
Based on our values:
6700/2000 x 9 = 30 cores
30 cores out of 144 (9 servers with 16 cores) process active mailboxes, in case 14 servers fail.
Based on this, we calculate the number of cores required for Exchange roles (as recommended by Microsoft).
Kernels Hub Transport = required number of cores / 5 = 30/5 = 6.
Kernels for Client Access server = required number of cores x 3/4 = 30 x 3/4 = 22.
Global catalog kernels = required number of cores / 8 = 30/8 = 4.
Similarly, the calculated value of RAM.
Based on the 6,700 mailboxes per server, in the event of a second site failure, and the 100 mailbox parameter, we get the ~ 40GB cache value for the Mailbox role. For the remaining roles, we leave ~ 40% RAM to the server. Thus, a server with 56GB of RAM should handle the load of 6,700 mailboxes. In our case, test servers with a large amount of RAM (64GB RAM) were used.
Based on the recommended Microsoft values, IOPS is calculated (the number of mailboxes is multiplied by the IOPS value for active and inactive copies): 60000 x (0.12 + 2 x 0.1) = ~ 19200 IOPS. In the current configuration, the array provides 21 363 IOPS 25read / 75write performance according to Microsoft recommendations for calculating the cache for storage systems.
The Microsoft Exchange 2010 product remains the most popular corporate email product today. Together with Microsoft, HP regularly tests its products and uploads performance data.
An example of such testing is the solution on the 3PAR StoreServ 7400 array for 60,000 users.
The test was launched for 24 hours and showed the ability of the 3PAR StoreServ array to cope with long-term high loads when working with Microsoft Exchange. Databases and log files were analyzed after testing for possible errors.
Solution Testing Report:
1. Were any errors in the event log?
- Not
2. Were any errors in the log files and databases as a result of checking checksums?
- Not
The average performance of the disk subsystem on the host side during peak loads:
Database I / O
Database Disk Transfers / sec 1921
Database Disk Reads / sec 1210
Database Disk Writes / sec 711
Average Database Disk Read Latency (ms) 19.79
Average Database Disk Write Latency (ms) 1.96
Transaction Log I / O
Log Disk Writes / sec 632
Average Log Disk Write Latency (ms) 0.58
When calculating the required configurations, you can use both the results of the already tested HP ESRP solutions and the recommended parameters for each roles on the Microsoft website.
Literature:
1. Calculation of the array for 140,000 Exchange users
2. Calculate an array of 60,000 Exchange users
3. Calculation of the array per 1000 Exchange users
4. Calculation per 1000 Exchange users on the ProLiant server disk subsystem
5. Microsoft role calculator
6. Performance and Scalability for Exchange Server 2010 SP1
Exchange 2010 Architecture Features
The Exchange 2010 product has several features related to replication mechanics and fault tolerance.
Exchange Server is tightly integrated with Active Directory: most user data is stored in Active Directory (link between user accounts and mailboxes, contact lists). Only mailboxes themselves are stored separately from Active Directory. Thanks to the Active Directory replication mechanism, when multiple Microsoft Exchange Servers are used, the data on all servers remains current.
To implement fault tolerance in Exchange 2010, a single DAG technology is offered, the former SCR, LCR, CCR technologies and the single storage cluster have been removed. DAG (Database Availability Group) - technology of asynchronous database replication between nodes with protection against data loss. DAG is not a pure cluster technology, since no virtual shared node. In this regard, the client connection point was moved from the Mailbox role to the Client Access role. In a DAG, only the base is clustered, which can move between servers included in the DAG. But the Windows Clustering service is still used to determine the quorum. DAG can work only on block devices (local disks and SAN) and cannot work on NAS network drives.
Based on the features of the Exchange 2010 architecture, as well as the advent of more efficient hardware (this applies primarily to processors), the approach to architecture sizing has changed.
')
Exchange 2010 Sizers
Microsoft publishes on its website a series of documents that describe the role of each component of the solution. Understanding the many articles is quite a non-trivial task. The following are theoretical sizingings.
1. Choosing the right processor
For a productive environment, you need a processor that works with the 64-bit version of Exchange 2010 on Windows Server 2008 or 2008 R2. Microsoft recommends using Intel processors that support Intel Extended Memory 64 Technology or AMD processors that support AMD64.
Hyperthreading
The use of the Hyperthreading option requires careful planning, and the effect of deployment in the Exchange environment is usually not significant.
By default, the Hyperthreading option should be disabled. Use is only possible in the form of a temporary measure to improve processor performance until the additional hardware is commissioned.
Below is a table with the minimum and recommended maximum settings for Exchange 2010.
| Exchange 2010 server role | Minimum
configuration | (recommended)
maximum configuration | example
(recommended) maximum configurations |
| Edge transport | 1x processor core | 2x processors | 12x processor cores |
| Hub transport | 1x processor core> | 2x processors | 12x processor cores |
| Client access | 2x processor cores | 2x processors | 12x processor cores |
| Unified messaging | 2x processor cores | 2x processors | 12x processor cores |
| Mailbox | 2x processor cores | 2x processors | 12x processor cores |
| Client Access / Hub Transport combined | 2x processor cores | 2x processors | 12x processor cores |
| Multiple (Client Access, Hub Transport, Mailbox roles) | 2x processor cores | 4x processors | 24x processor cores |
Hub Transport Role
The recommended Microsoft configuration is 8 processor cores, for an environment where several servers and several thousand mailboxes are supposed to be used. A server with a large number of cores can be effectively used when the Hub Transport role is configured to work with anti-virus and anti-spam software. The load factor on the processors depends on such indicators as the average message size, the number of forwarded messages, the number of involved transport agents, the configuration of the antivirus and other applications.
Client Access Server Role
In Microsoft Exchange 2010, most client functions were moved from the Mailbox Server role to the Client Access Server role. In the Exchange 2010 environment, messages are processed on the Client Access server if they are sent by a non-MAPI client (for example, POP3 or IMAP4). In addition, Microsoft Office Web App is rendered by the Client Access server role, and not by the Microsoft Exchange Information Store service, as in previous versions of Exchange.
These architectural features allow the Client Access server to seriously unload the Mailbox server and effectively use 8 processor cores. For organizations with a small number of mailboxes or with a low volume of traffic from non-MAPI clients, it is recommended to use 2 cores.
Unified Messaging Server role
Microsoft recommends the configuration of a Unified Messaging Server with 8 processor cores. Such a number of cores is necessary for the operation of voice mail, for example, the conversion of the .wav format to Windows Media Audio (WMA). A server with 2 cores can be used for organizations with a small number of mailboxes or in an environment with a small activity of the Unified Server Messaging role.
Mailbox server role
Microsoft recommends configuring this role based on the number of mailboxes and user profiles. 4 processor cores give a good balance between performance and cost, so many cores can handle up to several thousand mailboxes. Resource assessment requires an understanding of the average user profile. This image can be obtained using transport performance counters. The use of third-party utilities is also allowed.
2. The choice of RAM
The choice of RAM is determined by the following values
- RAM speed
- Volume of memory module / cost of memory module
- The number of slots in the server.
Exchange 2010 on the Windows Server 2010 operating system efficiently works with RAM up to 64 GB per server.
The table below shows the minimum and maximum values ​​for each role.
| Exchange 2010 server role | Minimum supported | Recommended maximum |
| Edge transport | 4GB | 1GB per core |
| Hub transport | 4GB | 1GB per core |
| Client access | 4GB | 2GB per core |
| Unified messaging | 4GB | 2GB per core |
| Mailbox | 4GB | 4GB + database cache |
| Client Access / Hub Transport combined | 4GB | 2GB per core |
| Multiple roles (Hub transport + Client Access + Mailbox) | 4GB | 4GB + 3-30 MB additional per mailbox. |
The role of Edge Transport and Hub Transport
These roles do not require a large amount of RAM, at least 4GB is enough to process all requests, the optimal value is 1GB per core, but not less than 4 GB in total.
Client Access Server Role
The consumption of RAM in this role is almost linearly dependent on the number of client connections and the number of transactions. Based on Microsoft recommendations, the role configuration from 2GB per core is balanced.
Mailbox Server Role
This role is quite complicated in the calculations, because The optimal parameters of RAM depend on the number of deployed roles, the number of mailboxes, the average user profile and the number of active databases.
Memory calculation of this role is critical for reducing the I / O load on the server disk subsystem.
3. Calculate Mailbox Database Cache
Exchange 2010 uses a mechanism to reduce the load on the disk subsystem of the server or data storage, based on the use of the Extensible Storage Engine (ESE) cache. The more cache available to the Exchange 2010 server, the lower the I / O load on the disk subsystem.
Cache performance in Microsoft Exchange 2010 has been enhanced during several technical changes. One of the most significant changes is the increase in the depth target check log. This log is used to ensure that the logs and database cache are written to the database within reasonable time limits. The size of the log depends on the number of copies and the type of database:
| Database configuration | Log checkpoint depth target |
| Single database | 20MB |
| Database with copies | 100MB |
| Inactive database | 5MB |
Using the Extensible Storage Engine, the I / O load for a database with two or more copies can be up to 40% lower than for a single database configuration. In situations with high load on the disk subsystem, the database stores change files in memory for a long time, this behavior reduces the number of sequential write operations (repeated write) and creates a merging effect (coalescing), the so-called hybrid storage model. This allows you to quickly restore the database in case of malfunction. At the same time, inactive copies of databases also use memory to store a log (5MB), this allows you to quickly translate an inactive database into active mode.
The cache size, and, accordingly, the amount of physical memory directly depends on the number of databases and can be calculated from the table:
| Number of databases | The minimum amount of RAM |
| 1-10 | 2 GB |
| 11-20 | 4 GB |
| 21-30 | 6 GB |
| 31-40 | 8 GB |
| 41-50 | 10 GB |
| 51-60 | 12 GB |
| 61-70 | 14 GB |
| 71-80 | 16 GB |
| 81-90 | 18 GB |
| 91-100 | 20 GB |
In previous versions of Echange, the key metric was the volume of I / O operations, now the current metrics are the volume of the database cache and the number of messages sent by the user per day.
The following table allows you to determine IOPS, based on the volume of the box 2GB and an average letter size of 75KB.
| Messages per mailbox per day (~ 75KB average message size) | Database Cache Per User (MB) | Single database (stand-alone): IOPS per box | Database copy: IOPS per box |
| 50 | 3 | .060 | .050 |
| 100 | 6 | .120 | .100 |
| 150 | 9 | .180 | .150 |
| 200 | 12 | .240 | .200 |
| 250 | 15 | .300 | .250 |
| 300 | 18 | .360 | .300 |
| 350 | 21 | .420 | .350 |
| 400 | 24 | .480 | .400 |
| 450 | 27 | .540 | .450 |
| 500 | thirty | .600 | .500 |
4. Active / inactive databases and database copies
Inactive databases.
In Exchange 2010, the same server can handle both active copies of databases and inactive ones in a configuration with a fault-tolerant solution. Processor requirements for processing inactive copies should also be considered at the design stage. Inactive database copies use processor resources to check replicated logs and process the database index. Each mailbox of an inactive database requires 15% of the CPU resources of the mailbox resources of the active database.
Copies of databases.
Each Exchange 2010 database can create up to 16 copies. Each copy of the database consumes 10% of the CPU from the main database.
Understanding the number of copies and the number of inactive databases is necessary in order to be able to create several models of the behavior of the Exchange 2010 environment depending on external conditions (for example, on the number of failed servers or platforms). Typical models for the Exchange 2010 environment are the model with 100% use of all mailboxes and the behavior model for the worst case scenario.
The planning steps are as follows:
- We determine the required level of fault tolerance, the number of copies required and the number of Data Availability Groups (DAGs) to protect against the failure of various components.
- If you use fault tolerance, we define a behavior model: activate all mailboxes or set the worst case scenario.
- Depending on the selected model, select values ​​from the table below, based on the type of profile (10 hours with a two-hour peak of loading in the morning):
| Number of letters per day per mailbox | Mailbox Database Cache (MB) | Single database with IOPS per drawer estimates | Database copies (IOPS on mailbox) | Megacycles for the active box | Megacycles for the inactive box |
| 50 | 3 | .06 | .05 | one | 0.15 |
| 100 | 6 | 0.12 | 0.1 | 2 | 0.3 |
| 150 | 9 | 0.18 | 0.15 | 3 | 0.45 |
| 200 | 12 | 0.24 | 0.2 | four | 0.6 |
| 250 | 15 | 0.3 | 0.25 | five | 0.75 |
| 300 | 18 | 0.36 | 0.3 | 6 | 0.9 |
| 350 | 21 | 0.42 | 0.35 | 7 | 1.05 |
| 400 | 24 | 0.48 | 0.4 | eight | 1.2 |
| 450 | 27 | 0.54 | 0.45 | 9 | 1.35 |
| 500 | thirty | 0.6 | 0.5 | ten | 1.5 |
The megacycles parameter is determined by the processor's core performance in MHz, for example, 2 x 4 core Intel Xeon x5470 3.33-gigahertz (GHz) is 3333 megacycles per core.
5. Storage System
The choice of storage system depends on many factors, but is primarily determined by the number of mailboxes and the requirement for expansion.
For smaller organizations with 1,000 mailboxes, HP and Microsoft recommend the use of an entry-level P2000G3 iSCSI storage or ProLiant disk subsystem .
For large installations, high-level storage is recommended, supporting a large number of disks. One of these arrays is 3PAR StoreServ, announced on December 4th, 2012. This array has already been tested in the Exchange 2010 Solution Reviewed Program (ESRP) - Storage v3.0.
As you know, 3PAR arrays are intended for use under virtualization tasks in an environment with unpredictable loads. Another possible destination is corporate mail.
In conjunction with Microsoft, a 3PAR StoreServ array of mail for 60,000 users was tested. The solution used HP Proliant Bl460c Gen8 blade servers, C7000 blade baskets, Brocade optical switches.
Test software: Microsoft Windows Server 2008 R2 and Microsoft Exchange 2010.
The document provides information about testing an array in an environment of 60,000 users.
High performance is organized through the use of wide-striping technology (using all drives to build fault-tolerant RAID) on a 3PAR array using the Database Availability Group (DAG), where each copy of DAG is served by 18 servers (9 active servers are served by 60,000 users in 2 DAGs). High data availability can be ensured by organizing data replication between two 3PAR arrays. The test solution used one array and 9 servers.
In the 3PAR disk array, all used disks are automatically divided into blocks of 1 GB (chunklet) in size, blocks with the same types of disks and one RAID level form a logical disk. Common Provisioning Group (CPG) - a set of policies responsible for the formation of RAID groups. CPG is also responsible for generating virtual volumes (LUNs) that are presented to the host.
In the described solution, 3PAR is configured with one CPG for all hosts in both DAGs containing active databases. In the event of a disk failure, the array will continue servicing all hosts; RAID rebuild processes will use spare chunklets on all remaining disks.
Figure 1 shows the process of forming logical drives in the 3PAR array.
Pic1
In the test environment, the configuration contains 18 servers connected to a 3PAR array with 216 2TB NL SAS. Virtual volumes are presented to hosts. From these virtual volumes, each server was provided with 12 volumes for databases / logs and one for recovery / maintenance.
In this solution, virtual volumes were created on the 3PAR array, which made it easy to increase the size of the volume as the databases grow. Active and inactive copies of the databases are placed on different arrays to ensure fault tolerance of the solution.
All database volumes, maintenance and recovery volumes were created using RAID protection. The actual size of databases and volumes for each specific customer depends on a large number of factors, including the Service Level Agreement (SLA), backup methodology, virtual volume management factors and the desired size of databases on disk.
Figure 2 shows the active data center with the layout of volumes and servers.
Pic2
Figure 3 shows a complete solution diagram with 18 servers, each DAG has 9 servers. In the event of a loss of communication or in the event of a power outage at a remote site, the 9 remaining servers are able to serve all the databases.
The scheme of the presented databases for each host:
In the event of failure of the primary site - the remaining servers will be able to continue processing requests:
6. Calculation for the scenario for 60,000 users
Consider a sample calculation scenario for the solution described.
Number of mailboxes: 60000
User profile: 100 emails per day (0.14 IOPS per mailbox user with a margin of 20%).
Load profile: 0.12 x 60000 = 8400 IOPS per active mailbox.
Mailbox size: 2GB.
Storage Architecture: RAID 10.
DAGs: 2.
Servers: 18 (9 servers for each DAG).
Calculate the number of mailboxes per server: 60000/18 = ~ 3300.
Number of mailboxes per database: 833.
The size of the virtual volume on the storage system for each database (LUN) Database + Log = 1740.88GB.
The storage system provides a raw capacity of at least 418,608 GB.
Storage Connection Type: Fiber Channel SAN.
Connection speed: 12x 8Gb = 96Gb / s
Switches: 2x Brocade 8/24
Northern platform: 18 servers 2 x 8 core Intel Xeon E5-2650 2.0 GHz, 64 GB RAM.
We start the calculation.
Since DAG should work in case of failure of 9 servers, then we start the calculation with 18 nodes. Assuming that all databases are equally distributed across 18 nodes, we get 3,300 mailboxes per server. We calculate the mailbox rate in case of failure of 9 nodes (60,000 / 9) = ~ 6700 mailboxes per server.
We calculate the number of megacycles per mailboxes: 6700 x 2 (according to the megacycle table per mailbox) = 13,400 megacycles. Multiply this value by 10% for each copy. In this example, for each active copy there are 2 inactive ones, therefore, the number of megacycles increases by 20%: 13400 x 1.2 = 17700 megacycles.
Multiply the number of inactive mailboxes (when the server processes the maximum number of mailboxes) by the megacycles index by the inactive mailbox from the table (3300 x 0.3 megacycles = 990 megacycles).
Add up megacycles values ​​for active mailboxes with values ​​for inactive mailboxes: 17700 + 990 = 18690 megacycles. We translate these values ​​to the platform.
Two eight-core Intel Xeon E5-2650 2.0 GHz processors correspond to a value of 16x 2000 = 32,000 megacycles. We consider the peak server utilization: 18690/32000 = 60%. This value shows that at the time of the peak of requests from all users, in case of failure of half of the physical servers, we get a load of 60% on the processor, this figure is considered acceptable. It is recommended to calculate the parameters so that the CPU utilization factor in a peak situation does not exceed 70%.
Calculate the required number of cores for active mailboxes.
For a fault tolerant solution:
Required number of cores = (required CPU performance for active mailboxes) / (megacycles per core) x (number of remaining servers) x (number of DAGs).
For a solution without fault tolerance:
Required number of cores = (required CPU performance for active mailboxes) / (megacycles per core) x (number of servers in the data center)
Based on our values:
6700/2000 x 9 = 30 cores
30 cores out of 144 (9 servers with 16 cores) process active mailboxes, in case 14 servers fail.
Based on this, we calculate the number of cores required for Exchange roles (as recommended by Microsoft).
Kernels Hub Transport = required number of cores / 5 = 30/5 = 6.
Kernels for Client Access server = required number of cores x 3/4 = 30 x 3/4 = 22.
Global catalog kernels = required number of cores / 8 = 30/8 = 4.
Similarly, the calculated value of RAM.
Based on the 6,700 mailboxes per server, in the event of a second site failure, and the 100 mailbox parameter, we get the ~ 40GB cache value for the Mailbox role. For the remaining roles, we leave ~ 40% RAM to the server. Thus, a server with 56GB of RAM should handle the load of 6,700 mailboxes. In our case, test servers with a large amount of RAM (64GB RAM) were used.
Based on the recommended Microsoft values, IOPS is calculated (the number of mailboxes is multiplied by the IOPS value for active and inactive copies): 60000 x (0.12 + 2 x 0.1) = ~ 19200 IOPS. In the current configuration, the array provides 21 363 IOPS 25read / 75write performance according to Microsoft recommendations for calculating the cache for storage systems.
Recommendations:
- Windows Server 2008R2 does not require the use of special utilities for formatting and “aligning” blocks on physical media, as required by Windows Server 2003. Windows Vista, Windows 7, Windows Server 2008 and later products by default align the partition with 1MB.
- Microsoft recommends using SAS disks for the mail environment, while SSDs for mail tasks are ineffective: In general, Exchange 2010 Mailbox servers .
- Use host-side MPIO with dual port HBA adapters.
- Always consider the load according to the worst scenario (as we considered the load on the host processors earlier).
- Use shared resources (Shared Logical Disks) during the deployment of Microsoft Exchange to get the most out of the 3PAR wide-striping technology.
- For SAS 7.2k drives, Microsoft recommends using RAID 1 + 0.
Findings:
The Microsoft Exchange 2010 product remains the most popular corporate email product today. Together with Microsoft, HP regularly tests its products and uploads performance data.
An example of such testing is the solution on the 3PAR StoreServ 7400 array for 60,000 users.
The test was launched for 24 hours and showed the ability of the 3PAR StoreServ array to cope with long-term high loads when working with Microsoft Exchange. Databases and log files were analyzed after testing for possible errors.
Solution Testing Report:
1. Were any errors in the event log?
- Not
2. Were any errors in the log files and databases as a result of checking checksums?
- Not
The average performance of the disk subsystem on the host side during peak loads:
Database I / O
Database Disk Transfers / sec 1921
Database Disk Reads / sec 1210
Database Disk Writes / sec 711
Average Database Disk Read Latency (ms) 19.79
Average Database Disk Write Latency (ms) 1.96
Transaction Log I / O
Log Disk Writes / sec 632
Average Log Disk Write Latency (ms) 0.58
When calculating the required configurations, you can use both the results of the already tested HP ESRP solutions and the recommended parameters for each roles on the Microsoft website.
Literature:
1. Calculation of the array for 140,000 Exchange users
2. Calculate an array of 60,000 Exchange users
3. Calculation of the array per 1000 Exchange users
4. Calculation per 1000 Exchange users on the ProLiant server disk subsystem
5. Microsoft role calculator
6. Performance and Scalability for Exchange Server 2010 SP1
Source: https://habr.com/ru/post/171045/
All Articles