Software raids: migration from Windows raid0 to Linux mdadm raid5
A report on how we migrated from Windows raid0 to Linux raid5, which pitfalls we met and how we overcame them.
Prehistory. The design department is served by a file server running Windows 2003. Initially, it was designed to exchange data, but over time it was used to store files for a long time, despite the fact that it had a software raid0. The time has come for the next hardware upgrade. After the purchase, “unexpectedly” it turned out that the new Windows 2003 motherboard does not start, the driver manufacturer does not release and recommends using a newer version of Windows, which is unacceptable for licensing reasons.
As a result, two decisions were made:
There are no questions about virtualization - it was done more than once. The main problem is the zero raid, organized by means of Windows. Time was spent trying to mount windows-raid0 in linux - failed, despite descriptions on the Internet that it was possible.
The second problem: on the Windows server, the hard drives for raid0 are two, 2TB each, the entire array is filled with data and there is nowhere to copy them. New winchesters also bought two but already at 3TB. The management of the department initially wanted to attach them to the existing zero raid and get a virtual disk for storing data in 10TB.
')
By explaining, we managed to convince of the incorrectness of this approach, but we could not convince the third hard drive to be purchased on 3TB, so we decided to act in sequence:
1) Build a raid5 with one missing hard drive
2) Transfer raid5 to Windows and copy it using its means
3) Upon completion of copying from the remaining disks, assemble raid0 and transfer it as the third block device to raid5
We tested this option in a virtual machine with a smaller amount of data - everything turns out great, after completing the test we have raid5, we perform several hard reboots, disable-enable virtual hard drives - the raid lives as expected - self-healing and no data loss.
It is very embarrassing that, after the data is copied, there is a degraded array and it is restored due to the two hard drives from which they were copied, in the event of a failure at this stage - the complete loss of all information. So you need the data somewhere previously backed up. But everything is complicated by the fact that there are only two 1TB of free hard drives. The situation is heating up, in the morning the data on the server should be at least read access.
The audit of other servers showed that we have a server with 2TB of free space, this saves the situation, we can save the image of one of the two 2TB hard drives and use raid5 to restore it in the stripe with the 1TB hard drive, in case of a failure all data is safe. The final scheme:
We assemble raid5 without one hard drive, run Windows 2003 for Linux in KVM, start copying and see that the write speed is very low 3-6MB / s, while writing from under Linux to the same degraded raid5 (three times the amount of data from RAM) at an average speed of 80MB / s, but due to the fact that the original data can be copied only from under Windows, this does not help. Yes, we know that the write speed on a degraded raid5 is reduced, but where does the difference between Linux and Windows come from?
For the test, we tried to copy data from a Windows disk to a single disk - 100MB / s. Maybe the implementation of mdadm is such that Windows feels bad in writing operations in the raid? It is unlikely - we have several Windows running on top of raid1 compiled into mdadm. For the sake of experiment we collected raid1, the speed of copying from under Windows in the same range as on a single disk is about 100MB / s.
Our mistakes are that we believed the article where it was stated from the context: “initially assembled in mdadm raid5 from two hard drives and one missing is zero-raid implementation”, and also that we only tested under Linux, we start copying during tests from - Under Windows, the problem was known before.
The test environment should be fully productive .
The decision is made to assemble raid0 from two 3TB hard drives, copy the data, and then rebuild this array into raid5.
1) Create raid0
2) Copy the data, this time from under Windows, the write speed is 60-100MB / s
3) Convert raid0 to raid1
This operation is performed instantly.
4) Add the third block device
We look at the status:
Tests completed successfully.
During raid5 rebuilding, optimization was performed to speed up, stripe_cache_size was increased to a maximum of 32768:
If windows-raid0 is filled to the limit, then there is a strong data fragmentation on it, to defragment to 4TB for a long time, we left the position by copying in several streams using Microsoft Richcopy
Very sensible tutorials on mdadm commands " Software RAID in Linux ".
Prehistory. The design department is served by a file server running Windows 2003. Initially, it was designed to exchange data, but over time it was used to store files for a long time, despite the fact that it had a software raid0. The time has come for the next hardware upgrade. After the purchase, “unexpectedly” it turned out that the new Windows 2003 motherboard does not start, the driver manufacturer does not release and recommends using a newer version of Windows, which is unacceptable for licensing reasons.
As a result, two decisions were made:
- Move server to virtual machine under KVM
- Use raid5 for storing data.
There are no questions about virtualization - it was done more than once. The main problem is the zero raid, organized by means of Windows. Time was spent trying to mount windows-raid0 in linux - failed, despite descriptions on the Internet that it was possible.
The second problem: on the Windows server, the hard drives for raid0 are two, 2TB each, the entire array is filled with data and there is nowhere to copy them. New winchesters also bought two but already at 3TB. The management of the department initially wanted to attach them to the existing zero raid and get a virtual disk for storing data in 10TB.
')
By explaining, we managed to convince of the incorrectness of this approach, but we could not convince the third hard drive to be purchased on 3TB, so we decided to act in sequence:
1) Build a raid5 with one missing hard drive
mdtest# mdadm --create /dev/md4 -l 5 -n 3 /dev/sda1 /dev/sdb1 missing 2) Transfer raid5 to Windows and copy it using its means
3) Upon completion of copying from the remaining disks, assemble raid0 and transfer it as the third block device to raid5
mdtest# mdadm --create /dev/md3 -l 0 -n 2 /dev/sdc1 /dev/sdd1 mdtest# mdadm --add /dev/md4 /dev/md3 We tested this option in a virtual machine with a smaller amount of data - everything turns out great, after completing the test we have raid5, we perform several hard reboots, disable-enable virtual hard drives - the raid lives as expected - self-healing and no data loss.
It is very embarrassing that, after the data is copied, there is a degraded array and it is restored due to the two hard drives from which they were copied, in the event of a failure at this stage - the complete loss of all information. So you need the data somewhere previously backed up. But everything is complicated by the fact that there are only two 1TB of free hard drives. The situation is heating up, in the morning the data on the server should be at least read access.
The audit of other servers showed that we have a server with 2TB of free space, this saves the situation, we can save the image of one of the two 2TB hard drives and use raid5 to restore it in the stripe with the 1TB hard drive, in case of a failure all data is safe. The final scheme:
/dev/md3 - raid0: 1TB + 2TB/dev/md4 - raid5: 2x3TB + /dev/md3Getting to action on real data
We assemble raid5 without one hard drive, run Windows 2003 for Linux in KVM, start copying and see that the write speed is very low 3-6MB / s, while writing from under Linux to the same degraded raid5 (three times the amount of data from RAM) at an average speed of 80MB / s, but due to the fact that the original data can be copied only from under Windows, this does not help. Yes, we know that the write speed on a degraded raid5 is reduced, but where does the difference between Linux and Windows come from?
For the test, we tried to copy data from a Windows disk to a single disk - 100MB / s. Maybe the implementation of mdadm is such that Windows feels bad in writing operations in the raid? It is unlikely - we have several Windows running on top of raid1 compiled into mdadm. For the sake of experiment we collected raid1, the speed of copying from under Windows in the same range as on a single disk is about 100MB / s.
Our mistakes are that we believed the article where it was stated from the context: “initially assembled in mdadm raid5 from two hard drives and one missing is zero-raid implementation”, and also that we only tested under Linux, we start copying during tests from - Under Windows, the problem was known before.
The test environment should be fully productive .
Back to the tests
The decision is made to assemble raid0 from two 3TB hard drives, copy the data, and then rebuild this array into raid5.
1) Create raid0
mdtest# mdadm --create /dev/md4 -l 0 -n 2 /dev/sda1 /dev/sdb1 2) Copy the data, this time from under Windows, the write speed is 60-100MB / s
3) Convert raid0 to raid1
mdtest# mdadm /dev/md4 --grow --level=5 mdadm: level of /dev/md4 changed to raid5 This operation is performed instantly.
mdadm -D /dev/md4 reports that we have raid5 mdtest# mdadm -D /dev/md4 /dev/md4: Raid Level : raid5 Raid Devices : 3 Total Devices : 2 State : clean, degraded Active Devices : 2 Working Devices : 2 Failed Devices : 0 Spare Devices : 0 Number Major Minor RaidDevice State 0 252 0 0 active sync /dev/dm-0 1 252 1 1 active sync /dev/dm-1 2 0 0 2 removed 4) Add the third block device
mstest# mdadm --add /dev/md4 /dev/md3 mdadm: added /dev/md3 We look at the status:
mdtest# mdadm -D /dev/md4 /dev/md4: Raid Level : raid5 Raid Devices : 3 Total Devices : 3 State : clean, degraded, recovering Active Devices : 2 Working Devices : 3 Failed Devices : 0 Spare Devices : 1 Rebuild Status : 2% complete Number Major Minor RaidDevice State 0 252 0 0 active sync /dev/dm-0 1 252 1 1 active sync /dev/dm-1 3 252 2 2 spare rebuilding /dev/dm-2 Tests completed successfully.
Actual migration
- We collected mdadm-raid0 [md4] from two 3TB hard drives.
- In Windows, the data from windows-raid0 was copied onto mdadm-raid0.
- Saved the image of the second hard drive from windows-raid0 on a nearby server (2TB).
- Converted mdadm-raid0 to mdadm-raid5.
- We collected mdadm-raid0 [md3] from a 1TB disk and a second disk from windows-raid0.
- Added mdadm-raid5 to the third block device mdadm-raid0 [md3].
Notes
During raid5 rebuilding, optimization was performed to speed up, stripe_cache_size was increased to a maximum of 32768:
echo 32768 > /sys/block/md4/md/stripe_cache_size . Write Intent Bitmap was disabled: mdadm --grow --bitmap=none /dev/md4 .If windows-raid0 is filled to the limit, then there is a strong data fragmentation on it, to defragment to 4TB for a long time, we left the position by copying in several streams using Microsoft Richcopy
Very sensible tutorials on mdadm commands " Software RAID in Linux ".
Source: https://habr.com/ru/post/185030/
All Articles