Re: Prepping to install

[David Brownlee <abs%absd.org@localhost>]

On 13 May 2015 at 16:03, William A. Mahaffey III <wam%hiwaay.net@localhost> wrote:
> On 05/13/15 08:48, David Brownlee wrote:
>>
>> On 12 May 2015 at 16:01, William A. Mahaffey III <wam%hiwaay.net@localhost> wrote:
>>>
>>> On 05/12/15 02:32, David Brownlee wrote:
>>>>
>>>> On 11 May 2015 at 23:46, William A. Mahaffey III <wam%hiwaay.net@localhost> wrote:
>>>>
>>>> If you are using RAID5 I would strongly recommend keeping to
>>>> "power-of-two + 1" components, to keep the stripe size as a nice power
>>>> of two, otherwise performance is... significantly impaired.
>>>
>>> Hmmmm .... Could you amplify on that point a bit ? I am intending to
>>> maximize available storage & have already procured the mbd & 6 drives,
>>> but I
>>> could rethink things if my possibly hasty choices would be too burdensome
>>
>> For RAID5 to perform efficiently data should be written in units which
>> re aligned with the RAID stripes and are a multiple of stripe in size,
>> otherwise a simple write changes into a read of  stripe, modification
>> of the affected part and then a write.
>>
>> Filesystems tend to have sectors and blocks which are powers of two,
>> so the easiest way to arrange this for ffs is for the filesystem block
>> size to be a multiple of the stripe size ("1" is a fine multiple in
>> this case).
>>
>> This is similar to the issue with sector drives which have 4K sectors
>> but present them 512byte sectors - if a filesystem is not 4K aligned
>> then write performance suffers horribly.
>
> Hmmmmm .... OK, I think I have it. The stripe size is N * <some underlying
> (disk|RAID) block size>, & if N (the # of active drives) is odd or prime (or
> both, as in my case), we would have/need bizarre filesystem block sizes (for
> alignment w/ RAID stripes) or unaligned FS blocks/sectors, which give crappy
> performance, right ? Could you estimate how crappy crappy really is ? 25%
> slower ? 50%, 100%, more ? Me scots sensibilities hate having almost 1 TiB
> of drive sitting around idle (although I do crave speed enough to override)
> :-/ ....

If you manage 25% of the performance (that is "only" 75% hit) I would
be surprised. I'd also be curious to see what number you do get :) -
I'm quite fond of pkgsrc/benchmarks/bonnie++ to get simple comparable
numbers. If you are testing, some things to vary
- Number of drives (5, 6)
- Stripe size, eg 4K per drive or 8K per drive
- Filesystem block size 32K, 64K (may not be able to use 64K for boot
partitions)
- mounting with '-o log' or not (generally you want this :)
Remember to ensure you have good (at least 4K) alignment on the base
partitions. If you have a modern '4K under the covers' drive and start
at sector 63... its not a good place to be

>>>> If you want to maximise space with some redundancy then as you say,
>>>> RAID5 is the way to go for the bulk of the storage.
>>>>
>>>> A while back I setup a machine with 5 * 2TB disks with netbsd-6, with
>>>> small RAID1 partitions for root and the bulk as RAID5
>>>>
>>>> http://abs0d.blogspot.co.uk/2011/08/setting-up-8tb-netbsd-file-server.html
>>>> (wow, was that really four years ago) - in your position I might keep
>>>> one 1TB as a scratch/build space and then RAID up the rest.
>>>>
>>>> If you have time definitely experiment, get a feel for the different
>>>> performance available from the different options.
>>>
>>> *Wow*, another fabulous resource. Your blog documents almost verbatim
>>> what I
>>> have in mind. I am going w/ 6 drives (already procured, 6 SATA3 slots on
>>> the
>>> mbd, done deal), but philosophically very close to what you describe. 1
>>> question: if you were doing this again today, would it be fdisk or GPT ?
>>
>> If I had >2TB drives it would be TB :) If not, I would still stick
>> with fdisk. The complexity of
>> gpt setup and wedge autoconfiguration is still greater than fdisk and
>> disklabel. I know I'm going to have to move to it at some point, but
>> I'm going to hold off until I need to
>>
>>> I think I am looking at 4 partitions per drive, ~16 GB for / (RAID1, 2
>>> drives)
>>> & /usr (4 drives, RAID10), 16 GB for swap (kernel driver, all 6 drives),
>>> 16
>>> - 32 GB for /var (RAID5, all 6 drives), & the rest for /home (RAID5, all
>>> 6
>>> drives). TIA & thanks again.
>>
>> I would definitely hold off on RAID5 for everything except the large
>> /home. RAID1 is much simpler and more performance for writes. I would
>> also try to avoid configuring multiple RAID5s across overlapping sets
>> of disks, while it theoretically provides more IO bandwidth, that
>> bandwidth will be having to compete with all the other filesystems and
>> swap usage on the system.
>>
>> If you wanted to use all six disks:
>> - 32G(RAID1 root+usr) 910G(non raid scratch space)
>> - 32G(RAID1 root+usr) 910G(RAID5 home)
>> - 32G(RAID1 var) 910G(RAID5 home)
>> - 32G(RAID1 var) 910G(RAID5 home)
>> - 32G(RAID1 swap+spare) 910G(RAID5 home)
>> - 32G(RAID1 swap+spare) 910G(RAID5 home)
>>
>> 32GB space notes:
>> - This gives you three 32GB RAID1 'pools' to allocate everything
>> outside of /home
>> - Can adjust the 32G up or down before partitioning, but all should be the
>> same
>> - In the suggestion, root+usr are kept on the same RAID (and could be
>> a single partition), so that the system can have all of the userland
>> available with only one disk attached, and a 'spare' partition is left
>> in case of later moderate additional space needs - maybe an extra
>> partition for /usr/pkg?, or for /var/pgsql, etc
>> - Obviously allocate usage within pools to taste - could put /usr on a
>> separate raid to provide more IO bandwidth for root & usr
>
> This is interesting. I kinda wanted swap spread out over all 6 drives for
> better swap I/O performance, an issue I am having with another box which is
> laid out sorta like this, with swap 'on top of' a RAID0 block (admittedly
> under Linux, not *BSD, but still), swap performance is horrible, several
> min. to page in 200-300 MB worth of paged out VM. I was planning on as much
> parallelization of each RAID as possible for max performance, & swap handled
> by the kernel driver. Others have suggested swap on a RAID 'partition', is
> that more de-rigeur for NetBSD, or the other BSD's for that matter ? This
> box, under FreeBSD 9.3R-p13, has 4 swap partitions under straight kernel
> management, & seems very spry, although it also has a lot of RAM & doesn't
> swap much (on purpose, BTW) ....

Separate swap devices will give the best performance, RAID1 or 5 will
give robustness in the face of a single component failure. You pays
your money... Of course, if you have dedicated partitions on the disk
which you could RAID then you can even change your mind after install,
swapctl off the swap, mess with the partitions and away you go (nerves
of steve advised, though not required :)

>> 910GB space notes:
>> - This gives 5* 910GB RAID5, which provides 4*910G (or 3640G) of space
>> - One disk is not included in the RAID5. This could be saved as a
>> spare for a RAID5 component failure (though a better approach might be
>> to have a disk on the desk next to the machine :), or used as non
>> raided scratch space. If it will not be active, then probably best to
>> put it on one of the components for the heaviest used 32G, or the most
>> important 32G
>
> Your assessments are quite persuasive, I think I now like the 5 drive RAID5
> for home, with that last partition as nebulous scratch space.
>
>>
>> Note in the above that IO to /home will hit (almost) all disks, and
>> will affect all of the 32GB pools, so if you have heavy IO to /home do
>> not expect blistering performance from any filesystem. On the other
>> hand when /home has very light IO then you should have relatively nice
>> multi spindle performance from the other filesystems.
>
> Yeah, but it would speed up access to *just* /home, right ? This box will be
> backing up other boxen on my LAN, initially to a directory under /home, so I
> want that I/O to be as swift as possible. I am maxing out the RAM (also
> already procured), so I hope I don't have too much contention between I/O to
> /home & swap ....

If you want home to be as fast a possible, then you really want to
prefer RAID1 to RAID5 (which conflicts with the space.. I know). I run
dirvish overnight from some machines to a RAID5 setup pretty much
identical to the one in my post, and it works well enough.

>> Having said all that, if I had the time to play I would install onto a
>> USB key, then script up the building and partitioning of the system in
>> many different forms and then chroot into the result and run some
>> tests to see how it performs.
>
> I *definitely* want to script the partitioning both for repeatability in the
> event of drive failure (or setting up another box) & to avoid fat-fingered
> screw-ups !!!!
>
> Thanks again for a fabulously informative reply.

Having discussed all this RAID5 goodness I feel obliged to comment
that when I finally run out of space and need a new build I'm probably
going to go for 4TB disks with six of them in RAID1 pairs for 12TB
with flexibility for adding more space (in 4TB units). If I *needed*
to get 16TB out of them I'd go the RAID5 route again, but I'm willing
to trade off the extra space for speed and simplicity. Of course I'm
really holding off having to actually *buy* six 4TB disks for as long
as humanly possible (by which point they may be 6TB disks, but who can
tell :)