Re: Vnode API change: add global vnode cache

["J. Hannken-Illjes" <hannken%eis.cs.tu-bs.de@localhost>]

On 09 Apr 2014, at 19:09, Chuck Silvers <chuq%chuq.com@localhost> wrote:

> On Tue, Apr 08, 2014 at 11:23:11AM +0200, J. Hannken-Illjes wrote:
>> 
>> On 07 Apr 2014, at 19:28, Chuck Silvers <chuq%chuq.com@localhost> wrote:
>> 
>>> On Sun, Apr 06, 2014 at 12:14:24PM +0200, J. Hannken-Illjes wrote:
>>>> Currently all file systems have to implement their own cache of
>>>> vnode / fs node pairs.  Most file systems use a copy and pasted
>>>> version of ufs_ihash.
>>>> 
>>>> So add a global vnode cache with lookup and remove:
>>> [...]
>>> 
>>> hi,
>>> 
>>> the principle of this is good, but I have a couple concerns about the 
>>> details:
>>> 
>>> - why use an rb tree instead of a hash table?
>>>  other people are saying that the lock contention is the same,
>>>  but there are two differences:
>>>   - updating an rb tree touches a lot more memory, so the lock is held for
>>>     longer.
>>>   - different parts of a hash table can easily be protected by
>>>     different locks, but that doesn't work for a tree.
>> 
>> The underlying data structure may change.  Once at least ufs, layerfs and nfs
>> use this cache it will be easier to do some measurements.
> 
> ok, no point in arguing this further without some actual data.
> but we should have that data before this is checked in.

On a 20-core amd64 (KVM virtualised on a 24-core CentOS 6.5) with
kernel GENERIC without DIAGNOSTIC I ran "dbench 30" on
a 2 GByte MD based UFS1 filesystem:

Throughput 45.5631 MB/sec 30 procs
Elapsed time: 721.75 seconds.

-- Adaptive mutex spin

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
100.00  390598   4034.18 vcache                 <all>
 87.65  343624   3535.86 vcache                 vcache_intern+44
  7.47   15761    301.37 vcache                 vcache_remove+21
  3.59   14706    144.67 vcache                 vcache_intern+1fc
  1.29   16503     52.19 vcache                 vcache_intern+14c
  0.00       4      0.09 vcache                 vcache_intern+b0

-- Adaptive mutex sleep

Total%  Count   Time/ms          Lock                       Caller
------ ------- --------- ---------------------- ------------------------------
100.00      12      5.43 vcache                 <all>
 63.34      11      3.44 vcache                 vcache_intern+44
 36.66       1      1.99 vcache                 vcache_intern+1fc

It is spinning about 0.6 % of the time -- doesn't look like contention to me.

--
J. Hannken-Illjes - hannken%eis.cs.tu-bs.de@localhost - TU Braunschweig 
(Germany)