>
> As Thor (I think it was) pointed out, going out to disk - even if you
> hit the disk's cache - is significantly slower than finding the data
> already in host RAM.
so it will be little slower as ONE process.
but making lower load which is more important on multiuser system.

> > or/and large blocks it's no difference.
>
> I don't see what difference the block size makes.  Could you explain?

when blocksize>>readahead size.

>
> >> (Unless the application provides such a place, which we agreed it
> >> shouldn't need to do.)
> > application can use O_ASYNC
>
> How does that help?  To get readahead, there has to be somewhere to put
> the readahead.  Direct I/O, by definition, doesn't keep kernel buffers,

you do this way:
request_read_first_block_and_wait
request_read_second_block
process_first block
wait_for_second_block_read
request_read_third_block
process_second_block
wait_for_third_block_read
request_read_fourth_block
process_third....