Subject: Re: progress(1) buffersize
To: None <tech-userlevel@netbsd.org>
From: Christos Zoulas <christos@astron.com>
List: tech-userlevel
Date: 06/04/2007 19:26:41
In article <20070604183552.GN26767@canolog.ninthwonder.com>,
Allen Briggs  <briggs@netbsd.org> wrote:
>-=-=-=-=-=-
>
>I found that the buffersize passed to progress can make a pretty
>sizable different in speed -- at least on a 'dd' test.  The 1k
>(BUFSIZ) default buffer size might be handy for really tight
>memory requirements, but a larger buffer size gets better pipe
>throughput.
>
>So I made a few changes (make the buffer dynamically allocated,
>make the buffer size a command-line option, and default the buffer
>size to 64k) that I'd like to propose.
>
>Diffs attached.  Comments welcome.
>
>-allen
>
>-- 
>Allen Briggs  |  http://www.ninthwonder.com/~briggs/  |  briggs@ninthwonder.com
>
>-=-=-=-=-=-
>
>? .gdbinit
>? progress
>? progress.cat1
>Index: progress.1
>===================================================================
>RCS file: /cvsroot/src/usr.bin/progress/progress.1,v
>retrieving revision 1.12
>diff -p -u -r1.12 progress.1
>--- progress.1	12 Apr 2007 06:31:20 -0000	1.12
>+++ progress.1	4 Jun 2007 18:30:02 -0000
>@@ -30,7 +30,7 @@
> .\" ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
> .\" POSSIBILITY OF SUCH DAMAGE.
> .\"
>-.Dd April 12, 2007
>+.Dd June 4, 2008
> .Dt PROGRESS 1
> .Os
> .Sh NAME
>@@ -39,6 +39,7 @@
> .Sh SYNOPSIS
> .Nm
> .Op Fl ez
>+.Op Fl b Ar buffersize
> .Op Fl f Ar file
> .Op Fl l Ar length
> .Op Fl p Ar prefix
>@@ -65,6 +66,11 @@ simply displays a count of the data and 
> .Pp
> The options are as follows:
> .Bl -tag -width XlXlengthXX
>+.It Fl b Ar buffersize
>+Read in buffers of the specified size (default 64k).
>+An optional suffix (per
>+.Xr strsuftoll 3 )
>+may be given.
> .It Fl e
> Display progress to standard error instead of standard output.
> .It Fl f Ar file
>Index: progress.c
>===================================================================
>RCS file: /cvsroot/src/usr.bin/progress/progress.c,v
>retrieving revision 1.14
>diff -p -u -r1.14 progress.c
>--- progress.c	7 Feb 2007 15:21:21 -0000	1.14
>+++ progress.c	4 Jun 2007 18:30:02 -0000
>@@ -73,8 +73,9 @@ static void
> usage(void)
> {
> 	fprintf(stderr,
>-	    "usage: %s [-ez] [-f file] [-l length] [-p prefix] cmd [args...]\n",
>-	    getprogname());
>+	    "usage: %s [-ez] [-b buffersize] [-f file] [-l length]\n"
>+	    "       %*.s [-p prefix] cmd [args...]\n",
>+	    getprogname(), (int) strlen(getprogname()), "");

Why %*.s? we never do this...

> 	exit(EXIT_FAILURE);
> }
> 
>@@ -82,7 +83,8 @@ usage(void)
> int
> main(int argc, char *argv[])
> {
>-	static char fb_buf[BUFSIZ];
>+	long long buffersize;

Why make buffersize long long, since malloc can only handle size_t?
Or at least check that buffersize < SIZE_T_MAX.

>+	char *fb_buf;
> 	char *infile = NULL;
> 	pid_t pid = 0, gzippid = 0, deadpid;
> 	int ch, fd, outpipe[2];
>@@ -97,10 +99,16 @@ main(int argc, char *argv[])
> 	/* defaults: Read from stdin, 0 filesize (no completion estimate) */
> 	fd = STDIN_FILENO;
> 	filesize = 0;
>+	buffersize = 64 * 1024;
> 	prefix = NULL;
> 
>-	while ((ch = getopt(argc, argv, "ef:l:p:z")) != -1)
>+	while ((ch = getopt(argc, argv, "b:ef:l:p:z")) != -1)
> 		switch (ch) {
>+		case 'b':
>+			/* 256MB seems absurdly large */
>+			buffersize = strsuftoll("buffer size", optarg,
>+			    0, 256 * 1024 * 1024 );

no space before paren.

>+			break;
> 		case 'e':
> 			eflag++;
> 			break;
>@@ -202,6 +210,10 @@ main(int argc, char *argv[])
> 	else
> 		ttywidth = ts.ts_cols;
> 
>+	fb_buf = malloc(buffersize);
>+	if (fb_buf == NULL)
>+		err(1, "malloc for buffersize");
>+
> 	if (pipe(outpipe) < 0)
> 		err(1, "output pipe");
> 	pid = fork();
>@@ -219,7 +231,7 @@ main(int argc, char *argv[])
> 	close(outpipe[0]);
> 
> 	progressmeter(-1);
>-	while ((nr = read(fd, fb_buf, BUFSIZ)) > 0)
>+	while ((nr = read(fd, fb_buf, buffersize)) > 0)
> 		for (off = 0; nr; nr -= nw, off += nw, bytes += nw)
> 			if ((nw = write(outpipe[1], fb_buf + off,
> 			    (size_t) nr)) < 0)
>@@ -255,5 +267,8 @@ main(int argc, char *argv[])
> 	}
> 
> 	progressmeter(1);
>+
>+	free(fb_buf);
>+
> 	exit(cmdstat ? cmdstat : gzipstat);
> }
>
>-=-=-=-=-=-