CVS commit: pkgsrc/parallel/parallel

["Filip Hajny" <fhajny%netbsd.org@localhost>]

Module Name:    pkgsrc
Committed By:   fhajny
Date:           Tue Sep 13 09:41:29 UTC 2016

Modified Files:
        pkgsrc/parallel/parallel: Makefile PLIST distinfo
Removed Files:
        pkgsrc/parallel/parallel/patches: patch-src_parallel.pod

Log Message:
Update parallel/parallel to 20160822

Upstream changes:

20160822

- Bug fixes and man page updates.

20160722

- env_parallel is now ready for wider testing. It is still beta
  quality.

- env_parallel is heavily modified for all shells and testing has been
  increased.

- Selectively choosing what to export using --env now works for
  env_parallel (bash, csh, fish, ksh, pdksh, tcsh, zsh).

- --round-robin now gives more work to a job that processes faster
  instead of same amount to all jobs.

- --pipepart works on block devices on GNU/Linux.

- Bug fixes and man page updates.

20160622

- $PATH can now be exported using --env PATH. Useful if GNU Parallel
  is not in your path on remote machines.

- If --block is left out, --pipepart will use a block size that will
  result in 10 jobs per jobslot.

- Bug fixes and man page updates.

20160522

- niceload --net pauses the program if the internet connection is
  overloaded.

- Bug fixes and man page updates.

20160422

- :::+ and ::::+ work like ::: and :::: but links this input source to
  the previous input source in a --xapply fashion. Contrary to
  --xapply values do not wrap: The shortest input source determines
  the length.

- --line-buffer --keep-order now outputs continously from the oldest
  job still running. This is more what you would expect than the
  earlier behaviour where --keep-order had no effect with
  --line-buffer.

- env_parallel supports tcsh, csh, pdksh. In fish it now supports
  arrays. In csh/tcsh it now supports variables, aliases, and arrays
  with no special chars. In pdksh it supports aliases, functions,
  variables, and arrays.

- Function exporting on Mac OS X works around old Bash version.

- Better CPU detection on OpenIndiana.

- Bug fixes and man page updates.

20160322

- env_parallel is a function that exports the environment (functions,
  aliases, variables, and arrays) to GNU Parallel. Run 'man
  env_parallel' for details.

- niceload --prg now searches for substrings if no process with the
  name is found.

- Bug fixes and man page updates.

20160222

- Bug fixes and man page updates.

20160122

- --sql DBURL uses DBURL as storage for jobs and output. It does not
  run any jobs so it requires at least one --sqlworker. DBURL must
  point to a table.

- --sqlworker DBURL gets jobs from DBURL and stores the result back to
  DBURL.

- --sqlandworker is a shorthand for --sql and --sqlworker.

- --sqlworker requires the output of a single job to fit in memory.

- --results now also saves a file called 'seq' containing the sequence
  number.

- If $PARALLEL_ENV is a file, then that file will be read into
  $PARALLEL_ENV.

- man parallel_tutorial has been given an overhaul.

- --workdir now accepts replacementstrings.

- Bug fixes and man page updates.

20151222

- --transfer is now an alias for --transferfile {}.

- --transferfile works like --transfer, but takes an argument like
  --return. This makes it possible to combine transferring files
  with multiple input sources: parallel -S server --tf {1} wc {2}
  {1} ::: * ::: -l -w -c

- total_jobs() can now be used in {= =}: parallel echo job {#} of '{=
  $_=total_jobs() =}' ::: {1..50}

- Bug fixes and man page updates.

20151122

- Bug fixes and man page updates.

20151022

- --plus makes it possible to use {##} as a shorthand for
  {= $_=$Global::JobQueue->total_jobs() =} which gives the the number
  of jobs to run in total.

- {= $_=$Global::JobQueue->total_jobs() =} is incompatible with -X,
  -m, and --xargs.

- GNU Parallel is now mostly compatible with lsh
  (http://www.lysator.liu.se/~nisse/lsh/) and somewhat compatible with
  autossh (http://www.harding.motd.ca/autossh/).

- --workdir ... now also works when run locally.

- Bug fixes and man page updates.

20150922

- Bug fixes and man page updates.

20150822

- If $job->skip() is called in {= =} the job will not be run.

- @arg can now be used in {= =}: parallel echo '{= $arg[2] < 5 and
  $_="j" =}' ::: 1 2 3 ::: 4 5 6

- Bug fixes and man page updates.

20150722

- If a record fits in a block, --block only reads upto the size of
  --block. If not: --block reads a full --block more.

- Due to widely spread distributions still using --tollef even after a
  year of being retired following a year of being obsolete, --tollef
  now gives an error unless used with --gnu.

- --nice now uses a perl wrapper instead of the nice command.

- Bug fixes and man page updates.

20150622

- --halt has been rewritten completely. You can now combine
  percentages with success or fail. See the man page.

- Exit values 102..254 have been removed. 101 means more than 100 jobs
  failed.

- Killing through --timeout, --memfree, or --halt is now done as a
  process group.

- --termseq determines which signals are sent when a job is killed.

- An empty argument would previously cause no string to be
  inserted. This is now changed to '' being inserted, thus prepending
  a space to the output of: parallel echo {} b ::: ''

- $PARALLEL_ENV can now be set to an environment prepending the
  command. Used in env_parallel as mentioned in the manpage.

- --retry-failed will retry all failed jobs in a joblog. It will
  ignore any command given.

- --ssh and $PARALLEL_SSH can be used to set the command used for
  ssh. The command is assumed to behave the same way as ssh.

- --fifo now works in csh, too.

- Q(...) can be used in {= =} to shell quote a string.

- Bug fixes and man page updates.

20150522

- Security: The security issue for --sshlogin + --fifo/--cat has been
  fixed. Thereby all issues with
  http://lists.gnu.org/archive/html/parallel/2015-04/msg00045.html
  have been fixed.

- Security: After further security analysis the issue fixed in
  20150422 also fixed the problem for --tmux.

- Bug fixes and man page updates.

20150422

New in this release:

- Security fix. An attacker on the local system could make you
  overwrite one of your own files with a single byte. The problem
  exist when you use --compress or --tmux or --pipe or --cat or
  --fifo. The attacker must figure out the randomly chosen file name
  and create a symlink within a time window of 15 ms.

- --shuf will generate all jobs, and shuffle them before running
  them. This is useful to get a quick preview of the results before
  running the full batch.

- Bug fixes and man page updates.

20150322

New in this release:

- --number-of-cores respects 'taskset' on GNU/Linux.

- --joblog --pipe gives the data send and received in the log.

- Bug fixes and man page updates.

20150222

- --tmux has gotten a major overhaul.

- Bug fixes and man page updates.

20150122

- Remote jobs now send stderr (standard error) to stderr (standard
  error) instead of stdout (standard output).

- Remote execution command is now packed using base64 encoded
  bzip2. This means that big environments (app. 100 KB) can be
  transferred. This change makes remote execution alpha quality.

- --semaphoretimeout now takes a negative value. This means give up if
  the semaphore is not gotten within the timeout.

- --halt -1 and -2 now means halt if a job succeeds (so the opposite
  of 1 and 2).

- --no-keep-order will reverse --keep-order.

- Bash's second fix of shellshock caused --env to break again when
  exporting functions. This has been fixed again.

- A semibig refactoring of big functions. All non-trivial functions
  are now less than 100 lines. The refactoring makes this release beta
  quality.

- A description of the design decisions for GNU Parallel can be found
  in 'man parallel_design'.

- A bug fix in replacement strings caused rewrite of the replacement
  function. This makes use of replacement strings alpha quality.

- Bug fixes and man page updates.

20141122

- Remote systems can be divided into hostgroups (e.g. web and db) by
  prepending '@groupname/' to the sshlogin. Multiple groups can be
  given by separating groups with '+'. E.g. @web/www1 @web+db/www2
  @db/mariadb

- Remote execution can be restricted to servers that are part of one
  or more groups by '@groupname' as an sshlogin. Multiple groups can
  be given by separating groups with '+'. E.g. -S @web or -S @db+web

- With --hostgroup you can restrict arguments to certain hostgroups by
  appending '@groupname' to the argument.  Multiple groups can be
  given by separating groups with '+'. E.g. my_web_arg@web
  db-or-web-arg@db+web db-only-arg@db Thanks to Michel Courtine for
  developing a prototype for this.

- Bug fixes and man page updates.

20141022

- --env was changed to deal with Bash's new way of exporting a function.

- Bug fixes and man page updates.

20140922

- If the file give as --sshloginfile is changed it will be re-read
  when a job finishes though at most once per second. This makes it
  possible to add and remove hosts while running.

20140822

- GNU Parallel now uses the same shell it was started from as the
  command shell for local jobs. So if GNU Parallel is started from
  tcsh it will use tcsh as its shell even if the login $SHELL is
  different. For remote jobs the login $SHELL will be used.

- The whole current environment in bash can be copied by using a shell
  wrapper function (Search manual for env_parallel).

- --plus adds the replacement strings {+/} {+.} {+..} {+...} {..}
  {...} {/..} {/...}. The idea being that '+foo' matches the opposite
  of 'foo' and {} = {+/}/{/} = {.}.{+.} = {+/}/{/.}.{+.} = {..}.{+..}
  = {+/}/{/..}.{+..} = {...}.{+...} = {+/}/{/...}.{+...}

- GNU Parallel now deals correctly with the combination
  rsync-3.1.X-client and rsync-2.5.7-server

- Bug fixes and man page updates.

20140722

- {= perl expression =} can be used as replacement string. The
  expression should modify $_. E.g. {= s/\.gz$// =} to remove .gz from
  the string. This makes replacement strings extremely flexible.

- Positional perl expressions (similar to {2}) are given as {=2 perl
  expression=} where 2 is the position.

- One small backwards incompatibility: {1}_{2} will replace {2} with
  the empty string if there is only one argument. Previously {2} would
  have been left untouched.

- Replacement strings can be defined using --rpl. E.g. parallel --rpl
  '{.gz} s/\.gz$//' echo {.gz} ::: *.gz

- The parenthesis around {= perl expression =} can be changed with
  --parens.

- --tmux will direct the output to a tmux session instead of
  files. Each running jobs will be in its own window.

- --halt 10% will stop spawning new jobs if 10% failed so far.

- Bug fixes and man page updates.

20140622

- --shebang and --shebang-wrap now works on FreeBSD.

- --pipepart works with --header, --regexp, --cat and --fifo.

- ./configure --disable-documentation will not build documentation.

- {%} works as job slot.

- Bug fixes and man page updates.

20140522

- Use --semaphoretimeout to ignore the semaphore lock after a while.

- {%} introduced as job slot replacement string. It has known bugs.

- --slotreplace changes slot replacement string.

- Bug fixes and man page updates.

20140422

- --pipepart is a highly efficient alternative to --pipe if the input
  is a real file and not a pipe.

- If using --cat or --fifo with --pipe the {} in the command will be
  replaced with the name of a physical file and a fifo respectively
  containing the block from --pipe. Useful for commands that cannot
  read from standard input (stdin).

- --controlmaster has gotten an overhaul and is no longer
  experimental.

- --env is now copied when determining CPUs on remote system. Useful
  for copying $PATH if parallel is not in the normal path.

- --results now chops the argument if the argument is longer than the
  allowed path length.

- Build now survives if pod2* are not installed.

- Bug fixes and man page updates.

20140322

- Bug fixes and man page updates.

20140222

- --tollef has been retired.

- --compress has be redesigned due to bugs.

- Format of --eta and --joblog has changed slightly.

- Bug fixes and man page updates.

20140122

- Bug fixes and man page updates.

20131222

- GNU Parallel now has a motto: For people who live life in the
  parallel lane.

- Detect if the buffer dir $TMPDIR runs out of space. This changes a
  central piece of code, making this release alpha quality.

- --keep-order --(n)onall will sort according to the sshlogin.

- Detect the number of CPUs on DEC Tru64.

- GNU sql: --list-tables for sqlite3.

- Bug fixes and man page updates.

20131122

- A citation notice is printed on stderr only if stderr is a terminal,
  the user has not specified --no-notice and the user has not run
  --bibtex once. This makes the release alpha quality.

- --compress will compress temporary files. If the output is big and
  very compressible this will take up less disk space in $TMPDIR and
  possibly be faster due to less disk I/O.

- --compress-program comtrols which program to use for compressing
  temporary files.

- --bar show progress as a progress bar compatible with zenity.

- --resume can now be used with --result: Jobs already run will be
  skipped.

- --transfer and --basefile support paths relative to the --workdir by
  inserting /./ into the path.

- Bug fixes and man page updates.

20131022

- --transfer files with /./ in the path will copy the files relative
  to the --workdir.

- The maximal command length is now cached in a file halfing the
  startup time.

- Bug fixes and man page updates.

20130922

- PDF-files of documentation is now included.

- Bug fixes and man page updates.

20130822

- A tutorial showing much of GNU Parallel's functionality. Spend an
  hour walking through the tutorial. Your commandline will love you
  for it. man parallel_tutorial

- --line-buffer will buffer output on line basis. --group keeps the
  output together for a whole job. --ungroup allows output to mixup
  with half a line coming from one job and half a line coming from
  another job. --line-buffer fits between these two; it prints a full
  line, but will allow for mixing lines of different jobs.

- --record-env records all variables and functions set. Useful to
  record the normal environment before using --env _.

- --env _ exports all functions and variables to the remote server
  except for the ones recorded.

- New signing key. Due to recommendations from NIST
  http://www.keylength.com/en/4/ the signing key was changed from
  1024D/ID:FFFFFFF1 to 9888R/ID:88888888.

- Bug fixes and man page updates.

20130722

- --round-robin with --pipe will write all blocks to already running
  jobs.

- --env can now transfer Bash function for remote execution. That is
  pretty cool!

- Quite a few bug fixes and man page updates.

20130622

- --xapply now recycles arguments if an input source has more
  arguments than others.

- The sleep time between jobs is now both increased and decreased
  exponentially.

- 10 seconds installation check the signature using GnuPG if GnuPG is
  installed.

- Bug fixes and man page updates.

20130522

- --ctrlc is now default if --pipe is not used; thus making it
  possible to kill remotely started jobs simply by pressing Ctrl-C.

- --timeout 200% now means kill jobs that take more than twice the
  time of the median run time of a job.

- Jobs are now distributed round robin when having mulitiple
  --sshlogin instead of filling up one --sshlogin at a time.

- niceload: darwin version of --io --mem --swap. Thanks to Anders F
  Bjorklund.

- Bug fixes and man page updates.

20130422

- 10 seconds installation: wget -O - pi.dk/3 | sh

- HPUX CPU/core detection. Thanks to Javier Tarradas.

- CPU detection for HURD, IRIX, SCO OpenServer and (old) HPUX.

- --ctrlc will send SIGINT to tasks running on remote computers and
  thus killing them.

- --load now uses 'ps' to see immediately see the number of running
  processes instead of 'uptime' thus making --load react much faster.

- Testing on Centos 3.9 and FreeBSD 7 revealed a lot of compability
  bugs. Some of these required quite extensive changes making this
  release beta quality.

-  --retries works with --onall.

- The new --load computation now works on FreeBSD 7.

- --nice works under tcsh.

- Bug fixes and man page updates.

20130222

- --resume works with --pipe.

- --resume-failed will go through --joblog, redo the failed jobs and
  then continue like --resume.

- Negative positional arguments count from the end: {-1} means the
  last argument, {-2} the second to last.

- NetBSD CPU detection.

- --blocksize increases exponentially if it smaller than a full
  record.

- Processing n-line records (--pipe -L n) is now much faster.

- --tollef is obsoleted after discussion on the mailing list.

- Bug fixes and man page updates.


To generate a diff of this commit:
cvs rdiff -u -r1.11 -r1.12 pkgsrc/parallel/parallel/Makefile
cvs rdiff -u -r1.2 -r1.3 pkgsrc/parallel/parallel/PLIST
cvs rdiff -u -r1.5 -r1.6 pkgsrc/parallel/parallel/distinfo
cvs rdiff -u -r1.1 -r0 \
    pkgsrc/parallel/parallel/patches/patch-src_parallel.pod

Please note that diffs are not public domain; they are subject to the
copyright notices on the relevant files.

Modified files:

Index: pkgsrc/parallel/parallel/Makefile
diff -u pkgsrc/parallel/parallel/Makefile:1.11 pkgsrc/parallel/parallel/Makefile:1.12
--- pkgsrc/parallel/parallel/Makefile:1.11      Sat Jul  9 06:38:48 2016
+++ pkgsrc/parallel/parallel/Makefile   Tue Sep 13 09:41:29 2016
@@ -1,8 +1,7 @@
-# $NetBSD: Makefile,v 1.11 2016/07/09 06:38:48 wiz Exp $
+# $NetBSD: Makefile,v 1.12 2016/09/13 09:41:29 fhajny Exp $
 #
 
-DISTNAME=      parallel-20130122
-PKGREVISION=   4
+DISTNAME=      parallel-20160822
 CATEGORIES=    parallel
 MASTER_SITES=  ${MASTER_SITE_GNU:=parallel/}
 EXTRACT_SUFX=  .tar.bz2
@@ -15,6 +14,7 @@ LICENSE=      gnu-gpl-v3
 USE_TOOLS+=    perl:run pod2man pod2html
 
 REPLACE_PERL+= src/parallel src/niceload src/sem src/sem.pod src/sql
+REPLACE_SH+=   src/env_parallel
 
 GNU_CONFIGURE= yes
 USE_LANGUAGES= # none
@@ -22,6 +22,8 @@ USE_LANGUAGES=        # none
 DOCDIR=                ${PREFIX}/share/doc/parallel
 DOCFILES=      NEWS README COPYING
 
+CHECK_INTERPRETER_SKIP+=       bin/env_parallel.*
+
 post-install:
 .for i in ${DOCFILES}
        ${INSTALL_DATA} ${WRKSRC}/${i} ${DESTDIR}${DOCDIR}/

Index: pkgsrc/parallel/parallel/PLIST
diff -u pkgsrc/parallel/parallel/PLIST:1.2 pkgsrc/parallel/parallel/PLIST:1.3
--- pkgsrc/parallel/parallel/PLIST:1.2  Sun Dec  2 10:38:51 2012
+++ pkgsrc/parallel/parallel/PLIST      Tue Sep 13 09:41:29 2016
@@ -1,20 +1,44 @@
-@comment $NetBSD: PLIST,v 1.2 2012/12/02 10:38:51 cheusov Exp $
+@comment $NetBSD: PLIST,v 1.3 2016/09/13 09:41:29 fhajny Exp $
+bin/env_parallel
+bin/env_parallel.bash
+bin/env_parallel.csh
+bin/env_parallel.fish
+bin/env_parallel.ksh
+bin/env_parallel.pdksh
+bin/env_parallel.tcsh
+bin/env_parallel.zsh
 bin/niceload
 bin/parallel
 bin/sem
 bin/sql
+man/man1/env_parallel.1
 man/man1/niceload.1
 man/man1/parallel.1
 man/man1/sem.1
 man/man1/sql.1
+man/man7/parallel_design.7
+man/man7/parallel_tutorial.7
 share/doc/parallel/COPYING
 share/doc/parallel/NEWS
 share/doc/parallel/README
+share/doc/parallel/env_parallel.html
+share/doc/parallel/env_parallel.pdf
+share/doc/parallel/env_parallel.texi
 share/doc/parallel/niceload.html
+share/doc/parallel/niceload.pdf
 share/doc/parallel/niceload.texi
 share/doc/parallel/parallel.html
+share/doc/parallel/parallel.pdf
 share/doc/parallel/parallel.texi
+share/doc/parallel/parallel_design.html
+share/doc/parallel/parallel_design.pdf
+share/doc/parallel/parallel_design.texi
+share/doc/parallel/parallel_tutorial.html
+share/doc/parallel/parallel_tutorial.pdf
+share/doc/parallel/parallel_tutorial.texi
 share/doc/parallel/sem.html
+share/doc/parallel/sem.pdf
 share/doc/parallel/sem.texi
 share/doc/parallel/sql.html
+share/doc/parallel/sql.pdf
 share/doc/parallel/sql.texi

Index: pkgsrc/parallel/parallel/distinfo
diff -u pkgsrc/parallel/parallel/distinfo:1.5 pkgsrc/parallel/parallel/distinfo:1.6
--- pkgsrc/parallel/parallel/distinfo:1.5       Mon Nov  2 23:46:13 2015
+++ pkgsrc/parallel/parallel/distinfo   Tue Sep 13 09:41:29 2016
@@ -1,7 +1,6 @@
-$NetBSD: distinfo,v 1.5 2015/11/02 23:46:13 agc Exp $
+$NetBSD: distinfo,v 1.6 2016/09/13 09:41:29 fhajny Exp $
 
-SHA1 (parallel-20130122.tar.bz2) = 0f7eac7c8a574e173e322f359f4f3804c9d05203
-RMD160 (parallel-20130122.tar.bz2) = 642064b2b06083a39d9156a5ee2733ae614c2d5f
-SHA512 (parallel-20130122.tar.bz2) = 7975c5058db03f586a2db88c27e86118efa61d0cc23ad2bd058c17335d80435ffc506d8a4f83ec9d5879f082665257ad435f4f59886eab1c65b5ccbe5a5858cb
-Size (parallel-20130122.tar.bz2) = 238646 bytes
-SHA1 (patch-src_parallel.pod) = 7c0ce9636c308ccc736ac2bbc34c961f4857ad2f
+SHA1 (parallel-20160822.tar.bz2) = eacae06b0e145559d9f9d2f027be5cfac28f76bf
+RMD160 (parallel-20160822.tar.bz2) = 6f56b1bca8f6f29fd712f2657cff9c44796bcb90
+SHA512 (parallel-20160822.tar.bz2) = 303715075f50f0a0e18b1b9e65d860ae5ba9901efc6de096406ee69b8d91d4d3a648a966b388156024d0177f7d7b510b7df437c212ba587c4636d415278188ab
+Size (parallel-20160822.tar.bz2) = 1370647 bytes