GNU parallel

Shell tool for executing jobs in parallel locally or on remote machines

This article relies excessively on references to primary sources. Please improve this article by adding secondary or tertiary sources.
Find sources: "GNU parallel" – news · newspapers · books · scholar · JSTOR (September 2015) (Learn how and when to remove this message)

|

Parallel

Developer(s)| GNU Project
Stable release|

20230522[1] / 22 May 2023; 13 months ago (2023-05-22)

Repository|

* git.savannah.gnu.org/cgit/parallel.git

Written in| Perl
Operating system| GNU
Type| Utility
License| GPLv3
Website| www.gnu.org/software/parallel/

GNU parallel is a command-line utility for Linux and other Unix-like operating systems which allows the user to execute shell scripts or commands in parallel. GNU parallel is free software, written by Ole Tange in Perl. It is available under the terms of GPLv3.[2]

## Usage

[edit]

Introduction video, Part 1 Introduction video, Part 2

The most common usage is to replace the shell loop, for example

while read x; do
do_something "$x"
done < list

to the form of

cat list | parallel do_something

where the file `list` contains arguments for `do_something` and where `process_output` may be empty.

Scripts using parallel are often easier to read than scripts using pexec.

The program parallel features also

* grouping of standard output and standard error so the output of the parallel running jobs do not run together;
* retaining the order of output to remain the same order as input;
* dealing nicely with filenames containing special characters such as space, single quote, double quote, ampersand, and UTF-8 encoded characters;

By default, parallel runs as many jobs in parallel as there are CPU cores.

## Examples

[edit]

find . -name "*.foo" | parallel grep bar

The above is the parallel equivalent to:

find . -name "*.foo" -exec grep bar {} +

This searches in all files in the current directory and its subdirectories whose name end in `.foo` for occurrences of the string `bar`. The parallel command will work as expected unless a file name contains a newline. In order to avoid this limitation one may use:

find . -name "*.foo" -print0 | parallel -0 grep bar

The above command uses the null character to delimit file names.

find . -name "*.foo" | parallel -X mv {} /tmp/trash

The above command expands `{}` with as many arguments as the command line length permits, distributing them evenly among parallel jobs if required. This can lower process overhead for short-lived commands that take less time to finish than they do to launch.

find . -maxdepth 1 -type f -name "*.ogg" | parallel -X -r cp -v -p {} /home/media

The command above does the same as:

cp -v -p *.ogg /home/media

However, the former command which uses `find`/`parallel`/`cp` is more resource efficient and will not halt with an error if the expansion of *.ogg is too large for the shell.

Wikimedia Commons has media related to GNU parallel.

## See also

[edit]

* xargs
* pexec
* GNU Queue

## References

[edit]

1. ^ Tange, Ole (22 May 2023). "GNU Parallel 20230522 ('Charles') released [stable]". parallel (Mailing list).
2. ^ "GNU Parallel". GNU.org.

## External links

[edit]

* Official website
* Project page and manual page of GNU parallel

* v
* t
* e

GNU Project

History|

* GNU Manifesto
* Free Software Foundation
* Europe
* India
* Latin America
* History of free software

Licenses|

* GNU General Public License
* linking exception
* font exception
* GNU Lesser General Public License
* GNU Affero General Public License
* GNU Free Documentation License

Software|

* GNU (variants)
* Hurd
* Linux-libre
* glibc
* Bash
* coreutils
* findutils
* Build system
* GCC
* binutils
* GDB
* GRUB
* GNUstep
* GIMP
* Jami
* GNU Emacs
* GNU TeXmacs
* GNU Octave
* GNU Taler
* GNU R
* GSL
* GMP
* GNU Electric
* GNU Archimedes
* GNUnet
* GNU Privacy Guard
* Gnuzilla (IceCat)
* GNU Health
* GNUmed
* GNU LilyPond
* GNU Go
* GNU Chess
* Gnash
* Guix
* more...

Contributors|

* Alexandre Oliva
* Benjamin Mako Hill
* Bradley M. Kuhn
* Brian Fox
* Federico Heinz
* Georg C. F. Greve
* John Sullivan
* Nagarjuna G.
* Richard M. Stallman

Other topics|

* GNU/Linux naming controversy
* Revolution OS
* Free Software Foundation anti-Windows campaigns
* Defective by Design

*[v]: View this template
*[t]: Discuss this template
*[e]: Edit this template