xargs reads space and line delimited strings from stdin and executes a specified command with the input as arguments.

This makes xargs a very powerful command when you want to run a command against arbitrary input.

For example, let's run the following command:

echo 'hello    there\nbig         world' | xargs echo

xargs automatically parses hello, there, big, and world as individual inputs. Then, these arguments are passed to the specified command (in this case, also echo).

echo 'hello    there\nbig         world' | xargs echo

# What gets executed:
# echo 'hello' 'there' 'big' 'world'

# Output:
hello there big world

Note: All computed input arguments were passed to a single echo command.

For more practical usage, you may find yourself wanting to run a command per single input line. For example, when the command you're using doesn't support multiple arguments.

A common command I find this useful for is aws s3. aws s3 doesn't support passing multiple arguments, so each command may only accept a single bucket name.

For example, using brace expansion and xargs to pipe bucket names to aws s3 rm doesn't work:

echo s3://my-bucket/partition-{1..3}/ | xargs aws s3 rm --recursive

# What gets executed:
# aws s3 rm --recursive 's3://my-bucket/partition-1/' 's3://my-bucket/partition-2/' 's3://my-bucket/partition-3/'

# Output:
Unknown options: s3://my-bucket/partition-2/,s3://my-bucket/partition-3/

To solve this, xargs accepts an -n <number> argument. The -n argument configures how many arguments should be per command invocation (the default is 5000).

So, since aws s3 rm only accepts one argument, we can pass -n 1 to xargs to execute a single command per line:

echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 aws s3 rm --recursive

# What gets executed (sequentially):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success

Note: The aws s3 rm command was invoked three times, once per input.

Success! 🎉

Now that we're running one command per line, it's important to note it's done so sequentially. However, you may also want to parallelize execution of long-running commands.

xargs has an argument for parallelizing execution: -P <number>.

The above aws s3 rm --recursive example may take very long per parition, so we can maximize efficency by running all the commands at once. To do this, we'll pass the -P 3 argument:

echo s3://my-bucket/partition-{1..3}/ | xargs -n 1 -P 3 aws s3 rm --recursive

# What gets executed (in parallel):
# aws s3 rm --recursive 's3://my-bucket/partition-1/'
# aws s3 rm --recursive 's3://my-bucket/partition-2/'
# aws s3 rm --recursive 's3://my-bucket/partition-3/'

# Output:
success
success
success

Tip: To run one command per CPU, you can use this handy shortcut:

xargs -n 1 -P $(getconf _NPROCESSORS_ONLN) command

Good luck! 🏎