We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

2

Create a visual report of the contents of a usb drive

 $ find /path/to/drive -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr

— by Anon1Qa6UsYT on April 15, 2012, 6:04 a.m.

Explanation

versorge asks:

I have a bunch of usb volumes lying around and I would like to get a quick summary of what is on the drives. How much space is taken up by pdf, image, text or executable files. This could be output as a text summary, or a pie chart.

This one-liner produces a list like this:

  5804731229 FLAC audio bitstream data
   687302212 MPEG sequence
    99487460 data
    60734903 PDF document
    55905813 Zip archive data
    38430192 ASCII text
    32892213 gzip compressed data
    24847604 PNG image data
    16618355 XML 1.0 document text
    13876248 JPEG image data

The find command locates all regular files (-type f) below the given directory, which could be a mounted USB stick or any other directory. For each one, it runs the file -b command with the filename to print the file type; if this succeeds, it also prints the file size (-printf '%s\n'). This results in a list containing a file type on one line, followed by the file size on the next.

The awk script takes this as input. The GNU file command often produces very specific descriptions such as GIF image data, version 87a, 640 x 480 - to generalize these, we set the field separator to be a comma with the -F option. Referencing $1 then only uses what's to the left of the first comma, giving us a more generic description like GIF image data.

In the awk script, the first pattern-action pair NR%2 {i=$1} applies to each odd-numbered line, setting the variable i to be the file type description. The even-numbered lines are handled by NR%2==0 {a[i]+=$1}, adding the value of the line (which is the file size) to the array variable a[i]. This results in an array indexed by file type, with each array member holding the cumulative sum of bytes for that type. The END { ... } pattern-action pair finally prints out a formatted list of the total size for each file type.

At the end of the line, the sort command sorts the list, putting the file types with the largest numbers at the top.

Limitations

This one-liner uses the -b option to file and the -printf primary of find - these are supported by the GNU utilities but may not work elsewhere. It can also take a long time to run, since it needs to open and analyze every file below the given directory.

2

How to send an http POST to a website with a file input field

 $ curl -L -v -F "value=@myfile" "http://domain.tld/whatever.php"

— by openiduser14 on Feb. 15, 2012, 11:26 p.m.

Explanation

  • curl read "man curl" if you need to info like using cookies,etc. you can also use wget
  • -L follow redirects
  • -v be verbose
  • -F an input field
  • value= the name of the input field
  • @myfile the file you want uploaded
  • "http://domain.tld/whatever.php" the url that will take the file

2

Rename all files in the current directory by capitalizing the first letter of every word in the filenames

 $ ls | perl -ne 'chomp; $f=$_; tr/A-Z/a-z/; s/(?<![.'"'"'])\b\w/\u$&/g; print qq{mv "$f" "$_"\n}'

— by Janos on Nov. 1, 2011, 12:51 p.m.

Explanation

  • When you pipe something to perl -ne, each input line is substituted into the $_ variable. The chomp, tr///, s/// perl functions in the above command all operate on the $_ variable by default.
  • The tr/A-Z/a-z/ will convert all letters to lowercase.
  • The regular expression pattern (?<![.'])\b\w matches any word character that follows a non-word character except a dot or a single quote.
  • The messy-looking '"'"' in the middle of the regex pattern is not a typo, but necessary for inserting a single quote into the pattern. (The first single quote closes the single quote that started the perl command, followed by a single quote enclosed within double quotes, followed by another single quote to continue the perl command.) We could have used double quotes to enclose the perl command, but then we would have to escape all the dollar signs which would make everything less readable.
  • In the replacement string $& is the letter that was matched, and by putting \u in front it will be converted to uppercase.
  • qq{} in perl works like double quotes, and can make things easier to read, like in this case above when we want to include double quotes within the string to be quoted.
  • After the conversions we print a correctly escaped mv command. Pipe this to bash to really execute the rename with | sh.

Limitations

The above command will not work for files with double quotes in the name, and possibly other corner cases.

2

Remove spaces recursively from all subdirectories of a directory

 $ find /path/to/dir -type d | tac | while read LINE; do target=$(dirname "$LINE")/$(basename "$LINE" | tr -d ' '); echo mv "$LINE" "$target"; done

— by Janos on Sept. 20, 2011, 4:52 p.m.

Explanation

  • find path_to_dir -type d finds all the subdirectories
  • tac reverses the order. This is important to make "leaf" directories come first!
  • target=... stuff constructs the new name, removing spaces from the leaf component and keeping everything before that the same
  • echo mv ... for safety you should run with "echo" first, if the output looks good then remove the "echo" to really perform the rename

Limitations

In UNIX or BSD there is no tac. There you can use tail -r instead.

2

Rename all files in a directory to lowercase names

 $ paste <(ls) <(ls | tr A-Z a-z) | while read OLD NEW; do echo mv -v $OLD $NEW; done

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

  • <(cmd) is the filename of a named pipe (FIFO), where the named pipe is filled by the output of cmd
  • paste puts together the named pipes to form two columns: first column with the original filenames, second column with the lowercased filenames
  • ... | tr abc ABC transforms stdin by replacing any characters that appear in the first set of letters to the second set of letters
  • while read old new; do ...; done for each line it reads the first column into $old and the second column into $new

Limitations

  • Won't work if there are spaces in a filename.

1

Have script run itself in a virtual terminal

 $ tty >/dev/null || { urxvt -e /bin/sh -c "tty >/tmp/proc$$; while test x; do sleep 1; done" & while test ! -f /tmp/proc$$; do sleep .1; done; FN=$(cat /tmp/proc$$); rm /tmp/proc$$; exec >$FN 2>$FN <$FN; }

— by openiduser111 on March 9, 2018, 2:56 a.m.

Explanation

  • We begin by testing if the script is not in a terminal with tty.
  • If it is not we start a terminal that runs tty and saves it to a filename. $$ was set by the original script and is its PID. That is opened in the background using & and then the original script waits for the filename to appear, then reads and removes it.
  • Finally, the main command is a special syntax of the bash builtin command exec that contains nothing but redirections (of stdout, stderr, and stdin) so they will apply to every command in the rest of the script file.

1

Big CSV > batches > JSON array > CURL POST data with sleep

 $ cat post-list.csv | split -l 30 - --filter='jq -R . | jq --slurp -c .' | xargs -d "\n" -I % sh -c 'curl -H "Content-Type: application/json" -X POST -d '"'"'{"type":1,"entries":%}'"'"' http://127.0.0.1:8080/purge-something && sleep 30'

— by pratham2003 on March 7, 2018, 12:12 p.m.

Explanation

post-list.csv contains list of URLs in my example.

  • split -l 30 Split by 30 lines

  • - Use stdin as input for split

  • --filter Couldn't find a way to easily pipe to stdout from split, hence --filter

  • jq -R . From the jq manual - Don’t parse the input as JSON. Instead, each line of text is passed to the filter as a string

  • jq --slurp -c . From the jq manual - Instead of running the filter for each JSON object in the input, read the entire input stream into a large array and run the filter just once. -c makes it easier to pipe and use it in the xargs that follows.

  • xargs -d "\n" -I % sh -c Execute a command for each array. Use "\n" as delimiter. Use % as a placeholder in the command that follows.

  • Single quotes inside sh -c ' ... ' are escaped as '"'"' single-double-single-double-single. You can do whatever you need to inside sh -c ' ... && sleep 123'

Limitations

You need jq installed, for example in Debian / Ubuntu:

apt-get install jq`

See also https://stedolan.github.io/jq/manual/

I suspect the input file (cat post-list.csv) may not contain double or single quotes but haven't tested it.

1

List all packages with at least a class defined in a JAR file

 $ jar tf "$1" | grep '/.*\.class$' | xargs dirname | sort -u | tr / .

— by stefanobaghino on Feb. 19, 2018, 12:13 p.m.

Explanation

The jar command allows you to read or manipulate JAR (Java ARchive) files, which are ZIP files that usually contain classfiles (Java compiled bytecode files) and possibly manifests and configuration files. We specify that we want to list file contents (t) that we provide as an argument (f, otherwise the jar will be read from stdin).

From the output, we get only the paths that contain a classfile (grep), then the path to the package that contains it (xargs dirname), we get the unique, sorted paths and translate /s to .s (to display their names as they would be shown in Java syntax).

Limitations

Will only exhaustively list the packages with a defined class for languages that require packages to map to the directory structure (e.g.: Java does, Scala doesn't). If this convention is respected, the command will output an exhaustive list of packages nonetheless.

1

Output an arbitrary number of open TCP or UDP ports in an arbitrary range

 $ comm -23 <(seq "$FROM" "$TO") <(ss -tan | awk '{print $4}' | cut -d':' -f2 | grep "[0-9]\{1,5\}" | sort | uniq) | shuf | head -n "$HOWMANY"

— by stefanobaghino on Feb. 9, 2018, 3:51 p.m.

Explanation

Originally published (by me) on unix.stackexchange.com.

comm is a utility that compares sorted lines in two files. It outputs three columns: lines that appear only in the first file, lines that only appear in the second one and common lines. By specifying -23 we suppress the latter columns and only keep the first one. We can use this to obtain the difference of two sets, expressed as a sequence of text lines. I learned about comm here.

The first file is the range of ports that we can select from. seq produces a sorted sequence of numbers from $FROM to $TO. The result is piped to comm as the first file using process substitution.

The second file is the sorted list of ports, that we obtain by calling the ss command (with -t meaning TCP ports, -a meaning all - established and listening - and -n numeric - don't try to resolve, say, 22 to ssh). We then pick only the fourth column with awk, which contains the local address and port. We use cut to split address and port with the : delimiter and keep only the latter (-f2). ss also output an header, that we get rid of by grepping for non-empty sequences of numbers that are no longer than 5. We then comply with comm's requirement by sorting numerically (-n) and getting rid of duplicates with uniq.

Now we have a sorted list of open ports, that we can shuffle to then grab the first "$HOWMANY" ones with head -n.

Example

Grab the three random open ports in the private range (49152-65535)

comm -23 <(seq 49152 65535) <(ss -tan | awk '{print $4}' | cut -d':' -f2 | grep "[0-9]\{1,5\}" | sort | uniq) | shuf | head -n 3

could return for example

54930
57937
51399

Notes

  • switch -t with -u in ss to get free UDP ports instead.
  • drop shuf if you're not interested in grabbing a random port

1

Ternary conditional clause

 $ [ test_statement ] && ( then_statement ) || ( else_statement );

— by dhsrocha on Jan. 22, 2018, 5:27 p.m.

Explanation

The test_statement is under a child process (a.k.a. subshell), which will return a boolean output. The then_statement depends on the former to be executed. Otherwise, it will be done upon the latter.

1

Get executed script's current working directory

 $ CWD=$(cd "$(dirname "$0")" && pwd)

— by dhsrocha on Jan. 22, 2018, 4:55 p.m.

Explanation

Will return excuting script's current working directory, wherever Bash executes the script containing this line.

1

Random Git Commit

 $ git commit -m "$(w3m whatthecommit.com | head -n 1)"

— by Jab2870 on Jan. 5, 2018, 4:55 p.m.

Explanation

This will commit a message pulled from What the Commit.

-m allows you to provide the commit message without entering your editor

w3m is a terminal based web browser. We basically use it to strip out all of the html tags

head -n 1 will grab only the first line

Limitations

This requires you to have w3m installed

1

Blackhole ru zone

 $ echo "address=/ru/0.0.0.0" | sudo tee /etc/NetworkManager/dnsmasq.d/dnsmasq-ru-blackhole.conf && sudo systemctl restart network-manager

— by olshek_ on Nov. 14, 2017, 2:12 p.m.

Explanation

It creates dnsmasq-ru-blackhole.conf file with one line to route all domains of ru zone to 0.0.0.0.

You might use "address=/home.lab/127.0.0.1" to point allpossiblesubdomains.home.lab to your localhost or some other IP in a cloud.

1

Retrieve dropped connections from firewalld journaling

 $ sudo journalctl -b | grep -o "PROTO=.*" | sed -r 's/(PROTO|SPT|DPT|LEN)=//g' | awk '{print $1, $3}' | sort | uniq -c

— by FoxBuru on Sept. 14, 2017, 5:10 p.m.

Explanation

We take the output of journalctl since the last boot (-b flag) and output from PROTO= until the EOL. Then, we remove identification tags (PROTO=/SPT=/DPT=/LEN=) and print just the protocol and destination port (cols 1 and 3). We sort the output properly so we can aggregate them on the call over uniq.

Limitations

  • Only works on Linux
  • You use firewalld and you have logging set on ALL (see firewalld.conf for details)
  • You use journald for logging
  • Your user has sudo privileges

1

Get the latest Arch Linux news

 $ w3m https://www.archlinux.org/ | sed -n "/Latest News/,/Older News/p" | head -n -1

— by Jab2870 on Aug. 15, 2017, 10:35 a.m.

Explanation

w3m is a terminal web browser. We use it to go to https://www.archlinux.org/

We then use sed to capture the text between Latest News and Older News.

We then get rid of the last line which is Older News.

Limitations

For this, w3m would need to be installed. It should be installable on most systems.

If Arch change the format of there website significantly, this might stop working.

1

Make a new folder and cd into it.

 $ mkcd(){ NAME=$1; mkdir -p "$NAME"; cd "$NAME"; }

— by PrasannaNatarajan on Aug. 3, 2017, 6:49 a.m.

Explanation

Paste this function in the ~/.bashrc file.

Usage:

mkcd name1

This command will make a new folder called name1 and cd into the name1.

I find myself constantly using mkdir and going into the folder as the next step. It made sense for me to combine these steps into a single command.

1

Listen to the radio (radio2 in example)

 $ mpv http://a.files.bbci.co.uk/media/live/manifesto/audio/simulcast/hls/uk/sbr_med/llnw/bbc_radio_two.m3u8

— by Jab2870 on July 19, 2017, 2:44 p.m.

Explanation

MPV is a terminal audio player. You could also use vlc or any media player that supports streams.

To find a stream for your favourite uk radio station, look here: UK Audio Streams. If you are outside of the uk, Google is your friend

Limitations

Requires an audio player that supports streams.

1

Go up to a particular folder

 $ alias ph='cd ${PWD%/public_html*}/public_html'

— by Jab2870 on July 18, 2017, 6:07 p.m.

Explanation

I work on a lot of websites and often need to go up to the public_html folder.

This command creates an alias so that however many folders deep I am, I will be taken up to the correct folder.

alias ph='....': This creates a shortcut so that when command ph is typed, the part between the quotes is executed

cd ...: This changes directory to the directory specified

PWD: This is a global bash variable that contains the current directory

${...%/public_html*}: This removes /public_html and anything after it from the specified string

Finally, /public_html at the end is appended onto the string.

So, to sum up, when ph is run, we ask bash to change the directory to the current working directory with anything after public_html removed.

Examples

If I am in the directory ~/Sites/site1/public_html/test/blog/ I will be taken to ~/Sites/site1/public_html/

If I am in the directory ~/Sites/site2/public_html/test/sources/javascript/es6/ I will be taken to ~/Sites/site2/public_html/

1

Generate a sequence of numbers

 $ perl -e 'print "$_\n" for (1..10);'

— by abhinickz6 on May 30, 2017, 2:47 p.m.

Explanation

Print the number with newline character which could be replaced by any char.

1

Delete static and dynamic arp for /24 subnet

 $ for i in {1..254}; do arp -d 192.168.0.$i; done

— by dennyhalim.com on Oct. 21, 2016, 5:07 a.m.

Explanation

Simply loop from 1 to 254 and run arp -d for each IP address in the 192.168.0.0/24 network.

1

Shuffle lines

 $ ... | perl -MList::Util=shuffle -e 'print shuffle <>;'

— by openiduser81 on Jan. 31, 2016, 9:02 p.m.

Explanation

Sorting lines is easy: everybody knows the sort command.

But what if you want to do the other way around? The above perl one-liner does just that:

  • -MList::Util=shuffle load the shuffle function from the List::Util package
  • -e '...' execute Perl command
  • print shuffle <> call List::Util::shuffle for the lines coming from standard input, read by <>

1

Convert all flac files in dir to mp3 320kbps using ffmpeg

 $ for FILE in *.flac; do ffmpeg -i "$FILE" -b:a 320k "${FILE[@]/%flac/mp3}"; done;

— by Orkan on Sept. 20, 2015, 5:45 p.m.

Explanation

It loops through all files in current directory that have flac extension and converts them to mp3 files with bitrate of 320kpbs using ffmpeg and default codec.

1

Preserve your fingers from cd ..; cd ..; cd..; cd..;

 $ upup(){ DEEP=$1; [ -z "${DEEP}" ] && { DEEP=1; }; for i in $(seq 1 ${DEEP}); do cd ../; done; }

— by andreaganduglia on June 9, 2015, 3:09 p.m.

Explanation

Include this function in your .bashrc and on the following line alias up='upup'

Now you are able to go back in your path simply with up N. So, for example:

Z:~$ cd /var/lib/apache2/fastcgi/dynamic/
Z:/var/lib/apache2/fastcgi/dynamic$ up 2
Z:/var/lib/apache2$ up 3 
Z:/$

1

Get number of all Python Behave scenarios (including all examples from Scenario Outlines)

 $ behave -d | grep "scenarios passed" | cut -d, -f4 | sed -e 's/^[[:space:]]*//' | sed 's/untested/scenarios/g'

— by openiduser188 on April 17, 2015, 2:21 p.m.

Explanation

behave -d

-d stands for dry-run, so behave invokes formatters without executing the steps.

grep "scenarios passed"

Then we grep for the summary line containing number of all scenarios

cut -d, -f4

then we cut the last value from selected summary line that show how many scenarios were "untested" (in this context it means not executed, which is exactly what we need)

sed -e 's/^[[:space:]]*//'

Trim leading space

sed 's/untested/scenarios/g'

Lastly simple sed to replace untested with scenarios

1

Print a flat list of dependencies of a Maven project

 $ mvn dependency:list | sed -ne s/..........// -e /patterntoexclude/d -e s/:compile//p -e s/:runtime//p | sort | uniq

— by Janos on Sept. 22, 2014, 9:02 p.m.

Explanation

The mvn dependency:list command produces a list of dependencies that's readable but not very program-friendly, looking like this:

[INFO] The following files have been resolved:
[INFO]    joda-time:joda-time:jar:2.3:compile
[INFO]    junit:junit:jar:4.11:test
[INFO]    log4j:log4j:jar:1.2.12:compile

A sed can shave off the extra formatting to turn this into:

joda-time:joda-time:jar:2.4
log4j:log4j:jar:1.2.12

Explanation:

  • -n don't print by default
  • -e s/..........// shave off the first 10 characters
  • -e /patterntoexclude/d you can exclude some unwanted patterns from the list using the d command like this
  • -e s/:compile//p -e s/:runtime//p replace and print :compile and :runtime

As multi-module projects may include duplicates, filter the result through | sort | uniq