We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

0

Get the octal, hexadecimal and decimal codes of the ASCII character set

 $ man ascii

— by Janos on March 29, 2012, 9:48 a.m.

Explanation

Knowing the octal, hexadecimal or decimal code of the ASCII character set can be handy at times. In the past, too often I did things like:

perl -e 'for my $n (1 .. 255) { print $n, chr($n), $n, "\n"; }'

... when a simple man ascii would have done the trick...

On a related note, these all print the letter "A":

echo -e '\0101'
printf '\101'
printf '\x41'
perl -e 'print "\x41"'

1

Sort and remove duplicate lines in a file in one step without intermediary files

 $ vi +'%!sort | uniq' +wq file.txt

— by Janos on March 22, 2012, 1:09 p.m.

Explanation

We open a file with vi and run two vi commands (specified with +):

  1. %!sort | uniq
    • % = range definition, it means all the lines in the current buffer.
    • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
  2. wq = write buffer contents to file and exit.

1

Show files containing "foo" and "bar" and "baz"

 $ grep -l 'baz' $(grep -l 'bar' $(grep -lr 'foo' *) )

— by Anon5eqErEbE on March 16, 2012, 5:37 a.m.

Explanation

Most people familiar with extended regular expressions know you can use the pipe symbol | to represent "or", so to see files containing any of "foo", "bar", or "baz" you could run:

grep -Elr 'foo|bar|baz' *

There is no corresponding symbol representing "and", but you can achieve the same effect by nesting invocations to grep. grep -lr 'foo' * returns a list of filenames in or below the current directory containing "foo". Via the $( ... ) syntax, this list is then operated on by grep -l 'bar', returning a list of filenames containing both 'foo' and 'bar', which finally is operated on by grep -l "baz". The end result is a list of filenames containing all three terms.

Limitations

This one-liner results in scanning files multiple times. You will want to put the term you expect to match the fewest number of times farthest to the right (that is, in the same position as "foo") and the one you expect to match most frequently farthest to the left (the same position as "baz"). This way, you will weed out the largest number of files sooner, making the one-liner complete more quickly.

0

List or edit and re-execute commands from the history list

 $ fc -l

— by Janos on March 15, 2012, 12:10 p.m.

Explanation

fc is a little known but very useful bash built-in.

  • fc -l will list the most recent 16 commands
  • fc will open the last command in a text editor defined in the environmental variable FCEDIT or EDITOR or else vi, and re-execute when you exit
  • fc 5 9 will open the history entries 5 to 9 in a text editor
  • fc -s pat=sub will run the last command after substituting pat with sub in it (does not open editor)
  • fc -s pat=sub cc is the same but on the last command starting with cc
  • fc -s cc will run the last command starting with cc

For more info see help fc.

0

Find the most recently modified files in a directory and all subdirectories

 $ find /path/to/dir -type f -mtime -7 -print0 | xargs -0 ls -lt | head

— by Janos on March 8, 2012, 5:10 p.m.

Explanation

  • find /path/to/dir -type f -mtime -7 -print0 prints all the files in the directory tree that have been modified within the last 7 days, with null character as the delimiter
  • xargs -0 ls -lt expects a null delimited list of filenames and will sort the files by modification time, in descending order from most recent to oldest
  • Since we are looking for the most recent files, with head we get the first 10 lines only

Note that if there are too many files in the output of find, xargs will run multiple ls -lt commands and the output will be incorrect. This is because the maximum command line length is getconf ARG_MAX and if this is exceeded xargs has to split the execution to multiple commands. So depending on your use case you may need to tweak the -mtime parameter to make sure there are not too many lines in the output.

0

List open files

 $ lsof -n

— by Janos on March 2, 2012, 10:01 a.m.

Explanation

With the -n flag it will not try to resolve network numbers to host names for network files, making it run a bit faster.

With the -c option you can select processes executing a matching command. And with the -t flag the output will be simply process ids without a header, suitable to use with kill. For example you can kill Google Chrome process gone crazy like this:

kill -HUP $(lsof -n -c /google/i -t)

Here /google/i is a regular expression pattern with case insensitive matching.

0

Set a colorful bash prompt per dev test prod environments

 $ PS1='\[\e[1;31m\][\u@\h \W]\$\[\e[0m\] '

— by Janos on Feb. 25, 2012, 2:46 p.m.

Explanation

It is useful to set a different color for the shell prompt in different deployment environments like dev/test/production, so that you don't mix up your multiple windows and do something by accident in the wrong window.

  • PS1 contains the format of the primary prompt
  • \[\e[1;31m\] sets the foreground color to red
  • \u will be substituted with the current username
  • \h will be substituted with the hostname
  • \W will be substituted with the current directory name
  • \[\e[0m\] is the end marker of the color setting

To make the color stand out even more for root users, the inverse color can be interesting too:

PS1='\[\e[7;31m\][\u@\h \W]\$\[\e[0m\] '

Other color examples:

#PS1='\[\e[1;32m\][\u@\h \W]\$\[\e[0m\] ' # green
#PS1='\[\e[1;33m\][\u@\h \W]\$\[\e[0m\] ' # yellow
#PS1='\[\e[1;34m\][\u@\h \W]\$\[\e[0m\] ' # blue

You can learn more in man bash, search for "PROMPTING".

Limitations

Your terminal program must support colors, of course ;-)

1

Find in files, recursively

 $ grep -rn 'nameserver' /etc 2>/dev/null

— by atpessoa on Feb. 19, 2012, 8:24 a.m.

Explanation

  • -r make a search recursively;
  • -n print line numbers;
  • -H is not need, is default;

Limitations

  • -i use for case insensitive search;

2

How to send an http POST to a website with a file input field

 $ curl -L -v -F "value=@myfile" "http://domain.tld/whatever.php"

— by openiduser14 on Feb. 15, 2012, 11:26 p.m.

Explanation

  • curl read "man curl" if you need to info like using cookies,etc. you can also use wget
  • -L follow redirects
  • -v be verbose
  • -F an input field
  • value= the name of the input field
  • @myfile the file you want uploaded
  • "http://domain.tld/whatever.php" the url that will take the file

1

Calculate the total disk space used by a list of files or directories

 $ du -cshx ./a ./b

— by openiduser14 on Feb. 15, 2012, 10:43 p.m.

Explanation

  • -s, --summarize; display only a total for each argument
  • -c, --total; produce a grand total
  • -x, --one-file-system; skip directories on different file systems
  • -h, --human-readable; print sizes in human readable format (e.g., 1K 234M 2G)

1

Create a compressed tar file that rsync will transfer efficiently

 $ GZIP='--rsyncable' tar cvzf bobsbackup.tar.gz /home/bob

— by Anon6y5E4Use on Feb. 15, 2012, 12:24 p.m.

Explanation

rsync works by comparing files on the local and remote machine and only sending those parts of the file that have changed. The normal way compression works, it results in everything after the modification changing, meaning lots of data ends up going over the network when you try to rsync compressed files.

The --rsyncable option to gzip changes the compression scheme so that modifications to the input file only affect the part of the file where they're located. This allows rsync to work its magic.

In this one-liner, the z option to tar calls gzip, which recognizes and uses any options specified in the GZIP environment variable.

Limitations

Using the --rsyncable option results in a slightly larger compressed file.

Not all versions of gzip include this feature - use the --help option to see if it's available on your system.

1

Cut select pages from a pdf file and create a new file from those pages.

 $ ps2pdf -dFirstPage=3 -dLastPage=10 input.pdf output.pdf

— by Anon6y5E4Use on Feb. 15, 2012, 11:08 a.m.

Explanation

ps2pdf is a script that comes with Ghostscript - despite the name, it can accept PDF files as input, not just postscript files.

Limitations

Only a single contiguous range of pages can be specified.

0

Calculate the total disk space used by a list of files or directories

 $ du -c

— by openiduser30 on Feb. 14, 2012, 1:34 a.m.

Explanation

-c option of du prints the total size of the arguments

0

View a file with line numbers

 $ cat -n /path/to/file | less

— by openiduser28 on Feb. 13, 2012, 5:14 p.m.

Explanation

cat -n will number all lines of a file.

Limitations

It will add some white spaces as padding.

1

Print the lines of file2 that are missing in file1

 $ comm -23 file2 file1

— by Anon9ge6A4uD on Feb. 13, 2012, 8:26 a.m.

Explanation

The POSIX-standard comm utility can do this also.

  • -2 suppresses lines from the second argument (file1)
  • -3 suppresses lines appearing in both files

Limitations

Assumes that file1 and file2 are already sorted. If they are not, you can use process substitution to do so:

comm -23 <(sort file2) <(sort file1)

Process substitution is a bash-specific feature (also available in zsh but with a different syntax).

0

Print the lines of file2 that are missing in file1

 $ grep -vxFf file1 file2

— by Janos on Feb. 8, 2012, 2:42 p.m.

Explanation

  • -f is to specify a file with the list of patterns: file1
  • -F is to treat the patterns fixed strings, without using regular expressions
  • -x is to match exactly the whole line
  • -v is to select non-matching lines

The result is effectively the same as:

diff file1 file2 | grep '^>' | sed -e s/..//

Limitations

The flags of grep might work differently depending on the system. So yeah you might prefer the second way which should work everywhere. Nonetheless the various of flags of grep are interesting.

1

Uses 'at' to run an arbitrary command at a specified time.

 $ echo 'play alarmclock.wav 2>/dev/null' | at 07:30 tomorrow

— by Anon5MAQumYj on Feb. 4, 2012, 11:03 a.m.

Explanation

at 07:30 tomorrow schedules a job for 7:30 AM the next day, running whatever command or script is fed to it as standard input. The format for specifying time and date is rather flexible. http://tinyurl.com/ibmdwat

echo 'play alarmclock.wav 2>/dev/null' | feeds the play alarmclock.wav command to at, while 2>/dev/null causes the text output of play to be thrown away (we are only interested in the alarm sound).

1

Calculate an h index from an EndNote export

 $ MAX=$(NUM=1;cat author.xml |perl -p -e 's/(Times Cited)/\n$1/g'|grep "Times Cited" |perl -p -e 's/^Times Cited:([0-9]*).*$/$1/g'|sort -nr | while read LINE; do if [ $LINE -ge $NUM ]; then echo "$NUM"; fi; NUM=$[$NUM+1]; done;); echo "$MAX"|tail -1

— by openiduser14 on Feb. 4, 2012, 1:06 a.m.

Explanation

EndNote?! I know but sometimes we have windows users as friends

1

Cut select pages from a pdf file and create a new file from those pages.

 $  pdftk input.pdf cat 2-4 7 9-10 output output.pdf

— by mmaki on Feb. 3, 2012, 6:50 a.m.

Explanation

pdftk is the PDF Toolkit

input.pdf is the input file.

cat 2-4 7 9-10 concatenate (combine) pages 2,3,4,7,9,10 of input.pdf.

output output.pdf the resulting pdf file containing the above pages.

0

Find in files, recursively

 $ find /etc -type f -print0 2>/dev/null | xargs -0 grep --color=AUTO -Hn 'nameserver' 2>/dev/null

— by openiduser21 on Feb. 2, 2012, 7:32 p.m.

Explanation

In the example above, find and display every file in /etc containing the string nameserver with the corresponding line, including line number, sample output:

/etc/ppp/ip-up.d/0dns-up:9:# Rev. Dec 22 1999 to put dynamic nameservers last.

/etc/ppp/ip-up.d/0dns-up:23:# nameservers given by the administrator. Those for which 'Dynamic' was chosen

/etc/ppp/ip-up.d/0dns-up:24:# are empty. 0dns-up fills in the nameservers when pppd gets them from the

/etc/ppp/ip-up.d/0dns-up:26:# 'search' or 'domain' directives or additional nameservers. Read the

/etc/ppp/ip-up.d/0dns-up:77:# nameserver lines to the temp file.

1

Re-compress a gzip (.gz) file to a bzip2 (.bz2) file

 $ time gzip -cd file1.tar.gz 2>~/logfile.txt | pv -t -r -b -W -i 5 -B 8M | bzip2 > file1.tar.bz2 2>>~/logfile .txt

— by DAVEB on Feb. 1, 2012, 6:02 p.m.

Explanation

*Requires PV (pipe viewer) if you want to monitor throughput; otherwise you can leave out the pv pipe.

Transparently decompresses an arbitrary .gz file (does not have to be a tar) and re-compresses it to bzip2, which has better compression and error recovery. Echoes error messages to a file named logfile.txt in your home directory.

NOTE: The original .gz file will NOT be deleted. If you want to save space, you will have to delete it manually.

1

Test your hard drive speed

 $ time (dd if=/dev/zero of=zerofile bs=1M count=500;sync);rm zerofile

— by DAVEB on Feb. 1, 2012, 5:35 p.m.

Explanation

Creates a 500MB blank file and times how long it takes to finish writing the entire thing to disk (sync)

time the entire dd + sync operation, and then remove the temporary file

Limitations

Works with Bash; not tested in other environments

1

Recursively remove all empty sub-directories from a directory tree

 $ find . -depth  -type d  -empty -exec rmdir {} \;

— by openiduser16 on Jan. 31, 2012, 11:15 p.m.

Explanation

Recursively remove all empty sub-directories from a directory tree using just find. No need for tac (-depth does that), no need for xargs as the directory contents changes on each call to rmdir. We're not reliant on the rmdir command deleting just empty dirs, -empty does that.

Limitations

Will make many calls to rmdir without using xargs, which bunches commands into one argument string, which is normally useful, but -empty /could/ end up being more efficient since only empty dirs will be passed to rmdir, so possibly fewer executions in most cases (searching / for example).

1

Group count sort a log file

 $ A=$(FILE=/var/log/myfile.log; cat $FILE | perl -p -e 's/.*,([A-Z]+)[\:\+].*/$1/g' | sort -u | while read LINE; do grep "$LINE" $FILE | wc -l | perl -p -e 's/[^0-9]+//g'; echo -e "\t$LINE"; done;);echo "$A"|sort -nr

— by openiduser14 on Jan. 31, 2012, 6:49 p.m.

Explanation

  • SQL: SELECT COUNT(x), x FROM y GROUP BY x ORDER BY count DESC;
  • BASH: a temp var for the last sort: $A=$(
  • the file you want: FILE=/var/log/myfile.log
  • dump the file to a stream: cat $FILE |
  • cut out the bits you want to count: perl -p -e 's/.*,([A-Z]+)[\:\+].*/$1/g' |
  • get a unique list: sort -u |
  • for each line/value in the stream do stuff: while read LINE; do
  • dump all lines matching the current value to an inner stream: grep "$LINE" $FILE |
  • count them: wc -l |
  • clean up the output of wc and drop the value on stdout: perl -p -e 's/[^0-9]+//g';
  • drop the current value to stdout: echo -e "\t$LINE";
  • finish per value operations on the outer stream: done;
  • finish output to the temp var: );
  • dump the temp var to a pipe: echo "$A" |
  • sort the list numerically in reverse: sort -nr

1

Use ghostscript to shrink PDF files

 $ gs -sDEVICE=pdfwrite -dCompatibilityLevel=1.4 -dPDFSETTINGS=/ebook -dNOPAUSE -dQUIET -dBATCH -sOutputFile=output.pdf input.pdf

— by openiduser10 on Jan. 31, 2012, 10:43 a.m.

Explanation

Replace input.pdf and output.pdf with the original PDF name and the new compressed version's file name respectively. The key to this is the PDFSETTINGS option which can be tuned for various levels of compression. For scanned text documents, I find the ebook setting works well enough for most purposes but you can experiment with the options below.

  • -dPDFSETTINGS=/screen (screen-view-only quality, 72 dpi images)
  • -dPDFSETTINGS=/ebook (low quality, 150 dpi images)
  • -dPDFSETTINGS=/printer (high quality, 300 dpi images)
  • -dPDFSETTINGS=/prepress (high quality, color preserving, 300 dpi imgs) '-dPDFSETTINGS=/default (almost identical to /screen)'