We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

0

Print the first instance of $foo that occurs after $bar

 $ awk '/interface Ethernet3\/1/ {instanza=1} /!/ {instanza=0} instanza && /ip address/ {print}' file...

— by Anon1UHy2ubE on May 11, 2012, 12:25 p.m.

Explanation

This can easily be done by setting a variable once the lead line of the stanza is found. Take the following file as input:

interface Ethernet3/1
 ip address 172.30.10.2 255.255.0.0
 ip router isis area1
 no ip route-cache
 isis metric 10
!
interface Ethernet3/2
 ip address 192.168.10.10 255.255.255.0
 ip router isis area1
 no ip route-cache
 isis metric 10

The first awk pattern-action pair in the one-liner /interface Ethernet3\/1/ {instanza=1} sets the instanza variable to 1 (true) when it encounters the "interface Ethernet 3/1" line. The second pattern-action /!/ {instanza=0} resets instanza back to 0 (false) once an exclamation point is encountered, ending the desired stanza. The third pattern-action instanza && /ip address/ {print} prints any line containing "ip address," but only if we are still in the desired stanza (instanza is true): in this case, ip address 172.30.10.2 255.255.0.0.

If you only wanted to see the first matching line of the first matching stanza, you could change the third action to be {print; exit} and awk would quit after printing the first match.

0

Sort du output in Human-readable format

 $ du -hsx * | sort -rh

— by Vaevictus on April 26, 2012, 9:08 p.m.

Explanation

sort supports -h for human readable number sorting.

Limitations

Probably a newer GNU only option for sort. :D

0

Replace symlinks with the actual files they are pointing at

 $ find /path/to/dir -type l -exec sh -c 'cp --remove-destination "$(readlink "{}")" "{}"' \; 

— by Janos on April 24, 2012, 3:29 p.m.

Explanation

  • All the double quoting is necessary to handle filenames with spaces.
  • Calling sh with -exec is necessary to evaluate readlink for each symlink

Limitations

The BSD implementation of cp does not have the --remove-destination flag.

0

Expire a user's password immediately

 $ chage -d 0 USERNAME

— by Janos on April 23, 2012, 11:05 p.m.

Explanation

This will effectively force the user to change his/her password at next login.

Limitations

Not in BSD. Yes in Linux. Don't know in UNIX.

0

Convert any 16:9 video to play on a QHD widescreen Android phone

 $ ffmpeg -i $1 -y -threads 0 -subq 6 -deinterlace -level 30 -f mp4 -acodec libfaac -ab 160k -ar 24000 -ac 2 -vcodec libx264 -b 1000k -maxrate 1000k -bufsize 2000k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -coder 0 -refs 2 -flags +loop -vol 256 -trellis 1 -me_method umh -async 1 $2

— by openiduser44 on April 18, 2012, 8:18 p.m.

Explanation

The Android video player is somewhat fussy about the formats it can play. This ffmpeg script will take any movie file in 16:9 widescreen format and convert it into a form that can be played on one of the current leading QHD phones such as HTC Sensation, Samsung Galaxy S2 or Motorola Atrix. Files that are merely marked as widescreen (e.g. DVD VOBs) will have to be processed into a true widescreen format such as .m2t first.

Parameters:

$1 the input file

$2 the output file

Output file suffix should be .mp4

Requires:

ffmpeg and codecs

1

Create a visual report of the contents of a usb drive

 $ find /path/to/drive -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr

— by Anon1Qa6UsYT on April 15, 2012, 6:04 a.m.

Explanation

versorge asks:

I have a bunch of usb volumes lying around and I would like to get a quick summary of what is on the drives. How much space is taken up by pdf, image, text or executable files. This could be output as a text summary, or a pie chart.

This one-liner produces a list like this:

  5804731229 FLAC audio bitstream data
   687302212 MPEG sequence
    99487460 data
    60734903 PDF document
    55905813 Zip archive data
    38430192 ASCII text
    32892213 gzip compressed data
    24847604 PNG image data
    16618355 XML 1.0 document text
    13876248 JPEG image data

The find command locates all regular files (-type f) below the given directory, which could be a mounted USB stick or any other directory. For each one, it runs the file -b command with the filename to print the file type; if this succeeds, it also prints the file size (-printf '%s\n'). This results in a list containing a file type on one line, followed by the file size on the next.

The awk script takes this as input. The GNU file command often produces very specific descriptions such as GIF image data, version 87a, 640 x 480 - to generalize these, we set the field separator to be a comma with the -F option. Referencing $1 then only uses what's to the left of the first comma, giving us a more generic description like GIF image data.

In the awk script, the first pattern-action pair NR%2 {i=$1} applies to each odd-numbered line, setting the variable i to be the file type description. The even-numbered lines are handled by NR%2==0 {a[i]+=$1}, adding the value of the line (which is the file size) to the array variable a[i]. This results in an array indexed by file type, with each array member holding the cumulative sum of bytes for that type. The END { ... } pattern-action pair finally prints out a formatted list of the total size for each file type.

At the end of the line, the sort command sorts the list, putting the file types with the largest numbers at the top.

Limitations

This one-liner uses the -b option to file and the -printf primary of find - these are supported by the GNU utilities but may not work elsewhere. It can also take a long time to run, since it needs to open and analyze every file below the given directory.

0

Sort du output in Human-readable format

 $ for i in G M K; do du -hsx * | grep "[0-9]$i\b" | sort -nr; done 2>/dev/null

— by Janos on April 14, 2012, 11:06 a.m.

Explanation

  • The reason to use a for loop is to sort results with G or M or K values separately, otherwise sort -n would just sort everything by the numbers regardless of G M K suffix.
  • grep "[0-9]$i\b" matches lines containing a digit followed by G or M or K followed by a "word boundary"

0

Sort du output in Human-readable format

 $ for i in $(echo -e 'G\nM\nK'); do du -hsx /* 2>/dev/null | grep '[0-9]'$i | sort -rn; done

— by jasembo on April 14, 2012, 6:02 a.m.

Explanation

  • echo -e prints G for Gigabytes, M for Megabytes and K for Kilobytes in a line each.
  • 2>/dev/null send stderr to /dev/null
  • sort -rn sorts in reverse numerical order. Largest first

1

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ bc <<< 'obase=2;1234'

— by openiduser43 on April 12, 2012, 8 p.m.

Explanation

<<< word is here-string syntax, a variant of here-documents.

0

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ echo 'obase=2;1234' | bc

— by Janos on April 11, 2012, 11:20 p.m.

Explanation

  • bc is an arbitrary precision calculator language.
  • obase defines the conversion base for output numbers, in this example 2 (binary)
  • ; is a statement separator in bc
  • 1234 is the decimal number to convert
  • By piping the command to bc we get 1234 in binary format

0

Convert from avi format to mp4 encoding

 $ ffmpeg -i file.avi file.mp4

— by Janos on April 11, 2012, 11:10 p.m.

Explanation

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Limitations

It is not a standard package in most systems and distros.

0

Format input into multiple columns, like a table, useful or pretty-printing

 $ mount | column -t

— by Janos on April 8, 2012, 4:08 p.m.

Explanation

column is a utility for formatting text. With the -t flag it detects the number of columns in the input so it can format the text into a table-like format.

For more details see man column.

0

Function to extract columns from an input stream

 $ col() { awk '{print $'$(echo $* | sed -e 's/ /,$/g')'}'; }

— by Janos on April 5, 2012, 11:36 p.m.

Explanation

Something I do a lot is extract columns from some input where cut is not suitable because the columns are separated by not a single character but multiple spaces or tabs. So I often do things like:

... | awk '{print $7, $8}'

... which is a lot of typing, additionally slowed down when typing symbols like '{}$ ... Using the simple one-line function above makes it easier and faster:

... | col 7 8

How it works:

  • The one-liner defines a new function with name col
  • The function will execute awk, and it expects standard input (coming from a pipe or input redirection)
  • The function arguments are processed with sed to use them with awk: replace all spaces with ,$ so that for example 1 2 3 becomes 1,$2,$3, which is inserted into the awk command to become the well formatted shell command: awk '{print $1,$2,$3}'

0

Resize an image proportionally to some specified width or height

 $ mogrify -geometry x31 path/to/image.gif

— by Janos on April 3, 2012, 9:48 p.m.

Explanation

  • mogrify is part of ImageMagick, an image manipulation software suite
  • mogrify manipulates the specified images. If you prefer to keep the original image untouched and write the manipulated image to a different file, simply replace mogrify with convert, the syntax is the same, but the last command line argument will be the target image to write to.
  • The -geometry flag is to resize the image, it requires a dimension parameter in the format WIDTHxHEIGHT
  • The dimension in this example has no width, which means the image will be resized to height=31 pixels, and the width will be proportional.

Limitations

ImageMagick is not a standard package, though it is open source and available in many systems.

0

Do something in another directory without going there

 $ (cd /path/to/somewhere; tar c .) > somewhere.tar

— by Janos on April 2, 2012, 10:24 p.m.

Explanation

As explained superbly in man bash:

   (list) list is executed in a subshell environment (see  COMMAND  EXECU-
          TION  ENVIRONMENT below).  Variable assignments and builtin com-
          mands that affect the  shell's  environment  do  not  remain  in
          effect  after  the  command completes.  The return status is the
          exit status of list.

In other words, this is a handy way to do something somewhere else without having to go there and coming back.

0

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ recode pc..l1 file1 file2 file3

— by Anon8MaLEqEp on March 31, 2012, 5:23 p.m.

Explanation

The recode utility installed on many systems converts between character sets. This command is shorthand for recode IBM-PC..latin1 file1 file2 file3 which converts the given files from CRLF to LF line endings.

0

Find the target path a symlink is pointing to

 $ readlink a_symbolic_link_to_somewhere

— by Janos on March 31, 2012, 3:23 p.m.

Explanation

Sure, you could figure out the link target from the output of ls -l a_symbolic_link_to_somewhere too, but the output of readlink is simply the target of the symbolic link itself, so it is cleaner and easier to read.

1

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ vi +'bufdo set ff=unix' +'bufdo %s/^M$//' +q file1 file2 file3

— by Janos on March 30, 2012, 3:50 p.m.

Explanation

  • The arguments starting with + are commands in vi that will be executed
  • set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
  • %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
  • bufdo means to run command in all buffers (each file is opened in a separate buffer)
  • q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1

0

Get the octal, hexadecimal and decimal codes of the ASCII character set

 $ man ascii

— by Janos on March 29, 2012, 9:48 a.m.

Explanation

Knowing the octal, hexadecimal or decimal code of the ASCII character set can be handy at times. In the past, too often I did things like:

perl -e 'for my $n (1 .. 255) { print $n, chr($n), $n, "\n"; }'

... when a simple man ascii would have done the trick...

On a related note, these all print the letter "A":

echo -e '\0101'
printf '\101'
printf '\x41'
perl -e 'print "\x41"'

1

Sort and remove duplicate lines in a file in one step without intermediary files

 $ vi +'%!sort | uniq' +wq file.txt

— by Janos on March 22, 2012, 1:09 p.m.

Explanation

We open a file with vi and run two vi commands (specified with +):

  1. %!sort | uniq
    • % = range definition, it means all the lines in the current buffer.
    • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
  2. wq = write buffer contents to file and exit.

1

Show files containing "foo" and "bar" and "baz"

 $ grep -l 'baz' $(grep -l 'bar' $(grep -lr 'foo' *) )

— by Anon5eqErEbE on March 16, 2012, 5:37 a.m.

Explanation

Most people familiar with extended regular expressions know you can use the pipe symbol | to represent "or", so to see files containing any of "foo", "bar", or "baz" you could run:

grep -Elr 'foo|bar|baz' *

There is no corresponding symbol representing "and", but you can achieve the same effect by nesting invocations to grep. grep -lr 'foo' * returns a list of filenames in or below the current directory containing "foo". Via the $( ... ) syntax, this list is then operated on by grep -l 'bar', returning a list of filenames containing both 'foo' and 'bar', which finally is operated on by grep -l "baz". The end result is a list of filenames containing all three terms.

Limitations

This one-liner results in scanning files multiple times. You will want to put the term you expect to match the fewest number of times farthest to the right (that is, in the same position as "foo") and the one you expect to match most frequently farthest to the left (the same position as "baz"). This way, you will weed out the largest number of files sooner, making the one-liner complete more quickly.

0

List or edit and re-execute commands from the history list

 $ fc -l

— by Janos on March 15, 2012, 12:10 p.m.

Explanation

fc is a little known but very useful bash built-in.

  • fc -l will list the most recent 16 commands
  • fc will open the last command in a text editor defined in the environmental variable FCEDIT or EDITOR or else vi, and re-execute when you exit
  • fc 5 9 will open the history entries 5 to 9 in a text editor
  • fc -s pat=sub will run the last command after substituting pat with sub in it (does not open editor)
  • fc -s pat=sub cc is the same but on the last command starting with cc
  • fc -s cc will run the last command starting with cc

For more info see help fc.

0

Find the most recently modified files in a directory and all subdirectories

 $ find /path/to/dir -type f -mtime -7 -print0 | xargs -0 ls -lt | head

— by Janos on March 8, 2012, 5:10 p.m.

Explanation

  • find /path/to/dir -type f -mtime -7 -print0 prints all the files in the directory tree that have been modified within the last 7 days, with null character as the delimiter
  • xargs -0 ls -lt expects a null delimited list of filenames and will sort the files by modification time, in descending order from most recent to oldest
  • Since we are looking for the most recent files, with head we get the first 10 lines only

Note that if there are too many files in the output of find, xargs will run multiple ls -lt commands and the output will be incorrect. This is because the maximum command line length is getconf ARG_MAX and if this is exceeded xargs has to split the execution to multiple commands. So depending on your use case you may need to tweak the -mtime parameter to make sure there are not too many lines in the output.

0

List open files

 $ lsof -n

— by Janos on March 2, 2012, 10:01 a.m.

Explanation

With the -n flag it will not try to resolve network numbers to host names for network files, making it run a bit faster.

With the -c option you can select processes executing a matching command. And with the -t flag the output will be simply process ids without a header, suitable to use with kill. For example you can kill Google Chrome process gone crazy like this:

kill -HUP $(lsof -n -c /google/i -t)

Here /google/i is a regular expression pattern with case insensitive matching.

0

Set a colorful bash prompt per dev test prod environments

 $ PS1='\[\e[1;31m\][\u@\h \W]\$\[\e[0m\] '

— by Janos on Feb. 25, 2012, 2:46 p.m.

Explanation

It is useful to set a different color for the shell prompt in different deployment environments like dev/test/production, so that you don't mix up your multiple windows and do something by accident in the wrong window.

  • PS1 contains the format of the primary prompt
  • \[\e[1;31m\] sets the foreground color to red
  • \u will be substituted with the current username
  • \h will be substituted with the hostname
  • \W will be substituted with the current directory name
  • \[\e[0m\] is the end marker of the color setting

To make the color stand out even more for root users, the inverse color can be interesting too:

PS1='\[\e[7;31m\][\u@\h \W]\$\[\e[0m\] '

Other color examples:

#PS1='\[\e[1;32m\][\u@\h \W]\$\[\e[0m\] ' # green
#PS1='\[\e[1;33m\][\u@\h \W]\$\[\e[0m\] ' # yellow
#PS1='\[\e[1;34m\][\u@\h \W]\$\[\e[0m\] ' # blue

You can learn more in man bash, search for "PROMPTING".

Limitations

Your terminal program must support colors, of course ;-)