We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

0

Write both stdout and stderr to the same file

 $ do_something.sh &> out.log

— by Janos on June 5, 2012, 11:34 a.m.

Explanation

Note: this is not always the same as:

do_something.sh >out.log 2>out.log

Using &> will ensure that the order of lines coming from stdout and stderr is preserved.

Limitations

This works in /bin/bash and may not work in /bin/sh or other stripped down variants of bash.

0

Create or mount an encrypted directory using encfs

 $ encfs -i 5 $PWD/raw $PWD/content

— by Janos on May 22, 2012, 12:21 a.m.

Explanation

The first time you run this, it will create an encrypted directory raw in the current working directory, and mount it on directory content in the same directory. It will ask a couple of questions:

  1. Create directory "raw" ? -- answer "y"
  2. Create directory "content" ? -- answer "y"
  3. Choose configuration mode -- press enter, or "p" for "paranoia mode"

The content directory will be visible and editable only by your user, and all the files you put inside will be saved encrypted under raw. The encrypted data is only visible when raw is mounted somewhere.

The -i 5 means the content directory will be automatically unmounted after being idle for 5 minutes.

You can manually unmount with umount ./content in Linux, and fusermount -u ./content in Mac OS.

To mount raw again, simply re-run the same command. This time it will only ask for the password.

Limitations

encfs is not a standard command/package. Major Linux distros have it in their repos, on Mac OS you can install using ports. (sudo port install encfs)

1

Faster disk imaging with dd

 $ dd if=/dev/sda bs=$(hdparm -i /dev/sda | grep BuffSize | cut -d ' ' -f 3 | tr [:lower:] [:upper:] | tr -d BUFFSIZE=,) conv=noerror | dd of=image.dd conv=noerror

— by austindcc on May 19, 2012, 3:28 a.m.

Explanation

GNU dd (disk dump) copies any block device to another block device or file. It's really useful for disk cloning, but its usual invocation isn't as fast as it could be. These settings, or settings like them, often improve copying speed by more than double.

  • Piping the input of dd into the output of another instance seems to always improve copying speed.
  • /dev/sda refers to your input device, which may vary. Check yours with fdisk -l.
  • image.dd refers to the copy stored in the current working directory. You can also use another block device, such as /dev/sdb. WARNING! Be sure you know what you set the output file to! A mistake here could do irreparable damage to your system.
  • The entire hdparm subshell sets dd's input block size to the buffer size of the source medium. This also usually improves copy speed, but may need adjustment (see limitations below).
  • conv=noerror tells dd to ignore read errors.

Check dd's progress with: kill -USR1 $(pidof dd)

Limitations

The hdparm subshell is not appropriate for block devices without buffers, like flash drives. Try block sizes from 512 bytes to 1 or 2MiB to get the best speed. dd usually requires root privileges to run, because it is very powerful and dangerous, and will not prompt when overwriting!. If you're not careful where dd outputs, you may permanently destroy all or part of your system. Use with care; double-check all parameters, especially the of file/device!

0

Run a never-ending process in the background in a way that you can check on its status anytime

 $ screen -d -m -S some_name ping my_router

— by Janos on May 17, 2012, 11:03 p.m.

Explanation

  • -d -m start screen in "detached" mode. This creates a new session but doesn't attach to it.
  • -S some_name gives the screen session a label, so you can easily re-attach to it with screen -R some_name
  • The shell prompt returns immediately, and of course you can logout too, the screen session will continue to run happily.
  • When the specified program exits (in this example ping), its screen window will also exit, and if there are no other windows in the screen session, then the screen session itself will also exit.

0

Make a hexdump or do the reverse with the xxd command

 $ xxd /path/to/binary/file

— by Janos on May 16, 2012, 10:22 a.m.

Explanation

This shows a very nice hexdump of the specified file. You can edit the output and convert it back to binary with xxd -r. But the best part is that you can configure vim to do all the converting back and forth for you, effectively turning vim into a binary editor, by adding this to your .vimrc:

augroup Binary
    au!
    au BufReadPre  *.bin let &bin=1
    au BufReadPost *.bin if &bin | %!xxd
    au BufReadPost *.bin set ft=xxd | endif
    au BufWritePre *.bin if &bin | %!xxd -r
    au BufWritePre *.bin endif
    au BufWritePost *.bin if &bin | %!xxd
    au BufWritePost *.bin set nomod | endif
augroup END

This will work for editing .bin files. To use it for other file extensions too, duplicate the lines within augroup and replace *.bin with *.exe for example.

This tip is from vim's :help xxd.

0

Really lazy way to print the first instance of $foo that occurs after $bar

 $ ifconfig | grep ^en1 -A5 | grep inet | head -n 1

— by Janos on May 12, 2012, 12:30 p.m.

Explanation

This is just for the sake of an example of finding $foo that occurs after $bar. Substitute ifconfig and the arguments of grep appropriately for your use case.

  • In the output of ifconfig there are several lines with inet. We want to get to the first one that comes after a line starting with en1
  • grep ^en1 -A5 will print the line starting with en1 and the next 5 lines that follow it
  • grep inet will print only the lines matching inet
  • head -n 1 will print only the first line

The value 5 in -A5 is really just a guess that the line we're interested in will be within the next 5 lines, the appropriate number depends on your use case.

Kind of a dumb technique, but it's easy to remember.

0

Print the first instance of $foo that occurs after $bar

 $ sed -n '\@interface Ethernet3/1@,/!/ s/ip address/&/p' file...

— by Anon5DuJaBeh on May 12, 2012, 6:27 a.m.

Explanation

Should have realized this can be done with sed, too - it's even shorter!

-n suppresses the normal printing of lines.

\@interface Ethernet3/1@,/!/ specifies the beginning and end of the stanza we want to work on, and addresses all lines in between. Because the string we want to match at the beginning contains a slash, we need to designate a different character as the delimiter for the regular expression (@ in this case - note the backslash before the first one). We are OK using slashes for the ending regex, which matches an exclamation mark.

While in the stanza of interest, we slightly abuse the s function with s/ip address/&/p. This matches the text "ip address" and replaces it with... exactly the same thing (& just represents the matched text). The reason we bother doing this is to take advantage of the p flag, which will print any line where a replacement has been made.

The result is that only lines which match "ip address" will be printed, and only when we are in the stanza of interest.

Limitations

sed is a little less flexible - it uses basic regular expressions while awk uses extended regular expressions. In addition, awk has a fuller set of functions and operators available, which can be used to create a wider variety of tests and actions.

0

Print the first instance of $foo that occurs after $bar

 $ awk '/interface Ethernet3\/1/ {instanza=1} /!/ {instanza=0} instanza && /ip address/ {print}' file...

— by Anon1UHy2ubE on May 11, 2012, 12:25 p.m.

Explanation

This can easily be done by setting a variable once the lead line of the stanza is found. Take the following file as input:

interface Ethernet3/1
 ip address 172.30.10.2 255.255.0.0
 ip router isis area1
 no ip route-cache
 isis metric 10
!
interface Ethernet3/2
 ip address 192.168.10.10 255.255.255.0
 ip router isis area1
 no ip route-cache
 isis metric 10

The first awk pattern-action pair in the one-liner /interface Ethernet3\/1/ {instanza=1} sets the instanza variable to 1 (true) when it encounters the "interface Ethernet 3/1" line. The second pattern-action /!/ {instanza=0} resets instanza back to 0 (false) once an exclamation point is encountered, ending the desired stanza. The third pattern-action instanza && /ip address/ {print} prints any line containing "ip address," but only if we are still in the desired stanza (instanza is true): in this case, ip address 172.30.10.2 255.255.0.0.

If you only wanted to see the first matching line of the first matching stanza, you could change the third action to be {print; exit} and awk would quit after printing the first match.

0

Sort du output in Human-readable format

 $ du -hsx * | sort -rh

— by Vaevictus on April 26, 2012, 9:08 p.m.

Explanation

sort supports -h for human readable number sorting.

Limitations

Probably a newer GNU only option for sort. :D

0

Replace symlinks with the actual files they are pointing at

 $ find /path/to/dir -type l -exec sh -c 'cp --remove-destination "$(readlink "{}")" "{}"' \; 

— by Janos on April 24, 2012, 3:29 p.m.

Explanation

  • All the double quoting is necessary to handle filenames with spaces.
  • Calling sh with -exec is necessary to evaluate readlink for each symlink

Limitations

The BSD implementation of cp does not have the --remove-destination flag.

0

Expire a user's password immediately

 $ chage -d 0 USERNAME

— by Janos on April 23, 2012, 11:05 p.m.

Explanation

This will effectively force the user to change his/her password at next login.

Limitations

Not in BSD. Yes in Linux. Don't know in UNIX.

0

Convert any 16:9 video to play on a QHD widescreen Android phone

 $ ffmpeg -i $1 -y -threads 0 -subq 6 -deinterlace -level 30 -f mp4 -acodec libfaac -ab 160k -ar 24000 -ac 2 -vcodec libx264 -b 1000k -maxrate 1000k -bufsize 2000k -rc_eq 'blurCplx^(1-qComp)' -qcomp 0.6 -qmin 10 -qmax 51 -qdiff 4 -coder 0 -refs 2 -flags +loop -vol 256 -trellis 1 -me_method umh -async 1 $2

— by openiduser44 on April 18, 2012, 8:18 p.m.

Explanation

The Android video player is somewhat fussy about the formats it can play. This ffmpeg script will take any movie file in 16:9 widescreen format and convert it into a form that can be played on one of the current leading QHD phones such as HTC Sensation, Samsung Galaxy S2 or Motorola Atrix. Files that are merely marked as widescreen (e.g. DVD VOBs) will have to be processed into a true widescreen format such as .m2t first.

Parameters:

$1 the input file

$2 the output file

Output file suffix should be .mp4

Requires:

ffmpeg and codecs

1

Create a visual report of the contents of a usb drive

 $ find /path/to/drive -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr

— by Anon1Qa6UsYT on April 15, 2012, 6:04 a.m.

Explanation

versorge asks:

I have a bunch of usb volumes lying around and I would like to get a quick summary of what is on the drives. How much space is taken up by pdf, image, text or executable files. This could be output as a text summary, or a pie chart.

This one-liner produces a list like this:

  5804731229 FLAC audio bitstream data
   687302212 MPEG sequence
    99487460 data
    60734903 PDF document
    55905813 Zip archive data
    38430192 ASCII text
    32892213 gzip compressed data
    24847604 PNG image data
    16618355 XML 1.0 document text
    13876248 JPEG image data

The find command locates all regular files (-type f) below the given directory, which could be a mounted USB stick or any other directory. For each one, it runs the file -b command with the filename to print the file type; if this succeeds, it also prints the file size (-printf '%s\n'). This results in a list containing a file type on one line, followed by the file size on the next.

The awk script takes this as input. The GNU file command often produces very specific descriptions such as GIF image data, version 87a, 640 x 480 - to generalize these, we set the field separator to be a comma with the -F option. Referencing $1 then only uses what's to the left of the first comma, giving us a more generic description like GIF image data.

In the awk script, the first pattern-action pair NR%2 {i=$1} applies to each odd-numbered line, setting the variable i to be the file type description. The even-numbered lines are handled by NR%2==0 {a[i]+=$1}, adding the value of the line (which is the file size) to the array variable a[i]. This results in an array indexed by file type, with each array member holding the cumulative sum of bytes for that type. The END { ... } pattern-action pair finally prints out a formatted list of the total size for each file type.

At the end of the line, the sort command sorts the list, putting the file types with the largest numbers at the top.

Limitations

This one-liner uses the -b option to file and the -printf primary of find - these are supported by the GNU utilities but may not work elsewhere. It can also take a long time to run, since it needs to open and analyze every file below the given directory.

0

Sort du output in Human-readable format

 $ for i in G M K; do du -hsx * | grep "[0-9]$i\b" | sort -nr; done 2>/dev/null

— by Janos on April 14, 2012, 11:06 a.m.

Explanation

  • The reason to use a for loop is to sort results with G or M or K values separately, otherwise sort -n would just sort everything by the numbers regardless of G M K suffix.
  • grep "[0-9]$i\b" matches lines containing a digit followed by G or M or K followed by a "word boundary"

0

Sort du output in Human-readable format

 $ for i in $(echo -e 'G\nM\nK'); do du -hsx /* 2>/dev/null | grep '[0-9]'$i | sort -rn; done

— by jasembo on April 14, 2012, 6:02 a.m.

Explanation

  • echo -e prints G for Gigabytes, M for Megabytes and K for Kilobytes in a line each.
  • 2>/dev/null send stderr to /dev/null
  • sort -rn sorts in reverse numerical order. Largest first

1

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ bc <<< 'obase=2;1234'

— by openiduser43 on April 12, 2012, 8 p.m.

Explanation

<<< word is here-string syntax, a variant of here-documents.

0

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ echo 'obase=2;1234' | bc

— by Janos on April 11, 2012, 11:20 p.m.

Explanation

  • bc is an arbitrary precision calculator language.
  • obase defines the conversion base for output numbers, in this example 2 (binary)
  • ; is a statement separator in bc
  • 1234 is the decimal number to convert
  • By piping the command to bc we get 1234 in binary format

0

Convert from avi format to mp4 encoding

 $ ffmpeg -i file.avi file.mp4

— by Janos on April 11, 2012, 11:10 p.m.

Explanation

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Limitations

It is not a standard package in most systems and distros.

0

Format input into multiple columns, like a table, useful or pretty-printing

 $ mount | column -t

— by Janos on April 8, 2012, 4:08 p.m.

Explanation

column is a utility for formatting text. With the -t flag it detects the number of columns in the input so it can format the text into a table-like format.

For more details see man column.

0

Function to extract columns from an input stream

 $ col() { awk '{print $'$(echo $* | sed -e 's/ /,$/g')'}'; }

— by Janos on April 5, 2012, 11:36 p.m.

Explanation

Something I do a lot is extract columns from some input where cut is not suitable because the columns are separated by not a single character but multiple spaces or tabs. So I often do things like:

... | awk '{print $7, $8}'

... which is a lot of typing, additionally slowed down when typing symbols like '{}$ ... Using the simple one-line function above makes it easier and faster:

... | col 7 8

How it works:

  • The one-liner defines a new function with name col
  • The function will execute awk, and it expects standard input (coming from a pipe or input redirection)
  • The function arguments are processed with sed to use them with awk: replace all spaces with ,$ so that for example 1 2 3 becomes 1,$2,$3, which is inserted into the awk command to become the well formatted shell command: awk '{print $1,$2,$3}'

0

Resize an image proportionally to some specified width or height

 $ mogrify -geometry x31 path/to/image.gif

— by Janos on April 3, 2012, 9:48 p.m.

Explanation

  • mogrify is part of ImageMagick, an image manipulation software suite
  • mogrify manipulates the specified images. If you prefer to keep the original image untouched and write the manipulated image to a different file, simply replace mogrify with convert, the syntax is the same, but the last command line argument will be the target image to write to.
  • The -geometry flag is to resize the image, it requires a dimension parameter in the format WIDTHxHEIGHT
  • The dimension in this example has no width, which means the image will be resized to height=31 pixels, and the width will be proportional.

Limitations

ImageMagick is not a standard package, though it is open source and available in many systems.

0

Do something in another directory without going there

 $ (cd /path/to/somewhere; tar c .) > somewhere.tar

— by Janos on April 2, 2012, 10:24 p.m.

Explanation

As explained superbly in man bash:

   (list) list is executed in a subshell environment (see  COMMAND  EXECU-
          TION  ENVIRONMENT below).  Variable assignments and builtin com-
          mands that affect the  shell's  environment  do  not  remain  in
          effect  after  the  command completes.  The return status is the
          exit status of list.

In other words, this is a handy way to do something somewhere else without having to go there and coming back.

0

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ recode pc..l1 file1 file2 file3

— by Anon8MaLEqEp on March 31, 2012, 5:23 p.m.

Explanation

The recode utility installed on many systems converts between character sets. This command is shorthand for recode IBM-PC..latin1 file1 file2 file3 which converts the given files from CRLF to LF line endings.

0

Find the target path a symlink is pointing to

 $ readlink a_symbolic_link_to_somewhere

— by Janos on March 31, 2012, 3:23 p.m.

Explanation

Sure, you could figure out the link target from the output of ls -l a_symbolic_link_to_somewhere too, but the output of readlink is simply the target of the symbolic link itself, so it is cleaner and easier to read.

1

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ vi +'bufdo set ff=unix' +'bufdo %s/^M$//' +q file1 file2 file3

— by Janos on March 30, 2012, 3:50 p.m.

Explanation

  • The arguments starting with + are commands in vi that will be executed
  • set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
  • %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
  • bufdo means to run command in all buffers (each file is opened in a separate buffer)
  • q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1