We aim to collect practical, well-explained bash one-liners, and promote best practices in shell scripting. To get the latest bash one-liners, follow @bashoneliners on Twitter, or subscribe to our rss feed. If you find any problems, report a bug on GitHub.

Tags

1

Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.

 $ uniq -u <(sort file1 file2)

— by EvaggelosBalaskas on March 6, 2013, 8:58 a.m.

Explanation

eg.

file1:

123456
234567
345678

file2:

234567
345678
456789

We want only the uniq lines from each file:

result

123456
456789
0

Get only the latest version of a file from across mutiple directories

 $ find . -name custlist\* | perl -ne '$path = $_; s?.*/??; $name = $_; $map{$name} = $path; ++$c; END { print $map{(sort(keys(%map)))[$c-1]} }'

— by openiduser3 on Feb. 23, 2013, 4:23 p.m.

Explanation

The purpose of the one-liner is to find the the "latest" version of the custlist_*.xls file from among multiple versions in directories and sub-directories, for example:

./c/custlist_v1.003.xls
./c/custlist_v2.001.xls
./d/b/custlist_v1.001.xls
./d/custlist_v1.002.xls

Let's decompose the one-liner to the big steps:

  • find . -name custlist\* -- find the files matching the target pattern
  • ... | perl -ne '...' -- run perl, with the input wrapped around in a while loop so that each line in the input is set in the variable $_
  • $path = $_; s?.*/??; $name = $_; -- save the full path in $path, and cut off the subdirectory part to get to the base name of the file and save it in $name
  • $map{$name} = $path; -- build a mapping of $name to $path
  • ++$c; -- we count the elements, to use it later
  • (sort(keys(%map)))[$c-1] -- sort the keys of the map, and get the last element, which is custlist_v2.001.xls in this example
  • END { print $map{$last} }' -- at the end of all input data, print the path of the latest version of the file

Limitations

Even if the latest version of the file appears multiple times in the directories, the one-liner will print only one of the paths. This could be fixed though if needed.

2

Create a thumbnail from the first page of a PDF file

 $ convert -thumbnail x80 file.pdf[0] thumb.png

— by openiduser3 on Feb. 6, 2013, 9:44 p.m.

Explanation

  • convert is part of ImageMagick image manipulation tool
  • -thumbnail x80 means create a thumbnail image of height 80 pixels, the width will be automatically chosen to make the image proportional
  • The [0] is to create a thumbnail for the first page only, without that a thumbnail image would be created for each page in the pdf file

To do this for all PDF files in a directory tree:

find /path/to/dir -name '*.pdf' -exec convert -thumbnail x80 {}[0] {}-thumb.png \;

Limitations

Requires the ImageMagick image manipulation tool.

0

Recreate or update an existing zip file and remove files that do not exist anymore

 $ zip --filesync -r /path/to/out.zip /path/to/dir

— by openiduser3 on Jan. 26, 2013, 8:48 p.m.

Explanation

zip does not have an explicit option to overwrite/recreate an existing zip file. If the specified destination file already exists, zip updates it. The problem is that files you did not specify to add to the zip but they already existed in the zip, will not be removed.

For example let's say you created a zip file from a directory with the command:

zip -r /path/to/out.zip /path/to/dir

Next you delete some files from the directory and repeat the command to recreate the zip. But that will not recreate the zip, it will only update it, and so the file you deleted from the directory will still be there in the zip.

One way to recreate the zip is to delete the file first. Another, better way is to use the --filesync or -FS flag. With this flag zip will remove files from the zip that do not exist anymore in the filesystem. This is more efficient than recreating the zip.

3

Remove offending key from known_hosts file with one swift move

 $ sed -i 18d .ssh/known_hosts

— by EvaggelosBalaskas on Jan. 16, 2013, 2:29 p.m.

Explanation

Using sed to remove a specific line

Adding -i parameter for inplace editing

0

How to expand a CIDR notation to its IPs

 $ for j in `seq 0 255`; do for i in `seq 0 255` ; do seq -f "10.$j.$i.%g" 0 255 ; done ; done 

— by EvaggelosBalaskas on Jan. 16, 2013, 11:53 a.m.

Explanation

Using two for loop to create second & third block of IP and finally through a formatted seq to printf the output

More efficient/using less memory than using {..} (range)

1

Get load average in a more parse-able format

 $ python -c 'import os;print os.getloadavg()[0]'

— by FoxWilson on Jan. 5, 2013, 3:32 a.m.

Explanation

In short, it runs a Python one-line script which imports the 'os' module and uses it to get the load average. It then indexes into a tuple, using the index of zero. The tuple is like this: (oneminuteaverage, fiveminuteaverage, fifteenminuteaverage), so you could substitute 0 for 1 to get the five minute load average, or 2 to get the fifteen minute load average. I find this really useful when writing larger bash one-liners which require this information, as I don't have to parse the output of uptime.

Limitations

Won't work unless you have Python installed.

1

A simple function to conveniently extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by openiduser3 on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'
4

Show dd status every so often

 $ watch --interval 5 killall -USR1 dd

— by FoxWilson on Dec. 6, 2012, 6:16 p.m.

Explanation

The dd command has no progress indicator. While copying large files it may seem like nothing is happening, as dd prints nothing until completed. However, when the dd process receives USR1 signal, it prints I/O statistics to standard error and resumes copying. Here we use killall to send the signal, and we call it with watch to repeat this every 5 seconds, effectively giving a progress indicator to good old dd.

Start in one window the watch:

$ watch --interval 5 killall -USR1 dd

Start copying in another:

$ dd if=/dev/random of=junk bs=1000 count=1000 
dd: warning: partial read (13 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
21 bytes (21 B) copied, 3.01687 s, 0.0 kB/s
0+3 records in
0+3 records out
29 bytes (29 B) copied, 8.02736 s, 0.0 kB/s
0

Make the output of the `time` builtin easier to parse

 $ TIMEFORMAT=%R

— by openiduser3 on Dec. 4, 2012, 10:43 p.m.

Explanation

The time builtin prints a summary of the real time, user CPU time and system CPU time spent executing commands, for example:

$ time sleep 1

real    0m1.002s
user    0m0.000s
sys     0m0.002s

If you need to parse this output, it helps to simplify it using the TIMEFORMAT variable. The value %R means "the elapsed time in seconds", for example:

$ TIMEFORMAT=%R
$ time sleep 1
1.004

The complete documentation of the format definition is in man bash, search for TIMEFORMAT.

1

Define an own watch(1)-like function (handy in OSX terminal)

 $ watch () { interrupted=false; trap "interrupted=true" INT; while ! $interrupted; do $*; sleep 1 || interrupted=true; done; }

— by ulidtko on Nov. 14, 2012, 10 p.m.

Explanation

Once I needed a classical watch(1) command, but all I got on Mac OS X was -bash: watch: command not found.

So this is a simple interruptable while loop which can poll anything you like, e.g. watch grep alert /var/log/syslog.

1

Remove offending key from known_hosts file with one swift move

 $ vi +18d +wq ~/.ssh/known_hosts

— by openiduser3 on Oct. 30, 2012, 9:28 p.m.

Explanation

When you try to ssh to a server where the host key has changed then you probably get an error message like this:

Offending key in /home/jack/.ssh/known_hosts:18

If you know for sure that it was the server's admins who changed the host key, then your next move is to remove line 18 from the file so that ssh lets you connect after accepting the new key.

The one-liner does this in one swift move by passing simple commands to vi:

  • +18d -- delete line 18
  • +wq -- save the file and exit
1

Replace the header of all files found.

 $ find . -type f -name '*.html' -exec sed -i -e '1r common_header' -e '1,/STRING/d' {} \;

— by jam on Oct. 25, 2012, 9:29 a.m.

Explanation

Replaces the lines from 1 to the first occurrence of a line starting with STRING of every file found with find.

  • find . -type f -name '*.html' returns a list of all files (not including directories) ending with .html. The ' ' in name is used to pass the literal wildcard * to the find command, instead of the * interpretation of bash, that is, repeat the command for every file in the current folder.
  • -exec execute the following command in each of the files found, using {} as the filename. the ; termination must be escaped with \;
  • sed -i replaces in file (output is the same file)
  • -e '1r common_header' -e '1,/STRING/d' {} reads common_header file, then finds the first occurrence of STRING and replaces it, deleting all the previous lines and putting the contents of common_header.
0

Remove EXIF data such as orientation from images

 $ mogrify -strip /path/to/image.jpg

— by openiduser3 on Oct. 24, 2012, 12:08 a.m.

Explanation

I use this mostly to remove orientation information from images. My problem with orientation information is that some viewers don't support it, and thus do not show the image correctly oriented. Rotating the image doesn't help, because if I make the image look correct in the viewer that doesn't support orientation, that will break it in the viewer that does support orientation. The solution is to remove the orientation information and rotate the image appropriately. That way the image will always look the same in all viewers, regardless of support for the orientation information.

The tool mogrify is part of ImageMagick, an image manipulation software. It manipulates image files and saves the result in the same file. A similar tool in ImageMagick that saves the result of manipulations is convert, you can use it like this:

convert -strip orig.jpg stripped.jpg

Limitations

The tool is part of ImageMagick, an image manipulation software.

0

Get the last modification date of a file in any format you want

 $ date -r /etc/motd +%Y%m%d_%H%M%S

— by openiduser3 on Oct. 17, 2012, 4:42 p.m.

Explanation

The -r flag is a shortcut of --reference and it is used to specify a reference file. Used in this way, the date command prints the last modification date of the specified file, instead of the current date.

The + controls the output format, for example:

  • %Y = 4-digit year
  • %m = 2-digit month
  • %d = 2-digit day
  • %H = 2-digit hour
  • %M = 2-digit minutes
  • %S = 2-digit seconds

So in this example +%Y%m%d_%H%M%S becomes 20121001_171233

You should be able to find all the possible format specifiers in man date.

Limitations

The default date command in Solaris does not support the --reference flag. Modern Solaris systems have the GNU tools installed, so you may be able to find the GNU implementation of date which supports this flag. Look for it in /usr/gnu/bin/date or /usr/local/bin/date, or do search the entire /usr with find /usr -name date.

In Solaris this may be a suitable substitute without using the date command:

ls -Ego /etc/motd | awk '{print $4 "_" $5}' | tr -d :- | sed -e 's/\..*//'

Or you can use good old perl:

perl -mPOSIX -e 'print POSIX::strftime("%Y%m%d_%H%M%S\n", localtime((stat("/etc/motd"))[9]))'
0

Forget all remembered path locations

 $ hash -r

— by openiduser3 on Oct. 14, 2012, 9:46 a.m.

Explanation

bash remembers the full path name of each command you enter, so it doesn't have to lookup in $PATH every single time you run the same thing. It also counts the number of times you used each command in the current session, you can see the list with hash.

Anyway, this behavior can poses a small problem when you reinstall an application at a different path. For example you reinstall a program that used to be in /usr/local/bin and now it is in /opt/local/bin. The problem is that if you used that command in the current shell session, then bash will remember the original location, which of course doesn't work anymore. To fix that, you can either run hash cmd which will lookup the command again, or run hash -r to forget all remembered locations (less efficient, but maybe faster to type ;-)

For more details, see help hash

0

Rename files with numeric padding

 $ perl -e 'for (@ARGV) { $o = $_; s/\d+/sprintf("%04d", $&)/e; print qq{mv "$o" "$_"\n}}'

— by openiduser3 on Oct. 6, 2012, 1:38 p.m.

Explanation

Basically a one-liner perl script. Specify the files to rename as command line parameters, for example:

perl -e '.....' file1.jpg file2.jpg

In this example the files will be renamed to file0001.jpg and file0002.jpg, respectively. The script does not actually rename anything. It only prints the shell commands to execute that would perform the renaming. This way you can check first that the script would do, and if you want to actually do it, then pipe the output to sh like this:

perl -e '.....' file1.jpg file2.jpg | sh

What's happening in the one-liner perl script:

  • for (@ARGV) { ... } is a loop, where each command line argument is substituted into the auto-variable $_.
  • $o = $_ :: save the original filename
  • s/// :: perform pattern matching and replacement on $_
  • print qq{...} :: print the mv command, with correctly quoted arguments

Limitations

The script does not cover all corner cases. For example it will not work with files that have double-quotes in their names. In any case, it is safe to review the output of the script first before piping it to sh.

If your system has the rename command (Linux), then a shortcut to do the exact same thing is with:

rename 's/\d+/sprintf("%04d", $&)/e' *.jpg

It handles special characters better too.

0

Copy or create files with specific permissions and ownership

 $ install -b -m 600 /dev/null NEWFILE

— by openiduser3 on Sept. 25, 2012, 2:20 p.m.

Explanation

This example creates a new (empty) file with permissions 600. You could also specify the owner and group using the -o and -g flags respectively.

Although you could accomplish the same for example by creating the file with touch and then change permissions with chmod and chown, or use umask to control the permissions of newly created files, those methods take multiple steps, while with install it is a single step.

You can also use install to copy multiple files to a directory with specified permissions like this:

install -m 600 -o jack -g wheel file1 file2 /path/to/existing/dir
1

Redirect stdout to a file you don't have write permission on

 $ echo hello | sudo tee -a /path/to/file

— by openiduser3 on Sept. 11, 2012, 9:24 a.m.

Explanation

  • The tee command copies standard input to standard output, making a copy in zero or more files.
  • If the -a flag is specified it appends instead of overwriting.
  • Calling tee with sudo makes it possible to write to files the current user has no permission to but root does.
0

`tail -f` a file until text is seen

 $ tail -f /path/to/file.log | sed '/^Finished: SUCCESS$/ q'

— by openiduser3 on Aug. 22, 2012, 8:29 a.m.

Explanation

tail -f until this exact line is seen:

Finished: SUCCESS

The exit condition does not have to be an exact line, it could just well be a simple pattern:

... | sed '/Finished/ q'
0

Run command multiple times with a for loop and a sequence expression

 $ for i in {1..10}; do date; sleep 1; done

— by openiduser3 on Aug. 19, 2012, 9:27 a.m.

Explanation

This is just a regular for loop with a sequence expression. A sequence expression has the form {x..y[..incr]}, where x and y are either integers or single characters, and incr an optional increment.

More examples:

  • {a..f} = a b c d e f
  • {a..f..2} = a c e
  • {0..1}{0..1} = 00 01 10 11

Limitations

Don't try a large range like {1..10000000000000000}, it may render your computer unusable until killed.

0

Recording SSH sessions

 $ ssh -l USER HOST | tee -a /path/to/file

— by LeandroToledo on Aug. 15, 2012, 5:04 p.m.

Explanation

tee is a command which displays or pipes the output of a command and copies it into a file or a variable.

  • -a option appends the output to the end of file instead of writing over it.

You can also create an alias in ~/.bashrc to record your session when using ssh:

function sshlog () { \ssh $@ 2>&1 | tee -a $(date +%Y%m%d).log; }
alias ssh=sshlog
0

Clear the swap space forcing everything back to main memory in Linux

 $ sudo swapoff -a; sudo swapon -a

— by openiduser3 on Aug. 14, 2012, 11:21 a.m.

Explanation

Note: if you don't have enough main memory the swapoff will fail.

Limitations

This works only in Linux.

0

Redirection operator to override the noclobber option

 $ some_command >| output.txt

— by openiduser3 on Aug. 11, 2012, 9:21 a.m.

Explanation

Normally the > operator overwrites the target file.

If the noclobber option is set (using: set -o noclobber), the > operator will fail if the target file exists.

The >| overrides the noclobber setting and overwrites the target file.

If the noclobber option is not set, then >| is equivalent to >, naturally.

0

How to set the ip address in Solaris 11

 $ ipadm create-addr -T static -a 192.168.1.10/24 eth0/staticaddr

— by openiduser3 on Aug. 3, 2012, 11:44 a.m.

Explanation

  • eth0 is the name of the network interface
  • ipadm show-if shows the list of network interfaces
  • staticaddr is a name you can choose

More details here: http://docs.oracle.com/cd/E19963-01/html/821-1458/gjwiq.html