We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

1

Md5sum the last 5 files in a folder

 $ find /directory1/directory2/ -maxdepth 1 -type f | sort | tail -n 5 | xargs md5sum

— by openiduser113 on Aug. 21, 2013, 3:26 p.m.

Explanation

  • find lists the files, no recursion, no directories, with full path
  • sort list files alphabetically
  • tail keep only the last 5 files
  • xargs send the list as arguments to md5sum
  • md5sum calculate the md5sum for each file

Limitations

Probably can't handle spaces in file or directory names.

1

Get mac address from default interface OS X

 $ netstat -rn | awk '/default/ { print $NF }' | head -1 | xargs -I {}  ifconfig {} | awk '/ether/ {print $2}'

— by spotmac on Aug. 21, 2013, 10:28 a.m.

Explanation

netstat -rn -> get routing table awk '/default/ { print $NF }' -> grep the default routes head -1 -> limit to the first result (is also the interface with the highest priority xargs -I {} ifconfig {} -> use the result to get data from ifconfig awk '/ether/ {print $2}' ->grep the mac address.

Limitations

Tested on OSX.

1

Print a random cat

 $ wget -O - http://placekitten.com/$[500 + RANDOM % 500] | lp

— by openiduser104 on July 26, 2013, 11:43 p.m.

Explanation

$RANDOM gives a random number.

http://placekitten.com is your cat place

wget -O - sends the output to stdout

lp prints

Limitations

Tested on OSX

Cat rules

1

Convert a music file (mp3) to a mp4 video with a static image

 $ ffmpeg -loop_input -i cover.jpg -i soundtrack.mp3 -shortest -acodec copy output_video.mp4

— by kowalcj0 on May 17, 2013, 8:09 p.m.

Explanation

Can come handy when you'd like to post a song to YT or somethin' :)

Can be easily wrapped up in a function:

function mp3tovidwithimg() {
  ffmpeg -loop_input -i $1 -i $2 -shortest -acodec copy $3
}

and used like that:

mp3tovidwithimg cover.jpeg music_track.mp3 output_vid.mp4

1

Find all of the distinct file extensions in a folder

 $ find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

— by kowalcj0 on May 17, 2013, 8:05 p.m.

Explanation

Will find all of the distinct file extensions in a folder hierarchy.

Originally posted at: http://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

1

Remove files and directories whose name is a timestamp older than a certain time

 $ ls | grep '....-..-..-......' | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"

— by openiduser95 on May 7, 2013, 8:54 a.m.

Explanation

Suppose you have a backup directory with backup snapshots named by timestamp:

$ ls
2013-05-03-103022
2013-05-04-103033
2013-05-05-103023
2013-05-06-103040
2013-05-07-103022

You want to remove snapshots older than 3 days. The one-liner does it:

$ date
Tue May  7 13:50:57 KST 2013
$ ls | grep '....-..-..-......' | sort | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"
removed directory: `2013-05-03-103022'
removed directory: `2013-05-04-103033'

Limitations

It doesn't work on OS X due to the differences between GNU date and BSD date).

1

Unhide all hidden files in the current directory.

 $ find . -maxdepth 1 -type f -name '\.*' | sed -e 's,^\./\.,,' | sort | xargs -iname mv .name name

— by openiduser93 on April 25, 2013, 7:46 a.m.

Explanation

This will remove the leading dot from all files in the current directory using mv, effectively "unhiding" them.

It will not affect subdirectories.

Limitations

Probably only works on GNU Linux, due to the specific usage of xargs.

1

Get only the latest version of a file from across mutiple directories.

 $ find . -name 'filename' | xargs -r ls -tc | head -n1

— by Anntoin on March 7, 2013, 11:39 p.m.

Explanation

Shows latest file (by last modification of file status information) for the given pattern. So in this example filename = custlist*.xls.

We use ls to do the sorting (-t) and head to pick the top one. xargs is given the -r option so that ls isn't run if there is no match.

Limitations

The filesystem needs to support ctime. Does not depend on a consistent naming scheme.

1

Get load average in a more parse-able format

 $ python -c 'import os; print os.getloadavg()[0]'

— by FoxWilson on Jan. 5, 2013, 3:32 a.m.

Explanation

In short, it runs a Python one-line script which imports the 'os' module and uses it to get the load average. It then indexes into a tuple, using the index of zero. The tuple is like this: (one-minute-average, five-minute-average, fifteen-minute-average), so you could substitute 0 for 1 to get the five minute load average, or 2 to get the fifteen minute load average. I find this really useful when writing larger bash one-liners which require this information, as I don't have to parse the output of uptime.

Limitations

Requires Python.

1

Function to extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by Janos on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'

1

Define an own watch(1)-like function

 $ watch () { interrupted=false; trap "interrupted=true" INT; while ! $interrupted; do $*; sleep 1 || interrupted=true; done; }

— by ulidtko on Nov. 14, 2012, 10 p.m.

Explanation

Once I needed a classical watch(1) command, but all I got on Mac OS X was -bash: watch: command not found.

So this is a simple interruptable while loop which can poll anything you like, e.g. watch grep alert /var/log/syslog.

1

Remove offending key from known_hosts file with one swift move

 $ vi +18d +wq ~/.ssh/known_hosts

— by Janos on Oct. 30, 2012, 9:28 p.m.

Explanation

When you try to ssh to a server where the host key has changed then you probably get an error message like this:

Offending key in /home/jack/.ssh/known_hosts:18

If you know for sure that it was the server's admins who changed the host key, then your next move is to remove line 18 from the file so that ssh lets you connect after accepting the new key.

The one-liner does this in one swift move by passing simple commands to vi:

  • +18d -- delete line 18
  • +wq -- save the file and exit

1

Replace the header of all files found.

 $ find . -type f -name '*.html' -exec sed -i -e '1r common_header' -e '1,/STRING/d' {} \;

— by jam on Oct. 25, 2012, 9:29 a.m.

Explanation

Replaces the lines from 1 to the first occurrence of a line starting with STRING of every file found with find.

  • find . -type f -name '*.html' returns a list of all files (not including directories) ending with .html. The ' ' in name is used to pass the literal wildcard * to the find command, instead of the * interpretation of bash, that is, repeat the command for every file in the current folder.
  • -exec execute the following command in each of the files found, using {} as the filename. the ; termination must be escaped with \;
  • sed -i replaces in file (output is the same file)
  • -e '1r common_header' -e '1,/STRING/d' {} reads common_header file, then finds the first occurrence of STRING and replaces it, deleting all the previous lines and putting the contents of common_header.

Limitations

The -i flag of sed requires a parameter in BSD sed, used as the suffix of a backup file. To not use a backup file, you can pass an empty value with -i''.

1

Redirect stdout to a file you don't have write permission on

 $ echo hello | sudo tee -a /path/to/file

— by Janos on Sept. 11, 2012, 9:24 a.m.

Explanation

  • The tee command copies standard input to standard output, making a copy in zero or more files.
  • If the -a flag is specified it appends instead of overwriting.
  • Calling tee with sudo makes it possible to write to files the current user has no permission to but root does.

1

`tail -f` a file until text is seen

 $ tail -f /path/to/file.log | sed '/^Finished: SUCCESS$/ q'

— by Janos on Aug. 22, 2012, 8:29 a.m.

Explanation

tail -f until this exact line is seen:

Finished: SUCCESS

The exit condition does not have to be an exact line, it could just well be a simple pattern:

... | sed '/Finished/ q'

1

Recording SSH sessions

 $ ssh -l USER HOST | tee -a /path/to/file

— by LeandroToledo on Aug. 15, 2012, 5:04 p.m.

Explanation

tee is a command which displays or pipes the output of a command and copies it into a file or a variable.

The -a option appends the output to the end of file instead of writing over it.

You can also create an alias in ~/.bashrc to record your session when using ssh:

function sshlog () { \ssh $@ 2>&1 | tee -a $(date +%Y%m%d).log; }
alias ssh=sshlog

1

Record audio from microphone or sound input from the console

 $ sox -t ossdsp -w -s -r 44100 -c 2 /dev/dsp -t raw - | lame -x -m s - File.mp3

— by Kleper on July 28, 2012, 8:55 p.m.

Explanation

sox is a software that lets you connect directly to the sound card and send what passes for it in raw format and the system memory through the concatenation of the linux command we can make real-time audio processing is generated by licks and be converted to mp3.

Limitations

Requires a plugin for alsa oss to run on modern distributions.

1

Use vim to pretty-print code with syntax highlighting

 $ vim +'hardcopy > output.ps' +q style.css 

— by Janos on July 21, 2012, 12:13 a.m.

Explanation

If you have syntax highlighting properly setup in vim, this command will pretty-print the specified file with syntax highlighting to output.ps.

If you prefer PDF, you can convert using ps2pdf output.ps.

1

Log and verify files received via FTP

 $ for i in $(cat /var/log/vsftpd.log | grep $DATE_TIME | grep UPLOAD | grep OK); do ls /FTP/HOME/$i >> /dev/null 2> \&1; if \[ $? = 0 \]; then echo "$i" >> $FILES_OK_UPLOADS.log; else  echo "$DATE ERROR: File $i not found" >> $FTP_FILES_NOTOK_$DATE_TIME.log; fi; done

— by dark_axl on July 10, 2012, 8:54 p.m.

Explanation

This one-liner checks and validates the received files via ftp, and generates a log of these files. To have a record of files received and be able to process, based on the successful transfer and the existence of the files.

1

Edit the Gimp launcher file to disable the splash screen

 $ printf '%s\n' ',s/^Exec=[^ ]*/& -s/' w q | ed /usr/share/applications/gimp.desktop

— by Anon8yhYNaVe on July 1, 2012, 12:57 a.m.

Explanation

sed is designed for editing streams - editing files is what ed is for! You can get consistent behavior on any UNIX platform with the above one-liner.

The printf command sends a series of editing commands to ed, each separated by a newline. In this case, the substitution command ,s/^Exec=[^ ]*/& -s/ is nearly the same as in sed, appending a space and a -s to the line starting with Exec=. The only difference is the comma at the beginning designating the lines to operate on. This is shorthand for 1,$, which tells ed to apply the command to the first through the last lines (i.e., the entire file). w tells ed to write the file, and q to quit.

1

Faster disk imaging with dd

 $ dd if=/dev/sda bs=$(hdparm -i /dev/sda | grep BuffSize | cut -d ' ' -f 3 | tr [:lower:] [:upper:] | tr -d BUFFSIZE=,) conv=noerror | dd of=image.dd conv=noerror

— by austindcc on May 19, 2012, 3:28 a.m.

Explanation

GNU dd (disk dump) copies any block device to another block device or file. It's really useful for disk cloning, but its usual invocation isn't as fast as it could be. These settings, or settings like them, often improve copying speed by more than double.

  • Piping the input of dd into the output of another instance seems to always improve copying speed.
  • /dev/sda refers to your input device, which may vary. Check yours with fdisk -l.
  • image.dd refers to the copy stored in the current working directory. You can also use another block device, such as /dev/sdb. WARNING! Be sure you know what you set the output file to! A mistake here could do irreparable damage to your system.
  • The entire hdparm subshell sets dd's input block size to the buffer size of the source medium. This also usually improves copy speed, but may need adjustment (see limitations below).
  • conv=noerror tells dd to ignore read errors.

Check dd's progress with: kill -USR1 $(pidof dd)

Limitations

The hdparm subshell is not appropriate for block devices without buffers, like flash drives. Try block sizes from 512 bytes to 1 or 2MiB to get the best speed. dd usually requires root privileges to run, because it is very powerful and dangerous, and will not prompt when overwriting!. If you're not careful where dd outputs, you may permanently destroy all or part of your system. Use with care; double-check all parameters, especially the of file/device!

1

Create a visual report of the contents of a usb drive

 $ find /path/to/drive -type f -exec file -b '{}' \; -printf '%s\n' | awk -F , 'NR%2 {i=$1} NR%2==0 {a[i]+=$1} END {for (i in a) printf("%12u %s\n",a[i],i)}' | sort -nr

— by Anon1Qa6UsYT on April 15, 2012, 6:04 a.m.

Explanation

versorge asks:

I have a bunch of usb volumes lying around and I would like to get a quick summary of what is on the drives. How much space is taken up by pdf, image, text or executable files. This could be output as a text summary, or a pie chart.

This one-liner produces a list like this:

  5804731229 FLAC audio bitstream data
   687302212 MPEG sequence
    99487460 data
    60734903 PDF document
    55905813 Zip archive data
    38430192 ASCII text
    32892213 gzip compressed data
    24847604 PNG image data
    16618355 XML 1.0 document text
    13876248 JPEG image data

The find command locates all regular files (-type f) below the given directory, which could be a mounted USB stick or any other directory. For each one, it runs the file -b command with the filename to print the file type; if this succeeds, it also prints the file size (-printf '%s\n'). This results in a list containing a file type on one line, followed by the file size on the next.

The awk script takes this as input. The GNU file command often produces very specific descriptions such as GIF image data, version 87a, 640 x 480 - to generalize these, we set the field separator to be a comma with the -F option. Referencing $1 then only uses what's to the left of the first comma, giving us a more generic description like GIF image data.

In the awk script, the first pattern-action pair NR%2 {i=$1} applies to each odd-numbered line, setting the variable i to be the file type description. The even-numbered lines are handled by NR%2==0 {a[i]+=$1}, adding the value of the line (which is the file size) to the array variable a[i]. This results in an array indexed by file type, with each array member holding the cumulative sum of bytes for that type. The END { ... } pattern-action pair finally prints out a formatted list of the total size for each file type.

At the end of the line, the sort command sorts the list, putting the file types with the largest numbers at the top.

Limitations

This one-liner uses the -b option to file and the -printf primary of find - these are supported by the GNU utilities but may not work elsewhere. It can also take a long time to run, since it needs to open and analyze every file below the given directory.

1

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ bc <<< 'obase=2;1234'

— by openiduser43 on April 12, 2012, 8 p.m.

Explanation

<<< word is here-string syntax, a variant of here-documents.

1

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ vi +'bufdo set ff=unix' +'bufdo %s/^M$//' +q file1 file2 file3

— by Janos on March 30, 2012, 3:50 p.m.

Explanation

  • The arguments starting with + are commands in vi that will be executed
  • set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
  • %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
  • bufdo means to run command in all buffers (each file is opened in a separate buffer)
  • q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1

1

Sort and remove duplicate lines in a file in one step without intermediary files

 $ vi +'%!sort | uniq' +wq file.txt

— by Janos on March 22, 2012, 1:09 p.m.

Explanation

We open a file with vi and run two vi commands (specified with +):

  1. %!sort | uniq
    • % = range definition, it means all the lines in the current buffer.
    • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
  2. wq = write buffer contents to file and exit.