We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

1

Find all of the distinct file extensions in a folder

 $ find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

— by kowalcj0 on May 17, 2013, 8:05 p.m.

Explanation

Will find all of the distinct file extensions in a folder hierarchy.

Originally posted at: http://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

1

Unhide all hidden files in the current directory.

 $ find . -maxdepth 1 -type f -name '\.*' | sed -e 's,^\./\.,,' | sort | xargs -iname mv .name name

— by openiduser93 on April 25, 2013, 7:46 a.m.

Explanation

This will remove the leading dot from all files in the current directory using mv, effectively "unhiding" them.

It will not affect subdirectories.

Limitations

Probably only works on GNU Linux, due to the specific usage of xargs.

1

Get only the latest version of a file from across mutiple directories.

 $ find . -name 'filename' | xargs -r ls -tc | head -n1

— by Anntoin on March 7, 2013, 11:39 p.m.

Explanation

Shows latest file (by last modification of file status information) for the given pattern. So in this example filename = custlist*.xls.

We use ls to do the sorting (-t) and head to pick the top one. xargs is given the -r option so that ls isn't run if there is no match.

Limitations

The filesystem needs to support ctime. Does not depend on a consistent naming scheme.

1

Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.

 $ sort file1 file2 | uniq -u

— by EvaggelosBalaskas on March 6, 2013, 8:58 a.m.

Explanation

The -u flag of uniq removes duplicate lines from the input.

Example file1:

123456
234567
345678

Example file2:

234567
345678
456789

Result:

123456
456789

1

Get load average in a more parse-able format

 $ python -c 'import os; print os.getloadavg()[0]'

— by FoxWilson on Jan. 5, 2013, 3:32 a.m.

Explanation

In short, it runs a Python one-line script which imports the 'os' module and uses it to get the load average. It then indexes into a tuple, using the index of zero. The tuple is like this: (one-minute-average, five-minute-average, fifteen-minute-average), so you could substitute 0 for 1 to get the five minute load average, or 2 to get the fifteen minute load average. I find this really useful when writing larger bash one-liners which require this information, as I don't have to parse the output of uptime.

Limitations

Requires Python.

1

Function to extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by Janos on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'

1

Define an own watch(1)-like function

 $ watch () { interrupted=false; trap "interrupted=true" INT; while ! $interrupted; do $*; sleep 1 || interrupted=true; done; }

— by ulidtko on Nov. 14, 2012, 10 p.m.

Explanation

Once I needed a classical watch(1) command, but all I got on Mac OS X was -bash: watch: command not found.

So this is a simple interruptable while loop which can poll anything you like, e.g. watch grep alert /var/log/syslog.

1

Remove offending key from known_hosts file with one swift move

 $ vi +18d +wq ~/.ssh/known_hosts

— by Janos on Oct. 30, 2012, 9:28 p.m.

Explanation

When you try to ssh to a server where the host key has changed then you probably get an error message like this:

Offending key in /home/jack/.ssh/known_hosts:18

If you know for sure that it was the server's admins who changed the host key, then your next move is to remove line 18 from the file so that ssh lets you connect after accepting the new key.

The one-liner does this in one swift move by passing simple commands to vi:

  • +18d -- delete line 18
  • +wq -- save the file and exit

1

Replace the header of all files found.

 $ find . -type f -name '*.html' -exec sed -i -e '1r common_header' -e '1,/STRING/d' {} \;

— by jam on Oct. 25, 2012, 9:29 a.m.

Explanation

Replaces the lines from 1 to the first occurrence of a line starting with STRING of every file found with find.

  • find . -type f -name '*.html' returns a list of all files (not including directories) ending with .html. The ' ' in name is used to pass the literal wildcard * to the find command, instead of the * interpretation of bash, that is, repeat the command for every file in the current folder.
  • -exec execute the following command in each of the files found, using {} as the filename. the ; termination must be escaped with \;
  • sed -i replaces in file (output is the same file)
  • -e '1r common_header' -e '1,/STRING/d' {} reads common_header file, then finds the first occurrence of STRING and replaces it, deleting all the previous lines and putting the contents of common_header.

Limitations

The -i flag of sed requires a parameter in BSD sed, used as the suffix of a backup file. To not use a backup file, you can pass an empty value with -i''.

1

Redirect stdout to a file you don't have write permission on

 $ echo hello | sudo tee -a /path/to/file

— by Janos on Sept. 11, 2012, 9:24 a.m.

Explanation

  • The tee command copies standard input to standard output, making a copy in zero or more files.
  • If the -a flag is specified it appends instead of overwriting.
  • Calling tee with sudo makes it possible to write to files the current user has no permission to but root does.

1

`tail -f` a file until text is seen

 $ tail -f /path/to/file.log | sed '/^Finished: SUCCESS$/ q'

— by Janos on Aug. 22, 2012, 8:29 a.m.

Explanation

tail -f until this exact line is seen:

Finished: SUCCESS

The exit condition does not have to be an exact line, it could just well be a simple pattern:

... | sed '/Finished/ q'

1

Recording SSH sessions

 $ ssh -l USER HOST | tee -a /path/to/file

— by LeandroToledo on Aug. 15, 2012, 5:04 p.m.

Explanation

tee is a command which displays or pipes the output of a command and copies it into a file or a variable.

The -a option appends the output to the end of file instead of writing over it.

You can also create an alias in ~/.bashrc to record your session when using ssh:

function sshlog () { \ssh $@ 2>&1 | tee -a $(date +%Y%m%d).log; }
alias ssh=sshlog

1

Record audio from microphone or sound input from the console

 $ sox -t ossdsp -w -s -r 44100 -c 2 /dev/dsp -t raw - | lame -x -m s - File.mp3

— by Kleper on July 28, 2012, 8:55 p.m.

Explanation

sox is a software that lets you connect directly to the sound card and send what passes for it in raw format and the system memory through the concatenation of the linux command we can make real-time audio processing is generated by licks and be converted to mp3.

Limitations

Requires a plugin for alsa oss to run on modern distributions.

1

Use vim to pretty-print code with syntax highlighting

 $ vim +'hardcopy > output.ps' +q style.css 

— by Janos on July 21, 2012, 12:13 a.m.

Explanation

If you have syntax highlighting properly setup in vim, this command will pretty-print the specified file with syntax highlighting to output.ps.

If you prefer PDF, you can convert using ps2pdf output.ps.

1

Log and verify files received via FTP

 $ for i in $(cat /var/log/vsftpd.log | grep $DATE_TIME | grep UPLOAD | grep OK); do ls /FTP/HOME/$i >> /dev/null 2> \&1; if \[ $? = 0 \]; then echo "$i" >> $FILES_OK_UPLOADS.log; else  echo "$DATE ERROR: File $i not found" >> $FTP_FILES_NOTOK_$DATE_TIME.log; fi; done

— by dark_axl on July 10, 2012, 8:54 p.m.

Explanation

This one-liner checks and validates the received files via ftp, and generates a log of these files. To have a record of files received and be able to process, based on the successful transfer and the existence of the files.

1

Edit the Gimp launcher file to disable the splash screen

 $ printf '%s\n' ',s/^Exec=[^ ]*/& -s/' w q | ed /usr/share/applications/gimp.desktop

— by Anon8yhYNaVe on July 1, 2012, 12:57 a.m.

Explanation

sed is designed for editing streams - editing files is what ed is for! You can get consistent behavior on any UNIX platform with the above one-liner.

The printf command sends a series of editing commands to ed, each separated by a newline. In this case, the substitution command ,s/^Exec=[^ ]*/& -s/ is nearly the same as in sed, appending a space and a -s to the line starting with Exec=. The only difference is the comma at the beginning designating the lines to operate on. This is shorthand for 1,$, which tells ed to apply the command to the first through the last lines (i.e., the entire file). w tells ed to write the file, and q to quit.

1

Faster disk imaging with dd

 $ dd if=/dev/sda bs=$(hdparm -i /dev/sda | grep BuffSize | cut -d ' ' -f 3 | tr [:lower:] [:upper:] | tr -d BUFFSIZE=,) conv=noerror | dd of=image.dd conv=noerror

— by austindcc on May 19, 2012, 3:28 a.m.

Explanation

GNU dd (disk dump) copies any block device to another block device or file. It's really useful for disk cloning, but its usual invocation isn't as fast as it could be. These settings, or settings like them, often improve copying speed by more than double.

  • Piping the input of dd into the output of another instance seems to always improve copying speed.
  • /dev/sda refers to your input device, which may vary. Check yours with fdisk -l.
  • image.dd refers to the copy stored in the current working directory. You can also use another block device, such as /dev/sdb. WARNING! Be sure you know what you set the output file to! A mistake here could do irreparable damage to your system.
  • The entire hdparm subshell sets dd's input block size to the buffer size of the source medium. This also usually improves copy speed, but may need adjustment (see limitations below).
  • conv=noerror tells dd to ignore read errors.

Check dd's progress with: kill -USR1 $(pidof dd)

Limitations

The hdparm subshell is not appropriate for block devices without buffers, like flash drives. Try block sizes from 512 bytes to 1 or 2MiB to get the best speed. dd usually requires root privileges to run, because it is very powerful and dangerous, and will not prompt when overwriting!. If you're not careful where dd outputs, you may permanently destroy all or part of your system. Use with care; double-check all parameters, especially the of file/device!

1

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ bc <<< 'obase=2;1234'

— by openiduser43 on April 12, 2012, 8 p.m.

Explanation

<<< word is here-string syntax, a variant of here-documents.

1

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ vi +'bufdo set ff=unix' +'bufdo %s/^M$//' +q file1 file2 file3

— by Janos on March 30, 2012, 3:50 p.m.

Explanation

  • The arguments starting with + are commands in vi that will be executed
  • set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
  • %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
  • bufdo means to run command in all buffers (each file is opened in a separate buffer)
  • q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1

1

Sort and remove duplicate lines in a file in one step without intermediary files

 $ vi +'%!sort | uniq' +wq file.txt

— by Janos on March 22, 2012, 1:09 p.m.

Explanation

We open a file with vi and run two vi commands (specified with +):

  1. %!sort | uniq
    • % = range definition, it means all the lines in the current buffer.
    • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
  2. wq = write buffer contents to file and exit.

1

Show files containing "foo" and "bar" and "baz"

 $ grep -l 'baz' $(grep -l 'bar' $(grep -lr 'foo' *) )

— by Anon5eqErEbE on March 16, 2012, 5:37 a.m.

Explanation

Most people familiar with extended regular expressions know you can use the pipe symbol | to represent "or", so to see files containing any of "foo", "bar", or "baz" you could run:

grep -Elr 'foo|bar|baz' *

There is no corresponding symbol representing "and", but you can achieve the same effect by nesting invocations to grep. grep -lr 'foo' * returns a list of filenames in or below the current directory containing "foo". Via the $( ... ) syntax, this list is then operated on by grep -l 'bar', returning a list of filenames containing both 'foo' and 'bar', which finally is operated on by grep -l "baz". The end result is a list of filenames containing all three terms.

Limitations

This one-liner results in scanning files multiple times. You will want to put the term you expect to match the fewest number of times farthest to the right (that is, in the same position as "foo") and the one you expect to match most frequently farthest to the left (the same position as "baz"). This way, you will weed out the largest number of files sooner, making the one-liner complete more quickly.

1

Find in files, recursively

 $ grep -rn 'nameserver' /etc 2>/dev/null

— by atpessoa on Feb. 19, 2012, 8:24 a.m.

Explanation

  • -r make a search recursively;
  • -n print line numbers;
  • -H is not need, is default;

Limitations

  • -i use for case insensitive search;

1

Calculate the total disk space used by a list of files or directories

 $ du -cshx ./a ./b

— by openiduser14 on Feb. 15, 2012, 10:43 p.m.

Explanation

  • -s, --summarize; display only a total for each argument
  • -c, --total; produce a grand total
  • -x, --one-file-system; skip directories on different file systems
  • -h, --human-readable; print sizes in human readable format (e.g., 1K 234M 2G)

1

Create a compressed tar file that rsync will transfer efficiently

 $ GZIP='--rsyncable' tar cvzf bobsbackup.tar.gz /home/bob

— by Anon6y5E4Use on Feb. 15, 2012, 12:24 p.m.

Explanation

rsync works by comparing files on the local and remote machine and only sending those parts of the file that have changed. The normal way compression works, it results in everything after the modification changing, meaning lots of data ends up going over the network when you try to rsync compressed files.

The --rsyncable option to gzip changes the compression scheme so that modifications to the input file only affect the part of the file where they're located. This allows rsync to work its magic.

In this one-liner, the z option to tar calls gzip, which recognizes and uses any options specified in the GZIP environment variable.

Limitations

Using the --rsyncable option results in a slightly larger compressed file.

Not all versions of gzip include this feature - use the --help option to see if it's available on your system.

1

Cut select pages from a pdf file and create a new file from those pages.

 $ ps2pdf -dFirstPage=3 -dLastPage=10 input.pdf output.pdf

— by Anon6y5E4Use on Feb. 15, 2012, 11:08 a.m.

Explanation

ps2pdf is a script that comes with Ghostscript - despite the name, it can accept PDF files as input, not just postscript files.

Limitations

Only a single contiguous range of pages can be specified.