We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

1

Convert a music file (mp3) to a mp4 video with a static image

 $ ffmpeg -loop_input -i cover.jpg -i soundtrack.mp3 -shortest -acodec copy output_video.mp4

— by kowalcj0 on May 17, 2013, 8:09 p.m.

Explanation

Can come handy when you'd like to post a song to YT or somethin' :)

Can be easily wrapped up in a function:

function mp3tovidwithimg() {
  ffmpeg -loop_input -i $1 -i $2 -shortest -acodec copy $3
}

and used like that:

mp3tovidwithimg cover.jpeg music_track.mp3 output_vid.mp4

1

Find all of the distinct file extensions in a folder

 $ find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

— by kowalcj0 on May 17, 2013, 8:05 p.m.

Explanation

Will find all of the distinct file extensions in a folder hierarchy.

Originally posted at: http://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

2

Dump network traffic with tcpdump to file with time-stamp in its filename

 $ date +"%Y-%m-%d_%H-%M-%Z" | xargs -I {} bash -c "sudo tcpdump -nq -s 0 -i eth0 -w ./dump-{}.pcap"

— by kowalcj0 on May 17, 2013, 8:03 p.m.

Explanation

will dump the traffic into a file with a time-stamp in its name. Example filename:

dump-2013-05-17_15-46-UTC.pcap

0

Get streamed FLV from Chrome with lsof

 $ export psid=$(pgrep -f libflashplayer.so); cp /proc/$psid/fd/$(lsof -p $psid | grep eleted | awk {' print $4 '} | sed -e "s/[a-z]//g") saved.flv

— by GNA on May 11, 2013, 10:55 p.m.

Explanation

first get the process id of the chome browser process which runs the flashplayer export it to a variable to be used later. Then we get in subshell the filedescriptor which is marked deleted and construct the /proc path for the in memory fileimage and copy it to the file named saved.flv

Limitations

IMPORTANT: only one video should be open to play in chrome

1

Remove files and directories whose name is a timestamp older than a certain time

 $ ls | grep '....-..-..-......' | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"

— by openiduser95 on May 7, 2013, 8:54 a.m.

Explanation

Suppose you have a backup directory with backup snapshots named by timestamp:

$ ls
2013-05-03-103022
2013-05-04-103033
2013-05-05-103023
2013-05-06-103040
2013-05-07-103022

You want to remove snapshots older than 3 days. The one-liner does it:

$ date
Tue May  7 13:50:57 KST 2013
$ ls | grep '....-..-..-......' | sort | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"
removed directory: `2013-05-03-103022'
removed directory: `2013-05-04-103033'

Limitations

It doesn't work on OS X due to the differences between GNU date and BSD date).

2

Tree-like output in ls

 $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'

— by clitips on April 26, 2013, 1:37 p.m.

Explanation

This one-liner initially does a recursive listing of the current directory: ls -R.

Any output other that the directory names, identified by : at the very end of each line (hence :$), is filtered out: grep ":$".

Finally there's a little of sed magic replacing any hierarchy level (/) with dashes (-).

Limitations

Works for me with Bash under Linux, Mac OS X, Solaris.

1

Unhide all hidden files in the current directory.

 $ find . -maxdepth 1 -type f -name '\.*' | sed -e 's,^\./\.,,' | sort | xargs -iname mv .name name

— by openiduser93 on April 25, 2013, 7:46 a.m.

Explanation

This will remove the leading dot from all files in the current directory using mv, effectively "unhiding" them.

It will not affect subdirectories.

Limitations

Probably only works on GNU Linux, due to the specific usage of xargs.

0

Rename all files in a directory to upper case

 $ for i in *; do mv "$i" "${i^^}"; done

— by EvaggelosBalaskas on April 20, 2013, 9:53 p.m.

Explanation

Loop over the items in the current directory, and use Bash built-in case modification expansion to convert to upper case.

Limitations

The case modification extension is available since Bash 4.

4

Rename all items in a directory to lower case

 $ for i in *; do mv "$i" "${i,,}"; done

— by EvaggelosBalaskas on April 20, 2013, 9:41 p.m.

Explanation

Loop over the items in the current directory, and use Bash built-in case modification expansion to convert to lower case.

Limitations

The case modification extension is available since Bash 4.

0

Print file owners and permissions of a directory tree

 $ find /path/to/dir1 -printf "%U %G %m %p\n" > /tmp/dir1.txt

— by Janos on March 19, 2013, 10:51 p.m.

Explanation

The command simply traverses the specified directory tree and for each file and directory it prints the UID of the owner, GID of the group, the permission bits and the path.

To compare file owners and permissions of two directory trees you can run this command for each directory, save the output in two files and then compare them using diff or similar.

See man find for more explanation of all the possible symbols you can use with -printf

Limitations

The -printf option does not exist in find on Solaris 10.

1

Get only the latest version of a file from across mutiple directories.

 $ find . -name 'filename' | xargs -r ls -tc | head -n1

— by Anntoin on March 7, 2013, 11:39 p.m.

Explanation

Shows latest file (by last modification of file status information) for the given pattern. So in this example filename = custlist*.xls.

We use ls to do the sorting (-t) and head to pick the top one. xargs is given the -r option so that ls isn't run if there is no match.

Limitations

The filesystem needs to support ctime. Does not depend on a consistent naming scheme.

0

Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.

 $ sort file1 file2 | uniq -u

— by EvaggelosBalaskas on March 6, 2013, 8:58 a.m.

Explanation

The -u flag of uniq removes duplicate lines from the input.

Example file1:

123456
234567
345678

Example file2:

234567
345678
456789

Result:

123456
456789

0

Get only the latest version of a file from across mutiple directories

 $ find . -name custlist\* | perl -ne '$path = $_; s?.*/??; $name = $_; $map{$name} = $path; ++$c; END { print $map{(sort(keys(%map)))[$c-1]} }'

— by Janos on Feb. 23, 2013, 4:23 p.m.

Explanation

The purpose of the one-liner is to find the the "latest" version of the custlist_*.xls file from among multiple versions in directories and sub-directories, for example:

./c/custlist_v1.003.xls
./c/custlist_v2.001.xls
./d/b/custlist_v1.001.xls
./d/custlist_v1.002.xls

Let's decompose the one-liner to the big steps:

  • find . -name custlist\* -- find the files matching the target pattern
  • ... | perl -ne '...' -- run perl, with the input wrapped around in a while loop so that each line in the input is set in the variable $_
  • $path = $_; s?.*/??; $name = $_; -- save the full path in $path, and cut off the subdirectory part to get to the base name of the file and save it in $name
  • $map{$name} = $path; -- build a mapping of $name to $path
  • ++$c; -- we count the elements, to use it later
  • (sort(keys(%map)))[$c-1] -- sort the keys of the map, and get the last element, which is custlist_v2.001.xls in this example
  • END { print $map{$last} }' -- at the end of all input data, print the path of the latest version of the file

Limitations

Even if the latest version of the file appears multiple times in the directories, the one-liner will print only one of the paths. This could be fixed though if needed.

2

Create a thumbnail from the first page of a PDF file

 $ convert -thumbnail x80 file.pdf[0] thumb.png

— by Janos on Feb. 6, 2013, 9:44 p.m.

Explanation

  • convert is part of ImageMagick image manipulation tool
  • -thumbnail x80 means create a thumbnail image of height 80 pixels, the width will be automatically chosen to make the image proportional
  • The [0] is to create a thumbnail for the first page only, without that a thumbnail image would be created for each page in the pdf file

To do this for all PDF files in a directory tree:

find /path/to/dir -name '*.pdf' -exec convert -thumbnail x80 {}[0] {}-thumb.png \;

Limitations

Requires the ImageMagick image manipulation tool.

0

Recreate or update an existing zip file and remove files that do not exist anymore

 $ zip --filesync -r /path/to/out.zip /path/to/dir

— by Janos on Jan. 26, 2013, 8:48 p.m.

Explanation

zip does not have an explicit option to overwrite/recreate an existing zip file. If the specified destination file already exists, zip updates it. The problem is that files you did not specify to add to the zip but they already existed in the zip, will not be removed.

For example let's say you created a zip file from a directory with the command:

zip -r /path/to/out.zip /path/to/dir

Next you delete some files from the directory and repeat the command to recreate the zip. But that will not recreate the zip, it will only update it, and so the file you deleted from the directory will still be there in the zip.

One way to recreate the zip is to delete the file first. Another, better way is to use the --filesync or -FS flag. With this flag zip will remove files from the zip that do not exist anymore in the filesystem. This is more efficient than recreating the zip.

3

Remove offending key from known_hosts file with one swift move

 $ sed -i 18d .ssh/known_hosts

— by EvaggelosBalaskas on Jan. 16, 2013, 2:29 p.m.

Explanation

Using sed to remove a specific line.

The -i parameter is to edit the file in-place.

Limitations

This works as posted in GNU sed. In BSD sed, the -i flag requires a parameter to use as the suffix of a backup file. You can set it to empty to not use a backup file:

sed -i'' 18d .ssh/known_hosts

0

How to expand a CIDR notation to its IPs

 $ for j in $(seq 0 255); do for i in $(seq 0 255) ; do seq -f "10.$j.$i.%g" 0 255; done; done

— by EvaggelosBalaskas on Jan. 16, 2013, 11:53 a.m.

Explanation

Using two for loops to create second & third block of IP and finally through a formatted seq to printf the output.

More efficient/using less memory than using {..} (range).

Limitations

seq is not available by default in some systems.

1

Get load average in a more parse-able format

 $ python -c 'import os; print os.getloadavg()[0]'

— by FoxWilson on Jan. 5, 2013, 3:32 a.m.

Explanation

In short, it runs a Python one-line script which imports the 'os' module and uses it to get the load average. It then indexes into a tuple, using the index of zero. The tuple is like this: (one-minute-average, five-minute-average, fifteen-minute-average), so you could substitute 0 for 1 to get the five minute load average, or 2 to get the fifteen minute load average. I find this really useful when writing larger bash one-liners which require this information, as I don't have to parse the output of uptime.

Limitations

Requires Python.

1

Function to extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by Janos on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'

4

Show dd status every so often

 $ watch --interval 5 killall -USR1 dd

— by FoxWilson on Dec. 6, 2012, 6:16 p.m.

Explanation

The dd command has no progress indicator. While copying large files it may seem like nothing is happening, as dd prints nothing until completed. However, when the dd process receives USR1 signal, it prints I/O statistics to standard error and resumes copying. Here we use killall to send the signal, and we call it with watch to repeat this every 5 seconds, effectively giving a progress indicator to good old dd.

Start in one window the watch:

$ watch --interval 5 killall -USR1 dd

Start copying in another:

$ dd if=/dev/random of=junk bs=1000 count=1000 
dd: warning: partial read (13 bytes); suggest iflag=fullblock
0+2 records in
0+2 records out
21 bytes (21 B) copied, 3.01687 s, 0.0 kB/s
0+3 records in
0+3 records out
29 bytes (29 B) copied, 8.02736 s, 0.0 kB/s

0

Make the output of the `time` builtin easier to parse

 $ TIMEFORMAT=%R

— by Janos on Dec. 4, 2012, 10:43 p.m.

Explanation

The time builtin prints a summary of the real time, user CPU time and system CPU time spent executing commands, for example:

$ time sleep 1

real    0m1.002s
user    0m0.000s
sys     0m0.002s

If you need to parse this output, it helps to simplify it using the TIMEFORMAT variable. The value %R means "the elapsed time in seconds", for example:

$ TIMEFORMAT=%R
$ time sleep 1
1.004

The complete documentation of the format definition is in man bash, search for TIMEFORMAT.

1

Define an own watch(1)-like function

 $ watch () { interrupted=false; trap "interrupted=true" INT; while ! $interrupted; do $*; sleep 1 || interrupted=true; done; }

— by ulidtko on Nov. 14, 2012, 10 p.m.

Explanation

Once I needed a classical watch(1) command, but all I got on Mac OS X was -bash: watch: command not found.

So this is a simple interruptable while loop which can poll anything you like, e.g. watch grep alert /var/log/syslog.

1

Remove offending key from known_hosts file with one swift move

 $ vi +18d +wq ~/.ssh/known_hosts

— by Janos on Oct. 30, 2012, 9:28 p.m.

Explanation

When you try to ssh to a server where the host key has changed then you probably get an error message like this:

Offending key in /home/jack/.ssh/known_hosts:18

If you know for sure that it was the server's admins who changed the host key, then your next move is to remove line 18 from the file so that ssh lets you connect after accepting the new key.

The one-liner does this in one swift move by passing simple commands to vi:

  • +18d -- delete line 18
  • +wq -- save the file and exit

1

Replace the header of all files found.

 $ find . -type f -name '*.html' -exec sed -i -e '1r common_header' -e '1,/STRING/d' {} \;

— by jam on Oct. 25, 2012, 9:29 a.m.

Explanation

Replaces the lines from 1 to the first occurrence of a line starting with STRING of every file found with find.

  • find . -type f -name '*.html' returns a list of all files (not including directories) ending with .html. The ' ' in name is used to pass the literal wildcard * to the find command, instead of the * interpretation of bash, that is, repeat the command for every file in the current folder.
  • -exec execute the following command in each of the files found, using {} as the filename. the ; termination must be escaped with \;
  • sed -i replaces in file (output is the same file)
  • -e '1r common_header' -e '1,/STRING/d' {} reads common_header file, then finds the first occurrence of STRING and replaces it, deleting all the previous lines and putting the contents of common_header.

Limitations

The -i flag of sed requires a parameter in BSD sed, used as the suffix of a backup file. To not use a backup file, you can pass an empty value with -i''.

0

Remove EXIF data such as orientation from images

 $ mogrify -strip /path/to/image.jpg

— by Janos on Oct. 24, 2012, 12:08 a.m.

Explanation

I use this mostly to remove orientation information from images. My problem with orientation information is that some viewers don't support it, and thus do not show the image correctly oriented. Rotating the image doesn't help, because if I make the image look correct in the viewer that doesn't support orientation, that will break it in the viewer that does support orientation. The solution is to remove the orientation information and rotate the image appropriately. That way the image will always look the same in all viewers, regardless of support for the orientation information.

The tool mogrify is part of ImageMagick, an image manipulation software. It manipulates image files and saves the result in the same file. A similar tool in ImageMagick that saves the result of manipulations is convert, you can use it like this:

convert -strip orig.jpg stripped.jpg

Limitations

The tool is part of ImageMagick, an image manipulation software.