We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

0

Insert lines from one text file to another one

 $ awk 'NR % 10 == 1 {getline f2 < "file1"; print f2} 1' file2 | cat -n

— by openiduser102 on June 22, 2013, 9:30 a.m.

Explanation

An alternative with line numbers.

0

Insert lines from one text file to another one

 $ sed -re ':a;Rfile1' -e 'x;s/^/./;/.{10}/!{x;ba};s/.*//;x' file2

— by openiduser102 on June 22, 2013, 9:29 a.m.

Explanation

This command reads the first line from file2 and then 10 lines from file1, then the second line from file2 and the next 10 lines from file1 and so on.

Limitations

Works in GNU sed.

0

Check that a directory is a parent of another

 $ is_parent() { [[ "$2" =~ $1/? ]]; }

— by Couannette on June 13, 2013, 11:03 p.m.

Explanation

The function expanded would look like this :

T() {
    if [[ "$2" =~ $1/? ]]; then
        echo "$2 is child of $1"
        return 0
    else
        echo "$2 is NOT child of $1 ($?)"
        return 1
    fi
}

0

Create fattal tone mapped images from a directory of raw images

 $ for img in /path/to/rawimages/*.RW2; do pfsin ${img} | pfssize -x 1024 -y 768 | pfstmo_fattal02 -v -s 1 | pfsout /path/to/finished/${img%%}.jpg; done

— by mmaki on June 3, 2013, 10:45 p.m.

Explanation

for img in /path/to/rawimages/*.RW2; do Loop through image directory

pfsin ${img} | read the raw image

pfssize -x 1024 -y 768 | resize it to 1024x768 because fattal looks better at low resolutions

pfstmo_fattal02 -v -s 1 | use the fattal tone mapping operator, be verbose and saturation value of 1

pfsout ./path/to/finished/${img%%}.jpg; done output and rename the file as a jpg.

Examples of fattal tone mapped images http://goo.gl/IayQQ

pfstools website http://pfstools.sourceforge.net/

Limitations

Portrait orientation images need to be processed -x 768 -y 1024

0

Send a file by email as attachment

 $ uuencode /var/log/messages messages.txt | mailx -s "/var/log/messages on $HOST" me@example.com

— by Janos on May 26, 2013, 9:37 a.m.

Explanation

  • uuencode /var/log/messages messages.txt -- the first parameter is the file to attach, the second is the filename to use for the attachment
  • mailx -s subject emailaddress -- takes standard input as the content of the email

0

Calculate md5sum from an input string

 $ md5sum <<< YOUR_TEXT | cut -f1 -d' '

— by kowalcj0 on May 17, 2013, 8:17 p.m.

Explanation

Calculate a MD5 sum/digest from an input string

Wrap it up in a function:

function md5() { md5sum <<< $1 | cut -f1 -d' '; }

Example usage:

md5 "this is a long string test_string"
md5 singleWordExample

2

Convert a music file (mp3) to a mp4 video with a static image

 $ ffmpeg -loop_input -i cover.jpg -i soundtrack.mp3 -shortest -acodec copy output_video.mp4

— by kowalcj0 on May 17, 2013, 8:09 p.m.

Explanation

Can come handy when you'd like to post a song to YT or somethin' :)

Can be easily wrapped up in a function:

function mp3tovidwithimg() {
  ffmpeg -loop_input -i $1 -i $2 -shortest -acodec copy $3
}

and used like that:

mp3tovidwithimg cover.jpeg music_track.mp3 output_vid.mp4

1

Find all of the distinct file extensions in a folder

 $ find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u

— by kowalcj0 on May 17, 2013, 8:05 p.m.

Explanation

Will find all of the distinct file extensions in a folder hierarchy.

Originally posted at: http://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

2

Dump network traffic with tcpdump to file with time-stamp in its filename

 $ date +"%Y-%m-%d_%H-%M-%Z" | xargs -I {} bash -c "sudo tcpdump -nq -s 0 -i eth0 -w ./dump-{}.pcap"

— by kowalcj0 on May 17, 2013, 8:03 p.m.

Explanation

will dump the traffic into a file with a time-stamp in its name. Example filename:

dump-2013-05-17_15-46-UTC.pcap

0

Get streamed FLV from Chrome with lsof

 $ export psid=$(pgrep -f libflashplayer.so); cp /proc/$psid/fd/$(lsof -p $psid | grep eleted | awk {' print $4 '} | sed -e "s/[a-z]//g") saved.flv

— by GNA on May 11, 2013, 10:55 p.m.

Explanation

first get the process id of the chome browser process which runs the flashplayer export it to a variable to be used later. Then we get in subshell the filedescriptor which is marked deleted and construct the /proc path for the in memory fileimage and copy it to the file named saved.flv

Limitations

IMPORTANT: only one video should be open to play in chrome

2

Remove files and directories whose name is a timestamp older than a certain time

 $ ls | grep '....-..-..-......' | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"

— by openiduser95 on May 7, 2013, 8:54 a.m.

Explanation

Suppose you have a backup directory with backup snapshots named by timestamp:

$ ls
2013-05-03-103022
2013-05-04-103033
2013-05-05-103023
2013-05-06-103040
2013-05-07-103022

You want to remove snapshots older than 3 days. The one-liner does it:

$ date
Tue May  7 13:50:57 KST 2013
$ ls | grep '....-..-..-......' | sort | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"
removed directory: `2013-05-03-103022'
removed directory: `2013-05-04-103033'

Limitations

It doesn't work on OS X due to the differences between GNU date and BSD date).

4

Tree-like output in ls

 $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'

— by clitips on April 26, 2013, 1:37 p.m.

Explanation

This one-liner initially does a recursive listing of the current directory: ls -R.

Any output other that the directory names, identified by : at the very end of each line (hence :$), is filtered out: grep ":$".

Finally there's a little of sed magic replacing any hierarchy level (/) with dashes (-).

Limitations

Works for me with Bash under Linux, Mac OS X, Solaris.

1

Unhide all hidden files in the current directory.

 $ find . -maxdepth 1 -type f -name '\.*' | sed -e 's,^\./\.,,' | sort | xargs -iname mv .name name

— by openiduser93 on April 25, 2013, 7:46 a.m.

Explanation

This will remove the leading dot from all files in the current directory using mv, effectively "unhiding" them.

It will not affect subdirectories.

Limitations

Probably only works on GNU Linux, due to the specific usage of xargs.

0

Rename all files in a directory to upper case

 $ for i in *; do mv "$i" "${i^^}"; done

— by EvaggelosBalaskas on April 20, 2013, 9:53 p.m.

Explanation

Loop over the items in the current directory, and use Bash built-in case modification expansion to convert to upper case.

Limitations

The case modification extension is available since Bash 4.

5

Rename all items in a directory to lower case

 $ for i in *; do mv "$i" "${i,,}"; done

— by EvaggelosBalaskas on April 20, 2013, 9:41 p.m.

Explanation

Loop over the items in the current directory, and use Bash built-in case modification expansion to convert to lower case.

Limitations

The case modification extension is available since Bash 4.

0

Print file owners and permissions of a directory tree

 $ find /path/to/dir1 -printf "%U %G %m %p\n" > /tmp/dir1.txt

— by Janos on March 19, 2013, 10:51 p.m.

Explanation

The command simply traverses the specified directory tree and for each file and directory it prints the UID of the owner, GID of the group, the permission bits and the path.

To compare file owners and permissions of two directory trees you can run this command for each directory, save the output in two files and then compare them using diff or similar.

See man find for more explanation of all the possible symbols you can use with -printf

Limitations

The -printf option does not exist in find on Solaris 10.

1

Get only the latest version of a file from across mutiple directories.

 $ find . -name 'filename' | xargs -r ls -tc | head -n1

— by Anntoin on March 7, 2013, 11:39 p.m.

Explanation

Shows latest file (by last modification of file status information) for the given pattern. So in this example filename = custlist*.xls.

We use ls to do the sorting (-t) and head to pick the top one. xargs is given the -r option so that ls isn't run if there is no match.

Limitations

The filesystem needs to support ctime. Does not depend on a consistent naming scheme.

1

Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.

 $ sort file1 file2 | uniq -u

— by EvaggelosBalaskas on March 6, 2013, 8:58 a.m.

Explanation

The -u flag of uniq removes duplicate lines from the input.

Example file1:

123456
234567
345678

Example file2:

234567
345678
456789

Result:

123456
456789

0

Get only the latest version of a file from across mutiple directories

 $ find . -name custlist\* | perl -ne '$path = $_; s?.*/??; $name = $_; $map{$name} = $path; ++$c; END { print $map{(sort(keys(%map)))[$c-1]} }'

— by Janos on Feb. 23, 2013, 4:23 p.m.

Explanation

The purpose of the one-liner is to find the the "latest" version of the custlist_*.xls file from among multiple versions in directories and sub-directories, for example:

./c/custlist_v1.003.xls
./c/custlist_v2.001.xls
./d/b/custlist_v1.001.xls
./d/custlist_v1.002.xls

Let's decompose the one-liner to the big steps:

  • find . -name custlist\* -- find the files matching the target pattern
  • ... | perl -ne '...' -- run perl, with the input wrapped around in a while loop so that each line in the input is set in the variable $_
  • $path = $_; s?.*/??; $name = $_; -- save the full path in $path, and cut off the subdirectory part to get to the base name of the file and save it in $name
  • $map{$name} = $path; -- build a mapping of $name to $path
  • ++$c; -- we count the elements, to use it later
  • (sort(keys(%map)))[$c-1] -- sort the keys of the map, and get the last element, which is custlist_v2.001.xls in this example
  • END { print $map{$last} }' -- at the end of all input data, print the path of the latest version of the file

Limitations

Even if the latest version of the file appears multiple times in the directories, the one-liner will print only one of the paths. This could be fixed though if needed.

2

Create a thumbnail from the first page of a PDF file

 $ convert -thumbnail x80 file.pdf[0] thumb.png

— by Janos on Feb. 6, 2013, 9:44 p.m.

Explanation

  • convert is part of ImageMagick image manipulation tool
  • -thumbnail x80 means create a thumbnail image of height 80 pixels, the width will be automatically chosen to make the image proportional
  • The [0] is to create a thumbnail for the first page only, without that a thumbnail image would be created for each page in the pdf file

To do this for all PDF files in a directory tree:

find /path/to/dir -name '*.pdf' -exec convert -thumbnail x80 {}[0] {}-thumb.png \;

Limitations

Requires the ImageMagick image manipulation tool.

0

Recreate or update an existing zip file and remove files that do not exist anymore

 $ zip --filesync -r /path/to/out.zip /path/to/dir

— by Janos on Jan. 26, 2013, 8:48 p.m.

Explanation

zip does not have an explicit option to overwrite/recreate an existing zip file. If the specified destination file already exists, zip updates it. The problem is that files you did not specify to add to the zip but they already existed in the zip, will not be removed.

For example let's say you created a zip file from a directory with the command:

zip -r /path/to/out.zip /path/to/dir

Next you delete some files from the directory and repeat the command to recreate the zip. But that will not recreate the zip, it will only update it, and so the file you deleted from the directory will still be there in the zip.

One way to recreate the zip is to delete the file first. Another, better way is to use the --filesync or -FS flag. With this flag zip will remove files from the zip that do not exist anymore in the filesystem. This is more efficient than recreating the zip.

3

Remove offending key from known_hosts file with one swift move

 $ sed -i 18d .ssh/known_hosts

— by EvaggelosBalaskas on Jan. 16, 2013, 2:29 p.m.

Explanation

Using sed to remove a specific line.

The -i parameter is to edit the file in-place.

Limitations

This works as posted in GNU sed. In BSD sed, the -i flag requires a parameter to use as the suffix of a backup file. You can set it to empty to not use a backup file:

sed -i'' 18d .ssh/known_hosts

0

How to expand a CIDR notation to its IPs

 $ for j in $(seq 0 255); do for i in $(seq 0 255) ; do seq -f "10.$j.$i.%g" 0 255; done; done

— by EvaggelosBalaskas on Jan. 16, 2013, 11:53 a.m.

Explanation

Using two for loops to create second & third block of IP and finally through a formatted seq to printf the output.

More efficient/using less memory than using {..} (range).

Limitations

seq is not available by default in some systems.

1

Get load average in a more parse-able format

 $ python -c 'import os; print os.getloadavg()[0]'

— by FoxWilson on Jan. 5, 2013, 3:32 a.m.

Explanation

In short, it runs a Python one-line script which imports the 'os' module and uses it to get the load average. It then indexes into a tuple, using the index of zero. The tuple is like this: (one-minute-average, five-minute-average, fifteen-minute-average), so you could substitute 0 for 1 to get the five minute load average, or 2 to get the fifteen minute load average. I find this really useful when writing larger bash one-liners which require this information, as I don't have to parse the output of uptime.

Limitations

Requires Python.

1

Function to extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by Janos on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'