We aim to collect practical, well-explained bash one-liners, and promote best practices in shell scripting. To get the latest bash one-liners, follow @bashoneliners on Twitter, or subscribe to our rss feed. If you find any problems, report a bug on GitHub.

Tags

0

Find which log files contain or don't contain a specific error message

 $ for i in *.log; do grep OutOfMemo $i >/dev/null && echo $i oom || echo $i ok; done

— by openiduser3 on Sept. 13, 2013, 3:43 p.m.

Explanation

In this example I was looking for a list of log files which contain or don't contain a stack trace of OutOfMemoryError events.

  • for i in *.log is to loop over the list of files.
  • For each file, I run grep, but redirect the output to /dev/null, as I don't need that, I just want to see a "yes or no" kind of summary for each file
  • grep exits with success if it found any matching lines, otherwise with failure. Using the pattern cmd && success || failure, I echo the filename and the text "oom" in case of a match, or "ok" otherwise

Remarks:

  • Using grep -q is equivalent to redirecting output to /dev/null, but might not be supported in all systems
  • grep -l can be used to list files with matches, and grep -L to list files without matches, but the latter does not exist in some implementations of grep, such as BSD
  • I realized it a bit late, but grep -c shows a count of the matches, so actually it could have been a suitable and simpler solution
1

Convert text from decimal to little endian hexadecimal

 $ echo $(printf %08X 256 | grep -o .. | tac | tr -d '\n')

— by openiduser111 on Aug. 21, 2013, 8:44 p.m.

Explanation

example of 256
printf %08X produces the 8 characters 00000100
grep breaks string by two characters
tac reverses
tr 00010000

Limitations

could be put in a loop like this
for A in $(printf %08X'\n' 256 255); do echo $A | grep -o .. | tac | tr -d '\n'; done

1

Md5sum the last 5 files in a folder

 $ find /directory1/directory2/ -maxdepth 1 -type f | sort | tail -n 5 | xargs md5sum

— by openiduser113 on Aug. 21, 2013, 3:26 p.m.

Explanation

  • find lists the files, no recursion, no directories, with full path
  • sort list files alphabetically
  • tail keep only the last 5 files
  • xargs send the list as arguments to md5sum
  • md5sum calculate the md5sum for each file

Limitations

Probably can't handle spaces in file or directory names.

1

Get mac address from default interface OS X

 $ netstat -rn | awk '/default/ { print $NF }' | head -1 | xargs -I {}  ifconfig {} | awk '/ether/ {print $2}'

— by spotmac on Aug. 21, 2013, 10:28 a.m.

Explanation

netstat -rn -> get routing table awk '/default/ { print $NF }' -> grep the default routes head -1 -> limit to the first result (is also the interface with the highest priority xargs -I {} ifconfig {} -> use the result to get data from ifconfig awk '/ether/ {print $2}' ->grep the mac address.

Limitations

Tested on OSX.

5

Convert directory of videos to MP4 in parallel

 $ for INPUT in *.avi ; do echo "${INPUT%.avi}" ; done | xargs -i -P9  HandBrakeCLI -i "{}".avi -o "{}".mp4

— by shavenwarthog on Aug. 13, 2013, 5:10 a.m.

Explanation

This oneliner uses the wonderful Handbrake program to convert videos. We convert a directory of AVIs at a time, in parallel.

The first three bits ("for INPUT...done |") lists the AVI files in the current directory, then uses a Bash function to strip off the suffix. It then sends each video file name to the next part.

The next part of the command (| xargs ...) runs our converter in parallel. The "-i" flag says take each input (video file name) and stick it in the "{}" parts of the xargs command. The parallel option lets us run up to 9 commands at the same time ("-P9").

The last part (HandBrakeCLI -i "{}".avi -o "{}".mp4) converts a single video to MP4 format. The two open-close curly braces are replaced with xargs, once per input video file. The first run through will be "HandBrakeCLI -i "input1".avi -o "input1".mp4", next will be "HandBrakeCLI -i "input2".avi -o "input2".mp4", etc.

Enjoy!

Another version of this writeup is on my blog: http://johntellsall.blogspot.com/2013/08/converting-video-for-media-player.html

Limitations

Converting videos in parallel is confusing as Handbrake overwrites the status for every file -- ignore the screen.

install Handbrake from http://handbrake.fr/

It also has a pretty GUI for those who don't like the terminal :)

0

Create a transparent image of given dimensions

 $ convert -size 100x100 xc:none transparency.png

— by openiduser3 on July 31, 2013, 11:32 p.m.

Explanation

  • convert is a tool that's part of the ImageMagick image manipulation library
  • -size 100x100 specifies the dimensions of the image to create
  • xc:none is a symbolic source image, indicating to convert "from nothing"
  • transparency.png is the destination filename, the image format is automatically determined by the extension

Limitations

Requires the ImageMagick image manipulation library.

1

Print a random cat

 $ wget -O - http://placekitten.com/$[500 + RANDOM % 500] | lp

— by openiduser104 on July 26, 2013, 11:43 p.m.

Explanation

$RANDOM gives a random number.

http://placekitten.com is your cat place

wget -O - sends the output to stdout

lp prints

Limitations

Tested on OSX

Cat rules

0

Create a heap dump of a Java process

 $ jmap -dump:format=b,file=/var/tmp/dump.hprof 1234

— by openiduser3 on July 8, 2013, 8:18 a.m.

Explanation

  • Create a heap dump from the running Java process with PID=1234
  • The heap dump will be saved in /var/tmp/dump.hprof in binary format
  • You can open the dump with "MAT", the Memory Analyzer Tool (based on Eclipse) and identify objects that use most of the memory and potential memory leaks

For more options see jmap -h

0

Insert lines from one text file to another one

 $ awk 'NR%10 == 1 {getline f2 < "file1"; print f2} 1' file2 | cat -n

— by openiduser102 on June 22, 2013, 9:30 a.m.

Explanation

An alternative with line numbers.

0

Insert lines from one text file to another one

 $ sed -re ':a;Rfile1' -e 'x;s/^/./;/.{10}/!{x;ba};s/.*//;x' file2

— by openiduser102 on June 22, 2013, 9:29 a.m.

Explanation

This command reads the first line from file2 and then 10 lines from file1, then the second line from file2 and the next 10 lines from file1 and so on.

0

Check that a directory is a parent of another

 $ is_parent() { [[ "$2" =~ ${1}['/'?] ]] && return 0 || return 1;}

— by Couannette on June 13, 2013, 11:03 p.m.

Explanation

The function expanded would look like this :

T() {
    if [[ "$2" =~ ${1}['/'?] ]] ; then
        echo "$2 is child of $1"
        return 0
    else
        echo "$2 is NOT child of $1 ($?)"
        return 1
    fi
}

Limitations

I'd like to know if this works in all cases :).

0

Creates fattal tone mapped images from a directory of raw images

 $ for img in /path/to/rawimages/*.RW2; do pfsin ${img} | pfssize -x 1024 -y 768 | pfstmo_fattal02 -v -s 1 | pfsout /path/to/finished/${img%%}.jpg; done

— by mmaki on June 3, 2013, 10:45 p.m.

Explanation

for img in /path/to/rawimages/*.RW2; do Loop through image directory

pfsin ${img} | read the raw image

pfssize -x 1024 -y 768 | resize it to 1024x768 because fattal looks better at low resolutions

pfstmo_fattal02 -v -s 1 | use the fattal tone mapping operator, be verbose and saturation value of 1

pfsout ./path/to/finished/${img%%}.jpg; done output and rename the file as a jpg.

Examples of fattal tone mapped images http://goo.gl/IayQQ

pfstools website http://pfstools.sourceforge.net/

Limitations

Portrait orientation images need to be processed -x 768 -y 1024

0

Send a file by email as attachment

 $ uuencode /var/log/messages messages.txt | mailx -s "/var/log/messages on $HOST" me@example.com

— by openiduser3 on May 26, 2013, 9:37 a.m.

Explanation

  • uuencode /var/log/messages messages.txt -- the first parameter is the file to attach, the second is the filename to use for the attachment
  • mailx -s subject emailaddress -- takes standard input as the content of the email
0

Calculate md5sum from an input string

 $ md5sum<<<YOUR_TEXT | cut -f1 -d' ';

— by kowalcj0 on May 17, 2013, 8:17 p.m.

Explanation

Will calculate a md5sum from an input string

Wrap it up in a function:

function md5() { md5sum<<<$1 | cut -f1 -d' '; }

Example usage:

md5 "this is a long string test_string"

md5 singleWordExample

0

Convert a music file (mp3) to a mp4 video with a static image

 $ ffmpeg -loop_input -i cover.jpg -i soundtrack.mp3 -shortest -acodec copy output_video.mp4; 

— by kowalcj0 on May 17, 2013, 8:09 p.m.

Explanation

Can come handy when you'd like to post a song to YT or somethin' :)

Can be easily wrapped up in a function:

function mp3tovidwithimg() { ffmpeg -loop_input -i $1 -i $2 -shortest -acodec copy $3; }

and used like that:

mp3tovidwithimg cover.jpeg music_track.mp3 output_vid.mp4

0

Find all of the distinct file extensions in a folder

 $ find . -type f | perl -ne 'print $1 if m/\.([^.\/]+)$/' | sort -u;

— by kowalcj0 on May 17, 2013, 8:05 p.m.

Explanation

Will find all of the distinct file extensions in a folder hierarchy.

Originally posted at: http://stackoverflow.com/questions/1842254/how-can-i-find-all-of-the-distinct-file-extensions-in-a-folder-hierarchy

0

Dump network traffic with tcpdump to file with time-stamp in its the filename

 $ date +"%Y-%m-%d_%H-%M-%Z" | xargs -I {} bash -c "sudo tcpdump -nq -s 0 -i eth0 -w ./dump-{}.pcap"

— by kowalcj0 on May 17, 2013, 8:03 p.m.

Explanation

will dump the traffic into a file with a time-stamp in its name. Example filename:

dump-2013-05-17_15-46-UTC.pcap

0

Get streamed FLV from Chrome with lsof

 $ export psid=$(pgrep -f libflashplayer.so); cp /proc/$psid/fd/$(lsof -p $psid | grep eleted | awk {' print $4 '} | sed -e "s/[a-z]//g") saved.flv

— by GNA on May 11, 2013, 10:55 p.m.

Explanation

first get the process id of the chome browser process which runs the flashplayer export it to a variable to be used later. Then we get in subshell the filedescriptor which is marked deleted and construct the /proc path for the in memory fileimage and copy it to the file named saved.flv

Limitations

IMPORTANT: only one video should be open to play in chrome

0

Remove files and directories whose name is a timestamp older than a certain time

 $ ls | grep '....-..-..-......' | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"

— by openiduser95 on May 7, 2013, 8:54 a.m.

Explanation

Suppose you have a backup directory with backup snapshots named by timestamp:

$ ls
2013-05-03-103022
2013-05-04-103033
2013-05-05-103023
2013-05-06-103040
2013-05-07-103022

You want to remove snapshots older than 3 days. The one-liner does it:

$ date
Tue May  7 13:50:57 KST 2013
$ ls | grep '....-..-..-......' | sort | xargs -I {} bash -c "[[ x{} < x$(date -d '3 days ago' +%Y-%m-%d-%H%M%S) ]] && rm -rfv {}"
removed directory: `2013-05-03-103022'
removed directory: `2013-05-04-103033'

Limitations

It doesn't work on OS X due to the differences between GNU date and BSD date).

1

Tree-like output in ls

 $ ls -R | grep ":$" | sed -e 's/:$//' -e 's/[^-][^\/]*\//--/g' -e 's/^/   /' -e 's/-/|/'

— by clitips on April 26, 2013, 1:37 p.m.

Explanation

This one-liner initially does a recursive listing of the current directory: ls -R.

Any output other that the directory names, identified by ":" at the very end of each line (hence ":$"), is filtered out: grep ":$".

Finally there's a little of "sed" magic replacing any hierarchy level ("/") with dashes ("-"): sed ....

Works for me with Bash under Linux, Mac OS X, Solaris.

0

Unhide all hidden files in the current directory.

 $ find . -maxdepth 1 -type f -name '\.*' | sed -e 's,^\./\.,,' | sort | xargs -iname mv .name name

— by openiduser93 on April 25, 2013, 7:46 a.m.

Explanation

This will remove the leading dot from all files in the current directory using mv, effectively "unhiding" them.

It will not affect subdirectories.

Limitations

Probably only works on GNU Linux, due to the specific usage of xargs.

0

Rename all files in a directory to upper case

 $ for i in `ls -1`; do mv $i "${i^^}" ; done

— by EvaggelosBalaskas on April 20, 2013, 9:53 p.m.

Explanation

using for, for looping the files through a directory and using bash build-in case modification expansion to upper case the files

4

An elegant way to rename all files in a directory to lowercase names

 $ for i in `ls -1`; do mv $i "${i,,}" ; done

— by EvaggelosBalaskas on April 20, 2013, 9:41 p.m.

Explanation

there is a lower/upper case modification build-in bash using for, for looping files through directory

Limitations

none

0

Compare file owners and permissions of two directory trees

 $ find /path/to/dir1 -printf "%U %G %m %p\n" > /tmp/dir1.txt

— by openiduser3 on March 19, 2013, 10:51 p.m.

Explanation

The command simply traverses the specified directory tree and for each file and directory it prints the UID of the owner, GID of the group, the permission bits and the path.

To compare file owners and permissions of two directory trees you can run this command for each directory, save the output in two files and then compare them using diff or similar.

See man find for more explanation of all the possible symbols you can use with -printf

Limitations

The -printf option does not exist in find on Solaris 10.

0

Get only the latest version of a file from across mutiple directories.

 $ find . -name 'filename' | xargs -r ls -tc | head -n1

— by Anntoin on March 7, 2013, 11:39 p.m.

Explanation

Shows latest file (by last modification of file status information) for the given pattern. So in this example filename = custlist*.xls.

We use ls to do the sorting (-t) and head to pick the top one. xargs is given the -r option so that ls isn't run if there is no match.

Limitations

The filesystem needs to support ctime. Does not depend on a consistent naming scheme.