We collect practical, well-explained Bash one-liners, and promote best practices in Bash shell scripting. To get the latest Bash one-liners, follow @bashoneliners on Twitter. If you find any problems, report a bug on GitHub.

Tags

0

Download a file from a webserver with telnet

 $ (echo 'GET /'; echo; sleep 1; ) | telnet www.google.com 80

— by Janos on Dec. 22, 2014, 11:31 p.m.

Explanation

If you are ever in a minimal headless *nix which doesn't have any command line utilities for downloading files (no curl, wget, lynx) but you have telnet, then this can be a workaround.

Another option is netcat:

/usr/bin/printf 'GET / \n' | nc www.google.com 80

Credit goes to this post: http://unix.stackexchange.com/a/83987/17433

0

Print the window title of current mpv session to display what is playing

 $ wmctrl -pl | grep $(pidof mpv) | cut -d- -f2-

— by openiduser171 on Dec. 15, 2014, 3:37 a.m.

Explanation

wmctrl -l lists all open windows (works with several window managers), -p includes the unique process ID of each window in the list. grep $(pidof mpv) matches the line that contains the process ID of mpv. cut -d'-' -f2- prints everything after the the first delimiter '-' (from the second onwards), which just leaves the title bit.

Limitations

Only works with one instance of mpv running. It's intended use is to share what film or series you are watching and you don't usually watch more than one thing at a time.

0

Shuffle lines

 $ ... | perl -MList::Util -e 'print List::Util::shuffle <>'

— by Janos on Oct. 25, 2014, 10:40 p.m.

Explanation

Sorting lines is easy: everybody knows the sort command.

But what if you want to do the other way around? The above perl one-liner does just that:

  • -MList::Util load the List::Util module (as if doing use List::Util inside a Perl script)
  • -e '...' execute Perl command
  • print List::Util::shuffle <> call List::Util::shuffle for the lines coming from standard input, read by <>

Another way would be sort -R if your version supports that (GNU, as opposed to BSD). In BSD systems you can install coreutils and try gsort -R instead. (For eample on OSX, using MacPorts: sudo port install coreutils.)

0

Open Windows internet shortcut (*.url) files in firefox

 $ firefox $(grep -i ^url='*' file.url | cut -b 5-)

— by tsjswimmer on Sept. 11, 2014, 10:03 a.m.

Explanation

Extract urls from a *.url file and open in Firefox. (Note that *.url files in Windows are basically just text files, so they can be parsed with a few commands.)

  • grep extracts lines starting with url=
  • The -i flag is to ignore case
  • cut extracts the range of characters from the 5th until the end of lines
  • The output of $(...) will be used as command line parameters for Firefox

Limitations

This only works with URLs that don't contain special characters that would be interpreted by the shell, such as spaces and others.

0

Check if a file exists and has a size greater than X

 $ [[ $(find /path/to/file -type f -size +51200c 2>/dev/null) ]] && echo true || echo false

— by Janos on Jan. 9, 2014, 12:34 p.m.

Explanation

  • The find takes care two things at once: checks if file exists and size is greater than 51200.
  • We redirect stderr to /dev/null to hide the error message if the file doesn't exist.
  • The output of find will be non-blank if the file matched both conditions, otherwise it will be blank
  • The [[ ... ]] evaluates to true or false if the output of find is non-blank or blank, respectively

You can use this in if conditions like:

if [[ $(find /path/to/file -type f -size +51200c 2>/dev/null) ]]; do
    somecmd
fi

0

Replace sequences of the same characters with a single character

 $ echo heeeeeeelllo | sed 's/\(.\)\1\+/\1/g'

— by Janos on Dec. 11, 2013, 7:58 p.m.

Explanation

That is, this will output "helo".

The interesting thing here is the regular expression in the s/// command of sed:

  • \(.\) -- capture any character
  • \1 -- refers to the last captured string, in our case the previous character. So effectively, \(.\)\1 matches pairs of the same character, for example aa, bb, ??, and so on.
  • \+ -- match one or more of the pattern right before it
  • ... and we replace what we matched with \1, the last captured string, which is the first letter in a sequence like aaaa, or bbbbbbb, or cc.

0

Counting the number of commas in CSV format

 $ perl -ne 'print tr/,//, "\n"' < file.csv | sort -u

— by Janos on Dec. 1, 2013, 1:03 p.m.

Explanation

Sometimes I need to know if a CSV file has the right number of columns, and how many columns there are.

The tr/// operator in perl is normally used to convert a set of characters to another set of characters, but when used in a scalar context like in this example, it returns the number of matches of the specified characters, in this case a comma.

The perl command above prints the number of commas in every line of the input file. sort -u sorts this and outputs only the unique lines. If all lines in the CSV file have the same number of commas, there should be one line of output. The number of columns in the file is this number + 1.

Limitations

This one-liner does not handle the more general case when the columns may have embedded commas within quotes. For that you would need a more sophisticated method. This simple version can still be very useful in many common cases.

0

Count the lines of each file extension in a list of files

 $ git ls-files | xargs wc -l | awk -F ' +|\\.|/' '{ sumlines[$NF] += $2 } END { for (ext in sumlines) print ext, sumlines[ext] }'

— by Janos on Nov. 9, 2013, 11:49 a.m.

Explanation

The pipeline:

  • git ls-files -- produces the list of files in a Git repository. It could be anything else that produces a list of filenames, for example: find . -type f
  • xargs wc -l -- run wc -l to count the lines in the filenames coming from standard input. The output is the line count and the filename
  • The final awk command does the main work: extract the extension name and sum the line counts:
  • -F ' +|\\.|/' -- use as field separator multiples of spaces, or a dot, or a slash
  • { sumlines[$NF] += $2 } -- $NF contains the value of the last field, which is the filename extension, thanks to the dot in the field separator, and $2 contains the value of the second field in the input, which is the line count. As a result, we are building the sumlines associative array, summing up the line counts of files with the same extension
  • END { for (ext in sumlines) print ext, sumlines[ext] }' -- After all lines have been processed, print the extension and the line count.

0

Add all unknown files in a Subversion checkout

 $ svn add . --force

— by Janos on Sept. 24, 2013, 7:59 a.m.

Explanation

Adding all unknown files in a working tree is usually very simple in other version control systems, for example:

git add .
bzr add

Not so simple in Subversion:

$ svn add .
svn: warning: '.' is already under version control

But if you add the --force flag, that will do!

Keep in mind that this is not the same as:

svn add * --force

That would add not only unknown files, but ignored files too, which is probably not your intention. Make sure to specify directories explicitly, avoid using * with this command.

0

Find files that are not executable

 $ find /some/path -type f ! -perm -111 -ls

— by Janos on Sept. 18, 2013, 9:14 p.m.

Explanation

The key is writing the parameter of -perm correctly. The value -111 means that all execution bits must be set: user and group and other too. By negating this pattern with ! we get files that miss any of the execution bits.

If you want to be more specific, for example find files that are not executable specifically by the owner, you could do like this:

find /some/path -type f ! -perm -100 -ls

The -ls option is to print the found files using a long listing format similar to the ls command.

0

Find which log files contain or don't contain a specific error message

 $ for i in *.log; do grep OutOfMemo $i >/dev/null && echo $i oom || echo $i ok; done

— by Janos on Sept. 13, 2013, 3:43 p.m.

Explanation

In this example I was looking for a list of log files which contain or don't contain a stack trace of OutOfMemoryError events.

  • for i in *.log is to loop over the list of files.
  • For each file, I run grep, but redirect the output to /dev/null, as I don't need that, I just want to see a "yes or no" kind of summary for each file
  • grep exits with success if it found any matching lines, otherwise with failure. Using the pattern cmd && success || failure, I echo the filename and the text "oom" in case of a match, or "ok" otherwise

Remarks:

  • Using grep -q is equivalent to redirecting output to /dev/null, but might not be supported in all systems
  • grep -l can be used to list files with matches, and grep -L to list files without matches, but the latter does not exist in some implementations of grep, such as BSD
  • I realized it a bit late, but grep -c shows a count of the matches, so actually it could have been a suitable and simpler solution

0

Create a transparent image of given dimensions

 $ convert -size 100x100 xc:none transparency.png

— by Janos on July 31, 2013, 11:32 p.m.

Explanation

  • convert is a tool that's part of the ImageMagick image manipulation library
  • -size 100x100 specifies the dimensions of the image to create
  • xc:none is a symbolic source image, indicating to convert "from nothing"
  • transparency.png is the destination filename, the image format is automatically determined by the extension

Limitations

Requires the ImageMagick image manipulation library.

0

Create a heap dump of a Java process

 $ jmap -dump:format=b,file=/var/tmp/dump.hprof 1234

— by Janos on July 8, 2013, 8:18 a.m.

Explanation

  • Create a heap dump from the running Java process with PID=1234
  • The heap dump will be saved in /var/tmp/dump.hprof in binary format
  • You can open the dump with "MAT", the Memory Analyzer Tool (based on Eclipse) and identify objects that use most of the memory and potential memory leaks

For more options see jmap -h

0

Insert lines from one text file to another one

 $ awk 'NR % 10 == 1 {getline f2 < "file1"; print f2} 1' file2 | cat -n

— by openiduser102 on June 22, 2013, 9:30 a.m.

Explanation

An alternative with line numbers.

0

Insert lines from one text file to another one

 $ sed -re ':a;Rfile1' -e 'x;s/^/./;/.{10}/!{x;ba};s/.*//;x' file2

— by openiduser102 on June 22, 2013, 9:29 a.m.

Explanation

This command reads the first line from file2 and then 10 lines from file1, then the second line from file2 and the next 10 lines from file1 and so on.

Limitations

Works in GNU sed.

0

Check that a directory is a parent of another

 $ is_parent() { [[ "$2" =~ $1/? ]]; }

— by Couannette on June 13, 2013, 11:03 p.m.

Explanation

The function expanded would look like this :

T() {
    if [[ "$2" =~ $1/? ]]; then
        echo "$2 is child of $1"
        return 0
    else
        echo "$2 is NOT child of $1 ($?)"
        return 1
    fi
}

0

Create fattal tone mapped images from a directory of raw images

 $ for img in /path/to/rawimages/*.RW2; do pfsin ${img} | pfssize -x 1024 -y 768 | pfstmo_fattal02 -v -s 1 | pfsout /path/to/finished/${img%%}.jpg; done

— by mmaki on June 3, 2013, 10:45 p.m.

Explanation

for img in /path/to/rawimages/*.RW2; do Loop through image directory

pfsin ${img} | read the raw image

pfssize -x 1024 -y 768 | resize it to 1024x768 because fattal looks better at low resolutions

pfstmo_fattal02 -v -s 1 | use the fattal tone mapping operator, be verbose and saturation value of 1

pfsout ./path/to/finished/${img%%}.jpg; done output and rename the file as a jpg.

Examples of fattal tone mapped images http://goo.gl/IayQQ

pfstools website http://pfstools.sourceforge.net/

Limitations

Portrait orientation images need to be processed -x 768 -y 1024

0

Send a file by email as attachment

 $ uuencode /var/log/messages messages.txt | mailx -s "/var/log/messages on $HOST" me@example.com

— by Janos on May 26, 2013, 9:37 a.m.

Explanation

  • uuencode /var/log/messages messages.txt -- the first parameter is the file to attach, the second is the filename to use for the attachment
  • mailx -s subject emailaddress -- takes standard input as the content of the email

0

Calculate md5sum from an input string

 $ md5sum <<< YOUR_TEXT | cut -f1 -d' '

— by kowalcj0 on May 17, 2013, 8:17 p.m.

Explanation

Calculate a MD5 sum/digest from an input string

Wrap it up in a function:

function md5() { md5sum <<< $1 | cut -f1 -d' '; }

Example usage:

md5 "this is a long string test_string"
md5 singleWordExample

0

Get streamed FLV from Chrome with lsof

 $ export psid=$(pgrep -f libflashplayer.so); cp /proc/$psid/fd/$(lsof -p $psid | grep eleted | awk {' print $4 '} | sed -e "s/[a-z]//g") saved.flv

— by GNA on May 11, 2013, 10:55 p.m.

Explanation

first get the process id of the chome browser process which runs the flashplayer export it to a variable to be used later. Then we get in subshell the filedescriptor which is marked deleted and construct the /proc path for the in memory fileimage and copy it to the file named saved.flv

Limitations

IMPORTANT: only one video should be open to play in chrome

0

Rename all files in a directory to upper case

 $ for i in *; do mv "$i" "${i^^}"; done

— by EvaggelosBalaskas on April 20, 2013, 9:53 p.m.

Explanation

Loop over the items in the current directory, and use Bash built-in case modification expansion to convert to upper case.

Limitations

The case modification extension is available since Bash 4.

0

Print file owners and permissions of a directory tree

 $ find /path/to/dir1 -printf "%U %G %m %p\n" > /tmp/dir1.txt

— by Janos on March 19, 2013, 10:51 p.m.

Explanation

The command simply traverses the specified directory tree and for each file and directory it prints the UID of the owner, GID of the group, the permission bits and the path.

To compare file owners and permissions of two directory trees you can run this command for each directory, save the output in two files and then compare them using diff or similar.

See man find for more explanation of all the possible symbols you can use with -printf

Limitations

The -printf option does not exist in find on Solaris 10.

0

Sort and remove duplicate lines from two (or more files). Display only uniq lines from files.

 $ sort file1 file2 | uniq -u

— by EvaggelosBalaskas on March 6, 2013, 8:58 a.m.

Explanation

The -u flag of uniq removes duplicate lines from the input.

Example file1:

123456
234567
345678

Example file2:

234567
345678
456789

Result:

123456
456789

0

Get only the latest version of a file from across mutiple directories

 $ find . -name custlist\* | perl -ne '$path = $_; s?.*/??; $name = $_; $map{$name} = $path; ++$c; END { print $map{(sort(keys(%map)))[$c-1]} }'

— by Janos on Feb. 23, 2013, 4:23 p.m.

Explanation

The purpose of the one-liner is to find the the "latest" version of the custlist_*.xls file from among multiple versions in directories and sub-directories, for example:

./c/custlist_v1.003.xls
./c/custlist_v2.001.xls
./d/b/custlist_v1.001.xls
./d/custlist_v1.002.xls

Let's decompose the one-liner to the big steps:

  • find . -name custlist\* -- find the files matching the target pattern
  • ... | perl -ne '...' -- run perl, with the input wrapped around in a while loop so that each line in the input is set in the variable $_
  • $path = $_; s?.*/??; $name = $_; -- save the full path in $path, and cut off the subdirectory part to get to the base name of the file and save it in $name
  • $map{$name} = $path; -- build a mapping of $name to $path
  • ++$c; -- we count the elements, to use it later
  • (sort(keys(%map)))[$c-1] -- sort the keys of the map, and get the last element, which is custlist_v2.001.xls in this example
  • END { print $map{$last} }' -- at the end of all input data, print the path of the latest version of the file

Limitations

Even if the latest version of the file appears multiple times in the directories, the one-liner will print only one of the paths. This could be fixed though if needed.

0

Recreate or update an existing zip file and remove files that do not exist anymore

 $ zip --filesync -r /path/to/out.zip /path/to/dir

— by Janos on Jan. 26, 2013, 8:48 p.m.

Explanation

zip does not have an explicit option to overwrite/recreate an existing zip file. If the specified destination file already exists, zip updates it. The problem is that files you did not specify to add to the zip but they already existed in the zip, will not be removed.

For example let's say you created a zip file from a directory with the command:

zip -r /path/to/out.zip /path/to/dir

Next you delete some files from the directory and repeat the command to recreate the zip. But that will not recreate the zip, it will only update it, and so the file you deleted from the directory will still be there in the zip.

One way to recreate the zip is to delete the file first. Another, better way is to use the --filesync or -FS flag. With this flag zip will remove files from the zip that do not exist anymore in the filesystem. This is more efficient than recreating the zip.