Twitter http://twitter.com/janosgyerik/
Blog http://janosgyerik.com/blog/
Website http://janosgyerik.com/

0

Kill a process running on port 8080

 $ lsof -i :8080 | awk 'NR > 1 {print $2}' | xargs --no-run-if-empty kill

— by Janos on Sept. 1, 2017, 8:31 p.m.

Explanation

lsof lists open files (ls-o-f, get it?). lsof -i :8080 lists open files on address ending in :8080. The output looks like this

COMMAND  PID     USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
chrome  2619 qymspace  149u  IPv4  71595      0t0  TCP localhost:53878->localhost:http-alt (CLOSE_WAIT)`

We use awk 'NR > 1 {print $2}' to print the second column for lines except the first. The result is a list of PIDs, which we pipe to xargs kill to kill.

Limitations

The --no-run-if-empty option of xargs is available in GNU implementations, and typically not available in BSD implementations. Without this option, the one-liner will raise an error if there are no matches (no PIDs to kill).

0

Get the HTTP status code of a URL

 $ curl -Lw '%{http_code}' -s -o /dev/null -I SOME_URL

— by Janos on June 19, 2017, 11:15 p.m.

Explanation

  • -w '%{http_code}' is to print out the status code (the meat of this post)
  • -s is to make curl silent (suppress download progress stats output)
  • -o /dev/null is to redirect all output to /dev/null
  • -I is to fetch the headers only, no need for the page content
  • -L is to follow redirects

0

List the content of a GitHub repository without cloning it

 $ svn ls https://github.com/user/repo/trunk/some/path

— by Janos on May 21, 2017, 6:01 p.m.

Explanation

Git doesn't allow querying sub-directories of a repository. But GitHub repositories are also exposed as Subversion repositories, and Subversion allows arbitrary path queries using the ls command.

Notice the /trunk/ between the base URL of the repository and the path to query. This is due to the way GitHub provides Subversion using a standard Subversion repository layout, with trunk, branches and tags sub-directories.

0

Download a file from a webserver with telnet

 $ (echo 'GET /'; echo; sleep 1; ) | telnet www.google.com 80

— by Janos on Dec. 22, 2014, 11:31 p.m.

Explanation

If you are ever in a minimal headless *nix which doesn't have any command line utilities for downloading files (no curl, wget, lynx) but you have telnet, then this can be a workaround.

Another option is netcat:

/usr/bin/printf 'GET / \n' | nc www.google.com 80

Credit goes to this post: http://unix.stackexchange.com/a/83987/17433

3

Change the encoding of all files in a directory and subdirectories

 $ find . -type f  -name '*.java' -exec sh -c 'iconv -f cp1252 -t utf-8 "$1" > converted && mv converted "$1"' -- {} \;

— by Janos on Nov. 20, 2014, 12:15 p.m.

Explanation

The parameters of find:

  • . -- search in the current directory, and its subdirectories, recursively
  • -type f -- match only files
  • -name '*.java' -- match only filenames ending with .java
  • -exec ... \; -- execute command

The command to execute is slightly complicated, because iconv doesn't rewrite the original file but prints the converted content on stdout. To update the original file we need 2 steps:

  1. Convert and save to a temp file
  2. Move the temp file to the original

To do these steps, we use a sh subshell with -exec, passing a one-liner to run with the -c flag, and passing the name of the file as a positional argument with -- {}.

Unfortunately the redirection will use UNIX style line endings. If the original files have DOS style line endings, add this command in the subshell:

vim +'set ff=dos' +wq converted

2

Generate a sequence of numbers

 $ for ((i=1; i<=10; ++i)); do echo $i; done

— by Janos on Nov. 4, 2014, 12:29 p.m.

Explanation

This is similar to seq, but portable. seq does not exist in all systems and is not recommended today anymore. Other variations to emulate various uses with seq:

# seq 1 2 10
for ((i=1; i<=10; i+=2)); do echo $i; done

# seq -w 5 10
for ((i=5; i<=10; ++i)); do printf '%02d\n' $i; done

0

Shuffle lines

 $ ... | perl -MList::Util -e 'print List::Util::shuffle <>'

— by Janos on Oct. 25, 2014, 10:40 p.m.

Explanation

Sorting lines is easy: everybody knows the sort command.

But what if you want to do the other way around? The above perl one-liner does just that:

  • -MList::Util load the List::Util module (as if doing use List::Util inside a Perl script)
  • -e '...' execute Perl command
  • print List::Util::shuffle <> call List::Util::shuffle for the lines coming from standard input, read by <>

Another way would be sort -R if your version supports that (GNU, as opposed to BSD). In BSD systems you can install coreutils and try gsort -R instead. (For eample on OSX, using MacPorts: sudo port install coreutils.)

1

Print a flat list of dependencies of a Maven project

 $ mvn dependency:list | sed -ne s/..........// -e /patterntoexclude/d -e s/:compile//p -e s/:runtime//p | sort | uniq

— by Janos on Sept. 22, 2014, 9:02 p.m.

Explanation

The mvn dependency:list command produces a list of dependencies that's readable but not very program-friendly, looking like this:

[INFO] The following files have been resolved:
[INFO]    joda-time:joda-time:jar:2.3:compile
[INFO]    junit:junit:jar:4.11:test
[INFO]    log4j:log4j:jar:1.2.12:compile

A sed can shave off the extra formatting to turn this into:

joda-time:joda-time:jar:2.4
log4j:log4j:jar:1.2.12

Explanation:

  • -n don't print by default
  • -e s/..........// shave off the first 10 characters
  • -e /patterntoexclude/d you can exclude some unwanted patterns from the list using the d command like this
  • -e s/:compile//p -e s/:runtime//p replace and print :compile and :runtime

As multi-module projects may include duplicates, filter the result through | sort | uniq

0

Find recent logs that contain the string "Exception"

 $ find . -name '*.log' -mtime -2 -exec grep -Hc Exception {} \; | grep -v :0$

— by Janos on July 19, 2014, 7:53 a.m.

Explanation

The find:

  • -name '*.log' -- match files ending with .log
  • -mtime -2 -- match files modified within the last 2 days
  • -exec CMD ARGS \; -- for each file found, execute command, where {} in ARGS will be replaced with the file's path

The grep:

  • -c is to print the count of the matches instead of the matches themselves
  • -H is to print the name of the file, as grep normally won't print it when there is only one filename argument
  • The output lines will be in the format path:count. Files that didn't match "Exception" will still be printed, with 0 as count
  • The second grep filters the output of the first, excluding lines that end with :0 (= the files that didn't contain matches)

Extra tips:

  • Change "Exception" to the typical relevant failure indicator of your application
  • Add -i for grep to make the search case insensitive
  • To make the find match strictly only files, add -type f
  • Schedule this as a periodic job, and pipe the output to a mailer, for example | mailx -s 'error counts' yourmail@example.com

Limitations

The -H flag of grep may not work in older operating systems, for example older Solaris. In that case use ggrep (GNU grep) instead, if it exists.

1

Extract your external IP address using dig

 $ dig +short myip.opendns.com @resolver1.opendns.com

— by Janos on Feb. 25, 2014, 7:50 a.m.

Explanation

This asks the IP address of myip.opendns.com from the name server resolver1.opendns.com (something you trust), which will return your external IP address.

If you don't have dig, you could use these other services instead:

curl ipecho.net/plain
curl icanhazip.com
curl curlmyip.com
curl l2.io/ip
curl ip.appspot.com
curl ifconfig.me/ip

Limitations

All these methods rely on external services, which might be sometimes temporarily or even permanently down. In that case, find an alternative service.

0

Check if a file exists and has a size greater than X

 $ [[ $(find /path/to/file -type f -size +51200c 2>/dev/null) ]] && echo true || echo false

— by Janos on Jan. 9, 2014, 12:34 p.m.

Explanation

  • The find takes care two things at once: checks if file exists and size is greater than 51200.
  • We redirect stderr to /dev/null to hide the error message if the file doesn't exist.
  • The output of find will be non-blank if the file matched both conditions, otherwise it will be blank
  • The [[ ... ]] evaluates to true or false if the output of find is non-blank or blank, respectively

You can use this in if conditions like:

if [[ $(find /path/to/file -type f -size +51200c 2>/dev/null) ]]; do
    somecmd
fi

0

Replace sequences of the same characters with a single character

 $ echo heeeeeeelllo | sed 's/\(.\)\1\+/\1/g'

— by Janos on Dec. 11, 2013, 7:58 p.m.

Explanation

That is, this will output "helo".

The interesting thing here is the regular expression in the s/// command of sed:

  • \(.\) -- capture any character
  • \1 -- refers to the last captured string, in our case the previous character. So effectively, \(.\)\1 matches pairs of the same character, for example aa, bb, ??, and so on.
  • \+ -- match one or more of the pattern right before it
  • ... and we replace what we matched with \1, the last captured string, which is the first letter in a sequence like aaaa, or bbbbbbb, or cc.

0

Counting the number of commas in CSV format

 $ perl -ne 'print tr/,//, "\n"' < file.csv | sort -u

— by Janos on Dec. 1, 2013, 1:03 p.m.

Explanation

Sometimes I need to know if a CSV file has the right number of columns, and how many columns there are.

The tr/// operator in perl is normally used to convert a set of characters to another set of characters, but when used in a scalar context like in this example, it returns the number of matches of the specified characters, in this case a comma.

The perl command above prints the number of commas in every line of the input file. sort -u sorts this and outputs only the unique lines. If all lines in the CSV file have the same number of commas, there should be one line of output. The number of columns in the file is this number + 1.

Limitations

This one-liner does not handle the more general case when the columns may have embedded commas within quotes. For that you would need a more sophisticated method. This simple version can still be very useful in many common cases.

0

Count the lines of each file extension in a list of files

 $ git ls-files | xargs wc -l | awk -F ' +|\\.|/' '{ sumlines[$NF] += $2 } END { for (ext in sumlines) print ext, sumlines[ext] }'

— by Janos on Nov. 9, 2013, 11:49 a.m.

Explanation

The pipeline:

  • git ls-files -- produces the list of files in a Git repository. It could be anything else that produces a list of filenames, for example: find . -type f
  • xargs wc -l -- run wc -l to count the lines in the filenames coming from standard input. The output is the line count and the filename
  • The final awk command does the main work: extract the extension name and sum the line counts:
  • -F ' +|\\.|/' -- use as field separator multiples of spaces, or a dot, or a slash
  • { sumlines[$NF] += $2 } -- $NF contains the value of the last field, which is the filename extension, thanks to the dot in the field separator, and $2 contains the value of the second field in the input, which is the line count. As a result, we are building the sumlines associative array, summing up the line counts of files with the same extension
  • END { for (ext in sumlines) print ext, sumlines[ext] }' -- After all lines have been processed, print the extension and the line count.

0

Add all unknown files in a Subversion checkout

 $ svn add . --force

— by Janos on Sept. 24, 2013, 7:59 a.m.

Explanation

Adding all unknown files in a working tree is usually very simple in other version control systems, for example:

git add .
bzr add

Not so simple in Subversion:

$ svn add .
svn: warning: '.' is already under version control

But if you add the --force flag, that will do!

Keep in mind that this is not the same as:

svn add * --force

That would add not only unknown files, but ignored files too, which is probably not your intention. Make sure to specify directories explicitly, avoid using * with this command.

0

Find files that are not executable

 $ find /some/path -type f ! -perm -111 -ls

— by Janos on Sept. 18, 2013, 9:14 p.m.

Explanation

The key is writing the parameter of -perm correctly. The value -111 means that all execution bits must be set: user and group and other too. By negating this pattern with ! we get files that miss any of the execution bits.

If you want to be more specific, for example find files that are not executable specifically by the owner, you could do like this:

find /some/path -type f ! -perm -100 -ls

The -ls option is to print the found files using a long listing format similar to the ls command.

0

Find which log files contain or don't contain a specific error message

 $ for i in *.log; do grep OutOfMemo $i >/dev/null && echo $i oom || echo $i ok; done

— by Janos on Sept. 13, 2013, 3:43 p.m.

Explanation

In this example I was looking for a list of log files which contain or don't contain a stack trace of OutOfMemoryError events.

  • for i in *.log is to loop over the list of files.
  • For each file, I run grep, but redirect the output to /dev/null, as I don't need that, I just want to see a "yes or no" kind of summary for each file
  • grep exits with success if it found any matching lines, otherwise with failure. Using the pattern cmd && success || failure, I echo the filename and the text "oom" in case of a match, or "ok" otherwise

Remarks:

  • Using grep -q is equivalent to redirecting output to /dev/null, but might not be supported in all systems
  • grep -l can be used to list files with matches, and grep -L to list files without matches, but the latter does not exist in some implementations of grep, such as BSD
  • I realized it a bit late, but grep -c shows a count of the matches, so actually it could have been a suitable and simpler solution

0

Create a transparent image of given dimensions

 $ convert -size 100x100 xc:none transparency.png

— by Janos on July 31, 2013, 11:32 p.m.

Explanation

  • convert is a tool that's part of the ImageMagick image manipulation library
  • -size 100x100 specifies the dimensions of the image to create
  • xc:none is a symbolic source image, indicating to convert "from nothing"
  • transparency.png is the destination filename, the image format is automatically determined by the extension

Limitations

Requires the ImageMagick image manipulation library.

0

Create a heap dump of a Java process

 $ jmap -dump:format=b,file=/var/tmp/dump.hprof 1234

— by Janos on July 8, 2013, 8:18 a.m.

Explanation

  • Create a heap dump from the running Java process with PID=1234
  • The heap dump will be saved in /var/tmp/dump.hprof in binary format
  • You can open the dump with "MAT", the Memory Analyzer Tool (based on Eclipse) and identify objects that use most of the memory and potential memory leaks

For more options see jmap -h

0

Send a file by email as attachment

 $ uuencode /var/log/messages messages.txt | mailx -s "/var/log/messages on $HOST" me@example.com

— by Janos on May 26, 2013, 9:37 a.m.

Explanation

  • uuencode /var/log/messages messages.txt -- the first parameter is the file to attach, the second is the filename to use for the attachment
  • mailx -s subject emailaddress -- takes standard input as the content of the email

0

Print file owners and permissions of a directory tree

 $ find /path/to/dir1 -printf "%U %G %m %p\n" > /tmp/dir1.txt

— by Janos on March 19, 2013, 10:51 p.m.

Explanation

The command simply traverses the specified directory tree and for each file and directory it prints the UID of the owner, GID of the group, the permission bits and the path.

To compare file owners and permissions of two directory trees you can run this command for each directory, save the output in two files and then compare them using diff or similar.

See man find for more explanation of all the possible symbols you can use with -printf

Limitations

The -printf option does not exist in find on Solaris 10.

0

Get only the latest version of a file from across mutiple directories

 $ find . -name custlist\* | perl -ne '$path = $_; s?.*/??; $name = $_; $map{$name} = $path; ++$c; END { print $map{(sort(keys(%map)))[$c-1]} }'

— by Janos on Feb. 23, 2013, 4:23 p.m.

Explanation

The purpose of the one-liner is to find the the "latest" version of the custlist_*.xls file from among multiple versions in directories and sub-directories, for example:

./c/custlist_v1.003.xls
./c/custlist_v2.001.xls
./d/b/custlist_v1.001.xls
./d/custlist_v1.002.xls

Let's decompose the one-liner to the big steps:

  • find . -name custlist\* -- find the files matching the target pattern
  • ... | perl -ne '...' -- run perl, with the input wrapped around in a while loop so that each line in the input is set in the variable $_
  • $path = $_; s?.*/??; $name = $_; -- save the full path in $path, and cut off the subdirectory part to get to the base name of the file and save it in $name
  • $map{$name} = $path; -- build a mapping of $name to $path
  • ++$c; -- we count the elements, to use it later
  • (sort(keys(%map)))[$c-1] -- sort the keys of the map, and get the last element, which is custlist_v2.001.xls in this example
  • END { print $map{$last} }' -- at the end of all input data, print the path of the latest version of the file

Limitations

Even if the latest version of the file appears multiple times in the directories, the one-liner will print only one of the paths. This could be fixed though if needed.

2

Create a thumbnail from the first page of a PDF file

 $ convert -thumbnail x80 file.pdf[0] thumb.png

— by Janos on Feb. 6, 2013, 9:44 p.m.

Explanation

  • convert is part of ImageMagick image manipulation tool
  • -thumbnail x80 means create a thumbnail image of height 80 pixels, the width will be automatically chosen to make the image proportional
  • The [0] is to create a thumbnail for the first page only, without that a thumbnail image would be created for each page in the pdf file

To do this for all PDF files in a directory tree:

find /path/to/dir -name '*.pdf' -exec convert -thumbnail x80 {}[0] {}-thumb.png \;

Limitations

Requires the ImageMagick image manipulation tool.

0

Recreate or update an existing zip file and remove files that do not exist anymore

 $ zip --filesync -r /path/to/out.zip /path/to/dir

— by Janos on Jan. 26, 2013, 8:48 p.m.

Explanation

zip does not have an explicit option to overwrite/recreate an existing zip file. If the specified destination file already exists, zip updates it. The problem is that files you did not specify to add to the zip but they already existed in the zip, will not be removed.

For example let's say you created a zip file from a directory with the command:

zip -r /path/to/out.zip /path/to/dir

Next you delete some files from the directory and repeat the command to recreate the zip. But that will not recreate the zip, it will only update it, and so the file you deleted from the directory will still be there in the zip.

One way to recreate the zip is to delete the file first. Another, better way is to use the --filesync or -FS flag. With this flag zip will remove files from the zip that do not exist anymore in the filesystem. This is more efficient than recreating the zip.

1

Function to extract columns from an input stream

 $ col() { awk '{print $('$(echo $* | sed -e s/-/NF-/g -e 's/ /),$(/g')')}'; }

— by Janos on Dec. 7, 2012, 4:14 p.m.

Explanation

A slightly improved version of the original one-liner to allow negative indexes to extract columns relative to the end of the line, for example:

$ echo a b c | col 1 -0 -1
a c b

In this example the function expands to:

awk '{print $(1), $(NF-0), $(NF-1)}'

0

Make the output of the `time` builtin easier to parse

 $ TIMEFORMAT=%R

— by Janos on Dec. 4, 2012, 10:43 p.m.

Explanation

The time builtin prints a summary of the real time, user CPU time and system CPU time spent executing commands, for example:

$ time sleep 1

real    0m1.002s
user    0m0.000s
sys     0m0.002s

If you need to parse this output, it helps to simplify it using the TIMEFORMAT variable. The value %R means "the elapsed time in seconds", for example:

$ TIMEFORMAT=%R
$ time sleep 1
1.004

The complete documentation of the format definition is in man bash, search for TIMEFORMAT.

1

Remove offending key from known_hosts file with one swift move

 $ vi +18d +wq ~/.ssh/known_hosts

— by Janos on Oct. 30, 2012, 9:28 p.m.

Explanation

When you try to ssh to a server where the host key has changed then you probably get an error message like this:

Offending key in /home/jack/.ssh/known_hosts:18

If you know for sure that it was the server's admins who changed the host key, then your next move is to remove line 18 from the file so that ssh lets you connect after accepting the new key.

The one-liner does this in one swift move by passing simple commands to vi:

  • +18d -- delete line 18
  • +wq -- save the file and exit

0

Remove EXIF data such as orientation from images

 $ mogrify -strip /path/to/image.jpg

— by Janos on Oct. 24, 2012, 12:08 a.m.

Explanation

I use this mostly to remove orientation information from images. My problem with orientation information is that some viewers don't support it, and thus do not show the image correctly oriented. Rotating the image doesn't help, because if I make the image look correct in the viewer that doesn't support orientation, that will break it in the viewer that does support orientation. The solution is to remove the orientation information and rotate the image appropriately. That way the image will always look the same in all viewers, regardless of support for the orientation information.

The tool mogrify is part of ImageMagick, an image manipulation software. It manipulates image files and saves the result in the same file. A similar tool in ImageMagick that saves the result of manipulations is convert, you can use it like this:

convert -strip orig.jpg stripped.jpg

Limitations

The tool is part of ImageMagick, an image manipulation software.

0

Get the last modification date of a file in any format you want

 $ date -r /etc/motd +%Y%m%d_%H%M%S

— by Janos on Oct. 17, 2012, 4:42 p.m.

Explanation

The -r flag is a shortcut of --reference and it is used to specify a reference file. Used in this way, the date command prints the last modification date of the specified file, instead of the current date.

The + controls the output format, for example:

  • %Y = 4-digit year
  • %m = 2-digit month
  • %d = 2-digit day
  • %H = 2-digit hour
  • %M = 2-digit minutes
  • %S = 2-digit seconds

So in this example +%Y%m%d_%H%M%S becomes 20121001_171233

You should be able to find all the possible format specifiers in man date.

Limitations

The default date command in Solaris does not support the --reference flag. Modern Solaris systems have the GNU tools installed, so you may be able to find the GNU implementation of date which supports this flag. Look for it in /usr/gnu/bin/date or /usr/local/bin/date, or do search the entire /usr with find /usr -name date.

In Solaris this may be a suitable substitute without using the date command:

ls -Ego /etc/motd | awk '{print $4 "_" $5}' | tr -d :- | sed -e 's/\..*//'

Or you can use good old perl:

perl -mPOSIX -e 'print POSIX::strftime("%Y%m%d_%H%M%S\n", localtime((stat("/etc/motd"))[9]))'

0

Forget all remembered path locations

 $ hash -r

— by Janos on Oct. 14, 2012, 9:46 a.m.

Explanation

bash remembers the full path name of each command you enter, so it doesn't have to lookup in $PATH every single time you run the same thing. It also counts the number of times you used each command in the current session, you can see the list with hash.

Anyway, this behavior can poses a small problem when you reinstall an application at a different path. For example you reinstall a program that used to be in /usr/local/bin and now it is in /opt/local/bin. The problem is that if you used that command in the current shell session, then bash will remember the original location, which of course doesn't work anymore. To fix that, you can either run hash cmd which will lookup the command again, or run hash -r to forget all remembered locations (less efficient, but maybe faster to type ;-)

For more details, see help hash

0

Rename files with numeric padding

 $ perl -e 'for (@ARGV) { $o = $_; s/\d+/sprintf("%04d", $&)/e; print qq{mv "$o" "$_"\n}}'

— by Janos on Oct. 6, 2012, 1:38 p.m.

Explanation

Basically a one-liner perl script. Specify the files to rename as command line parameters, for example:

perl -e '.....' file1.jpg file2.jpg

In this example the files will be renamed to file0001.jpg and file0002.jpg, respectively. The script does not actually rename anything. It only prints the shell commands to execute that would perform the renaming. This way you can check first that the script would do, and if you want to actually do it, then pipe the output to sh like this:

perl -e '.....' file1.jpg file2.jpg | sh

What's happening in the one-liner perl script:

  • for (@ARGV) { ... } is a loop, where each command line argument is substituted into the auto-variable $_.
  • $o = $_ :: save the original filename
  • s/// :: perform pattern matching and replacement on $_
  • print qq{...} :: print the mv command, with correctly quoted arguments

Limitations

The script does not cover all corner cases. For example it will not work with files that have double-quotes in their names. In any case, it is safe to review the output of the script first before piping it to sh.

If your system has the rename command (Linux), then a shortcut to do the exact same thing is with:

rename 's/\d+/sprintf("%04d", $&)/e' *.jpg

It handles special characters better too.

0

Copy or create files with specific permissions and ownership

 $ install -b -m 600 /dev/null NEWFILE

— by Janos on Sept. 25, 2012, 2:20 p.m.

Explanation

This example creates a new (empty) file with permissions 600. You could also specify the owner and group using the -o and -g flags respectively.

Although you could accomplish the same for example by creating the file with touch and then change permissions with chmod and chown, or use umask to control the permissions of newly created files, those methods take multiple steps, while with install it is a single step.

You can also use install to copy multiple files to a directory with specified permissions like this:

install -m 600 -o jack -g wheel file1 file2 /path/to/existing/dir

1

Redirect stdout to a file you don't have write permission on

 $ echo hello | sudo tee -a /path/to/file

— by Janos on Sept. 11, 2012, 9:24 a.m.

Explanation

  • The tee command copies standard input to standard output, making a copy in zero or more files.
  • If the -a flag is specified it appends instead of overwriting.
  • Calling tee with sudo makes it possible to write to files the current user has no permission to but root does.

1

`tail -f` a file until text is seen

 $ tail -f /path/to/file.log | sed '/^Finished: SUCCESS$/ q'

— by Janos on Aug. 22, 2012, 8:29 a.m.

Explanation

tail -f until this exact line is seen:

Finished: SUCCESS

The exit condition does not have to be an exact line, it could just well be a simple pattern:

... | sed '/Finished/ q'

0

Run command multiple times with a for loop and a sequence expression

 $ for i in {1..10}; do date; sleep 1; done

— by Janos on Aug. 19, 2012, 9:27 a.m.

Explanation

This is just a regular for loop with a sequence expression. A sequence expression has the form {x..y[..incr]}, where x and y are either integers or single characters, and incr an optional increment.

More examples:

  • {a..f} = a b c d e f
  • {a..f..2} = a c e
  • {0..1}{0..1} = 00 01 10 11

Limitations

Don't try a large range like {1..10000000000000000}, it may render your computer unusable until killed.

0

Clear the swap space forcing everything back to main memory in Linux

 $ sudo swapoff -a; sudo swapon -a

— by Janos on Aug. 14, 2012, 11:21 a.m.

Explanation

Note: if you don't have enough main memory the swapoff will fail.

Limitations

This works only in Linux.

0

Redirection operator to override the noclobber option

 $ some_command >| output.txt

— by Janos on Aug. 11, 2012, 9:21 a.m.

Explanation

Normally the > operator overwrites the target file.

If the noclobber option is set (using: set -o noclobber), the > operator will fail if the target file exists.

The >| overrides the noclobber setting and overwrites the target file.

If the noclobber option is not set, then >| is equivalent to >, naturally.

0

How to set the ip address in Solaris 11

 $ ipadm create-addr -T static -a 192.168.1.10/24 eth0/staticaddr

— by Janos on Aug. 3, 2012, 11:44 a.m.

Explanation

  • eth0 is the name of the network interface
  • ipadm show-if shows the list of network interfaces
  • staticaddr is a name you can choose

More details here: http://docs.oracle.com/cd/E19963-01/html/821-1458/gjwiq.html

1

Use vim to pretty-print code with syntax highlighting

 $ vim +'hardcopy > output.ps' +q style.css 

— by Janos on July 21, 2012, 12:13 a.m.

Explanation

If you have syntax highlighting properly setup in vim, this command will pretty-print the specified file with syntax highlighting to output.ps.

If you prefer PDF, you can convert using ps2pdf output.ps.

0

Edit the Gimp launcher file to disable the splash screen

 $ sudo sed -i 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop

— by Janos on June 30, 2012, 9:06 p.m.

Explanation

  • The -i flag of sed means to perform the command "in place", that is, save any changes in the input file. Use this flag with extreme caution, one wrong move and you can completely break the original file.
  • The regex pattern /^Exec=[^ ]*/ will match the line starting with Exec= followed by zero or more non-space characters.
  • In the replacement string, & -s, the & is replaced with whatever was matched, in this example probably something like Exec=gimp-2.8, after which we add a space and the -s flag which will effectively disable the splash screen when starting Gimp.

Limitations

The -i flag of sed works differently in GNU and BSD systems. This example works in GNU systems only. The equivalent in BSD is:

sudo sed -i '' 's/^Exec=[^ ]*/& -s/' /usr/share/applications/gimp.desktop

In any case, always be very careful when using the -i flag of sed.

0

`less` is more convenient with the `-F` flag

 $ less -F FILE1

— by Janos on June 25, 2012, 6:47 p.m.

Explanation

less is a "pager" program like more, with a lot of added features. By default, to exit less you have to press q. This can be annoying when viewing a small file that would fit on the screen.

The -F flag to the rescue! When started with the -F flag, less will quit if the entire input (whether from stdin or a file) fits on a single screen.

It has no effect whatsoever for longer input, so it is safe to add an alias for this:

alias less='less -F'

0

Append to a file text, a blank line, and the last line of another file

 $ { echo some text; echo; tail -n1 /var/log/apache2/error.log; } >> /path/to/file

— by Janos on June 22, 2012, 5:29 p.m.

Explanation

All the standard output from all the commands between the braces will be redirected.

-1

Run a local shell script on a remote server without copying it there

 $ ssh user@server bash < /path/to/local/script.sh

— by Janos on June 21, 2012, 12:06 a.m.

Explanation

Yes this is almost trivial: a simple input redirection, from a local shell script to be executed by bash on the remote server.

The important point being, if you have a complex and very long chain of commands to run on a remote server, it is better to put the commands in a shell script, break the long one-liner to multiple lines for readability and easier debugging.

Replace bash accordingly depending on the language of the script, for example for python:

ssh user@server python < /path/to/local/script.py

0

Convert a list of terms in slug format to capitalized words

 $ sed -e 's/^./\U&/' -e 's/_./\U&/g' -e 's/_/ /g' /path/to/input

— by Janos on June 17, 2012, 7:54 a.m.

Explanation

The goal here is to take an input like this:

police_station
post_office
real_estate_agency

... and convert it to an output like this:

Police Station
Post Office
Real Estate Agency
  • -e ... the sed command can take several -e parameters, which will be executed one by one when processing each line in the input
  • The s/// command is a pattern replacement, and has the general format s/pattern/replacement/flags
  • s/^./\U&/ - replace the first letter of the line with uppercase version of the letter: \U means convert to uppercase, & is the matched string
  • s/_./\U&/g- replace _ and any letter followed by it. The g flag at the end means a "global" replacement, so all occurrences of the pattern _. will be replaced
  • s/_/ /g - replace all underscores with spaces
  • Input to sed can come from a list of files, or input redirection with <, or from a pipe.

0

Execute different commands with find depending on file type

 $ find /path/to/dir -type d -exec chmod 0755 '{}' \; -o -type f -exec chmod 0644 '{}' \;

— by Janos on June 17, 2012, 12:01 a.m.

Explanation

  • -type d -exec chmod 0755 '{}' \; for each directory, run chmod 0755
  • \; is to mark the end of the -exec
  • {} is replaced with the filename, we enclosed it in single quotes like this '{}' to handle spaces in filenames
  • -ological OR operator
  • -type f -exec chmod 0644 '{}' \; for each regular file, run chmod 0644

0

Convert m4a files to mp3 using faad and lame

 $ faad -o tmp.wav music.m4a && lame -b 192 tmp.wav music.mp3

— by Janos on June 14, 2012, 9:29 a.m.

Explanation

  • Step 1: convert m4a to wav using faad
  • Step 2: convert wav to mp3 using lame
  • -b 192 is the bitrate

Limitations

Neither faad nor lame are standard commands.

0

Write both stdout and stderr to the same file

 $ do_something.sh &> out.log

— by Janos on June 5, 2012, 11:34 a.m.

Explanation

Note: this is not always the same as:

do_something.sh >out.log 2>out.log

Using &> will ensure that the order of lines coming from stdout and stderr is preserved.

Limitations

This works in /bin/bash and may not work in /bin/sh or other stripped down variants of bash.

0

Create or mount an encrypted directory using encfs

 $ encfs -i 5 $PWD/raw $PWD/content

— by Janos on May 22, 2012, 12:21 a.m.

Explanation

The first time you run this, it will create an encrypted directory raw in the current working directory, and mount it on directory content in the same directory. It will ask a couple of questions:

  1. Create directory "raw" ? -- answer "y"
  2. Create directory "content" ? -- answer "y"
  3. Choose configuration mode -- press enter, or "p" for "paranoia mode"

The content directory will be visible and editable only by your user, and all the files you put inside will be saved encrypted under raw. The encrypted data is only visible when raw is mounted somewhere.

The -i 5 means the content directory will be automatically unmounted after being idle for 5 minutes.

You can manually unmount with umount ./content in Linux, and fusermount -u ./content in Mac OS.

To mount raw again, simply re-run the same command. This time it will only ask for the password.

Limitations

encfs is not a standard command/package. Major Linux distros have it in their repos, on Mac OS you can install using ports. (sudo port install encfs)

0

Run a never-ending process in the background in a way that you can check on its status anytime

 $ screen -d -m -S some_name ping my_router

— by Janos on May 17, 2012, 11:03 p.m.

Explanation

  • -d -m start screen in "detached" mode. This creates a new session but doesn't attach to it.
  • -S some_name gives the screen session a label, so you can easily re-attach to it with screen -R some_name
  • The shell prompt returns immediately, and of course you can logout too, the screen session will continue to run happily.
  • When the specified program exits (in this example ping), its screen window will also exit, and if there are no other windows in the screen session, then the screen session itself will also exit.

0

Make a hexdump or do the reverse with the xxd command

 $ xxd /path/to/binary/file

— by Janos on May 16, 2012, 10:22 a.m.

Explanation

This shows a very nice hexdump of the specified file. You can edit the output and convert it back to binary with xxd -r. But the best part is that you can configure vim to do all the converting back and forth for you, effectively turning vim into a binary editor, by adding this to your .vimrc:

augroup Binary
    au!
    au BufReadPre  *.bin let &bin=1
    au BufReadPost *.bin if &bin | %!xxd
    au BufReadPost *.bin set ft=xxd | endif
    au BufWritePre *.bin if &bin | %!xxd -r
    au BufWritePre *.bin endif
    au BufWritePost *.bin if &bin | %!xxd
    au BufWritePost *.bin set nomod | endif
augroup END

This will work for editing .bin files. To use it for other file extensions too, duplicate the lines within augroup and replace *.bin with *.exe for example.

This tip is from vim's :help xxd.

0

Really lazy way to print the first instance of $foo that occurs after $bar

 $ ifconfig | grep ^en1 -A5 | grep inet | head -n 1

— by Janos on May 12, 2012, 12:30 p.m.

Explanation

This is just for the sake of an example of finding $foo that occurs after $bar. Substitute ifconfig and the arguments of grep appropriately for your use case.

  • In the output of ifconfig there are several lines with inet. We want to get to the first one that comes after a line starting with en1
  • grep ^en1 -A5 will print the line starting with en1 and the next 5 lines that follow it
  • grep inet will print only the lines matching inet
  • head -n 1 will print only the first line

The value 5 in -A5 is really just a guess that the line we're interested in will be within the next 5 lines, the appropriate number depends on your use case.

Kind of a dumb technique, but it's easy to remember.

0

Replace symlinks with the actual files they are pointing at

 $ find /path/to/dir -type l -exec sh -c 'cp --remove-destination "$(readlink "{}")" "{}"' \; 

— by Janos on April 24, 2012, 3:29 p.m.

Explanation

  • All the double quoting is necessary to handle filenames with spaces.
  • Calling sh with -exec is necessary to evaluate readlink for each symlink

Limitations

The BSD implementation of cp does not have the --remove-destination flag.

0

Expire a user's password immediately

 $ chage -d 0 USERNAME

— by Janos on April 23, 2012, 11:05 p.m.

Explanation

This will effectively force the user to change his/her password at next login.

Limitations

Not in BSD. Yes in Linux. Don't know in UNIX.

0

Sort du output in Human-readable format

 $ for i in G M K; do du -hsx * | grep "[0-9]$i\b" | sort -nr; done 2>/dev/null

— by Janos on April 14, 2012, 11:06 a.m.

Explanation

  • The reason to use a for loop is to sort results with G or M or K values separately, otherwise sort -n would just sort everything by the numbers regardless of G M K suffix.
  • grep "[0-9]$i\b" matches lines containing a digit followed by G or M or K followed by a "word boundary"

0

Convert a decimal number to octal, hexadecimal, binary, or anything

 $ echo 'obase=2;1234' | bc

— by Janos on April 11, 2012, 11:20 p.m.

Explanation

  • bc is an arbitrary precision calculator language.
  • obase defines the conversion base for output numbers, in this example 2 (binary)
  • ; is a statement separator in bc
  • 1234 is the decimal number to convert
  • By piping the command to bc we get 1234 in binary format

0

Convert from avi format to mp4 encoding

 $ ffmpeg -i file.avi file.mp4

— by Janos on April 11, 2012, 11:10 p.m.

Explanation

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Limitations

It is not a standard package in most systems and distros.

0

Format input into multiple columns, like a table, useful or pretty-printing

 $ mount | column -t

— by Janos on April 8, 2012, 4:08 p.m.

Explanation

column is a utility for formatting text. With the -t flag it detects the number of columns in the input so it can format the text into a table-like format.

For more details see man column.

0

Function to extract columns from an input stream

 $ col() { awk '{print $'$(echo $* | sed -e 's/ /,$/g')'}'; }

— by Janos on April 5, 2012, 11:36 p.m.

Explanation

Something I do a lot is extract columns from some input where cut is not suitable because the columns are separated by not a single character but multiple spaces or tabs. So I often do things like:

... | awk '{print $7, $8}'

... which is a lot of typing, additionally slowed down when typing symbols like '{}$ ... Using the simple one-line function above makes it easier and faster:

... | col 7 8

How it works:

  • The one-liner defines a new function with name col
  • The function will execute awk, and it expects standard input (coming from a pipe or input redirection)
  • The function arguments are processed with sed to use them with awk: replace all spaces with ,$ so that for example 1 2 3 becomes 1,$2,$3, which is inserted into the awk command to become the well formatted shell command: awk '{print $1,$2,$3}'

0

Resize an image proportionally to some specified width or height

 $ mogrify -geometry x31 path/to/image.gif

— by Janos on April 3, 2012, 9:48 p.m.

Explanation

  • mogrify is part of ImageMagick, an image manipulation software suite
  • mogrify manipulates the specified images. If you prefer to keep the original image untouched and write the manipulated image to a different file, simply replace mogrify with convert, the syntax is the same, but the last command line argument will be the target image to write to.
  • The -geometry flag is to resize the image, it requires a dimension parameter in the format WIDTHxHEIGHT
  • The dimension in this example has no width, which means the image will be resized to height=31 pixels, and the width will be proportional.

Limitations

ImageMagick is not a standard package, though it is open source and available in many systems.

0

Do something in another directory without going there

 $ (cd /path/to/somewhere; tar c .) > somewhere.tar

— by Janos on April 2, 2012, 10:24 p.m.

Explanation

As explained superbly in man bash:

   (list) list is executed in a subshell environment (see  COMMAND  EXECU-
          TION  ENVIRONMENT below).  Variable assignments and builtin com-
          mands that affect the  shell's  environment  do  not  remain  in
          effect  after  the  command completes.  The return status is the
          exit status of list.

In other words, this is a handy way to do something somewhere else without having to go there and coming back.

0

Find the target path a symlink is pointing to

 $ readlink a_symbolic_link_to_somewhere

— by Janos on March 31, 2012, 3:23 p.m.

Explanation

Sure, you could figure out the link target from the output of ls -l a_symbolic_link_to_somewhere too, but the output of readlink is simply the target of the symbolic link itself, so it is cleaner and easier to read.

1

Remove carriage return '\r' character in many files, without looping and intermediary files

 $ vi +'bufdo set ff=unix' +'bufdo %s/^M$//' +q file1 file2 file3

— by Janos on March 30, 2012, 3:50 p.m.

Explanation

  • The arguments starting with + are commands in vi that will be executed
  • set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
  • %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
  • bufdo means to run command in all buffers (each file is opened in a separate buffer)
  • q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1

0

Get the octal, hexadecimal and decimal codes of the ASCII character set

 $ man ascii

— by Janos on March 29, 2012, 9:48 a.m.

Explanation

Knowing the octal, hexadecimal or decimal code of the ASCII character set can be handy at times. In the past, too often I did things like:

perl -e 'for my $n (1 .. 255) { print $n, chr($n), $n, "\n"; }'

... when a simple man ascii would have done the trick...

On a related note, these all print the letter "A":

echo -e '\0101'
printf '\101'
printf '\x41'
perl -e 'print "\x41"'

1

Sort and remove duplicate lines in a file in one step without intermediary files

 $ vi +'%!sort | uniq' +wq file.txt

— by Janos on March 22, 2012, 1:09 p.m.

Explanation

We open a file with vi and run two vi commands (specified with +):

  1. %!sort | uniq
    • % = range definition, it means all the lines in the current buffer.
    • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
  2. wq = write buffer contents to file and exit.

0

List or edit and re-execute commands from the history list

 $ fc -l

— by Janos on March 15, 2012, 12:10 p.m.

Explanation

fc is a little known but very useful bash built-in.

  • fc -l will list the most recent 16 commands
  • fc will open the last command in a text editor defined in the environmental variable FCEDIT or EDITOR or else vi, and re-execute when you exit
  • fc 5 9 will open the history entries 5 to 9 in a text editor
  • fc -s pat=sub will run the last command after substituting pat with sub in it (does not open editor)
  • fc -s pat=sub cc is the same but on the last command starting with cc
  • fc -s cc will run the last command starting with cc

For more info see help fc.

0

Find the most recently modified files in a directory and all subdirectories

 $ find /path/to/dir -type f -mtime -7 -print0 | xargs -0 ls -lt | head

— by Janos on March 8, 2012, 5:10 p.m.

Explanation

  • find /path/to/dir -type f -mtime -7 -print0 prints all the files in the directory tree that have been modified within the last 7 days, with null character as the delimiter
  • xargs -0 ls -lt expects a null delimited list of filenames and will sort the files by modification time, in descending order from most recent to oldest
  • Since we are looking for the most recent files, with head we get the first 10 lines only

Note that if there are too many files in the output of find, xargs will run multiple ls -lt commands and the output will be incorrect. This is because the maximum command line length is getconf ARG_MAX and if this is exceeded xargs has to split the execution to multiple commands. So depending on your use case you may need to tweak the -mtime parameter to make sure there are not too many lines in the output.

0

List open files

 $ lsof -n

— by Janos on March 2, 2012, 10:01 a.m.

Explanation

With the -n flag it will not try to resolve network numbers to host names for network files, making it run a bit faster.

With the -c option you can select processes executing a matching command. And with the -t flag the output will be simply process ids without a header, suitable to use with kill. For example you can kill Google Chrome process gone crazy like this:

kill -HUP $(lsof -n -c /google/i -t)

Here /google/i is a regular expression pattern with case insensitive matching.

0

Set a colorful bash prompt per dev test prod environments

 $ PS1='\[\e[1;31m\][\u@\h \W]\$\[\e[0m\] '

— by Janos on Feb. 25, 2012, 2:46 p.m.

Explanation

It is useful to set a different color for the shell prompt in different deployment environments like dev/test/production, so that you don't mix up your multiple windows and do something by accident in the wrong window.

  • PS1 contains the format of the primary prompt
  • \[\e[1;31m\] sets the foreground color to red
  • \u will be substituted with the current username
  • \h will be substituted with the hostname
  • \W will be substituted with the current directory name
  • \[\e[0m\] is the end marker of the color setting

To make the color stand out even more for root users, the inverse color can be interesting too:

PS1='\[\e[7;31m\][\u@\h \W]\$\[\e[0m\] '

Other color examples:

#PS1='\[\e[1;32m\][\u@\h \W]\$\[\e[0m\] ' # green
#PS1='\[\e[1;33m\][\u@\h \W]\$\[\e[0m\] ' # yellow
#PS1='\[\e[1;34m\][\u@\h \W]\$\[\e[0m\] ' # blue

You can learn more in man bash, search for "PROMPTING".

Limitations

Your terminal program must support colors, of course ;-)

0

Print the lines of file2 that are missing in file1

 $ grep -vxFf file1 file2

— by Janos on Feb. 8, 2012, 2:42 p.m.

Explanation

  • -f is to specify a file with the list of patterns: file1
  • -F is to treat the patterns fixed strings, without using regular expressions
  • -x is to match exactly the whole line
  • -v is to select non-matching lines

The result is effectively the same as:

diff file1 file2 | grep '^>' | sed -e s/..//

Limitations

The flags of grep might work differently depending on the system. So yeah you might prefer the second way which should work everywhere. Nonetheless the various of flags of grep are interesting.

0

Find all the unique 4-letter words in a text

 $ cat ipsum.txt | perl -ne 'print map("$_\n", m/\w+/g);' | tr A-Z a-z | sort | uniq | awk 'length($1) == 4 {print}'

— by Janos on Jan. 29, 2012, 10:28 p.m.

Explanation

  • The perl regex pattern m/\w+/g will match consecutive non-word characters, resulting in a list of all the words in the source string
  • map("$_\n", @list) transforms a list, appending a new-line at the end of each element
  • tr A-Z a-z transforms uppercase letters to lowercase
  • In awk, length($1) == 4 {print} means: for lines matching the filter condition "length of the first column is 4", execute the block of code, in this case simply print

0

Concatenate PDF files using GhostScript

 $ gs -dNOPAUSE -sDEVICE=pdfwrite -sOUTPUTFILE=output.pdf -dBATCH file1.pdf file2.pdf file3.pdf

— by Janos on Jan. 26, 2012, 8:51 a.m.

Explanation

Free PDF editing software might become more and more available, but this method has been working for a long time, and likely will continue to do so.

Limitations

It may not work with all PDFs, for example files that don't conform to Adobe's published PDF specification.

0

Format text with long lines to text with fixed width

 $ fmt -s -w80 file.txt

— by Janos on Jan. 22, 2012, 10:08 a.m.

Explanation

  • It will break lines longer than 80 characters at appropriate white spaces to make them less than 80 characters long.
  • The -s flag will collapse multiple consecutive white spaces into one, or at the end of a sentence a double space.

0

Come back quickly to the current directory after doing some temporary work somewhere else

 $ pushd /some/where/else; work; cd /somewhere; work; cd /another/place; popd

— by Janos on Jan. 15, 2012, 11:12 p.m.

Explanation

  • pushd, popd and dirs are bash builtins, you can read about them with help dirs
  • bash keeps a stack of "remembered" directories, and this stack can be manipulated with the pushd and popd builtins, and displayed with the dirs builtin
  • pushd will put the current directory on top of the directory stack. So, if you need to change to a different directory temporarily and you know that eventually you will want to come back to where you are, it is better to change directory with pushd instead of cd. While working on the temporary task you can change directories with cd several times, and in the end when you want to come back to where you started from, you can simply do popd.

0

Export a git project to a directory

 $ git archive master | tar x -C /path/to/dir/to/export

— by Janos on Jan. 12, 2012, 11:04 a.m.

Explanation

The git archive command basically creates a tar file. The one-liner is to create a directory instead, without an intermediate tar file. The tar command above will untar the output of git archive into the directory specified with the -C flag. The directory must exist before you run this command.

0

Delete all tables of a mysql database

 $ mysql --defaults-file=my.cnf -e 'show tables' | while read t; do mysql --defaults-file=my.cnt  -e 'drop table '$t; done

— by Janos on Jan. 8, 2012, 7:53 a.m.

Explanation

If you have a root access to the database, a drop database + create database is easiest. This script is useful in situations where you don't have root access to the database.

First prepare a file my.cnf to store database credentials so you don't have to enter on the command line:

[client]

database=dbname

user=dbuser

password=dbpass

host=dbhost

Make sure to protect this file with chmod go-rwx.

The one-liner will execute show tables on the database to list all tables. Then the while loop reads each table name line by line and executes a drop table command.

Limitations

The above solution is lazy, because not all lines in the output of show tables are table names, so you will see errors when you run it. But hey, shell scripts are meant to be lazy!

0

Calculate the total disk space used by a list of files or directories

 $ du -s file1 dir1 | awk '{sum += $1} END {print sum}'

— by Janos on Dec. 28, 2011, 8:42 p.m.

Explanation

  • This is really simple, the first column is the size of the file or the directory, which we sum up with awk and print the sum at the end.
  • Use du -sk to count in kilobytes, du -sm to count in megabytes (not available in some systems)

1

Concatenate two or more movie files into one using mencoder

 $ mencoder cd1.avi cd2.avi -o movie.avi -ovc copy -oac copy

— by Janos on Dec. 24, 2011, 3:51 p.m.

Explanation

  • You can specify as many files as you want on the command line to process them in sequence.
  • -ovc copy simply means to copy the video exactly
  • -oac copy simply means to copy the audio exactly
  • -o movie.avi is the output file, with all the source files concatenated

Limitations

  • mencoder is usually not a standard package
  • mencoder may be in the same package as mplayer, or maybe not
  • mencoder has binary packages for Linux, Mac and Windows

See the MPlayer homepage for more info: http://www.mplayerhq.hu/

1

Calculate the average execution time (of short running scripts) with awk

 $ for i in {1..10}; do time some_script.sh; done 2>&1 | grep ^real | sed -e s/.*m// | awk '{sum += $1} END {print sum / NR}'

— by Janos on Dec. 21, 2011, 8:50 a.m.

Explanation

  • The for loop runs some_script.sh 10 times, measuring its execution time with time
  • The stderr of the for loop is redirected to stdout, this is to capture the output of time so we can grep it
  • grep ^real is to get only the lines starting with "real" in the output of time
  • sed is to delete the beginning of the line up to minutes part (in the output of time)
  • For each line, awk adds to the sum, so that in the end it can output the average, which is the total sum, divided by the number of input records (= NR)

Limitations

The snippet assumes that the running time of some_script.sh is less than 1 minute, otherwise it won't work at all. Depending on your system, the time builtin might work differently. Another alternative is to use the time command /usr/bin/time instead of the bash builtin.

0

Check the performance of a script by re-running many times while measuring the running time

 $ for i in {1..10}; do time curl http://localhost:8000 >/dev/null; done 2>&1 | grep real

— by Janos on Dec. 17, 2011, 1:49 a.m.

Explanation

  • {1..10} creates a sequence from 1 to 10, for running the main script 10 times
  • 2>&1 redirects stderr to stdout, this is necessary to capture the "output" of the time builtin

0

A convenient way to re-run the previous command with sudo

 $ sudo !!

— by Janos on Dec. 14, 2011, 11:26 p.m.

Explanation

!! (bang bang!) is replaced with the previous command.

You can read more about it and other history expansion commands in man bash in the Event Designators section.

0

Put an ssh session in the background

 $ ~^z

— by Janos on Dec. 9, 2011, 7:44 p.m.

Explanation

  • Normally, ^z (read: ctrl-z) pauses the execution of the current foreground task. That doesn't work in an ssh session, because it is intercepted by the remote shell. ~^z is a special escape character for this case, to pause the ssh session and drop you back to the local shell.
  • For all escape characters see ~?
  • The ~ escape character must always follow a newline to be interpreted as special.
  • See man ssh for more details, search for ESCAPE CHARACTERS

1

Rotate a movie file with mencoder

 $ mencoder video.avi -o rotated-right.avi -oac copy -ovc lavc -vf rotate=1

— by Janos on Dec. 2, 2011, 10:30 p.m.

Explanation

mencoder is part of mplayer.

Other possible values of the rotate parameter:

  • 0: Rotate by 90 degrees clockwise and flip (default).
  • 1: Rotate by 90 degrees clockwise.
  • 2: Rotate by 90 degrees counterclockwise.
  • 3: Rotate by 90 degrees counterclockwise and flip.

0

Recursively remove all empty sub-directories from a directory tree

 $ find . -type d | tac | xargs rmdir 2>/dev/null

— by Janos on Nov. 29, 2011, 8:01 p.m.

Explanation

  • find will output all the directories
  • tac reverses the ordering of the lines, so "leaf" directories come first
  • The reordering is important, because rmdir removes only empty directories
  • We redirect error messages (about the non-empty directories) to /dev/null

Limitations

In UNIX and BSD systems you might not have tac, you can try the less intuitive tail -r instead.

0

Remove all the versioned-but-empty directories from a Subversion checkout

 $ find . -name .svn -type d | while read ss; do dir=$(dirname "$ss"); test $(ls -a "$dir" | wc -l) == 3 && echo "svn rm \"$dir\""; done

— by Janos on Nov. 27, 2011, 8:38 a.m.

Explanation

Empty directories in version control stink. Most probably they shouldn't be there. Such directories have a single subdirectory in them named ".svn", and no other files or subdirectories.

  • The "find" searches for files files named .svn that are directories
  • The "while" assigns each line in the input to the variable ss
  • The "dirname" gets the parent directory of a path, the quotes are necessary for paths with spaces
  • ls -a should output 3 lines if the directory is in fact empty: ".", "..", and ".svn"
  • If the test is true and there are precisely 3 files in the directory, echo what we want to do
  • If the output of the one-liner looks good, pipe it to | sh to really execute

0

Create a sequence of integer numbers

 $ echo {4..-9}

— by Janos on Nov. 24, 2011, 10:07 p.m.

Explanation

  • Useful for counters. For example, to do something 10 times, you could do for i in {1..10}; do something; done
  • Be careful, there cannot be spaces between the braces
  • As the example shows, can count backwards too

Limitations

Does not work in /bin/sh, this is bash specific.

0

Redirect the output of the time builtin command

 $ { time command; } > out.out 2> time+err.out

— by Janos on Nov. 20, 2011, 8:34 p.m.

Explanation

  • time is a bash builtin command, and redirecting its output does not work the same way as with proper executables
  • If you execute within braces like above, the output of time will go to stderr (standard error), so you can capture it with 2>time.out
  • An alternative is to use the /usr/bin/time executable, by referring to its full path. (The path may be different depending on your system.)

0

Copy a directory with a large number of files to another server

 $ tar cp -C /path/to/dir . | ssh server2 'tar x -C /path/to/target'

— by Janos on Nov. 17, 2011, 12:19 p.m.

Explanation

With a large number of files, scp or rsync can take very very long. It's much faster to tar up on one side and extract on the other. Without the -f flag tar writes output to standard output and expects input from standard input, so piping to ssh can work this way, without creating any intermediary files.

You may (or may not) gain an extra speed boost by compression, either with the z flag for tar, or with the -C flag for ssh, or with gzip pipes in the middle, like this:

tar cp -C /path/to/dir . | gzip | ssh server2 'gzip -cd | tar x -C /path/to/target'

Limitations

Depending on your system and version of tar, you may need to hyphenate the flags, for example tar -cp, and tar -x. The -C flag might also not work, but that shouldn't be too difficult to work around.

0

Redirect the output of multiple commands

 $ { cmd1 ; cmd2 ; cmd3 ; } > out.out 2> err.out

— by Janos on Nov. 14, 2011, 10:08 a.m.

Explanation

  • Curly braces are very helpful for grouping several commands together
  • Be careful with the syntax: 1. there must be whitespace after the opening brace 2. there must be a semicolon after the last command and before the closing brace
  • Another practical use case: test something || { echo message; exit 1; }

0

View a file with line numbers

 $ grep -n ^ /path/to/file | less

— by Janos on Nov. 9, 2011, 11:05 p.m.

Explanation

  • grep ^ will match all lines in a file
  • grep -n will prefix each line of output with the line number within its input file

Limitations

In some systems you might have to use egrep instead of grep.

0

Print the n-th and m-th line of a file

 $ sed -ne '101 p' -e '106 p' /path/to/the/file

— by Janos on Nov. 6, 2011, 11:20 p.m.

Explanation

  • The above command will print the 101th and 106th lines of the specified file.
  • The -n switch will make sed not print all lines by default.
  • The -e switch is to specify a sed command, you can use it multiple times at once.

Some sed command examples:

  • 45 p - print line #45
  • 34,55 p - print lines #34-#55
  • 99,$ p - print lines #99-end of the file

0

Repeat the previous command but with a string replacement

 $ ^geomtry^geometry

— by Janos on Nov. 4, 2011, 7:10 a.m.

Explanation

This can be very useful for example after a mistyped command like this:

convert -crop 745x845+0+150 my_logo.png -geomtry 400x my_logo2.png
^geomtry^geometry

"-geomtry" should have been "-geometry", the one-liner above will re-run the command with a replacement that fixes the typo.

1

Rename all files in the current directory by capitalizing the first letter of every word in the filenames

 $ ls | perl -ne 'chomp; $f=$_; tr/A-Z/a-z/; s/(?<![.'"'"'])\b\w/\u$&/g; print qq{mv "$f" "$_"\n}'

— by Janos on Nov. 1, 2011, 12:51 p.m.

Explanation

  • When you pipe something to perl -ne, each input line is substituted into the $_ variable. The chomp, tr///, s/// perl functions in the above command all operate on the $_ variable by default.
  • The tr/A-Z/a-z/ will convert all letters to lowercase.
  • The regular expression pattern (?<![.'])\b\w matches any word character that follows a non-word character except a dot or a single quote.
  • The messy-looking '"'"' in the middle of the regex pattern is not a typo, but necessary for inserting a single quote into the pattern. (The first single quote closes the single quote that started the perl command, followed by a single quote enclosed within double quotes, followed by another single quote to continue the perl command.) We could have used double quotes to enclose the perl command, but then we would have to escape all the dollar signs which would make everything less readable.
  • In the replacement string $& is the letter that was matched, and by putting \u in front it will be converted to uppercase.
  • qq{} in perl works like double quotes, and can make things easier to read, like in this case above when we want to include double quotes within the string to be quoted.
  • After the conversions we print a correctly escaped mv command. Pipe this to bash to really execute the rename with | sh.

Limitations

The above command will not work for files with double quotes in the name, and possibly other corner cases.

0

Do not save command history of current bash session

 $ HISTFILE=

— by Janos on Oct. 29, 2011, 2:41 p.m.

Explanation

The command history of the current bash session is saved to $HISTFILE when you exit the shell. If this variable is blank, the command history will not be saved.

0

Use rsync instead of cp to get a progress indicator when copying large files

 $ rsync --progress largefile.gz somewhere/else/

— by Janos on Oct. 19, 2011, 12:48 a.m.

Explanation

Although rsync is famous for synchronizing files across machines, it also works locally on the same machine. And although the cp command doesn't have progress indicator, which can be annoying when copying large files, but rsync does have it, so there you go.

Limitations

When copying directories be careful that the meaning of a trailing slash when specifying directories can be slightly different from cp.

0

Create and restore backups using cpio

 $ find . -xdev -print0 | cpio -oa0V | gzip > path_to_save.cpio.gz

— by Janos on Oct. 17, 2011, 11:06 p.m.

Explanation

To restore:

gzip -cd path_to_save.cpio.gz | cpio -imV

Why not use tar instead? cpio is slightly more accurate!

  • find . -xdev -print0 finds all files and directories without crossing over to other partitions and prints a null delimited list of filenames
  • cpio -oa0V takes the list of files to archive from stdin and creates an archive file preserving timestamps and permissions
  • cpio -imV extracts the files and directories from stdin while preserving timestamps and permissions

0

Get the available space on a partition as a single numeric value

 $ df /path/to/dir | sed -ne 2p | awk '{print $4}'

— by Janos on Oct. 2, 2011, 5:41 p.m.

Explanation

  • sed -ne 2p prints the 2nd line
  • awk '{print $4}' prints the 4th column

Limitations

The output of the df command might be different depending on the system, and the available space might not be the 4th column. Make the necessary adjustments depending on your system.

0

Schedule a one-time task using "at" command and intuitive date specifications

 $ at now + 30 minutes

— by Janos on Sept. 25, 2011, 11:30 a.m.

Explanation

  • The example will run something 30 minutes from now.
  • Another example: at 0815 wed -- run something at 8:15am on the next Wednesday.
  • With atq you can see the list of scheduled jobs, this is good to confirm that you specified the date correctly and the job will run when you want.
  • With atrm you can cancel scheduled jobs, using the job id as parameter, which you can find in the atq output.

1

Remove spaces recursively from all subdirectories of a directory

 $ find /path/to/dir -type d | tac | while read LINE; do target=$(dirname "$LINE")/$(basename "$LINE" | tr -d ' '); echo mv "$LINE" "$target"; done

— by Janos on Sept. 20, 2011, 4:52 p.m.

Explanation

  • find path_to_dir -type d finds all the subdirectories
  • tac reverses the order. This is important to make "leaf" directories come first!
  • target=... stuff constructs the new name, removing spaces from the leaf component and keeping everything before that the same
  • echo mv ... for safety you should run with "echo" first, if the output looks good then remove the "echo" to really perform the rename

Limitations

In UNIX or BSD there is no tac. There you can use tail -r instead.

1

Replace a regexp pattern in many files at once

 $ vi +'bufdo %s/pattern/replacement/g | update' +q $(grep -rl pattern /path/to/dir)

— by Janos on Sept. 15, 2011, 11:50 p.m.

Explanation

  • The inner grep will search recursively in specified directory and print the filenames that contain the pattern.
  • All files will be opened in vi, one buffer per file.
  • The arguments starting with + will be executed as vi commands:
    • bufdo %s/pattern/replacement/g | update = perform the pattern substitution in all files and save the changes
    • q = exit vi

Limitations

The :bufdo command might not be there in old versions of vim.

0

The first command you should run right after you login to a remote server ;-)

 $ screen

— by Janos on Sept. 15, 2011, 10:24 p.m.

Explanation

With screen you can have multiple shells in the same window. You don't need to open new windows for a second connection to the same remote server, you can simply press C-a c inside the screen session. And if your computer crashes, you screen session on the remote server survives, you can re-attach to a previous screen session with screen -R.

0

Find video files cached by the flash plugin in browsers

 $ file /proc/*/fd/* 2>/dev/null | grep Flash | cut -f1 -d:

— by Janos on Aug. 27, 2011, 1:46 p.m.

Explanation

Recent versions of the flash plugin hide the temporary file by marking it deleted. Practically the video stream is downloaded to a "deleted file". However, even when a file is deleted, if the file is opened by a process then you can find its file descriptor and consequently the file contents.

This simple script prints out the file descriptors of opened Flash videos:

file /proc/*/fd/* 2>/dev/null | grep Flash | cut -f1 -d:

And, you probably want to create a regular file from the file descriptor, for example:

cp $(file /proc/*/fd/* 2>/dev/null | grep Flash | cut -f1 -d: | head -n 1) video.avi

Otherwise the file descriptor is not very convenient (remember, it's a deleted file!)

The method should work regardless of your browser.

0

Force the preferred language when downloading a web page with wget

 $ wget -–header='Accept-Language: en-us' http://www.timeanddate.com/calendar/index.html?year=2008&country=26 -O calendar.html

— by Janos on Aug. 21, 2011, 1:40 p.m.

Explanation

When downloading web pages with wget, some websites try to be smart and detect your preferred language based on geographical location of your IP address. This can be a problem if for example you are in Japan but you want to download a page in English.

0

Burn the contents of a directory to dvd without needing a gui application

 $ growisofs -dvd-compat -Z /dev/scd0 -R -J -pad /path/to/dir

— by Janos on Aug. 14, 2011, 9:35 p.m.

Explanation

Useful if you have a directory full of data to burn to DVD without using a GUI. The growisofs tool is in the "dvd+rw-tools" package in Ubuntu, install it with: sudo apt-get install dvd+rw-tools

0

Convert all flac files in the current directory to mp3 format using "lame"

 $ for i in *.flac; do flac -c -d "$i" | lame -m j -b 192 -s 44.1 - "${i%.flac}.mp3"; done

— by Janos on Aug. 9, 2011, 9:57 p.m.

Explanation

0

Halt the system in Linux without the halt command or gui

 $ echo o > /proc/sysrq-trigger

— by Janos on Aug. 9, 2011, 9:55 p.m.

Explanation

  • First you need to enable the sysrq interface with: echo 1 > /proc/sys/kernel/sysrq
  • echo o > /proc/sysrq-trigger halts
  • echo b > /proc/sysrq-trigger reboots

0

Change to the previous directory, a shortcut for "cd $OLDPWD"

 $ cd -

— by Janos on Aug. 9, 2011, 1:31 a.m.

Explanation

0

Create an encrypted tar file with openssl

 $ tar c paths_to_files_and_dirs | gzip -c | openssl des3 > encrypted.tar.gz

— by Janos on Aug. 9, 2011, 1:30 a.m.

Explanation

Decrypt with: openssl des3 -d < encrypted.tar.gz | tar zx

0

Make another user superuser in Ubuntu

 $ for i in $(grep :boss /etc/group | cut -f1 -d:); do adduser wife $i; done

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

In Ubuntu the first user (created during installation) has special privileges. The privileges come from the fact that the user was automatically added to various system groups. To make another user have the same privileges, all you need to do is add the user to the same groups.

  • grep :boss /etc/group gets the group records where the user boss is the first member.
  • cut -f1 -d: gets the first column, where ":" is the column separator.
  • for i in ...; do ... ; done for each group it adds the user to the group.

0

Change the label of a USB drive in Linux without a gui

 $ sudo mlabel -i /dev/sdd1 ::NewLabel

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

Replace /dev/sdd1 with whatever your USB stick is mounted on.

0

Mirror from one Subversion repository to another Subversion repository

 $ bzr co https://repo1/proj1/trunk proj1 && cd proj1 && bzr push https://repo2/vendor/proj1/trunk

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

  • The commit history in repo1 will be copied to repo2.
  • The temporary Bazaar repository (proj1) will have the full history of changes.
  • The command above initializes the mirror. To update the mirror, run this script, you can schedule it to run periodically to keep the mirror up to date: cd proj1 && bzr up && bzr push

Limitations

  • The author information will get lost in the process.

1

Rename all files in a directory to lowercase names

 $ paste <(ls) <(ls | tr A-Z a-z) | while read OLD NEW; do echo mv -v $OLD $NEW; done

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

  • <(cmd) is the filename of a named pipe (FIFO), where the named pipe is filled by the output of cmd
  • paste puts together the named pipes to form two columns: first column with the original filenames, second column with the lowercased filenames
  • ... | tr abc ABC transforms stdin by replacing any characters that appear in the first set of letters to the second set of letters
  • while read old new; do ...; done for each line it reads the first column into $old and the second column into $new

Limitations

  • Won't work if there are spaces in a filename.

0

Find Flash videos stored by browsers on a Mac

 $ find /private/ 2>/dev/null | grep /Flash

— by Janos on Aug. 5, 2011, 8:57 p.m.

Explanation

When you watch a flash video like youtube in a browser, the video file is saved on your harddisk at a temporary location. And, if you watch a video and then another video in the same window, the first one will be deleted.

Limitations

  • Might not work with all browsers.
  • Does not work with all websites (for example IMDB).
  • Does not work with an anonymous window in Chrome.