openiduser3

Explanation

• -d -m start screen in "detached" mode. This creates a new session but doesn't attach to it.
• -S some_name gives the screen session a label, so you can easily re-attach to it with screen -R some_name
• The shell prompt returns immediately, and of course you can logout too, the screen session will continue to run happily.
• When the specified program exits (in this example ping), its screen window will also exit, and if there are no other windows in the screen session, then the screen session itself will also exit.

Explanation

This shows a very nice hexdump of the specified file. You can edit the output and convert it back to binary with xxd -r. But the best part is that you can configure vim to do all the converting back and forth for you, effectively turning vim into a binary editor, by adding this to your .vimrc:

augroup Binary
    au!
    au BufReadPre  *.bin let &bin=1
    au BufReadPost *.bin if &bin | %!xxd
    au BufReadPost *.bin set ft=xxd | endif
    au BufWritePre *.bin if &bin | %!xxd -r
    au BufWritePre *.bin endif
    au BufWritePost *.bin if &bin | %!xxd
    au BufWritePost *.bin set nomod | endif
augroup END

This will work for editing .bin files. To use it for other file extensions too, duplicate the lines within augroup and replace *.bin with *.exe for example.

This tip is from vim's :help xxd.

Explanation

This is just for the sake of an example of finding $foo that occurs after $bar. Substitute ifconfig and the arguments of grep appropriately for your use case.

• In the output of ifconfig there are several lines with inet. We want to get to the first one that comes after a line starting with en1
• grep ^en1 -A5 will print the line starting with en1 and the next 5 lines that follow it
• grep inet will print only the lines matching inet
• head -n 1 will print only the first line

The value 5 in -A5 is really just a guess that the line we're interested in will be within the next 5 lines, the appropriate number depends on your use case.

Kind of a dumb technique, but it's easy to remember.

Explanation

• All the double quoting is necessary to handle filenames with spaces.
• Calling sh with -exec is necessary to evaluate readlink for each symlink

Explanation

This will effectively force the user to change his/her password at next login.

Explanation

• The reason to use a for loop is to sort results with G or M or K values separately, otherwise sort -n would just sort everything by the numbers regardless of G M K suffix.
• grep "[0-9]$i\b" matches lines containing a digit followed by G or M or K followed by a "word boundary"

Explanation

• bc is an arbitrary precision calculator language.
• obase defines the conversion base for output numbers, in this example 2 (binary)
• ; is a statement separator in bc
• 1234 is the decimal number to convert
• By piping the command to bc we get 1234 in binary format

Explanation

FFmpeg is a complete, cross-platform solution to record, convert and stream audio and video. It includes libavcodec - the leading audio/video codec library.

Explanation

column is a utility for formatting text. With the -t flag it detects the number of columns in the input so it can format the text into a table-like format.

For more details see man column.

Explanation

Something I do a lot is extract columns from some input where cut is not suitable because the columns are separated by not a single character but multiple spaces or tabs. So I often do things like:

... | awk '{print $7, $8}'

... which is a lot of typing, additionally slowed down when typing symbols like '{}$ ... Using the simple one-line function above makes it easier and faster:

... | col 7 8

How it works:

• The one-liner defines a new function with name col
• The function will execute awk, and it expects standard input (coming from a pipe or input redirection)
• The function arguments are processed with sed to use them with awk: replace all spaces with ,$ so that for example 1 2 3 becomes 1,$2,$3, which is inserted into the awk command to become the well formatted shell command: awk '{print $1,$2,$3}'

Explanation

• mogrify is part of ImageMagick, an image manipulation software suite
• mogrify manipulates the specified images. If you prefer to keep the original image untouched and write the manipulated image to a different file, simply replace mogrify with convert, the syntax is the same, but the last command line argument will be the target image to write to.
• The -geometry flag is to resize the image, it requires a dimension parameter in the format WIDTHxHEIGHT
• The dimension in this example has no width, which means the image will be resized to height=31 pixels, and the width will be proportional.

Explanation

As explained superbly in man bash:

   (list) list is executed in a subshell environment (see  COMMAND  EXECU-
          TION  ENVIRONMENT below).  Variable assignments and builtin com-
          mands that affect the  shell's  environment  do  not  remain  in
          effect  after  the  command completes.  The return status is the
          exit status of list.

In other words, this is a handy way to do something somewhere else without having to go there and coming back.

Explanation

Sure, you could figure out the link target from the output of ls -l a_symbolic_link_to_somewhere too, but the output of readlink is simply the target of the symbolic link itself, so it is cleaner and easier to read.

Explanation

• The arguments starting with + are commands in vi that will be executed
• set ff=unix is a shortcut for set fileformat=unix and means to use "unix" file format, i.e. without the carriage return \r character.
• %s/^M$// is a pattern substitution for all lines in the entire buffer, the pattern is "carriage return at end of the line", where ^M is not two literal characters but actually one, to enter it on the command line press Ctrl followed by Enter/Return
• bufdo means to run command in all buffers (each file is opened in a separate buffer)
• q is to quit vi

Note: the set ff=unix is necessary, otherwise the pattern substitution will not do anything if all the lines end with \r = the file is in dos format, because in that case the line ending character will not be considered as part of the line.

Note: if a shell-script has "accidentally" some carriage returns in it, then when you try to execute you may get an error: bad interpreter: No such file or directory. This one-liner fixes that problem. If you know that all the lines in the file have the carriage return, and there is only one file to fix, then a simplified version of the one-liner is enough:

vi +'set ff=unix' +wq file1

Explanation

Knowing the octal, hexadecimal or decimal code of the ASCII character set can be handy at times. In the past, too often I did things like:

perl -e 'for my $n (1 .. 255) { print $n, chr($n), $n, "\n"; }'

... when a simple man ascii would have done the trick...

On a related note, these all print the letter "A":

echo -e '\0101'
printf '\101'
printf '\x41'
perl -e 'print "\x41"'

Explanation

We open a file with vi and run two vi commands (specified with +):

1. %!sort | uniq
   • % = range definition, it means all the lines in the current buffer.
   • ! = run filter for the range specified. Filter is an external program, in this example sort | uniq
2. wq = write buffer contents to file and exit.

Explanation

fc is a little known but very useful bash built-in.

• fc -l will list the most recent 16 commands
• fc will open the last command in a text editor defined in the environmental variable FCEDIT or EDITOR or else vi, and re-execute when you exit
• fc 5 9 will open the history entries 5 to 9 in a text editor
• fc -s pat=sub will run the last command after substituting pat with sub in it (does not open editor)
• fc -s pat=sub cc is the same but on the last command starting with cc
• fc -s cc will run the last command starting with cc

For more info see help fc.

Explanation

• find /path/to/dir -type f -mtime -7 -print0 prints all the files in the directory tree that have been modified within the last 7 days, with null character as the delimiter
• xargs -0 ls -lt expects a null delimited list of filenames and will sort the files by modification time, in descending order from most recent to oldest
• Since we are looking for the most recent files, with head we get the first 10 lines only

Note that if there are too many files in the output of find, xargs will run multiple ls -lt commands and the output will be incorrect. This is because the maximum command line length is getconf ARG_MAX and if this is exceeded xargs has to split the execution to multiple commands. So depending on your use case you may need to tweak the -mtime parameter to make sure there are not too many lines in the output.

Explanation

With the -n flag it will not try to resolve network numbers to host names for network files, making it run a bit faster.

With the -c option you can select processes executing a matching command. And with the -t flag the output will be simply process ids without a header, suitable to use with kill. For example you can kill Google Chrome process gone crazy like this:

kill -HUP $(lsof -n -c /google/i -t)

Here /google/i is a regular expression pattern with case insensitive matching.

Explanation

It is useful to set a different color for the shell prompt in different deployment environments like dev/test/production, so that you don't mix up your multiple windows and do something by accident in the wrong window.

• PS1 contains the format of the primary prompt
• \[\e[1;31m\] sets the foreground color to red
• \u will be substituted with the current username
• \h will be substituted with the hostname
• \W will be substituted with the current directory name
• \[\e[0m\] is the end marker of the color setting

To make the color stand out even more for root users, the inverse color can be interesting too:

PS1='\[\e[7;31m\][\u@\h \W]\$\[\e[0m\] '

Other color examples:

#PS1='\[\e[1;32m\][\u@\h \W]\$\[\e[0m\] ' # green
#PS1='\[\e[1;33m\][\u@\h \W]\$\[\e[0m\] ' # yellow
#PS1='\[\e[1;34m\][\u@\h \W]\$\[\e[0m\] ' # blue

You can learn more in man bash, search for "PROMPTING".

Explanation

• -f is to specify a file with the list of patterns: file1
• -F is to treat the patterns fixed strings, without using regular expressions
• -x is to match exactly the whole line
• -v is to select non-matching lines

The result is effectively the same as:

diff file1 file2 | grep '^>' | sed -e s/..//

Explanation

• The perl regex pattern m/\w+/g will match consecutive non-word characters, resulting in a list of all the words in the source string
• map("$_\n", @list) transforms a list, appending a new-line at the end of each element
• tr A-Z a-z transforms uppercase letters to lowercase
• In awk, length($1) == 4 {print} means: for lines matching the filter condition "length of the first column is 4", execute the block of code, in this case simply print

Explanation

Free PDF editing software might become more and more available, but this method has been working for a long time, and likely will continue to do so.

Explanation

• It will break lines longer than 80 characters at appropriate white spaces to make them less than 80 characters long.
• The -s flag will collapse multiple consecutive white spaces into one, or at the end of a sentence a double space.

Explanation

• pushd, popd and dirs are bash builtins, you can read about them with help dirs
• bash keeps a stack of "remembered" directories, and this stack can be manipulated with the pushd and popd builtins, and displayed with the dirs builtin
• pushd will put the current directory on top of the directory stack. So, if you need to change to a different directory temporarily and you know that eventually you will want to come back to where you are, it is better to change directory with pushd instead of cd. While working on the temporary task you can change directories with cd several times, and in the end when you want to come back to where you started from, you can simply do popd.

Explanation

The git archive command basically creates a tar file. The one-liner is to create a directory instead, without an intermediate tar file. The tar command above will untar the output of git archive into the directory specified with the -C flag. The directory must exist before you run this command.

Explanation

If you have a root access to the database, a drop database + create database is easiest. This script is useful in situations where you don't have root access to the database.

First prepare a file my.cnf to store database credentials so you don't have to enter on the command line:

[client]

database=dbname

user=dbuser

password=dbpass

host=dbhost

Make sure to protect this file with chmod go-rwx.

The one-liner will execute show tables on the database to list all tables. Then the while loop reads each table name line by line and executes a drop table command.

Explanation

• This is really simple, the first column is the size of the file or the directory, which we sum up with awk and print the sum at the end.
• Use du -sk to count in kilobytes, du -sm to count in megabytes (not available in some systems)

Explanation

• You can specify as many files as you want on the command line to process them in sequence.
• -ovc copy simply means to copy the video exactly
• -oac copy simply means to copy the audio exactly
• -o movie.avi is the output file, with all the source files concatenated

Explanation

• The for loop runs some_script.sh 10 times, measuring its execution time with time
• The stderr of the for loop is redirected to stdout, this is to capture the output of time so we can grep it
• grep ^real is to get only the lines starting with "real" in the output of time
• sed is to delete the beginning of the line up to minutes part (in the output of time)
• For each line, awk adds to the sum, so that in the end it can output the average, which is the total sum, divided by the number of input records (= NR)

Explanation

• {1..10} creates a sequence from 1 to 10, for running the main script 10 times
• 2>&1 redirects stderr to stdout, this is necessary to capture the "output" of the time builtin

Explanation

!! (bang bang!) is replaced with the previous command.

You can read more about it and other history expansion commands in man bash in the Event Designators section.

Explanation

• Normally, ^z (read: ctrl-z) pauses the execution of the current foreground task. That doesn't work in an ssh session, because it is intercepted by the remote shell. ~^z is a special escape character for this case, to pause the ssh session and drop you back to the local shell.
• For all escape characters see ~?
• The ~ escape character must always follow a newline to be interpreted as special.
• See man ssh for more details, search for ESCAPE CHARACTERS

Explanation

mencoder is part of mplayer.

Other possible values of the rotate parameter:

• 0: Rotate by 90 degrees clockwise and flip (default).
• 1: Rotate by 90 degrees clockwise.
• 2: Rotate by 90 degrees counterclockwise.
• 3: Rotate by 90 degrees counterclockwise and flip.

Explanation

• find will output all the directories
• tac reverses the ordering of the lines, so "leaf" directories come first
• The reordering is important, because rmdir removes only empty directories
• We redirect error messages (about the non-empty directories) to /dev/null

Explanation

Empty directories in version control stink. Most probably they shouldn't be there. Such directories have a single subdirectory in them named ".svn", and no other files or subdirectories.

• The "find" searches for files files named .svn that are directories
• The "while" assigns each line in the input to the variable ss
• The "dirname" gets the parent directory of a path, the quotes are necessary for paths with spaces
• ls -a should output 3 lines if the directory is in fact empty: ".", "..", and ".svn"
• If the test is true and there are precisely 3 files in the directory, echo what we want to do
• If the output of the one-liner looks good, pipe it to | sh to really execute

Explanation

• Useful for counters. For example, to do something 10 times, you could do for i in {1..10}; do something; done
• Be careful, there cannot be spaces between the braces
• As the example shows, can count backwards too

Explanation

• time is a bash builtin command, and redirecting its output does not work the same way as with proper executables
• If you execute within braces like above, the output of time will go to stderr (standard error), so you can capture it with 2>time.out
• An alternative is to use the /usr/bin/time executable, by referring to its full path. (The path may be different depending on your system.)

Explanation

With a large number of files, scp or rsync can take very very long. It's much faster to tar up on one side and extract on the other. Without the -f flag tar writes output to standard output and expects input from standard input, so piping to ssh can work this way, without creating any intermediary files.

You may (or may not) gain an extra speed boost by compression, either with the z flag for tar, or with the -C flag for ssh, or with gzip pipes in the middle, like this:

tar cp -C /path/to/dir . | gzip | ssh server2 'gzip -cd | tar x -C /path/to/target'

Explanation

• Curly braces are very helpful for grouping several commands together
• Be careful with the syntax: 1. there must be whitespace after the opening brace 2. there must be a semicolon after the last command and before the closing brace
• Another practical use case: test something || { echo message; exit 1; }

Explanation

• grep ^ will match all lines in a file
• grep -n will prefix each line of output with the line number within its input file

Explanation

• The above command will print the 101th and 106th lines of the specified file.
• The -n switch will make sed not print all lines by default.
• The -e switch is to specify a sed command, you can use it multiple times at once.

Some sed command examples:

• 45 p - print line #45
• 34,55 p - print lines #34-#55
• 99,$ p - print lines #99-end of the file

Explanation

This can be very useful for example after a mistyped command like this:

convert -crop 745x845+0+150 my_logo.png -geomtry 400x my_logo2.png
^geomtry^geometry

"-geomtry" should have been "-geometry", the one-liner above will re-run the command with a replacement that fixes the typo.

Explanation

• When you pipe something to perl -ne, each input line is substituted into the $_ variable. The chomp, tr///, s/// perl functions in the above command all operate on the $_ variable by default.
• The tr/A-Z/a-z/ will convert all letters to lowercase.
• The regular expression pattern (?<![.'])\b\w matches any word character that follows a non-word character except a dot or a single quote.
• The messy-looking '"'"' in the middle of the regex pattern is not a typo, but necessary for inserting a single quote into the pattern. (The first single quote closes the single quote that started the perl command, followed by a single quote enclosed within double quotes, followed by another single quote to continue the perl command.) We could have used double quotes to enclose the perl command, but then we would have to escape all the dollar signs which would make everything less readable.
• In the replacement string $& is the letter that was matched, and by putting \u in front it will be converted to uppercase.
• qq{} in perl works like double quotes, and can make things easier to read, like in this case above when we want to include double quotes within the string to be quoted.
• After the conversions we print a correctly escaped mv command. Pipe this to bash to really execute the rename with | sh.

Explanation

The command history of the current bash session is saved to $HISTFILE when you exit the shell. If this variable is blank, the command history will not be saved.

Explanation

Although rsync is famous for synchronizing files across machines, it also works locally on the same machine. And although the cp command doesn't have progress indicator, which can be annoying when copying large files, but rsync does have it, so there you go.

Explanation

To restore:

gzip -cd path_to_save.cpio.gz | cpio -imV

Why not use tar instead? cpio is slightly more accurate!

• find . -xdev -print0 finds all files and directories without crossing over to other partitions and prints a null delimited list of filenames
• cpio -oa0V takes the list of files to archive from stdin and creates an archive file preserving timestamps and permissions
• cpio -imV extracts the files and directories from stdin while preserving timestamps and permissions

Explanation

• sed -ne 2p prints the 2nd line
• awk '{print $4}' prints the 4th column

Explanation

• The example will run something 30 minutes from now.
• Another example: at 0815 wed -- run something at 8:15am on the next Wednesday.
• With atq you can see the list of scheduled jobs, this is good to confirm that you specified the date correctly and the job will run when you want.
• With atrm you can cancel scheduled jobs, using the job id as parameter, which you can find in the atq output.

Explanation

• find path_to_dir -type d finds all the subdirectories
• tac reverses the order. This is important to make "leaf" directories come first!
• target=... stuff constructs the new name, removing spaces from the leaf component and keeping everything before that the same
• echo mv ... for safety you should run with "echo" first, if the output looks good then remove the "echo" to really perform the rename

Explanation

• The inner grep will search recursively in specified directory and print the filenames that contain the pattern.
• All files will be opened in vi, one buffer per file.
• The arguments starting with + will be executed as vi commands:
  • bufdo %s/pattern/replacement/g | update = perform the pattern substitution in all files and save the changes
  • q = exit vi

Explanation

With screen you can have multiple shells in the same window. You don't need to open new windows for a second connection to the same remote server, you can simply press C-a c inside the screen session. And if your computer crashes, you screen session on the remote server survives, you can re-attach to a previous screen session with screen -R.

Explanation

Recent versions of the flash plugin hide the temporary file by marking it deleted. Practically the video stream is downloaded to a "deleted file". However, even when a file is deleted, if the file is opened by a process then you can find its file descriptor and consequently the file contents.

This simple script prints out the file descriptors of opened Flash videos:

file /proc/*/fd/* 2>/dev/null | grep Flash | cut -f1 -d:

And, you probably want to create a regular file from the file descriptor, for example:

cp $(file /proc/*/fd/* 2>/dev/null | grep Flash | cut -f1 -d: | head -n 1) video.avi

Otherwise the file descriptor is not very convenient (remember, it's a deleted file!)

The method should work regardless of your browser.

Explanation

When downloading web pages with wget, some websites try to be smart and detect your preferred language based on geographical location of your IP address. This can be a problem if for example you are in Japan but you want to download a page in English.

Explanation

Useful if you have a directory full of data to burn to DVD without using a GUI. The growisofs tool is in the "dvd+rw-tools" package in Ubuntu, install it with: sudo apt-get install dvd+rw-tools

Explanation

Explanation

• First you need to enable the sysrq interface with: echo 1 > /proc/sys/kernel/sysrq
• echo o > /proc/sysrq-trigger halts
• echo b > /proc/sysrq-trigger reboots

Explanation

Explanation

Decrypt with: openssl des3 -d < encrypted.tar.gz | tar zx

Explanation

In Ubuntu the first user (created during installation) has special privileges. The privileges come from the fact that the user was automatically added to various system groups. To make another user have the same privileges, all you need to do is add the user to the same groups.

• grep :boss /etc/group gets the group records where the user boss is the first member.
• cut -f1 -d: gets the first column, where ":" is the column separator.
• for i in ...; do ... ; done for each group it adds the user to the group.

Explanation

Replace /dev/sdd1 with whatever your USB stick is mounted on.

Explanation

• The commit history in repo1 will be copied to repo2.
• The temporary Bazaar repository (proj1) will have the full history of changes.
• The command above initializes the mirror. To update the mirror, run this script, you can schedule it to run periodically to keep the mirror up to date: cd proj1 && bzr up && bzr push

Explanation

• <(cmd) is the filename of a named pipe (FIFO), where the named pipe is filled by the output of cmd
• paste puts together the named pipes to form two columns: first column with the original filenames, second column with the lowercased filenames
• ... | tr abc ABC transforms stdin by replacing any characters that appear in the first set of letters to the second set of letters
• while read old new; do ...; done for each line it reads the first column into $old and the second column into $new

Explanation

When you watch a flash video like youtube in a browser, the video file is saved on your harddisk at a temporary location. And, if you watch a video and then another video in the same window, the first one will be deleted.