Linux Commands
From TechWiki
To understand the directory structure of Linux, consult [1], [2].
Keybindings
Making the shell more fun...
Really usefull:
- tab -> word completion
- arrow up -> cycle through history
- ctrl c -> cancle
- ctrl z -> suspend
Good to know:
- alt b -> back one word
- alt f -> forward one word
- alt r -> undo all changes
- ctrl k -> kill to eol
- ctrl w -> kill beginning of line
- ctrl a -> start of line
- ctrl e -> end of line
- ctrl u -> delete line
- esc backspace -> delete last word
See more stuff here:
- http://www.ice2o.com/bash.php, http://gerard.st/~jeff/unix-essentials.html.
- http://onlyubuntu.blogspot.com/2007/03/bash-shell-keyboard-shortcuts-for-linux.html
Permissions
Permissions for Linux are very powerful and often annoying. There is the owner, group and others who can have permissions to read/write/execute files.
ls -la
shows all the relevant information.
chown name file
makes name owner of file. And similarly
chgrp group file
Permissions are set with chmod
chmod 666 file
makes file read/write for owner/group/other. Equivalent command
chmod a+rw file
7 makes the file read/write/executable.
a : all u : owner g : group o : others +- r : read w : write x : execute
Problems
You set the file permissions so that user X can read the file. However, if user X tries to copy the file
cp path/File newDir/
he gets
cp: cannot stat 'path/File': Permission denied
This because the directory path is not accessible for the user X.
System Commands
lspci lspci -v
shows all the devices on the motherboard.
dmesg
prints bootup messages.
lsmod
displays the modules.
Environment Variables
Each (bash) shell's environment is loaded from one of the following files in the user's home directory
.bashrc .login .profile
After changing something, each new shell will have the updated version. For the current shell, use
source .bashrc
to force the changes.
Check all environment variobles with
set
or preferably
set | less
Check individual variable's values with using
echo $MYVAR
for the variable MYVAR.
Very Useful
See more below.
sudo su cd pwd ls ssh scp wget tar diff grep tail df du ps top w kill killall find locate which whoami ln lpr env echo source bg man more less tail head uptime cat zcat ps2pdf * . .. ./ ~ & !
More system commands under Networking.
Pipes
| > < >>
Connect output from one command as input to second:
[cmd1] | [cmd2]
Redirect output to file
[cmd1] > [filename]
Note
> redirects standard output &> redirects standard output and error 2> redirects error
Append output to file
[cmd1] >> [filename]
Read input from file
[cmd1] < [filename]
Usage
Note that more examples of how to use the commands are given in Bash Script Examples.
Standard
!
![some letters]
will execute last command starting with [some letters]. e.g.
xemacs & !x
grep
grep pattern <filename or path> grep -v pattern (excludes pattern) <filename or path> grep -l pattern *.ps (as an example: "grep -l '' *.ps" searches fro all files ending with ps) grep -C4 "pattern" <filename or path> grep -C4 "pattern" * grep -r "pattern" *
don't use
grep pattern
which will just freeze.
find
find .
find . -name "*"
find . -name "*.c" -exec grep -i "find me" {} /dev/null \;
find . -name "directoryName" -type d
Example: remove all Subversion subdirectories:
find . -name '.svn' -type d -exec rm -rf {} \;
SSH
ssh -X [host] ssh user@host.domain.net
will automatically open an X-window tunnel for your Secure SHell connection. I.e., all graphical applications started on the remote host will be displayed in your local session.
Get SSH on your mobile:
Download:
SSH and tar
tar -c myDir | ssh my.host tar -x -C /newDir
ls
ls -ltr -la -sbF -s -F -T 0 -b -H -1 -C --color=auto
lsof
List of open files.
E.g., if you have a device blocking a .nfsxxx file, i.e., you cna't delete it, check which process is using it:
lsof |grep \.nfs
and kill the process...
mv
mv -i dir/fileTMove destination
the i ensures that you will be prompted before an existing file with the same name won't be overwritten.
mv -rf dir/fileTMove destination
recursive and forced (no prompting).
du
du -s * du -hs
df
df -h df -hT
ps
ps -ef ps auxww ps fax ps -eo '%C %c' --sort -c
If you need more info on a PID, go to
/proc
and cd into directory named with the PID.
scp
scp filename host:dir/newname scp -r dir host:dir2
Or, if scp is too slow, using tar and ssh:
tar -c tempDir/ | ssh moxie tar x -C /targetDir
Just make sure ssh doesn't prompt for a password see: http://www.cs.umd.edu/~arun/misc/ssh.html).
tar
tar -vczf name.tar.gz dir/ tar -vxzf name.tar.gz
See also under SSH.
Splitting a GNU tar archive across multiple files, i.e., create a tar archive files not larger than some size in kb:
tar -cv -M -L [size kb] -f backup.tar ~myDir
When the maximum size is reached, you'll be asked for the next file, you will b prompted with
Prepare volume #2 for `/dir/backup.tar' and hit return:
Type
n /dir/backup2.tar [Enter]
Note that if you're using a local directory you don't need to bother with the directory name.
To recover content from a multi-volume tar file
tar -xv -M -f backup.tar -f backup2.tar
Use -z to gzip the files.
Taken from this blog.
dar
setenv
setenv DISPLAY localhost:0
EDITOR xemacs
PATH dir/
ln
ln -s /dir nameOfLinkToDir
tail
tail -100 filename
-f filename
/
In less or vi use
/ pattern
to search.
uname
Kernel version
uname -r
ulimit
Show memory limits of a bash shell
ulimit -a
On ubuntu systems
/etc/security/limits.conf
gives you more options to set...
dd
[[3][Low level backup with dd (German)]], [[4][dd]]
time
time [command]
computes the processing time to execute command. Can be used within bash scripts to compute the processing time.
host
host whatever.domain.com
resolves IP number.
convert
Convert anything. E.g., convert a PDF to JPG, PNG, whatever...
strace
Trace system calls of process with PID
strace -p [PID]
or simply
strace [nameOfProgram]
Good for debugging, when you don't know why a program is failing.
smbmount
/usr/bin/smbmount //WINNAME /mnt/point/ -o guest,uid=www-data,gid=whoever,file_mode=0775,dir_mode=0775,rw mount
file
file myFile.bla
shows type of file myFile.bla.
wc
Wordcount
wc -l myDoc.txt
number of lines.
If you don't want to count LaTeX stuff (i.e., remove LaTeX commands), use
detex myTex.tex | wc -w
Execution
To run a process which is in the current directory, but not in the environment path, simply typing
MyExec.sh
will result in command not found. To specify that the executable is here, i.e. ".", use
./MyExec.sh
To run a process in the background, i.e., to not block the console after execution, run
COMMAND &
If you forget to use "&" press
ctrl z and type bg
To run multiple commands, use "|"
ps -ef | grep PATTERN1 | grep PATTERN2
To redirect output into a file, use ">"
ps -ef > filename
Note:
> redirects standard output &> redirects standard output and error 2> redirects error
Use
2> /dev/null
to eliminate error output.
Partitions
Show partitions:
sudo fdisk -l
sed
See also the examples below.
Replace STRING1 with STRING2 globally in my.file
sed 's/STRING1/STRING2/g' my.file
Replace whole word and not only string
sed -e 's/\bcat\b/dog/g'
so "catastrophe" won't become "dogastrophe".
sed 's/[:.:]0//' < in.file
removes '.0' from entries in in.file, e.g., 200.0 becomes 200.
To get the output into a file, use '> file.name'.
sed -e '1,5d' old > new
deletes lines 1 to 5. If you want to do it without having two files, use this script
#!/bin/bash if [ -z $1 ]; then echo "Usage: ./myexec.sh <file>.asc" exit fi time sed -e '1,3d' < $1 > temp.asc mv temp.asc $1
Replace tab delimiter with comma:
sed 's/\t/,/g' < in.asc > out.asc
For multiple commands
sed -e [c1] -e [c2] ... < in.file > out.file
Delete whole line containing a keyword in file orig.txt:
sed '/keyword/d' orig.txt > mod.txt
awk
See also the examples below.
To extract a column from some output, e.g., from ps -ef which outputs
UID PID PPID C STIME TTY TIME CMD
use:
ps -ef | awk '{print $2}'
for a list of the PIDs.
To extract lines containing PATTERN from a file:
cat filename | awk '/PATTERN/'
Equivalent to cat filename | grep PATTERN.
To extract lines containing PATTERN and output the 1st and 10th entry thereof
cat filename | awk '/PATTERN/ {var=$1 " " $10} {print "Var:" var}'
Get 2nd, 3rd and 4th column and comma-separate (-F ",")
awk -F "," '{print $2 "," $3 "," $4}'
i.e.,
cat myFile.txt | awk -F "," '{print $2 "," $3 "," $4}' > newFile.txt
Using
'
prevents awk from substituting any variables. Using
"
will allow the variables to be changed, so in order to avoid problems, every symbol must explicitly be declared with a preceding \
\$ \"
cut
Get characters from a string.
E.g.
cut -c31-100
prints the 31st to 100th characters.
And
cut -f1 -d\<
prints all the characters before the delimiter <.
xargs
Executes piped output.
E.g.
ps -ef | grep PROCESS | awk '{print $2}'
will give a list of PIDs. To use kill on them
ps -ef | grep PROCESS | awk '{print $2}' | xargs kill
If you have a text file with a list of things to execute
cat file_with_names | xargs -n 1 COMMAND
-n 1 ensures that only one entry from the file will be executed at a time. Omitting this option will execute all entries simultaneously.
Copy the .jar files from the directory /mylib/ to /home/me/jars/
find /mylib/ -name "*.jar" | xargs -n 1 cp --target=/home/me/jars/
uniq
Report or omit repeated lines.
Network
ifconfig
netstat
netstat -i
RX-OK: received OK; TX-OK: transmitted OK
System
cat /proc/cpuinfo
apt-get
Package installation manager. More under Installing Debian GNU/Linux.
Alternative: aptitude
The directory
etc/apt/sources.list
has source lists.
Usual
sudo apt-get update sudo apt-get upgrade
Distribution update
sudo apt-get update sudo apt-get upgrade sudo apt-get dist-upgrade
Remove
sudo apt-get remove pkg sudo apt-get remove --purge pkg sudo dpkg -r pkg sudo apt-get install pkg --reinstall
If all fails
cd /var/lib/dpkg/info sudo dpkg --remove --force-remove-reinstreq pkg
Problems
If the package manager locks, i.e.,
sudo apt-get update
gives
E: Could not get lock /var/lib/apt/lists/lock - open (11 Resource temporarily unavailable) E: Unable to lock the list directory
then
sudo apt-get clean cd /var/lib/apt/lists sudo mv lock lock.old
Other stuff
ps -ef | grep dpkg sudo dpkg --configure -a sudo apt-get -f install sudo apt-get autoremove
dpkg
dpkg -l | grep <packag>
to search for <package> in the list of installed packages.
Get Xserver to look at hardware resolutions
dpkg-reconfigure xserver-xorg
Tools
Screen
screen screen -ls screen -r [pid.]tty.host
C-a ?: list commands C-a c : create C-a C-a: switch C-a ": to get a full-screen list C-a A: to give it a name C-a K: kill C-a d: detach screen -r: reattach C-a [: enable copy mode - scroll in terminal with arrow keys C-a ]: back to
To scroll up and down in the terminal:
C-a Esc: switch to 'copy mode' C-u: scroll up C-d: scroll down q: quit 'copy' mode
Vi
- Esc -> command mode
- :wq -> write and quit
- :wq! -> force write and quit
- i -> insert (edit) mode
- dd -> delete line
- dw -> delete word
Or vim (vi improved).
To open two files in a split screen and diff the two
vim file.1 -d file.2
Bandwidth Monitoring
See also ubuntugeek.com, debian-administration.org, ntop.org and linux.com.
I use
vnstat iftop bwbar
N.B.
sudo /etc/init.d/bwbar start
Shells
See here for customization and here for extended examples.
Every shell script needs a special first line, e.g.,
#!/bin/bash
that sets the shell.
Debuging:
sh -x myScript
Bash
Remove Duplicate Lines in Bash History
Put
export HISTCONTROL=erasedups
in your .bashrc or .pofile file.
Examples
Checking for input variables, e.g, ./myBash.sh input:
if [ -z $1 ]; then echo "Usage: myBash.sh <file name>" exit fi
Checking for OS:
OS=$(uname)
if [ "$OS" = "SunOS" ]; then
VER=$(uname -r)
if [ "$VER" = "5.8" ]; then
echo "Solaris 5.8"
else
echo "Solaris 5.9"
fi
else
echo "Linux"
fi
If statement:
if [ "${#a}" -gt "22" ]; then
echo $a
perc=`echo $a|cut -f 2 -d '_'`
tot=`echo $tot + $perc|bc`
fi
Checks if the length of $a, namely ${#a} is greater than (-gt) 22.
Arrays:
arr=(0.75 0.75 0.777 1.211)
echo ${arr[2]}
Incrementing
i=$(($i+1))
Or
for n in $DAT; do
arr[i]=$n
done
echo ${arr[4]}
Useful
To debug, use
bash -x myBash.sh
Annoying
Wrong:
var = 0 if ["$a" = "ttt"]
Right:
var=0 if [ "$a" = "ttt" ]
C Shell
if (`uname -r` == 5.9) then
echo "Solaris 5.9"
endif
Applescript
Apple (Mac OS X) has its own script language, but you can integrate normal shell scripts. Open
Finder -> Applications -> AppleScript -> Scripteditor
and put in the following lines:
do shell script ("/Users/me/Desktop/myScript.sh")
which points to the myScript.sh executable.
Save as application or program creating myScript.app which can be but in the dock or wherever.
To make shell scripts executable type
chmod a+x myScript.sh
Bash Script Collection
A Bash script lets one execute Unix commands in a text program. There are various Unix shells next to Bash with different syntax. However, it is a very common Linux shell.
To write a simple shell, go to a directory you have write permission for and type
vi myScript.sh
The vi editor will open an empty session. To edit press i and enter
#!/bin/bash echo "hello"
To exit and save, press Esc and :wq (write, quit) followed by Enter.
To execute, the script needs to be executable, which by default it won't be:-(
ls -la myScript.sh
will show you the permissions. To make it executable for everyone, i.e., owner, group, all, type
chmod a+x myScript.sh
To execute, and this can be very annoying for beginners, you can't just enter
myScript.sh
The result will be an error
bash: myScript.sh: command not found
This is because the system is looking for myScript.sh in all the paths specified by the environment variables (type env and look for the PATH entry) and unless your script is in one of them, the system can't find it.
To specifiy the execution of a file in the current directory, you need to use ".", the dot in Unix speak meaning this directory:
./myScript.sh
And voila...
0
Outputs the executed command:
#!/bin/bash X="ls -la" eval $X
Outputs the executed command:
#!/bin/bash X="ls" $X
Loops over the values of the executed command:
#!/bin/bash
DIRS=`find .`
for n in $DIRS
do
echo $n
done
creates tarballs of the directories /etc and /usr/local:
#!/bin/bash
DIRS=( /etc /usr/local)
NAMES=( etc usr-local)
for (( i = 0 ; i < ${#DIRS[@]} ; i++ ))
do
tar -czf $DSTDIR/${NAMES[i]}.tar.gz ${DIRS[i]}
done
1
Script replacing a text file with entries like
Wed Oct 1 00:00:10 2003, 10383778, 0, 123, xxx, yyy, 250, 111.2000000000, 9999.0000000000, 110.9100000000, 0.0000000000, 0.0000000000,
to
2003-10-1 00:00:10, ...
Code:
#!/bin/bash
# remove first word (caps, no caps) from input file ($1) and save in temporary file ($2)
sed 's/[A-Z][a-z]*//' <$1 >temporary.file
# read every line from file
while read line
do
# get month in file
FILE_MTH=`echo $line | awk '{print $1}'`
# set up variables and loop
NR=1
for MONTH in Jan Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec
do
# check if looped month is equal to file month
case "$FILE_MTH" in
"$MONTH" )
# replace month in file with number
`echo $line | eval "sed -n 's/$MONTH/$NR/p' >> temporary.file2"`
;;
esac
((NR=NR+1))
done
done < "temporary.file"
rm temporary.file
# change order to yyy-mm-dd and save in output file ($2)
cat temporary.file2 | awk '{print $4-0"-"$1"-"$2,$3",",$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16,$17,$18,$19}' > $2
rm temporary.file2
2
This script, called update.sh, takes a filename as argument and executes MySQL commands.
#!/bin/bash
sed 's/[:.:]0//' <$1 >xy.z
# read every line from file
while read line
do
echo "update table set f1='`echo $line | awk '{print $2}'`' where f2='`echo $line | awk '{print $1}'`';"
|mysql -uuser -ppassword -t -hhost db
done < "xy.z"
rm xy.z
It is started as ./update.sh tmp.txt, where tmp.txt contains records like:
219 3429.0 291 7000.0 359 10.0
The MySQL statements to be executed are:
update table set f1='3429' where f2='219'; update table set f1='7000' where f2='291'; update table set f1='10' where f2='359';
3
This script computes bookkeeper statistics from an input file (csv) containing entries like
Mon Aug 1 01:01:18 2005, 61000587, , BuyMarket, USD/CHF, 6, 1.28240, 6.0014 , 13677.29 Mon Aug 1 01:02:23 2005, 61000867, , SellMarket, NZD/USD, 3, 0.68240, 2.0472 , 13677.29
#!/bin/bash
equity=10000
if [ -z $1 ]; then
echo "Usage: bookkeeper.sh <file name> <account equity>"
exit
fi
if [ -z $2 ]; then
echo "Default <account equity> = 10000"
else
equity=$2
fi
echo "Start:" `cat $1 | head -1 | cut -f1 -d,`
echo "Stop: " `cat $1 | tail -1 | cut -f1 -d,`
real=0
unreal=0
tot=0
ccys="AUD/JPY AUD/USD CHF/JPY"
for ccy in $ccys
do
a=`echo -n "$ccy: "
cat $1 | grep $ccy | awk '
BEGIN {l=0; s=0; v=0; real=0}
/BuyMarket/ {l+=$10; s-=$10*$11; price=$11; v+=$10; ex=$12/$10} /SellMarket/ {l-=$10; s+=$10*$11; price=$11; v+=$10; ex=$12/$10}
{if ( l == 0 )
real+=l+s/price
fi}
END {pl=l+s/price; perc=pl/"'"$equity"'"*100*ex;percreal=real/"'"$equity"'"*100*ex
if ( l != 0 )
av=s/-l
else
av=0
fi
print " Open Pos " l ", Real " real ", Unreal " pl "/" s+l*price ", Average " av ", Vol/2 " v/2 ";
Unreal " perc " % home, Real " percreal " % home"}
' 2> /dev/null`
if [ "${#a}" -gt "22" ]; then
echo $a
perc=`echo $a|cut -f 14 -d' '`
perc=`echo $perc | awk '{printf "%6.7f",$1}'`
unreal=`echo $unreal + $perc|bc -l`
percreal=`echo $a|cut -f 18 -d' '`
percreal=`echo $percreal | awk '{printf "%6.7f",$1}'`
real=`echo $real + $percreal|bc -l`
else
echo $a "no data"
fi
done
r=`echo $unreal | awk '{printf "%6.7f",$1}'`
u=`echo $real | awk '{printf "%6.7f",$1}'`
tot=`echo $u + $r|bc -l`
echo "Total unrealized:" $unreal "% (of $equity home currency)"
echo "Total realized:" $real "% (of $equity home currency)"
echo "TOTAL:" $tot "% (of $equity home currency)"
4
Create directories with index.html file from google xml sitemap in /var/www.
Get map (e.g., http://j-node.homeip.net/index.php?id=134)
Cut:
cat map.xml | grep 'j-node.homeip.net' | awk '{print $1}' | cut -c31-200 | cut -f1 -d\<
Use this as input to create directories with files:
cat map.xml | grep 'j-node.homeip.net' | awk '{print $1}' | cut -c31-200 | cut -f1 -d\< | xargs -n 1 mkdir
Copy file to directories:
find . | xargs -n 1 cp index.html
5
Search and replace:
rgrep -l http://www.old.ch/level1 dir/ | while read f; do sed 's&www\.old\.ch/level1&www\.mysite\.ch&g' $f > $f.new; mv $f.new $f; done
Searches recursively in the directory dir/ for the pattern 'http://www.old.ch/level1' and replaces it with 'www.mysite.ch'.
5.1
Rename files in a directory. E.g., rename ps file names containing '...00:00:00...' to '...00-00-00...':
for i in *.ps; do mv "$i" `echo "$i" | sed 's/:/-/g'`; done
6
Creates multiple files containing R code that plot a linear estimation model from data in a dat file containing x and y values.
DIR="/home/some/dir"
DAT=`ls $DIR | grep '.dat'`
rm MAIN.R
# Read and copy files ending in .dat in DIR
# Produce custom R file for each one
for n in $DAT
do
# Use first 7 chars of filename to define CCY
CCY=`echo $n | cut -c1-7`
# Use '-' as delimiter and take second occurrence of string up to the delimiter to define SL
SL=`echo $n | cut -f 2 -d '-'`
# Copy file
echo `cp $DIR/$n .`
echo $n
# Take sl.R file and replace globally the string NAME with the values of "SL-CCY"
# and the string FILE with the name of the current file and output to a file named "$SL-$CCY.R"
sed -e "s/NAME/$SL-$CCY/g" -e "s/FILE/$n/" < ./sl.R > $SL-$CCY.R
# Append the command "source('$SL-$CCY.R')" to MAIN.R (which outputs .ps plot, used below)
echo source\(\"$SL-$CCY.R\"\) >> MAIN.R
done
echo "****"
echo Start R and execute:
echo source\(\'MAIN.R\'\)
echo ...and press [ENTER] to continue
read nothing
echo Convert ps to eps
DAT=`grep -l '' *.ps`
for n in $DAT
do
ps2epsi $n
done
echo done
7
Keyboard input with else if statement checking input string
echo Type 't' or 'x', then [ENTER] to continue read sl if [ "$sl" = "t" ]; then echo "ttt" elif [ "$sl" = "x" ]; then echo "xxx" elif echo "Wrong input..." exit fi
eclipse_to_classpath.sh
#!/bin/bash
# Eclipse to Classpath Script, Markus M. Geipel
# Lists all "bin" directories of Eclipse projects.
# Usage: eclipse_to_classpath.sh <workspace-directory>
cd $1
pwd=$(pwd)
folders=$(ls -l | grep '^drw*' | awk '{print $8}')
for folder in $folders
do
TEMP=$TEMP:${pwd}/${folder}/bin
done
echo $TEMP
cd -
Used in Java Stuff. As usual make to file executable after downloading:
chmod a+x filename.sh
jar_to_classpath.sh
#!/bin/bash
# Jar to Classpath Script, Markus M. Geipel
# Lists all "jars" in a directory.
# Usage: jar_to_classpath.sh <directory>
cd $1
pwd=$(pwd)
jars=$(ls *.jar)
TEMP=
for jar in $jars
do
TEMP=$TEMP:${pwd}/${jar}
done
echo $TEMP
cd - > /dev/null
Used in Java Stuff. As usual make to file executable after downloading:
chmod a+x filename.sh
10
Creates R code which plots multiple linear regression estimations from files called .dat containing x and y values with a legend. Output either to the screen or a postscript file.
#!/bin/bash
DIR="."
DAT=`ls $DIR | grep '.dat'`
rm MAIN.*
rm *.R
rm *.ps
rm *.epsi
# Read and copy files ending in .dat in DIR
# Produce R code plotting all scaling laws in one plot
# User input
echo Type 'p' for .ps or 'x' for X11 output
read OPT
if [ "$OPT" = "p" ]; then
echo "postscript"
elif [ "$OPT" = "x" ]; then
echo "X11"
else
echo "Wrong input..."
#exit
fi
# Name
SL=`echo {$DAT[1]} | cut -f 2 -d '-'`
# Header
echo 'library(Hmisc)' >> $SL.R
if [ "$OPT" = "p" ]; then
echo 'postscript("'$SL'.ps")' >> $SL.R
fi
echo 'Min <- -10.0' >> $SL.R
echo 'Max <- -2.0' >> $SL.R
echo 'MinY <- 2.0' >> $SL.R
echo 'MaxY <- 18.0' >> $SL.R
# Some vars
COL=(gray green blue black cyan yellow magenta pink brown red)
#LGTH=${#COL[@]}
LGTH=`echo $DIR | xargs ls | grep .dat | wc -l`
i=0
# Init
CCY=`echo ${DAT[0]} | cut -c1-7`
echo 'col="'${COL[0]}'"' >> $SL.R
echo 'leg="'${CCY}'"' >> $SL.R
# Cycle thru input
for n in $DAT
do
CCY=`echo $n | cut -c1-7`
# Start with R code
echo $CCY ' <- read.table("'$n'")' >> $SL.R
echo 'l'$CCY ' <- log('$CCY')' >> $SL.R
echo $CCY'_X <- l'$CCY'$V1[l'$CCY'$V1 >= Min & l'$CCY'$V1 <= Max]' >> $SL.R
echo $CCY'_Y <- l'$CCY'$V2[l'$CCY'$V1 >= Min & l'$CCY'$V1 <= Max]' >> $SL.R
echo 'plot('$CCY'_X, '$CCY'_Y, xlab = "'$SL'-x", ylab = "'$SL'-y", main = "'$SL'-t", cex.axis=1.5, ylim=c(MinY,MaxY), col="'${COL[i]}'", pch=10, cex=0.15)' >> $SL.R
echo $CCY'res <- lm('$CCY'_Y~'$CCY'_X)' >> $SL.R
echo 'abline('$CCY'res$coef, col="'${COL[i]}'")' >> $SL.R
echo $CCY'A <- summary('$CCY'res)$coefficients[1,1]' >> $SL.R
echo $CCY'dA <- summary('$CCY'res)$coefficients[1,2]' >> $SL.R
echo $CCY'B <- summary('$CCY'res)$coefficients[2,1]' >> $SL.R
echo $CCY'dB <- summary('$CCY'res)$coefficients[2,2]' >> $SL.R
echo $CCY'C <- exp(-'$CCY'A/'$CCY'B)' >> $SL.R
echo $CCY'C.err <- sqrt((-'$CCY'C/'$CCY'B*'$CCY'dA)^2+('$CCY'A/'$CCY'B^2*'$CCY'C*'$CCY'dB)^2)' >> $SL.R
echo $CCY'Etxt <- c("'$CCY'E =", '$CCY'B, "+/-", '$CCY'dB)' >> $SL.R
echo $CCY'Ctxt <- c("'$CCY'C =", '$CCY'C, "+/-", '$CCY'C.err)' >> $SL.R
echo 'sink(file="MAIN.para", append=TRUE)' >> $SL.R
echo 'cat("'$CCY'", "\t", '$CCY'Etxt, "\t", '$CCY'Ctxt, "\t", "#data ", length('$CCY'_X), "\trange ", exp(Min), " - " , exp(Max), "\n")' >> $SL.R
echo 'sink()' >> $SL.R
if [ "$i" -lt "`echo $LGTH - 1|bc -l`" ] ; then
echo 'par(new=TRUE)' >> $SL.R
fi
if [ "$i" -gt "0" ] ; then
echo 'col=c(col,"'${COL[i]}'")' >> $SL.R
echo 'leg=c(leg,"'${CCY}'")' >> $SL.R
fi
i=`echo $i + 1|bc -l`
done
echo 'legend(Min+1,MaxY,leg, col)' >> $SL.R
# Footer
#echo 'grid(col="black")' >> $SL.R
echo 'minor.tick(nx=10, ny=10, tick.ratio=0.6)' >> $SL.R
if [ "$OPT" = "p" ]; then
echo 'dev.off()' >> $SL.R
fi
echo "****"
echo Start R and execute:
echo 'source("'$SL.R'")'
11
Create symbolic links to files called dat in the directory /whatever/.
#!/bin/bash
DIR="/whatever/"
echo $DIR
DAT=`ls $DIR | grep '.dat'`
for n in $DAT
do
`echo ln -s $DIR$n`
done
12
List all directories in current directory and remove them:
ls -l | egrep '^d'| awk '{print $8}' | xargs -n 1 rm -rf
13
Batch execution of LaTeX files in the current directory:
ls -l | egrep 'tex'| awk '{print $8}' | xargs -n 1 ps4pdf
14
Move files having the elements of CCYS in their name in DIR to a newly created directory called CCYS
#!/bin/bash
CCYS=(DC_FREQU DELTA_DC DELTA_FREQU DELTA_T TCK_T)
DIR=/home/whatever/
for sl in ${CCYS[@]} ; do
echo `mkdir $sl`
DAT=`grep -l $DIR*$sl*`
echo `mv $DAT $sl/.`
done
15
A bash script replacing values in a current file and generating an new files is given in R_and_Bash.
16
#!/bin/bash
Read file content of individual files and merge into one file. '-e' option translates '\n' into line break etc.
CTRY=`ls -la DATA/dataCBTotALLT_80_50U10/indir/*mat | cut -f4 -d/ | cut -c1-2`
rm CVind.rank
for c in $CTRY ; do
if [ "$c" != "fi" ]; then
echo $c >> CVind.rank
echo -e `cat "DATA/dataCBTotALLT_80_50U10/indir/$c-WiW-aHind.txt"` >> CVind.rank
fi
done
17
Show sizes of only the subdirectories in the current directory
ls -lap | grep / | cut -f 2 -d ':' | cut -c4-200 | xargs -n 1 du -sh
18
Replace a string in some tex-files in the current directory
#!/bin/bash
if [ -z $1 ]; then
echo "Usage: ./translate.sh <Old String, e.g. 'DC_CNT'> <New String, e.g. 'OS_DXX'>"
exit
fi
FILE=`ls *.tex`
for n in $FILE
do
echo "Replacing $1 in $n"
sed -e "s/$1/$2/g" < $n > temp.tex
mv temp.tex $n
done
However, if you need to replace symbols which are also regex (e.g., \, -, ...) consult unix.com/.
19
Use a LaTeX template file myDir/stats.template with the tags TAG1, TAG2, ... and replace them with a file with numbers myDir/myStatsFile.txt, looking like
value1 = 500 value2 = 1.2345
and generate a LaTeX file called myDir/stats.tex which is then compiled.
#!/bin/bash
if [ -z $1 ]; then
echo "Usage: ./RUN.sh [name]"
exit
fi
## Config
DIR="myDir/"
TEMPL="stats.template"
OUT="stats.tex"
TO=$DIR$OUT
### Read from DATA/stats.txt and format
DAT=`cat myDir/myStatsFile.txt | cut -f2 -d= | cut -c2-120`
### Generate latex file by replacing tags in template file
TEX=$DIR$TEMPL
# Name
echo $1
sed -e "s/NAME/$1/g" < $TEX > temp.tex
# Stats
no=0
IN=(TAG1 TAG2)
for n in $DAT
do
echo "Replacing ${IN[no]} with $n"
sed -e "s/${IN[no]}/$n/g" temp.tex > temp2.tex
mv temp2.tex temp.tex
no=`echo $no + 1|bc -l`
done
mv temp.tex $TO
### Compile
cd $DIR
latex $OUT
latex $OUT
latex $OUT
cd ../..
20
Convert dot file of a simple graph to an edge-list file. I.e., delete first and last line of file and all the entries defining the nodes:
#!/bin/bash
FILE=`ls *.dot`
for n in $FILE
do
echo "Reading $n"
links=`grep '>' $n | wc -l`
tot=`wc -l $n| awk '{print $1}' `
nodes=`echo $tot-$links|bc -l`
nodes=`echo $nodes-1|bc -l`
sed -e "$tot d" $n > tmp
sed -e "1,$nodes d" < tmp > $n
#echo $nodes
done
21
Need to summarize the results of multiple text files in a table? E.g., multiple files like this one
<k_i> = 2.045 <k_o> = 2.045 <k_tot> = 4.090 alpha k_i = 3.420 alpha k_o = 1.850 <links> = 137.0 nodes = 67.0 #CC = 1.0 % LCC = 100.000 #SCC = 1.0 % LSCC = 2.985 % IN = 8.209 % OUT = 41.045 % TT = 47.761 % DC = 0.000
called stats-v1.txt.
Step 1: Create LaTeX template table.template
\begin{sidewaystable}[p]
\centering \scriptsize
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
Name & $\langle k^{\textrm{out}}_i \rangle$ & $\langle k^{\textrm{in}}_i \rangle$ & $\langle k^{\textrm{tot}}_i \rangle$ & $\alpha_{k^{\textrm{out}}}$ & $\alpha_{k^{\textrm{in}}}$ & \# links & \# nodes & \#CC & \% LCC &\#SCC & \% LSCC & \% IN & \% OUT & \% T\&T & \% DC \\
\hline
%Marker
\hline
\end{tabular}
\caption{
Simulation results: averaged statistics over SIM network realizations, each network generated in IT time steps.}
\end{sidewaystable}
Step 2: Write bash script GenerateTable.sh that generates a row in the column of the table for each statsXXX.txt file and fills in the values
#!/bin/bash
if [ -z $1 ]; then
echo "Usage: ./GenerateTable.sh [#iterations] [#simulations]"
exit
fi
## Config
TEMPL="table.template"
OUT="table.tex"
TO=$OUT
# Generate latex file by replacing tags in template file
TEX=$TEMPL
cp $TEX temp.tex
sed -e "s/IT/$1/g" < temp.tex > temp2.tex
sed -e "s/SIM/$2/g" < temp2.tex > temp.tex
### Read from
FILE=`ls *.txt`
for n in $FILE
do
echo "Reading $n"
NAME=`ls $n | sed -e 's/\b.txt\b//g'`
NAME=`echo $NAME | sed -e 's/\bstats-\b//g'`
DAT=`cat $n | cut -f2 -d= | cut -c2-120`
#echo "$NAME"
#echo "$DAT"
### Stats
# New line
sed '/^%Marker/iNAM & KI & KO & KAI & KAO & KT & LI & NO & TCC & TLCC & TSCC & TLSCC & TIN & TOUT & TTT & TDC \\\\' temp.tex > temp2.tex
sed -e "s/NAM/$NAME/g" < temp2.tex > temp.tex
no=0
IN=(KI KO KAI KAO KT LI NO TCC TLCC TSCC TLSCC TIN TOUT TTT TDC)
for n in $DAT
do
echo "Replacing ${IN[no]} with $n"
sed -e "s/${IN[no]}/$n/g" temp.tex > temp2.tex
mv temp2.tex temp.tex
no=`echo $no + 1|bc -l`
done
done
mv temp.tex $TO
pdflatex overview.tex
pdflatex overview.tex
pdflatex overview.tex
Step 3: Embed the table in a LaTeX document using the longtables package
\documentclass[10pt,a4paper]{article}
\usepackage{lscape}
\usepackage{rotating}
\begin{document}
\include{table}
\end{document}
Step 4: Run
.\GenerateTable.sh 100 10
and get a nice table from table.tex
\begin{sidewaystable}[p]
\centering \scriptsize
\begin{tabular}{|l|c|c|c|c|c|c|c|c|c|c|c|c|c|c|c|}
\hline
Name & $\langle k^{\textrm{out}}_i \rangle$ & $\langle k^{\textrm{in}}_i \rangle$ & $\langle k^{\textrm{tot}}_i \rangle$ & $\alpha_{k^{\textrm{out}}}$ & $\alpha_{k^{\textrm{in}}}$ & \# links & \# nodes & \#CC & \% LCC &\#SCC & \% LSCC & \% IN & \% OUT & \% T\&T & \% DC \\
\hline
v1 & 2.045 & 2.045 & 4.090 & 3.420 & 1.850 & 137.0 & 67.0 & 1.0 & 100.000 & 1.0 & 2.985 & 8.209 & 41.045 & 47.761 & 0.000 \\
v2 & 2.069 & 2.069 & 4.138 & 2.940 & 2.080 & 135.5 & 65.5 & 1.0 & 100.000 & 1.0 & 4.580 & 9.924 & 48.855 & 36.641 & 0.000 \\
v3 & 1.972 & 1.972 & 3.944 & 3.500 & 1.860 & 142.0 & 72.0 & 1.0 & 100.000 & 1.0 & 2.778 & 0.694 & 71.528 & 25.000 & 0.000 \\
v4 & 1.972 & 1.972 & 3.944 & 3.145 & 2.090 & 142.0 & 72.0 & 1.0 & 100.000 & 1.0 & 2.778 & 7.639 & 50.694 & 38.889 & 0.000 \\
%Marker
\hline
\end{tabular}
\caption{
Simulation results: averaged statistics over 10 network realizations, each network generated in 100 time steps.}
\end{sidewaystable}
Sys Admin Scripts
Mount external HD, mount samba HD, and back up (tar gz) various direcories on the samba mount to the external HD:
#!/bin/bash
/bin/mount /dev/sdc1 /mnt/toshiba/
/usr/bin/smbmount //DOMAIN/name /mnt/myFS/ -o guest
BASE="/mnt/myFS/data/"
DSTDIR="/mnt/toshiba/"
# Directories to be backed up
DIRS=(pix videos music data_volatile)
for (( i = 0 ; i < ${#DIRS[@]} ; i++ ))
do
tar -czf $DSTDIR${DIRS[i]}.tar.gz $BASE${DIRS[i]}
done
/bin/umount /mnt/toshiba/
Dump MySQL DBs:
#!/bin/bash
DSTDIR="/myBackups"
DB=( mysql techwikidb )
DB_PASSWD="rootpwd"
for (( i = 0 ; i < ${#DB[@]} ; i++ ))
do
/usr/bin/mysqldump --password=$DB_PASSWD -u root ${DB[i]} > $DSTDIR/${DB[i]}.sql
done
Log your IP number:
#!/bin/bash
IP_NOW=`cat /home/ipcheck/webip.out | awk '{print $6}' | cut -f1 -d\<`
IP_LAST=`tail -1 /home/log/ip.log | cut -f1 -d\ `
if [ $IP_NOW != $IP_LAST ]; then
echo $IP_NOW `date` >> /home/log/ip.log
fi
where /home/ipcheck/webip.out contains the current IP saved as "<html><head><title>Current IP Check</title></head><body>Current IP Address: 123.456.789.123</body></html>" using dyndns.org's ipcheck (sudo apt-get install ipcheck) script.
Save data to USB HD:
!/bin/bash
mount /dev/sdb1 /mnt/usb
DSTDIR="/mnt/usb/core1"
DIRS=( /var/www /home )
NAMES=( var-www myHome)
for (( i = 0 ; i < ${#DIRS[@]} ; i++ ))
do
tar -czf $DSTDIR/${NAMES[i]}.tar.gz ${DIRS[i]}
done
[[Category:Computer Science]]
