Unix/Linux Shell Scripting

Test Mode Scripts.

If you want to have shell scripts that can run in a test mode - with more output or a different processing result, you can have a symbolic link to you existing script:

ln -s emailCheck.sh emailCheck_test.sh

An add to the script:

if [ `expr $0 : '.*test\.sh'` -gt 0 ]; then
  testMode=1
  testModeStr=test
else
  testMode=0
  testModeStr=
fi

Awk.

Awk (also known as gawk) predates perl and many other scripting languagues.
For more information see: www.gnu.org gawk manual
It is a stream processor - and can be considered a halfway house between sed and perl.
An example:

cat wstest2 | awk -F '\' '{printf "%s\\%s\\%s\\%s\\%s\\s\\%s\\%s echo %s >>\\temp\\wsarchivecheck\n", $1, $2, $3, $4, $5, $6, $7, $7;}'

This generates a dos shell script based on the source input.

Another - similar example, extract portion of the dos file name:

 cat t1|awk -F '\' '{ printf "%s\\%s\\%s\\%s\\%s %s\n", $1, $2, $3,$4,$5, gensub(/^([0-9]*)v.*/, "\\1", "g", $5);}'

for this input:

G:\mm4file09\105913\0000002\112693514v1.doc
G:\mm4file09\105913\0000002\112710470v1.DOC
G:\mm4file09\105913\0000002\112731114v1.DOC

it will generate this:

G:\mm4file09\105913\0000002\112693514v1.doc 112693514
G:\mm4file09\105913\0000002\112710470v1.DOC 112710470
G:\mm4file09\105913\0000002\112731114v1.DOC 112731114

gensub is a general purpose substitution function.

Generation of Awk Scripts.

Awk can be used as a pattern matcher similar to grep.
What is can also do is a multiple match and extraction.
For example, if you have a large webserver access log and want to do a mutliway split based on a path:

60.242.60.172 - - [27/Nov/2012:16:16:53 +1100] "GET /company/premium.htm HTTP/1.1" 200 15089 "-" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB7.4; .NET4.0C; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; .NET CLR 1.1.4322; BRI/2)"
66.249.74.152 - - [27/Nov/2012:16:18:14 +1100] "GET /company/zipindustries.htm HTTP/1.1" 404 307 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
66.249.74.4 - - [27/Nov/2012:16:18:15 +1100] "GET /company/zipindustries.htm HTTP/1.1" 404 309 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

This awk script will do this:

BEGIN {  }
$7 ~ /premium/ { print $0 >> "cologs/premium.log" }
$7 ~ /zipindustries/ { print $0 >> "cologs/zipindustries.log" }
$7 ~ /aaf/ { print $0 >> "cologs/aaf.log" }

END { }

If a source file lists the web sites to extact:

3ablinds  3a Blinds
premium    Premium Carpets
zipindustries Zip Industries

This script will generate the awk extraction file:

echo "BEGIN {  }" > extract.awk
while read shortname sitename; do
  #echo "Building awk script $shortname: $sitename"
  echo "\$7 ~ /${shortname}/ { print \$0 >> \"cologs/${shortname}.log\" }" \
    >> extract.awk
done < $liblisthome/local.pages
echo "END { }" >> extract.awk

To run it:

awk -f extract.awk /var/log/httpd/access_log