Skip to content

Contrarian thoughts on Big Data

I recently read an interesting presentation given by Maciej Ceglowski on the pitfalls of Big Data.  It likens our present Big Data era with the early times of the Atomic age where all anyone saw was the upside with very little downside.
I won’t paraphrase the presentation further as it’s quite succinct and easy to understand so I’ll only suggest you read it yourself.

Haunted By Data



Solaris: Can’t login after password reset, what the …

The linux faillog command doesn’t work on solaris of course.  The lad you need in that case is passwd -d <username>.  This command deletes any lock and also any password that was there.  You then need to run passwd <username> to give it a new password.

Some good free unix pdfs and docs

I recently joined a linkedIn group called shell scripting enthusiasts and on the forum one of the members linked to a page on his site where he had amalgamated a number of documents to do with *nix and scripting. I’m a firm believer in having a good book handy when you are doing something.

At the moment on my desk I have a book I can’t live without unix in a nutshell the amount of unix problems this has gotten me out makes this invaluable. Everytime I pick it up and browse through it I find something I didn’t know before.

Another book I have on my desk and constantly reference is Expert Shell Scripting by apress. What I like about this book is that it teaches a structured approach to shell scripting and uses worked examples and source code. What is even better is that the examples it uses are for sys admin scripts that you can actually use.

Anyway now I have to think about whether it is worth my money to buy a secondhand e-reader (don’t want to add to the waste stream don’t you know) to put some of these books on to peruse. I’ve heard that e-readers aren’t great for reference books as nothing beats putting a post it on your favourite page and just picking it out instantly. Anyway something to muse whilst I do some out of hours work.


I was recently asked to set up snoop so that it would create fifty 50MB files and keep doing so by deleting the oldest one, once the new one was created.  I said, sure tcpdump is your man for that job so I logged in ran the tcpdump command and what would you know but solaris doesn’t support tcpdump.  Now I could go off and upload the tcpdump command but I work in a very bureaucratic organisation and that would involve a lot of paperwork and sign off so I decided to use the already present snoop command and cross breed it with my script from my post keep an eye on that to allow me to do something approximating the relevant tcpdump command.

Below is the text of the script I have running at the moment on two of our boxes.  You can also find the code in my site.


# Author        :bonstechblog
# Filename      : rotateSnoop
# Description   : This script runs a snoop trace and outputs a new file
#                 every 130000 packets.  It also keeps check of the number
#                 of output files and deletes the oldest once over that
#                 number.
# Created       :
#       VER     DATE            ORIGINATOR      DESCRIPTION
#       1.0     23/01/2013      bonstechblog   First version
while :;
 nice snoop -d e1000g1 -c 210000 -o /opt/`hostname`Snoop_`date +%Y%m%d%H%M%S`.cap udp > /dev/null 2>&1
 fileCount=$( ls -al /opt/`hostname`Snoop*|/bin/wc -l )
 if [ $fileCount -gt 50 ]
  rm $( ls -alt /opt/`hostname`Snoop*|tail -1|awk ‘{print$NF}’ )

You will have to have root permissions to run snoop.  How it works is that it uses the script keep an eye on that to continually loop through the commands I’ve put in the while loop.  I’ve taken out the sleep command as I don’t need it.  The snoop itself is niced to stop it from potentially hogging all the resources on the box.   -d e1000g1 is the device, -c 210000 is the maximum number of packets that the snoop instance will take, I found through trial and error that this created trace files of approximately 50MB for the trace I was running.  But you should experiment to find out your file size.  -o identifies where it’s outputting the trace file to and finally udp specifies that only udp packets be snoop’d as I was only concerned with radius protocol traffic at the time but you should alter the snoop to do what you require.

When the snoop hits 210000 packets it stops snooping, the rest of the script is concerned with keeping the number of snoop files below fifty in number and deleting the last one if over.  Finally the while command loops again and kicks off a new snoop command to a new output file and it’ll do this forever until you ctrl-c or kill it.

I run this script in the background and whilst I’m logged out by doing the following

nohup ./rotateSnoop &

If you don’t understand anything here drop me a comment I’ll get back to you eventually I promise, or remember google is your friend (sorta, maybe in limited circumstances)

How is your interface set up?

I’ll be coming back to this post over the next while but suffice it to say that the commands that follow will be useful until I can clean it up

netstat -i
Name  Mtu  Net/Dest      Address        Ipkts  Ierrs Opkts  Oerrs Collis Queue
lo0   8232 loopback      localhost      1243752432 0     1243752432 0     0      0
e1000g0 1500 host-e1000g0 host-e1000g0 30610700 0     21764448 0     0      0
e1000g1 1500 host-e1000g1 host-e1000g1 3677041974 0     2055910348 0     0      0
e1000g2 1500 host-e1000g2 host-e1000g2 156726556 83    21748741 0     0      0
e1000g3 1500 host-e1000g3 host-e1000g3 661509549 0     4218142992 0     0      0

kstat -m e1000g -i 3|fgrep link
        link_asmpause                   1
        link_autoneg                    1
        link_duplex                     2
        link_pause                      1
        link_state                      1
        link_up                         1
        link_speed                      1000



keep an eye on that

You often want to keep an eye on something on a box like the size of a file, who is on the box or a process is up or down.  You could type the command over and over again, you could do the up key and return if you use bash as your shell or you could run the following

while :; do


 <insert command(s) here>

 sleep <insert amount of time between iterations in seconds here>


I hope this helps

How to check what linux build/version you have

[root@someserver ~]# lsb_release -a
LSB Version:    :core-3.1-amd64:core-3.1-ia32:core-3.1-noarch:graphics-3.1-amd64:graphics-3.1-ia32:graphics-3.1-noarch
Distributor ID: RedHatEnterpriseServer
Description:    Red Hat Enterprise Linux Server release 5.3 (Tikanga)
Release:        5.3
Codename:       Tikanga