Now, NetFlix is a pretty big company and I’m sure they have some pretty smart people working for them.  The question for people entering this contest is, just how smart are the Netflix engineers?

From a business point of view, this move by NetFlix is extremely smart.  It will make thousands of people try to solve an extremely difficult problem.  Since the problem has been worked on for years now and little progress has been made recently, it is more than likely that the next major advancement will come by moving away from the current top solution and using a completely different technique.  This contest should fuel new thought and innovation and I believe NetFlix should be commended for funding research in this way.

Now the question is… should I try?  A seemingly impossible task with a million dollar reward… that’s right up my alley.

I’ll have to look into it a little more…

In order for an operating system to be used by “the public”, it has to be relatively simple.  If they want something, they should have to click, answer a few questions and move on.  This is what Windows provides and this is what will be required of Linux before it is able to start taking significant market share.

Now, this won’t be a long ranting post, but what I do want to say is that Linux is almost ready.  I recently setup a drive array (software RAID 5) in Linux and found that with the most recent version of the Kernel, it was able to detect all of my hardware and required very little “hacking” to get working.  In my opinion hardware detection was the previous downfall of Linux, however from my recent experience, this problem is well on its way to being fixed.  More vendors are releasing Linux drivers and more legacy drivers for older hardware are being released all the time.  As the userbase grows, so will the support.

I now have a spare computer which I can use as my Linux desktop test.  I’m going to see how much of my day to day computer work I could do using a Linux desktop and what problems I run into.  I’ll be recording my experiences here.

The underlying problem here is that the big software companies (you know who) have tried to make the computer a friendly device which acts more like a co-worker than a typewriter.  People treat the computer like another person because it asks them questions and cute little paper-clips talk to them.  The function of asking questions and talking to them is simply a method of extracting information from people who can’t tell the computer what they want.  But from the user’s point of view, the computer is talking to them and therefore is doing things for them.  This is where things break down.

You see, so the user ends up thinking the computer is doing something for them when really they are using the computer to do something for themselves.  So when something goes wrong, it’s the computer’s fault, because the computer was doing it.

Now there are some cases where software or hardware is defective and in those cases, yes, the computer is to blame.  But when someone is using a program that is known to crash often, and they still continue to use it… can they really blame it when they lose their file?

1. Keep up to date
The best tip I can give you to have better results is to use the latest version of SPAMassassin.  You don’t need to have bleeding edge, but just make sure you aren’t using a version which is a year or two old.  Update your installation every few months if you can.  The newer versions of SPAMassassin are always adding new detection techniques and rules which help cut out a vast majority of SPAM.  I’ve found that newer features such as the URIBL are able to cut your false negatives down by 80%.

2. Use a third party ruleset.
The SPAMers find ways around the base rules in SPAMassassin all the time, so the more rules you can add, the better (as long as they are good rules).  SARE is a great resource for a constantly updated set of third party rules.  These rules are all tested and help to greatly increase the accuracy of your filter.  A script called RulesDuJour is also available which automatically downloads these rules for you and updates your installation with the latest ruleset at a scheduled time every day.  Having a constantly changing ruleset makes it difficult for the SPAMers to get around the filter.

3. Train your Bayes engine
SPAMassassin comes with a bayesian learning engine which allows it to use statistical methods to categorize a message a SPAM or HAM (not SPAM).  This engine is extremely accurate as long as it has the proper training.  In order to train the bayes engine you need to provide it with messages that have been reviewed by a human and are confirmed to be either SPAM or HAM.  You can manually train the engine using the sa-learn command, however it is far more efficient to use a front end to SPAMassassin like Maia Mailguard.  With Maia, it keeps a copy of all messages that pass through it so that if a message either gets filtered when it shouldn’t have or doesn’t get filtered when it should have, you can go back and mark the message as either SPAM or HAM.  This will then train the Bayes engine and it will then filter similar messages in the future.  I’ve really just touched on the Bayes engine in SPAMassassin and therefore you should look it up online and read more about it to find out what the best method would be for your environment.

4. Use the SPAM network detection methods
There are several blacklists and hash detection methods that can be used in SPAMassassin.  The two that I highly recommend are DCC and Razor.  These two products will help to increase the accuracy of your SPAM filter again by a large percentage.  Support for the products is builtin to SPAMassassin by default but they do require a client to be installed in order for them to start working.  A quick search online will provide many tutorials for getting them going.

Those are the big four tips I have.  There is a lot of other tuning you can do, but by making sure you are doing the above four things, your accuracy should easily be above 98%.  Using this configuration, I am currently seeing about 99.4% accuracy for detecting SPAM and only about 0.01% false positives.  If you have any other good tips for SPAMassassin, please leave a comment.

“Remember two months ago when you were fixing my printer? Well, now my Internet Explorer doesn’t work.”

This is a common example of a user with “related blame” syndrome.  What the user doesn’t quite understand is that they are creating a correlation based on the fact that both of these things are in the “technology” category and that there was a problem with each of them.  If we look at example which is not in the technology realm, it helps to demonstrate the depth of this issue.

“Remember how last week at lunch your soup was cold? Well now my fridge doesn’t work.”

This statement is almost exactly equivalent.  Soup is related to food, fridge is related to food, there was a problem with both of them, so the cold soup must be related to the broken fridge.

Now, I understand this is because technology is difficult to understand for some and that they don’t understand the underlying systems involved, but this does not explain why they need to create any correlation at all.  If you don’t know anything about the subject at hand, why would you attempt to make a correlation? If I get the flu the day after I had a salad containing cucumber, I don’t draw a correlation between cucumbers and the flu.  I don’t know anything about the flu or how a cucumber might be related to it and therefore making that correlation doesn’t even enter my mind.

This problem affects the mental well-being of computer professionals around the world and therefore must be stopped.  Please help fight related blame syndrome in the technology industry by allowing those afflicted with this problem to read this article and understand exactly what they are saying.

Many people in the tech industry are fans of one company or another, but I’m a fan of one product over another.  For example, I’m a big fan of Google’s recent move into the office productivity arena because it gives Microsoft, who is currently dominating that market, some healthy competition.  This forces Microsoft to make a better Office suite and maybe even lower their prices.

Unlike a lot of tech people however, I’m not all about sticking it to Microsoft.  I like the fact that Microsoft is trying to improve their search engine to compete with Google.  This forces Google to step up to the plate and deliver a top quality search engine.  I also like the fact that Microsoft is releasing a portable media player named Zune which forces Apple to make sure the iPod is always moving forward and improving.

So when people ask me why I like Linux so much, I tell them that it’s because Linux is one of the only operating systems that has a serious shot and competing with Windows…. and competition is a good thing.

5. Doing day-to-day work as an administrator
Logging into your system as an administrator/root just to check your email or surf the web is a very bad thing.  You should only be using administrative privileges when you need them.  Using commands like su in *nix or runas in Windows is the best way to get admin privileges only when you need them.  If you’re logging in as an administrator, an attacker simply has to send you a malicious email or lure you to a bad website to gain complete control over your system and maybe even your network.

4. Vulnerable email client
If your e-mail client is not fully patched with the latest vendor security patches, you’re asking for a world of trouble.  When someone is able to simply send you an email and have it auto-execute whatever code they want, this is a bad thing!  Patch your email client often and if you’re using a common email client like Microsoft Outlook, it’s even more important.

3. FTP server with simple account passwords.
In one of the companies I used to work for, we ran a small webhosting server farm.  The logs on these servers would show invalid FTP login attempts.  At least once a week (if not more), we would see an attacker probe usernames/passwords.  They would try a few hundred common username/password combinations to attempt to get in.  The result when they do get in is usually several hidden directories in which they put movies/music/pirated software.  Make sure your FTP server doesn’t have any “default” accounts and all of your common accounts have strong passwords.

2. Vulnerable web browser.
Using an unpatched web browser is like leaving the keys to your parked car on the hood.  You won’t definitely have your car stolen, but the odds are good.  The internet has become a breading ground for spyware and viruses.  A large majority of spyware infects people’s system using web browser vulnerabilities.  Make sure to always download and install recent Firefox updates, visit Windows Update to patch Internet Explorer or download all security updates for whatever browser you do use.

1. Blank local administrator or root password.
The number one easiest way to get hack is to have a blank local administrator or root password set on your system.  You may think that it is quite obvious, but it can be easily overlooked.  I’ve seen systems hacked in less than an day when the Citrix server went online with a blank local administrator password.  This kind of thing was easily missed because the system was only logged into using domain accounts and the manufacturer installation CD had set the administrator password to blank.  Always double check all accounts that have access to your system for reasonable passwords.  No system in your network should have a blank administrative password!

The problem is that these people honestly believe they are computer experts. Why? I’m not too sure.  At what point do these people or their relatives award them the “computer expert” designation?

I just wonder what makes computers so different that experts are so easy to come by?  It doesn’t seem to apply to other professions, so why does it apply to this one?

I guess in the end, if you can’t beat ‘em, join ‘em. So I here by declare myself the following:

• Plumber – I can use a plunger
• Doctor – I’ve applied one or more band-aids
• Lawyer – One time I got in an argument
• Stunt driver – I hit a curb once
• Accountant - I pay my bills
• Clown – Sometimes I dress funny

I hope that everyone from all of those professions accepts me into their field with open arms the same way I have accepted all of the “computer experts” into my field.  Hey wait, now that I think of it, I think I fit into one more category… Gynecologist!