Are the weather forecasts in Ireland really as bad as people complain?

The weather in Ireland is notoriously changeable. As a result of this the common perceptions is that the professional weather forecasters often fail to accurately predict the weather even a few days into the future. However, it is hard to find definitive data about how accurate or not the weather forecasts are.

A quick search of the Internet will reveal several sites giving confident predictions for what the weather will be in the future and/or listing what the weather was at various dates in the past. However, very few of them reveal what their past predictions were and so it is difficult to find an objective measure for how much confidence you should place in a particular forecast.

I thought this would make a very interesting project for the BT Young Scientist competition for my daughter and a few of her friends. Unfortunately the judges did not agree and they rejected their entry into the competition. However, in anticipation of their project being accepted, I created a simple batch job that fetched weather forecasts from three different Internet sites each day and saved them in files for later analysis. Since the data was being collected anyway, I thought it would be a shame not to do anything with it and so I decided to a short bit of analysis which I write up here. If you don't want to bother reading all of the blog post the short summary is that the forecasts are indeed not very accurate.

When choosing the sites to use I was more influenced by how easy the data was to collect than by whether or not the source was authoritative. For example, the Met Éireann are the official forecasting service of the Irish Government, but their forecasts are deliberately translated from numerical predictions into a forecast that humans can easily understand e.g. "rain will spread from the west and become heavy by nightfall". It is very hard to do any statistical analysis on forecasts like that, so I deliberately chose three services which provided numerical forecasts in a format that was easy to parse:

1. The Yahoo weather service is widely used. By fetching the contents of this URL each day I was able to retrieve an XML file with details of current weather conditions in Dublin, Ireland as well as their forecast for the weather the next 2 days.
2. Weather.com is the weather service provided by the well known Weather channel and it provides weather data and predictions for all parts of the globe. By fetching this URL I got an XML file with their current weather data for Dublin as well as a prediction for the next 4 days.
3. WeatherOnline is not quite so well known a weather prediction site, but they make their data very easy to retrieve. By fetching this URL, I was able to get a CSV formatted file with current weather conditions in Dublin and a forecast for the next 5 days.

I have collected about 2 months worth of data at this stage. Initially I decided to analyse the accuracy of the WeatherOnline data for the practical reason that it was easier to transform CSV files into spreadsheets than to transform XML files.

The first thing I looked at was rainfall predictions. The following chart shows the predicted rainfall (on the Y-Axis in millimetres) plotted against the actual observed rainfall (on the X-Axis). If the forecast was perfect all the dots would be on a straight line with a 45 degree slope. I don't think that anyone would expect the forecast to be perfect, but I must admit that I was personally surprised at how poor this forecast is. I calculated the correlation coefficient between  the forecast and actual data and it came out at 0.28 - the general rule of thumb would be to interpret such a low correlation figure as "there may be some small association between the figures". If I looked at the prediction from 5 days before rather than the prediction from the day before the correlation coefficient goes down to 0.07 - this is normally interpreted to mean that there is no association between the prediction and actual values.

Rainfall Prediction v Actual (mm)

The next parameter I looked at was temperature. The following chart show the actual temperature plotted against the predicted temperature from the day before and from 5 days before.

I think you would agree that the temperature predictions seem to be a little better than the rain predictions and this next chart shows the predicted temperature readings from the day before (in degrees Celsius on the Y-Axis)  plotted against the actual temperature on the X-Axis. This is not the straight 45 degree line we would hope for, but at least there is some association between the two. Indeed the correlation coefficient is 0.33 which is jut high enough to indicate that there is a medium strength correlation.

Predicted Temp v Actual Temp

However, the team at WeatherOnline should not be too proud of this result because the temperature in Ireland does not swing very much from day to day so it should be easy to predict. The average error in their temperature prediction was 2.66 degrees.  I am sure that they use a very sophisticated prediction methodology, but if instead they simply predicted that the temperature tomorrow will be the same as today, their average error would only increase to 2.76 degrees.

I have only scratched the surface of this topic. If the girls' project had been accepted they would probably have done a much more extensive analysis. Areas that would be interesting to tackle would be:

• Analysing the other factors of the prediction e.g. wind speed and direction, pressure etc.
• Looking at different weather prediction services to see if some are better than others.
• Looking at longer time scales. Because of the way I am collecting the data it is not possible to go back into the past and collect historical data, but if anyone knows of a data source showing old weather predictions I would love to analyse this.
• Looking at similar prediction accuracy in other parts of the world. For example, the weather is an extremely popular topic of conversation among Irish people, but an Egyptian colleague assures me that Egyptian people rarely discuss weather among themselves. I guess a discussion of the weather among Egyptians would quickly become boring since most days are warm and dry. Presumably weather predictions in Egypt are probably more accurate than in Ireland (but maybe nobody bothers to read them).

If anyone wants to learn more about this you can download the raw data here.

Sleeping and still consumping electricity

I know that many people are worried about the amount of power being consumed by electronic devices which are in stand-by mode and not actually doing anything useful. For example this article from the Economist magazine suggests that devices on stand-by could be consuming either 5% or 10% of total domestic electric demand (depending upon what part of the article you believe). I recently found out about the One Watt initiative which aims to ensure that in future all devices will be legally obliged to consume a maximum of 1 watt each while on stand-by.

The current cost meter that I have installed in my home provides me with an indication of the lowest power consumption rate each day. This figure is normally around 140-150 watts. Presumably this rate is reached when all of the household is asleep and not actively using any of the many devices we have in the house. My total power consumption varies from day to day, but on a normal day this background usage level would account for roughly 20% of my total consumption.

Not all of the devices consuming power  could or should be switched off at night e.g. it would not be a good idea to unplug the fridge/freezer before going to bed each night. However, I am sure that we should be able to reduce this figure significantly.

In order to identify which devices are consuming power at night, I would need to take measurements of the consumption of the various devices left on at night. The amount of power that each device consumes would probably be quite small and so the current cost meter (which only measures to the nearest watt) is probably not accurate enough. So I bought a plug in energy monitor from Maplins. which was capable of monitoring to an accuracy of 0.1 watts.

My first impressions are that this is a great device and wonderful value for money. However, the device has two main drawbacks which make it difficult for me to use:

1. The display has no back light and so it cannot be read unless I have a bright light shining on it. However, I can understand this since they don't want the device itself consuming too much power. And in any case, the use of a flash-light can overcome this problem.
2. The digits on the display are so small that they are barely legible. The unit has a number of different modes. In each mode one measurement is displayed in large digits, a secondary measurement is displayed in smaller digits an unfortunately I can only guess what value us being displayed. It also uses an even smaller symbol to indicate what mode it is in an this sysmbol is so small that I have no hope of even guessing what it us. However, I have developed a little trick to work around this problem - the voltage reading will always be close to 240 and  it would be very bad luck if any other measurement gave a similar reading. If I cycle through the modes until I see a reading close to 240, then I can use the manual to see how many more times I have to switch through modes to get to the mode I want. It is a pity that a great device should be ruined by such a simple defect.

When I complete my measurements with this new device I hope to be able to report back here on which devices I was able to switch off and how much electricity I saved.

Microphone adjustment on the Thinkpad W510

As I blogged about before, my current work laptop is a Lenovo W510 which is a really great machine once you get it working, but the setup can be a little bit tricky. Because I work on Sametime and most of my collagues are remote from me, I spend most of my days in Sametime Audio/Video meetings. When I got the laptop initially one of the factors that impressed me was the quality of the built in microphone. In fact the quality of the built in microphone was so high that I no longer had to use a dedicated headset.

Unfortunately, after a few months people began to complain that they could not  hear me clearly. Initially I was able to solve this by simply turning up the microphone gain with the "sound preferences" application, but eventually people complained that they could no longer hear me even with the gain turned up to the maximum. I was so desperate to find a solution that I even tried using Windows on my laptop, but this didn't seem to behave any differently so I was convinced that the problem was a hardware one.

I raised a ticket with our local hardware support team to see if they could repair it. They reported that when they tested it the volume coming from the microphone was OK, but there was a lot of background hiss and so they replaced the microphone control board. Initially the microphone worked OK, but then it started to tun itself off again randomly after a reboot or a suspend/resume. Now that I was convinced that it was unlikely to be a hardware problem so I did a little bit more digging on the Internet.

Eventually I found a solution. It seems that although the "sound preferences" application treats the microphone as a single device, there are actually two different devices contained within it. The alsamixer application sees these two devices as "Capture" and "Analog Mic Boost" each with their separate settings (see picture). The sensitivity of the microphone is effectively a combination of both settings, but the "sound preferences" application only adjusts the level of the "capture" device. For some reason the "Analog Mic Boost" had been turned down to 0 and when I adjusted this back up I was then able to make myself heard at team meetings.

An example of Foursquare's sense of fun

I blogged before about how FourSquare seems to have managed to inject the right amount of fun into their platform. Just today, I saw an example when I checked in this morning at a local coffee shop I got the following email from FourSquare. It is hard not to smile when you read this :-)

Congrats! Your recent check-in at Insomnia just unlocked the Fresh Brew badge, Level 2!

Look at you, Juan Valdez! That's a lot of coffee. Now that you've had your caffeine fix, get out there and conquer the day - one twitchy step at a time.


Nice! You've hit up 5 different coffee spots. 5 more new ones and you'll unlock Level 3! Foam moustache party!

It is leaving your laptop in sleep mode that consumes energy, not whether or not it is plugged in

I blogged before about how leaving your laptop charger attached to your laptop can cause it to consume a small amount of electricity if the laptop is in sleep mode as distinct from being either switched off or in hibernate mode.

When I was thinking again about this I realised that this is probably caused by the fact that the laptop's battery needed to be constantly topped up. Therefore I decided to leave the laptop unplugged in sleep mode overnight. When I plugged in charger to the laptop the following morning it started to consume about 24 watts of energy until the battery was back to full power again and then it went back down to the 1-2 watts of trickle power again.

As a result I had not saved any power overall by leaving the charger unplugged overnight.

Laptop Chargers consume much more power than phone chargers

Previously I blogged about the power consumed by my phone charger and I concluded that it was nothing to get too worried about. However, laptops typically consume much more power than mobile phones and so it is to be expected that laptop chargers will consume significantly more power than phone chargers. Therefore I decided to do a similar measurement on the power consumed by my laptop charger.

The laptop I used for the test was a reasonably old Compaq and the charger is not the original one which came with it (the original died a few years ago). I leave my laptop charger almost constantly plugged into the electrical socket in the wall, but for the purposes of the test I connected it through the Individual Appliance Monitor as shown in the picture.

1. I checked the power consumed by the laptop charger when there was no laptop attached and like the phone charger it was consuming 0 watts.
2. I waited until the laptop battery was well drained and then I plugged it in. The charger immediately started to consume about 63/64 watts of power.
3. I put the laptop into sleep mode, but the power consumption of the charger did not decrease (presumably because it was still busy charging the battery). After 2 hours the power consumption gradually decreased to 1/2 watts but not to 0. I assume that this is because the laptop is still consuming power in sleep mode. To eliminate all power consumption I would need to either unplug the laptop or else put it into hibernate mode.
4. The charger consumed roughly 0.05 KWh to charge the battery fully. At current electricity prices in Ireland this would cost slightly less than a cent.
5. When I woke the laptop from standby the consumption went back up to 24 watts. I assume this is because the charger is providing live power to run the laptop, but not needing to charge the battery.

Overall I think the lesson is that leaving the laptop charger plugged into the wall is does not waste electricity, but leaving the laptop plugged into it and in sleep mode does consume electricity. However, if you leave your laptop unplugged in sleep mode, you will still need to charge up the battery when you do plug it in later. To be really effecient on power you need to turn off your laptop completely or else put it into hibernate mode if you are not going to be using it for a long time.

Does leaving chargers plugged in really waste electricty?

I have often heard people say that you can save significant amounts of electricity by unplugging your phone and laptop chargers when they are not in active use. However, I have also heard people say that this is rubbish because modern well designed chargers will consume virtually no electricity when not actively charging a device. I was not sure which advice to trust and so I decided to run some tests with my new current cost meter in order to check it out.

In order to test out how much electricity is used charging my phone, I made sure to make heavy use of it today and I completely drained its battery (not an unusual thing for me to do). Then I plugged it in to my phone charger and observed the electricity usage using my Current Cost monitor. Despite the fact that my charger was plugged in all day, it consumed no measurable amount of electricity until I plugged in my phone. As soon as I plugged in my phone the consumption jumped up to 4 watts for 2 hours before the phone was fully charged and the usage went down to zero again.

Based upon this measurement, I calculated that if I was paying 14 cent per KiloWattHour, I would spend roughly 50 cent on electricity to charge my phone for a whole year. Leaving the phone charger plugged in while not in use would make absolutely no difference to my bill.

What do I blog about and why

Recently I was interviewed by a member of the IBM internal communications team for their "meet a colleague" series. The interview, which was published on the IBM intranet went reasonably well, but there was one question that she asked me that I found difficult. She asked me "what do you blog about and why?". The reason why I didn't answer very clearly is because I don't really know this myself.

It might seem surprising that I write a blog without a clear idea of why I am blogging or even a clear plan for what to write about it. However, to understand how this situation arose you need to know how I got started blogging. I initially started blogging on the IBM internal blog platform as part of an effort to understand the usefulness of the Lotus Connections product which at the time we were proposing to develop. My motivation was simply to get a better understanding of how/if people could derive value from using these tools because without this understanding I could not really judge if the platform we were developing was a good one or not.

Towards the end of 2008, I had a discussion about my blogging activity with one of my mentors (who was an avid blogger). She advised me that blogging inside the IBM firewall was somewhat like cycling with stabilizer wheels - a useful training exercise, but nothing like the real thing. As a result of her advice, I established an external blog. Initially I wrote very few blog posts on my external but gradually I gained confidence and now I would say that I am reasonably comfortable with my blogging activity. I still have an IBM internal blog, but I post to that blog relatively rarely because I focus my efforts on maintaining the external blog.

I still can't really say that I fully understand exactly how blogging can provide value for the blogger, but the fact that I am still actively blogging years later is evidence that I must be getting some value from all this effort. In general blogging can be used for spreading information and/or for facilitating discussion. As you will see below, I use my blog for both purposes (with mixed success).

One of the advantages of a blog is that you have a record of your activity so I decided that I would do a detailed analysis on what I was actually writing about and whether or not it was achieving my goals. In terms of spreading information, I guess that it is successful if a reasonable number of people are reading my posts. On the other have, the number of comments left on my blog would probably be a better measure of whether or not it is helping facilitate discussion. Here is the detailed analysis of my blogging activity for the last year (apologies for the length of this post, but since you have read this far you might as well read the whole post):

Why blog on IBM internal platform rather than external blog?: During the last year I wrote 103 posts to my external blog which means that I blog on average twice per week. However, I only posted 6 times on my internal blog . Three of the of the posts on the internal blog were cross-posts form the external one (to increase the chance of IBM employees seeing the post). Of the remaining three internal only posts, two were on a topic that would not be of any interest to someone outside IBM and one was expressing an opinion on an IBM policy. As I said earlier, I knew my external blog was taking over from the internals one, but I was surprised at this statistic. On reflection I should probably redress this balance slightly (even if this is only done by cross posting more frequently).

What do I blog about?: In general I blog about whatever I am thinking about. When a though enters my mind about what would be a topic worth blogging about, I generally create a draft blog entry with that title to remind me. I revisit my blog when I have time available and work on polishing the draft entries until they are ready for publishing. At the moment there are 15 draft entries in my blog, but if I need more than a few weeks to polish them I take this as an indication that I don't have a very clear opinion on the topic to express and so I delete the draft.
I analysed the content of the posts over the last year in detail and I could see that slightly less than half (45%) were expressing an opinion and hence were inviting comments. The remaining posts were split between passing in news about stuff that was happening (35%) and posts which were giving technical information on how to do something (20%),

Do people read my blog?: My first ever first blog post got 7 views, as my blog became better known the number of readers settled into the low double figures. In the last year, none of my posts got fewer than 10 reads, roughly 2/3 of the posts received between 10 and 50 readers and the remainder were split evenly between posts that received 50-100 readers and posts that received over 100 readers. In general I am happy with this level of readership, after all I am not a celebrity and in reality there are ways people can read the content without appearing in the statistics so if anything the readership would be higher than these statistics indicate.

Why do some posts receive a lot of readers?: This is a bit of a mystery for me because sometimes I write a post that I expect to be popular and it gets hardly any attention, but other times  a post will be an unexpected hit. The most popular post over the last year (getting over 1,200 hits) was a description of how I recorded OSSBarCamp and converted the content into a format suitable for posting online so apparently this is a subject that people struggle with. Most blog posts are mainly only read in the week immediately after they are posted, but occasionally a popular post will continue to be read for a long time afterwards. For example, shortly after I started my blog I posted a test entry from my mobile phone. The entry did not have much content because it is difficult to type much on my phone and in any case I intended to delete it once I verified that it was successfully posted (but obviously I forgot).  This turns out to be my second most popular post ever and interestingly almost  all of the readers come from mobile devices (the content describes how I posted from my mobile and hence it is more interesting for this audience).

Where are my readers?: Surprisingly, more of my readers come from USA (4,587) than Ireland (3,285) and my readers come from developed countries all over the world. As you can see from this map, Africa and South America are not at all represented among the readers of my blog..

Does the blog help start discussion?: Unfortunately I get very few comments on my blog. Roughly 80% of my posts get no comments at all. Half of the remainder (10%) get 1 comment an only 10% receive more than one comment. At first glance this is very disappointing and would seem to indicate that the blog has very little success in generating conversation. However, I do find that I get quite a few comments that are not left on the blog itself. Sometimes people will comment on the Facebook of LinkedIn platforms where all of my entries get cross-posted. What really surprises me is how often I will be standing in the queue for coffee in work when the person next to me will turn around and say "I was reading what you wrote on your blog about X and I was thinking ..." - this can often lead to a very interesting discussion.

Am I really a human?

Lots of web sites implement a CAPTCHA system to stop access to their site from automated programs. Normally this involves showing a distorted picture of a few letters and asking the user to read them and type them into a text box to verify that they are a real human. The general idea is that reading such a distorted picture should be an easy task for a human, but it is a surprisingly difficult task for automated image recognition programs.

When CAPTCHAs were first introduced the distortion in the picture was relatively minor and so it was not a major inconvenience for user to type in the letters. However, it should not have been a major surprise that the bad guys invested in improved software for reading distorted letters and the CAPTCHA developers were then forced to implement even greater distortion of their sample letters to trick the automated software. As a result I now find that the letters in most CAPTCHAs are extremely difficult for me to read.

Maybe it is simply the fact that my sight is failing because I am getting older, but the last few times I have failed to complete it at the first attempt. I normally require at least 2 or 3 attempts before I manage to get the letters right which probably makes the site suspect I am not a real human. Do other people find the same problem with completing CAPTCHAs?

Much of FourSuqre's attraction comes from its mystery

FourSquare is clearly the leading location based social network. The reason seems to be that most people agree that it is the platform that provides most fun. Unfortunately for their competitors it is hard to define exactly why it provides most fun. Personally I think that much of the enjoyment comes from an element of mystery about how exactly the rewards are handed out.

If you have used FourSquare you will probably know that they award badges for various activity. Once you unlock a badge, they give you a page explaining exactly why you were awarded it. However, they don't tell you in advance exactly what you will have to do tho earn badges that you have not yet been awarded. It is not really hard to guess that for example the Newbie badge is earned for your first ever check-in and if you hover over the logo for an unearned badge they will give you a hint, but you still don't know the rules for sure.

For me, I know that FourSquare often gives me a pleasant surprise when I earn badges or points that I was not expecting. For example, this morning when I checked into my local Lidl I got a surprise to see that I earned 25 bonus points for being the anniversary of my first check-in. If the FourSquare team keep up the good work  I will probably keep using it for a feew more years.

Automatically starting the VNC server at boot time on your Linux virtual machines

[Note: check this update before following these instructions]

These days, many people in IBM (and I am sure many other companies) are using Linux virtual machines for their normal work. The hard core geeks are happy to connect to their virtual machines via ssh and do all of their work on the command line. However, the rest of use appreciate the convenience of a graphical interface and so like to use VNC. Unfortunately, the default configuration is that the VNC server does not automatically start every time you reboot. If you reboot your server regularly, it can be a pain to continually have to log into the server and start VNC.

After a bit of digging I found this excellent blog post which describes how to auto-start the VNC server on Ubuntu.  However, most people I know tend to install Ubuntu on their laptop (which doesn't really need VNC server to be running) and use either SUSE or RedHat on their real servers. I had to make a few minor tweaks to get the script working on these Linux variants. The script below works on SUSE and RedHat variants of Linux. I have tested it on SLES 11 and RHEL5.6 - it should work on pretty much any Linux variant, but I would love to hear feedback from people if there are any issues.

As described in andrew's blog, it is necessary to start VNC manually the first time so that you can enter the security password. The first time you launch the program will create a default ~/.vnc/xstartup script which you can customize to meet your preferences. You should download the script to /etc/init.d/vncserver (making sure that the script is executable with the command "chmod +x /etc/init.d/vncserver") and then use the command "chkconfig vncserver on" to configure the server to start at boot time.

The bulk of the script is identical to Andrew's so you can read his description of how it works. I highlighted in red the places where I needed to alter it:

1. The original script declared a dependency on the networking service, but this service is called network on other Linux variants. Changing the dependency to $network allows the script to be more portable.
2. At the start of the script you can see some specially formatted comments which are interpreted as directives by the chkconfig command. There are several variants of this command and most systems do not have detailed documentation on what directives are used. The Debian wiki seems to have a complete list of possible directives. You don't need to worry about putting in special directives which are not understood by your variant of chkconfig because they will simply be treated as normal comments. The original script has enough directives to keep Ubuntu happy, but SUSE seems to insist on a "Required-Stop:" directive and RedHat seems to insist on the service description being included.
3. The original script used log_action_begin_msg, but this seems to be a command only supported on Debian derivatives so I changed them to simple echo commands.

#!/bin/sh -e
### BEGIN INIT INFO
# Provides:          vncserver
# Required-Start:    $network
# Default-Start:     3 4 5
# Default-Stop:      0 6
# Required-Stop:
# Short-Description: Starts and stops VNC server
# Description: Starts and stops VNC server
### END INIT INFO
PATH="$PATH:/usr/X11R6/bin/"

# The Username:Group that will run VNC
export USER="root"
#${RUNAS}

# The display that VNC will use
DISPLAY="1"

# Color depth (between 8 and 32)
DEPTH="16"

# The Desktop geometry to use.
#GEOMETRY="x"
#GEOMETRY="800x600"
GEOMETRY="1024x768"
#GEOMETRY="1280x1024"

# The name that the VNC Desktop will have.
NAME="my-vnc-server"

OPTIONS="-name ${NAME} -depth ${DEPTH} -geometry ${GEOMETRY} :${DISPLAY}"

. /lib/lsb/init-functions

case "$1" in
start)
echo "Starting vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver ${OPTIONS}"
;;

stop)
echo "Stoping vncserver for user '${USER}' on localhost:${DISPLAY}"
su ${USER} -c "/usr/bin/vncserver -kill :${DISPLAY}"
;;

restart)
$0 stop
$0 start
;;
esac

exit 0

Can you run 64bit Virtual Machines on a 32 bit host operating system

I was recently trying to use VMWARE Player to run some 64 bit virtual machines which a colleague had built for me. I used Google to find out if it would be possible to run the 64 bit guest OS on a 32 bit host - unfortunately the results seemed to be split almost 50-50 between saying yes and no, so I had to try it for myself.

When I tried to power on the virtual machine I got the following error screen. I went to the link suggested by the error page and I downloaded the utility to see if my CPU was capable of running in 64 bit mode. The tool from VMWARE told me that my CPU was not capable of running a 64 bit operating system. This puzzled me because until recently I was running a 64 bit operating system (RHEL6) on the same laptop.

I still thought that the problem might be that it is not possible to run a 64 bit guest operating system on a 32 bit host operating system. Then I remembered that I still had kept the boot partition for the 64 bit OS, so I booted this partition and tried again to run the virtual machines. Puzzlingly I still got the same result. The VMWARE test utility was also still telling me that my CPU was not capable of operating in 64 bit mode which was deifnietly not true since it was running in 64 bit mode when I ran the test.

I did a bit more digging and I found a utility from KVM which is supposed to check if your system can run 64bit virtual machines. It also told me that I couldn't, but it gave a very different error message. As the error message suggested, I went into my BIOS settings and enabled support for "Intel Virtualized Technology" and hey-presto I was able to run the 64 bit virtual machines. Unfortunately I don't really know what "Intel Virtualized Technology" is, but this article seems to have a feasible explanation.