How accurate are temperature forecasts for Dublin

I got a very positive reaction to my presentation about my weather forecast tracking application on Bluemix. Most people who saw this presentation asked which weather forecast service was best. Unfortunately I had to reply to them that I didn't know because I had only just built the app and never used it yet to do any detailed analysis. This post is an attempt start rectifying that error by looking at the accuracy of the various services for forecasting temperature in Dublin over the last month.

At the innovation showcase event a number of people spoke highly of  the Norwegian forecast service yr.no and claimed that it was the most accurate in their experience. Therefore, I decided to add this service into my data collection service (it is a testament to the flexible architecture that this was a relatively easy task) and wait until a month had passed before beginning analysis so that we had  enough data from them to compare.

I chose to initially analyse data from Dublin since this is where I live. I analysed temperature readings and forecasts for the period from 22nd of November to 21st of December 2016. I looked at the actual weather reported as well as the forecast for next day and the forecast for 3 days time. Some of the services offered forecasts for further into the future, but I chose to pick the 3 day forecast because this was supported by all services. If you want, you can see the actual data here.

The thing that most people commented upon at the innovation showcase was the fact that actual temperature data reported by each service did not match up exactly. When I analysed this, I noticed that some of this difference was down to data rounding. The temperature data reported by OpenWeatherMap  and DarkSky is given in celcius with 2 decimal places, in contrast WeatherUnderground and yr.no only give a single decimal place and both WeatherOnline and the BlueMix weather data are reported in whole numbers.

People asked me which service was actually correct, but the truth is that I have no way to tell. What I decided to do was to compute the average of the temperature reported by all services and then comparing the number reported by each service to this average. As you can see from the table below, the average error was actually quite small - not much more  than the average of 0.25 degrees which would be expected from rounding error. Interestingly, the yr.no service has the greatest level of difference from the average.

For assessing the forecasts I looked at the difference between the temperature that was forecast 1 or 3 days earlier and the actual temperature reported on the date in question. I also looked at the forecasts for one week ahead, although this could only be done for 4 of the 6 services since WeatherOnline and OpenWeatherMap don't supply 1 week forecasts.

The average level of error was slightly higher for 3 days in advance  (1.74 degrees) than it was for 1 day in advance (1.41 degrees) and higher again (1.95 degrees) for 7 day forecasts. The most accurate forecasts were from the BlueMix weather service across all 3 timespans. The second most accurate 1 daya forecasts were from the yr.no service although the 3 day and 7 forecasts from yr.no were definitely worse - the 7 day forecasts from yr.no were the worst of all services..


-->

Provider	Decimal Places	Ave Err	Fcst1 Err	Fcst3 Err	Fcst7 Err
weatheronline	0	0.39	1.20	1.37
openweathermap	2	0.37	1.93	2.21
weatherunderground	1	0.58	1.96	2.01	2.30
darksky	2	0.52	1.59	1.79	1.30
BlueMix	0	0.52	0.80	0.80	1.30
yr.no	1	0.75	0.96	2.28	2.90
	Average	0.52	1.41	1.74	1.95

-->

Overall, I think the conclusion is that the temperature forecasts are actually quite close to the  actual values, but the details might vary if I looked at a longer time period and/or more cities.

In reality temperature forecasts are relatively easy since temperature doesn't vary widely from day to day in this part of the world. Next I will look at wind speed forecasts since these might be more variable and also because I have been told that they are closely watched by insurance companies since wind speed correlates with weather damage,



-->-->

Where the streets have too many names

U2 had a big hit with a song named "Where the streets have no name", but in Ireland many streets have more than one name. Sometimes the change is minor, like when "Baggot Street Upper" becomes "Baggot Street Lower", but other times the streets have totally unconnected names. This is very confusing for visitors and locals alike.

One of the worst examples starts off in Rathmines, where you can travel 3 km along a straight road but officially be walking on 9 different street names:

1. The journey starts on "Rathmines Road Upper".
2. As you head north you will see that the road becomes "Rathmines Road Lower" which probably is no surprise.
3. Continue straight and as you pass over the canal and you will see that for a short time you are travelling on "Richmond Street South". If this was not confusing enough, after a short while you will come to a junction where you can either continue straight or turn right and in both cases you would still be on  "Richmond Street South".
4. Assuming you keep straight, the road will transform itself into "Camden Street Upper".
5. Unsurprisingly this road changes its name into "Camden Street Lower".
6. However, if you continue straight on the same road, you might be a little confused to find that the road transforms again into "Wexford Street".
7. Many Dubliners know "Wexford Street", but relatively few realize that if you continue walking straight along this street you will be on "Redmond's Hill". If you are looking for this piece of the road, don't be fooled by the name into thinking that there should be some slope in the road - this is a perfectly flat stretch of road.
8. If you are confused to find yourself on "Redmond's Hill", don't worry - it only lasts about 100 meters or so before transforming into "Aungier Street".
9. "Aungier Street" is reasonably long, but clearly some city planners decided that it was too long since it changes without warning into "South Great George's Street". Don't be fooled into thinking that walking north on "South Great George's Street" would bring you to "North Great George's Street" - that street is several km away in a different part of town. Instead the road comes to an abrupt end in a T-junction with "Dame Street" (needless to say "Dame Street" also has several names)..

Is it any wonder that tourists get confused when they ask someone what street they are walking along and they are told "Aungier Street" and then 5 minutes later they ask someone  else and they are assured that they are walking along "South Great George's Street" despite the fact that they have been walking straight all the time.

Confusing presidential ballot

I am lucky enough to have an absentee ballot in US presidential elections. Of course I knew that Clinton and Trump were battling it out for the Democrats and Republicans. I vaguely heard that there were additional candidates for the Libertarian and Green parties, so I was expecting to see 4 names on the ballot, However, when I got my absentee ballot, I was surprised to see 8 candidates that I could vote for, Closer scrutiny revealed that many of the candidates were listed more than once.

Hillary Clinton is the candidate for the Democratic Party, but is also listed separately as the candidate for the Women's Equality Party and the candidate for the Working Families Party (neither of which I heard of before). Likewise Donald Trump is listed as the Republican Party candidate and separately listed as the candidate for the Conservative Party (which I thought was based in the UK).

I had vaguely heard that Gary Johnson was running for the Libertarian candidate, but the ballot also lists him as the candidate for the Independence Party. Poor Jil Stein of the Green Party is the only candidate who has to make do with a single listing on the ballot.

What I am not sure of is if it makes any difference if I tick the box beside the Democratic Party or the box beside the Working Families Party? Could it be that Hillary and/or Donald are splitting their vote or do the counters amalgamate both sets of ticks?

My weather forecast tracking project.on BlueMix

In Ireland people are obsessed about the accuracy (or otherwise) of weather forecasts. This is why I previously did a small project to compare the accuracy of a three different online weather forecast services for Ireland. This project was limited and, only looked at Ireland over a short time period. However, because the way I collected data was only partially automated, it was impractical to extend it without a radical change.

IBM has recently launched the BlueMix cloud platform with lots of cool features which make it ideal to do what I wanted i.e. collect, store and process weather data. Therefore I decided to re-implement my earlier idea with a much wider scope.

There are lots of services available on the internet for current weather data and forecasts. As I mentioned before, there are even some which offer historic weather data, but none which offer access to historic forecasts - i.e. they will tell you what they predict tomorrow's weather is going to be, but they won't tell you what they predicted yesterday for today's weather. Therefore I need to collect and store the weather forecasts so I can later analyse them.

There are so many different services offering weather forecasts on the internet that it was hard to decide which ones to use. I picked the services to use mainly based upon the ease of use of their API:

• Weatheronline
• OpenWeatherMap
• WeatherUnderground
• DarkSky
• BlueMix - i.e. the weather company data bundled with BlueMix

I started by collecting data for cities with which I had some association, but this collection didn't cover the globe very well. So I then added in a few random cities which increased my geographic and climate type distribution. (e.g. Antarctica covered an extreme in both directions).

The current list of cities monitored are:

• Antarctica (not really a city, but I picked a point beside the south pole).
• Canterbury, New Zealand
• Cape Town, South Africa
• Coral Springs, Florida, USA
• Clemmons, North Carolina, USA
• Dublin, Ireland
• Galway, Ireland
• Honolulu, Hawaii, USA
• Isle of Wight, UK
• Lanzarote, Canary Islands, Spain
• Luxor, Egypt
• Perth, Western Australia
• Rio de Janeiro, Brazil

I know that most of these services update their forecasts several times a day, but I decided to make things simple by collecting weather data and forecasts once a day at midday. Working out midday in local time would be complex, especially since some of the cities implement daylight savings time at different times of the year and in different directions. Instead I decided to use the concept of solar midday - i.e. the time when the sun is highest at the city. This had the advantage of spreading out the sampling so that the system is not overloaded by taking many samples at the same time - e.g. the Isle of Wight has a solar midday at 5 minutes past 12 UTC, while Lanzarote has a solar midday of 54 minutes past 12 UTC despite the fact that the two locations use the exact same local time.

All of the weather services I used allow me to specify the latitude and longitude of where I want data or a forecast for. For larger cities I have a reasonably wide choice of where exactly within the city boundary I specify. However, the services don't have enough weather stations to provide coverage of everywhere. When I request data about a specific location the services will instead return information about the closest location that they have information about. This possibly explains some of the differenced between the services.

I added Antarctica to the list of cities so that I would have an extremely cold climate to compare with warmer locations like Luxor, Egypt. In fact, Antarctica  is not a city as such: there are no cities in such an inhospitable place. Unsurprisingly, not all of the weather services provide data or forecasts for Antarctica. Only OpenWeatherMap and DarkSky provide weather reports for Antarctica, but surprisingly BlueMix does also provide forecasts for here.

Technology Overview

As a computer geek, I am naturally most interested in the inner workings of how the data gets collected and processed. Therefore I decided to include a brief overview here, with the detail saved for a later post. However, if you are only interested in weather then jump to the next section which describes how to access the data.

• As I mentioned, the project used the Bluemix platform. More specifically, I used the "Internet-of-Things (IOT) template" which provided me with the Node-Red programming environment and a Cloudant NoSQL database service to store my data. This was my first significant application to develop with Node-Red, but I had no problems getting to grips to this programming environment. It was also my first significant use of a NoSQL database. I created a database for the forecasts, but I also created additional ones to store logs of the various steps in the process.  Putting data into the database was really easy, but getting it back out again was not so easy - on a few occasions I was tempted to use a traditional SQL database.
• All of the weather services report the weather data in different formats and so I needed to extract the data into a common format so they could be compared. This inevitably meant that I needed to discard some interesting information which is only supplied by one service. An example of the type of record I store is shown below:

  {
    "provider": "darksky",
    "request_date": "2016-11-01T00:32:00.980Z",
    "city": "Canterbury",
    "days_in_advance": 7,
    "temp_max": 18.26,
    "temp_min": 7.54,
    "temp": 18.26,
    "pressure": 997.09,
    "humidity": 68,
    "wind_speed": 11.62,
    "rain": true,
    "forecast_date": "2016-11-08T00:32:00.980Z"
  }
  

  The sample shown is a forecast from the darksky service for the city of Canterbury, New Zealand. The forecasts was retrieved shortly after midnight GMT (which is close to midday locally) on the 1st of November, Since the forecasts is for 7 days in advance, it is a prediction of what weather would be like on 8th of November. Most of the fields are self explanatory, but it should be noted that the precision is different from service to service. DarkSky gives 2 decimal places of accuracy for its temperature prediction while most give only 1 decimal place or even round them to the nearest degree centigrade. The value "rain": true implies that they predicted it would be raining on 8th of November. However, the value of the rain field currently comes with a serious health warning because I don't have a good grasp of the different ways that the various services report on rain. This will be improved in the future.
• The example I gave is of a forecast. A historical weather reading is similar with the only difference being that it won't contain days_in_advance or forecast_date fields.
• Most of the effort went into collecting the data, but I also needed to provide a way to get the data back out again. I implemented a simple web page (described below) which contains a Google chart widget and some buttons to control what data is displayed on the chart. In addition to this web page (accessed by the /ui url) I also needed to implement 2 other urls. The /data url is a service that gets called by the /ui page which returns raw weather data in JSON format. In addition I implemented the /check url which counts the number of weather reports and forecasts which have been retrieved today for each city. This url is used by a simple script that executes on a machine running on the Amazon cloud which sends me an email if the count doesn't match what is expected. I thought it was important to have this running outside of BlueMix because it is better not to have a service monitoring itself.

How to Access the Data

The simple instruction is to go to http://bodonovan.mybluemix.net/ui and if
 I did a reasonable job you should be able to figure it out yourself. So just go ahead and try it, the description here is just in case you get stuck.

The bulk of the page comprises a google chart widget with a line chart for what data you have chosen.  Below it there are a number of controls to select data for charting.

• You must specify the date range you want to chart by means of a to and from date fields. If you view the page in Google Chrome or Internet Explorer browser, these fields will have a fancy date picker associated with them, but if you use Firefox they act like simple text entry fields. You can specify dates as far back as August 2016 which is when I started collecting data, but the data is very patchy before October 2016 when I implemented the full range of providers and cities.
• There are drop downs to specify which provider you want data from, what city you are interested in and what feature you want t o chart. The features supported are temperature, pressure, humidity, wind speed and wind direction.In the future, I hope to support rainfall, but this is not working yet because of the different ways that the different services report rainfall. You also have an option to select actual data or forecasts. If you select forecasts, you also must specify how many days in advance you want the forecast for (not all service providers provide forecasts as many days in advance)
• When you have made all of the selections, you can click on the "load data" button to add the relevant line to the chart. You can then change your selections and add more lines to compare anything you want. For example you might want to compare actual data for your city to what the forecast was 1/5/7 days in advance. You can also compare what the different providers are forecasting to see if there is consensus - or you can compare the actual data they report to see if they even agree on this. You can even go crazy and compare temperature in one city with pressure in another city, but comparing different feature is unlikely to be satisfactory because I don't support rescaling the different measurement scales.
• If the database is missing data about some of the days you requested, it will draw a line with gaps. However, if select something which we have no data about (e.g. asking for a 7 day forecast from a service which only supports 5 days) you with get a "no data" warning.
• With so many possibilities it is inevitable that you will make some mistakes. Therefore there is a "remove data" button to remove the most recently added line from the chart. There is also a "clear chart" button which removed all lines from the chart.
• When you have interesting data displayed, there is an "export data" button which allows you to export the displayed readings to a CSV file so you can analyse them further with other tools.

Future work

• The UI for accessing the data is very basic so I intend to implement an improved UI soon.
• The check service regularly informs me that certain services have failed to report data for a particular city, but I have no idea why. If I implement better error tracking and logging I should be able to reduce the error rate.
• From an Irish perspective, one of the most important weather features is rain. Therefore, I intend to improve the way rain data and forecasts are handled.
• I reckon that there is a reasonably wide spread of cities and weather forecast services supported, but I will add more if there is a demand.
• Last, but not least, I hope to get some real users so I can learn how the service can be useful.

Do people really decide who to vote for AFTER they see the debates

A lot of people have been discussing whether or not people make up their mind about who to vote for after they watch the presidential debates. Some commentators say that the debates are a crucial factor in helping swing voters make up their mind, while other commentators think that the vast majority of people have already decided whom to vote for and the debates only serve to confirm their existing opinions.

I tend to agree with the later opinion. Most of the people I know are Clinton supporters and their consensus was that Clinton was the clear winner of the first debate. As a Clinton supporter, I was glad to hear this. However, my good humour was burst when I spoke with the few Trump supporters that I know and I realised that they were equally convinced that Trump had won the debate.

I was amused to read that Jimmy Kimmel ran an interesting experiment to test this phenomenon. He went out on the streets to ask random passers by how they felt the two candidates had performed on the first presidential debate. This is a pretty normal thing to do - the twist was the fact that he conducted the interviews several hours before the debate happened. I wasn't surprised that most of the interviewees had a strong opinions about who had won the debate. It is well worth checking out this episode of the Jimmy Kimmel show.

Getting full use out of the touchpad of my new mini-laptop

I am the only person in my house who doesn't have an iPad because I never thought I would find one useful.  Instead I recently splashed out and bought myself a HP mini laptop. It is a little bit like an iPad because it is roughly the same size and hence it fits the gap between a full size laptop and a smartphone. However, it has a full keyboard so there is no awkward touch typing and it runs a real operating system rather than Apple's restricted iOS.

The machine came with Windows 10 and since they place a few restrictions making it difficult to install another operating system (e.g. Ubuntu) I have been forced to learn how to use it. Overall the experience was not too bad - but that will be the subject of another blog post. This article will focus on the touch pad.

The laptop came with a sheet of paper which (among other things) described how to make best use of the touchpad. I read this with great interest, but I was disappointed to find that some of the gestures did not work as they suggested. I did a bit of googling and found an article giving a different set of gestures and a video with even  more suggestions for how it might work, but again many gestures they described didn't work when I tried them. Eventually I found an article on how-to-geek which explained that apparently the touchpad on my laptop is not of high enough quality for all of the gestures to work (Windows apparently disables many gestures when it detects that the touchpad hardware is not of brand that it has certified).

This list shows the gestures that I found to work - this might work on other laptops or might not, but it is definitely worth trying them.

• Drag a finger gently across the touchpad to move the mouse.
• Press on the left-bottom corner of the touchpad to emulate a click on the left mouse button.
• Press with a single finger on the right-bottom corner of the touchpad or press anywhere on the touchpad with two fingers to emulate a click on the right mouse button.
• Press hard anywhere on the touchpad apart from the bottom right corner to emulate a left click. I guess this gesture requires a considerable amount of pressure to avoid the user accidentally triggering a left mouse click while dragging the mouse
• While clicking the mouse with one finger, you can move the mouse with another finger. This allows you to do drag-and-drop.
• When you press on the touchpad with two fingers, you can zoom in by moving your fingers apart or zoom out by moving them closer together. This pinch/stretch gesture is the same as is commonly available on smartphones.
• If you click on the touchpad with two fingers and maintain a constant distance between them you can scroll text up or down by moving your fingers up or down. You can even use this two-fingered gesture to scroll left or right if there is a horizontal scroll option in your application.
• If you find two finger scrolling too awkward, you can configure your system to scroll by dragging a single finger up and down on the right hand side of the touchpad. (by default this option is disabled).

None of the three-finger gestures seemed to work for me (possibly due to limitations of my hardware).  In any case I think the above list of gestures is complicated enough to remember.

This shower is definitely closed

At work they are currently doing some maintenance on the toilets and shower facilities. I think they expect that people might ignore the instructions not to use the facilities while the works are in progress. As a result they put warning signs telling people to keep out which make it look like someone died in there.

Is it OK to talk to strangers?

When I was younger my parents used to tell me "don't talk to strangers", but what I observed them doing was "talk to strangers at every opportunity". This attitude was quite typical in Ireland and it still is a cultural norm.

The Irish Tourist Board is very proud of the fact that surveys of visitors to Ireland report that the people are very friendly. However, the reason for the friendliness is simply a desire to meet strangers.

The reason for this is because Irish culture evolved in a situation where people lived in small villages and everyone knew each other very well In this situation when you met a local person, there was a good chance that you already knew a lot about that person's life so it is not surprising that the conversation might tend to be a little boring. In contrast, on the rare occasions when you met a stranger/tourist you made a special effort to speak to that stranger because it would be an interesting conversation where you might learn something new.

So you see the famous Irish friendliness is in fact just a way to relieve the boredom of living in what until recently used to be a very mono-cultural society.

My First ParkRun

A colleague at work has been  trying to encourage me to join the ParkRun movement. He correctly predicted that I would find the event really fun, but the timing on Saturday morning wasn't ideal and so it was this week before I got a chance to join in the Porterstown event which is less than 2km from where I live.

It was a very icy morning so I had to be careful of my step as I ran to the venue. When I got there I could see there was a big crowd and a great atmosphere - partly due to the fact that it was associated with Operation Transformation, I was immediately put in a good mood by the positive vibe coming from the other participants.

I knew the focus was more on competing than on speed, but I checked the previous results to see the pace. I was delighted to see that the leader in 50-54 male group was someone I already knew. The atmosphere of the race encouraged me to ran faster than normal. I was timed at 28 minutes and 32 seconds. My normal is between 30 and 35 minutes for 5km.

It may be my first ParkRun, but it definitely won't be my last.

My new Headphone Hat

While I appreciate all of the Christmas presents I got this year, I was particularly taken by the usefulness of a present of a Bluetooth Headphone Hat.This is a regular knitted hat which has two small speakers hidden in the seam which allow it to act as a Bluetooth Headset.

.My initial impression was that this was a fun gimmick - an impression reinforced by the fact that the brand name is Techfun. However, I now realise it is a really useful and practical gadget.

I normally wear a Bluetooth Headset while running so that I can listen to podcasts. However, the action of running tends to shake many headsets off my head - which means that I need to choose a headset that wraps around my ears and can be quite uncomfortable to wear for extended periods. The headphone hat is quite comfortable to wear and while the sound quality is not great, it is certainly good enough for my needs. In addition I would need to wear a hat like this anyway while running at this time of year, this means that I can reduce my number of pieces of equipment by combining a set of headphones with a hat.

Fitbit steps for running, walking and cycling

I recently got a FitBit activity tracker and have been having great fun competing with family members who live abroad to see which of us can take the most steps. It is a fun way for me  feel connected to them by looking at their activity logs even if time zone differences make it impractical to actually speak to them.

As part of a discussion about one of our step challenges I was asked if the FitBit was giving me credit for taking steps while I cycled. (I think I might have been subtly accused of cheating in the challenge).

The honest answer was that I didn't know how/if the FitBit was giving me step credit for cycling. So, in order to test this out, I checked what steps I was given credit for when involved in various different activities.

This is my results:

• Running - I checked my FitBit before and after a few runs and found that it gives me credit for almost exactly 1,000 steps per km. This seems consistent with the fact that I probably have a stride length of roughly 1 metre while jogging.
  
• Walking - when I go for a walk with the dog, I get credit for 8-10% more steps than if I ran the same distance. This probably means that my stride is shorter while walking than it is when running. The fact that it is slightly more variable is probably because I wander around a little while walking the dog, but rigidly put one foot in front of the other when running.
  
• Cycling - Twice daily I cycle the route between home and work so I have lots of chances to measure how many steps I get credit for when doing this journey. My FitBit seems to give me credit for somewhere between 700 and 1,800 steps for this 8 km journey. This is a strange reading from a few points of view:
  
• I do tend to take a few real steps at the start and end of the journey as I wheel the bike in and out of the shed, but this would probably be 10s of steps not 100s.
  
• On the other hand, it could be argued that each time I push the pedals up and down this is effectively the same motion as a step. But, if was what the FitBit counts as steps, then it is seriously underestimating the number of pedal turns that I would do.
  
• The other thing that is strange is that there is so much variation in step count - some journeys were more than twice as many steps as others despite the fact that I was going on exactly the same route.
  
  I think (but can't prove) that the key factor is how often I wobble. If I was cycling nice and steady on a flat road, my wrist would be staying still on the handlebars and the FitBit would count that as resting. However if I was wobbling from side to side, the up and down motion of my left wrist would probably move up and down in a similar way to if I was walking. The amount of wobbling I do would probably vary from day to day depending upon traffic and other factors.
  
• Working - I work as a software engineer so my work involves virtually no physical movement (which is why it is a very unhealthy occupation). The FitBit should not give me credit for any steps while I work. I tested this one day by looking at my step count when I came into work and then checking it 1 hour later.
  
  The FitBit said I took 7 steps in that hour - initially I thought it was giving me false credit, but then I remembered that during the hour I had left my seat once to look at a colleague's screen to help solve a problem he was having (his desk is beside mine so 7 steps sounds right for there and back). Therefore I think that it correctly gives me no credit for activity while working.
  
• Sleeping - I know that I tend to toss and turn a lot during my sleep. In fact my FitBit sleep data tends to show frequent periods of red in the middle of the deep blue sleep time (the red periods are times when the FitBit thinks I am awake/restless - I am not sure how it decides that). I thought that the FitBit might give me credit for steps when I was really just tossing about restlessly during the night. However, it seems that it doesn't.
  
  The FitBit resets its step counter at midnight and I would normally be asleep by then (or at least lying in bed trying to sleep). So I decided to check my FitBit on a few mornings straight away after I got up out of bed. It would normally give me credit for a small number of steps, but never more than about 10-15 steps per night.

Overall I think that the FitBit gives a reasonably accurate measurement of activity level even if it is off significantly when measuring cycling.

I have read on-line about a legal case over how inaccurate the FitBit heart rate monitoring function can be. I don't doubt that the readings are often wrong, but I think the real problem is why were people expecting that a simple wrist-watch with two LEDs built into it could really provide accurate readings? (Medical professionals use complex and expensive devices to take the same reading.)