How big does your city have to be in order to make Sentiment Analysis worthwhile

I wrote earlier about a solution I helped develop which allows city leaders monitor the sentiment being expressed online about their city. As we present this solution to the leaders of various cities, one of the questions that is always asked is whether their city is well known enough to generate enough mentions so that the sentiment charts will be statistically significant.

The general rule of thumb we have been using is that a city must have a population of at least .25 million in order to make the tool feasible. The thing that matters is the number of mentions of the city online (we would hope for at least 5k per week) and many times (but not always) the population of a city can be a rough guide to how many mentions that are likely to be made. Therefore I decided to run a quick test to see how many mentions I would find for a pseudo-random selection of cities with both large and small populations (some of the smaller places were not technically cities) in the first week of April this year.

This table summarises the results:

City	Population	Mentions/week
Loughrea	5,057	102
Birr	5,818	291
Nazareth	14,123	3,873
Clemmons	18,627	224
Bethlehem	25,266	3,661
Navan	28,158	775
Dundalk	31,149	898
Lorient	58,135	1,971
Galway	75,529	4,758
Cergy-Pontoise	183,430	235
Bordeaux	235,891	9,555
Montpellier	255,080	6,539
Toulouse	449,328	11,933
Dublin	527,612	29,859
Boston	625,087	111,035
Jerusalem	801,000	20,767
Paris	2,234,105	178,406
Sydney	4,627,345	52,791
London	8,173,194	257,094
Bangalore	8,474,970	16,455
Moscow	11,503,501	49,534
Tokyo	13,185,502	216,606
New York	19,570,261	440,535
Beijing	20,693,000	274,062

This can be visualised by the following chart:

I think you can see that there is a correlation between city size and the number of mentions (correlation coefficient = 0.83). You can also see that Galway is getting roughly enough mentions to make sentiment analysis useful despite only having a population of 75k, while Cergy-Panoise has more than double the population but is not getting enough internet mentions to make sentiment monitoring useful.

A few examples of where the city gets a number of mentions very different from what would be predicted for their population:

• Both Bethlehem and Nazareth get many more mentions than would be predicted by their population (e.g. they are both mentions significantly more than Navan and Dundalk which have larger populations). This is probably due to the biblical significance of the towns - in fact this is why I chose them for inclusion in the test and I don't know the names of any other towns in the middle-east with such small population.
• Where the name of the city in the local language was different from the name in English I searched on both versions of the name. In general the local language version received more hits (e.g. there were 6.5 times as many mentions of 北京 as there were for Beijing. However, for the Israeli cities it was the other way around. For example the word "Jerusalem" got 20,711 mentions while the Arabic and Hebrew translations of the city name only had 17 and 39 mentions respectively. Perhaps this is an indication that people in other parts of the world are talking about the city much more than the locals.
• Cergy-Ponoise only gets 235 mentions,while Lorient gets 1,971 mentions despite having a smaller population. I am not sure why this should be the case, but perhaps it is due to the fact that Cergy-Ponoise is so close to Paris that local residents consider themselves to be Parisians. Lorient has no similar large city nearby to overshadow it.
• The statistics will vary over time.For example,if I has run my test for the 3rd week in April rather than the first, the number of mentions for Boston would have been 1,893,159 rather than 111,035 - probably due to coverage of the marathon bombing.

Notes:

• In the case of some of the cities I chose, there are multiple cities with same name - for example, the wikipedia disambiguation page for Boston lists several cities with this name, but I only counted the population of the capital of Massachusetts (the population of the other cities would probably not be very large). 
• Wikipedia sometimes has several different estimates of population because of ambiguity of how large an area to include. I only considered the first number listed(which is typically the smallest). In some cases the difference might be minor, but in others it would be very significant. For example the estimates of the population of Boston vary by an order of magnitude from 625 thousand to 7.6 million.