Scraping Weather Data

April 8, 2015/0/0
Home / Scraping Weather Data / Scraping Weather Data

We recently had a requirement for UK weather data. A little digging around revealed that hourly weather observation data is available, free of charge from the UK met office, via the uk gov open data site.

However, this link above only allows one to download one hour of results at a time, which is a little restrictive… a year of data for all hours of each day (just 1.2M rows) would take 8760 separate downloads…

Next I took a look at the met office’s own API (Datapoint), which is supposed to provide data for a given datetime for it’s weather stations. Unfortunately, although I happily pull xml and json data feeds from their API, every time I polled for a historical date (not today), it returned ‘no matching records’.

In the end I decided to return to the first form, and instead write a python script to loop through every day and hour I was interested in on the data gov site, pulling back each csv into a folder so I could combine them to get the full dataset.

Code below for your use!

Navigation

Add comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.