Get up-to-date graphs and data on your website in five minutes. Or your money back. ^*

1. Find a subject you’re interested in.

Timetric has over a million statistics from sources including the World Bank, the European Union’s Eurostat, the European Central Bank, the St. Louis Fed, the UK’s Office of National Statistics, Amazon, and many others. We have quite a bit of data: around 1.1 million indicators at time of writing…

Start by;

• picking a subject you’re interested in from our list of topics
• looking at the data we’ve got from one of our sources, or
• doing a search from our front page.

2. Pick the data you’re interested and add it to the graph.

Filtering through the dataset is really easy: just search, then tick any data you want to see in the graph. Want to take it out again? Untick it.

3. Tweak the graph and grab the embed code.

Right now, you can change the size of the graph or the line thickness. (We’re adding more options here soon). Once you’ve finished tweaking, grab the embed code and paste it into your site or blog, just like you would with a Youtube movie.

4. Not done yet? Let us help.

Email us at help@timetric.com, catch us on Twitter, or leave a comment! We’d be glad to help you out.

(*) timetric.com is free to use. Find us and we’ll buy you a drink, though.

How the Guardian is pioneering data journalism with new tools — Simon Rogers, editor of the Guardian Data Blog, talks about the tools he uses to produce what, for our money, is one of the most forward-thinking bits of newspaperland. Of course, we're biased…

“Okay, so this is the data blog today. So obviously we’ve got Afghanistan at the top. Afghanistan is often at the top at the moment. This is a full list of everybody who’s died, every British casualty who’s died and been wounded over time. So you’ve got this data here. We use, I tend to use a lot of third party services. This is a company called Timetric, who are very good at visualizing time series data. It takes about five minutes to create that, and you can roll over and get more information.”

…but it’s definitely worth your time to watch this if you're interested in where journalism's going. (You might be interested in this SxSW panel too.)

From Simon, Florence Nightingale: data visualization pioneer.

Some counterpoint to that: lies, damn lies and visualizations, from O’Reilly Radar.

Real-time charts, from Tom Taylor, of your odds of getting a bike from London's new bike hire scheme. Our local station's the 13th busiest.

Young Rewired State was last week, and as ever, the kids put us all to shame. We're really proud to have been sponsors, and we wish we'd thought of GovSpark, a real-time monitor of the electricity usage of government departments, first!

The Economist's daily charts are really well done. They've been tracking the price of breakfast, which is shooting up thanks to a global wheat shortage. Our suggestion's to treat yourself to a fried breakfast:

On a very different area of economics, Asymco — a new blog, largely writing about the mobile phone market. It's using data really, really smartly; this post on Nokia's woes, for instance.

Slate’s been doing a bunch of terrific long-form writing. It’s the kind of writing Instapaper's made for. This family of essays on risk by John Dickerson is a great example.

The tools we use shape how we solve the problems we face. Timetric’s designed to help you solve problems through data. So what does this mean for services like ours?

Tools Shape Thought

With Timetric, we aim to make it easy for you to ask questions of, and draw conclusions from, the world’s statistics. In that light, here’s the key quote:

Our tools shape the questions we ask; therefore they shape the answers we get, and therefore they shape the conclusions we draw.

This is a hard problem. Several of us at Timetric have a background in scientific research. Our products are used by journalists. Journalists and scientists have a responsibility to be honest and trustworthy. We don’t take the responsibility this places on us lightly.

Google famously coined “don’t be evil”, but it goes further than that. Your credibility, if you rely on us, depends on our credibility, which makes credibility our business. We want you to trust the conclusions you draw using Timetric. That’s why we built it. So here’s how we work, and what we promise to you:

• We’ll always publish the original source of our data directly alongside it.
• We’ll give our data the best and most helpful titles we can, and we’ll surround it with as much supplementary information as we can find, so that you can work out if it’s really the data you need.
• We won’t editorialize or fudge data. The data on Timetric is the data as we received it — all we do is transform it into Timetric’s native format and clean up any mechanical errors we find in it. What you get is the best and most transparent version of the statistics we serve that we can give you.
• We’ll keep our visualization and analysis tools as simple, as easy to use, and as transparent as we can make them.
• If you find mistakes in our data, and you tell us about them, we’ll fix them — and what’s more we’ll tell you what we did to fix them.
• When we make mistakes — as everyone does — we expect you to call us on them. We’ll discuss them openly with you, and again, once we’ve diagnosed the problems, we’ll tell you what we’re doing to fix them.

We take your integrity very seriously. So we can do that, we take our integrity very seriously. Without that commitment, no information service deserves your trust, and your trust is the most important thing you can give us. Every day, we come to work knowing we need to earn that trust, and we’re grateful that you choose to use Timetric. We won’t forget what that means.

My topic for the night was “Scaling search to a million pages, with Solr, Python and Django”. I’ve put the slides up at SlideShare (direct PDF link) if anyone wants them.

The tl;dr is summarized on the last-but-one-slide. If you want to be able to scale your search across millions of pages, and still get good results from your users, then you need to pay attention to some details at the small scale, and some details at the large scale.

At the small scale, you need to spend time thinking about how to construct your index schema. What queries do you want to be able to run, and what information do you need to present when your search results come back? The shape of your index schema needs to be driven entirely by the answers to these two questions, and that depends heavily on the shape of your data, and the way your users want to interact with it.

On the large scale, each installation will have its own problems, but three things you’ll almost certainly need to pay attention to are:

• Decoupling reading from and writing to the index. They have very different performance characteristics (and writing presents special problems if you’re updating documents as well as adding brand new documents).
• Working out the right balance of adding/commiting/optimizing data. This will be driven by the frequency with which you add data, and how soon you need to be able to serve results from newly-added data. Must it be immediate, or can you wait seconds/minutes/hours?
• Fine-tuning your tokenizers/analyzers. Although small and fiddly, this is an issue which will bite you more and more as a corpus of data grows. You’ll need to tweak your indexing algorithms away from the defaults; extracting relevant results from a pile of a million documents is much harder than from a few thousand.

I also took the opportunity to plug my Python/Solr library, sunburnt. It’s a work in progress, but it’s battle-tested here at Timetric. If you’re trying to use Solr in any interesting Python project, I think its API is worth a look.

All the data in Timetric is uploaded by a subsystem we call the “Big Dataset Uploader”. This goes away and pulls in data from various organization’s websites, FTP servers, or wherever else it’s to be found, beats it around until it’s in the right shape, then uploads it. Ideally we’d get all our data from consistent and well defined web services; The World Bank’s API is a good example for others to follow in this regard.

In general, though, the biggest help has been using proven, open source software components. We’ve been able to draw on the wealth of knowledge available and contribute back wherever possible; Toby’s sunburnt library, a Python interface to the Solr search engine, for instance.

We’ve been building Timetric almost entirely in Python. You might be surprised by that – there are much faster compiled languages – but it’s worked out well for us. Python has great libraries. In particular, with Numpy, it has very good numeric performance for a scripting language. With a small team the productivity and maintainability advantages more than compensate for any performance hit.

We’re also making use of Ubuntu, Postgres, jQuery, Tokyo Cabinet, Memcached, Git and RabbitMQ. Best-of-breed software throughout the stack, which makes our lives so much easier!

A lot of our pages now have an email button:

If you click that button where you see it, and fill out the form, we’ll handle the rest. Here’s how it works:

And your friend will get an email like this:

Really simple and, we hope, really useful. Let us know what you think.

Today’s browser landscape looks quite different. Javascript performance his improved significantly, and IE6′s market share has halved. More significantly, there are a lot of browsers out there that aren’t even Flash-capable. Hello, Apple!

I’ll go into the techie details below, but the short version is this: if you used to see a big blank space in the middle of the page on Timetric, you’ll probably now see an attractive, interactive graph; if you’re used to seeing a “Loading…” indicator then a Flash graph you’ll probably still see that. However, if you’re feeling adventurous you can try adding ‘?noflash’ to the end of the URL and you’ll get the cool new thing everyone’s raving about.

Timetric's SVG graphing from an iPad

Techie Bits

We were looking for a plotting solution which matched the following criteria:

• Standards compliant
• Fast & light-weight
• Cross-browser
• Interactive (to both mouse events and touch)
• Attractive

SVG vs. Canvas

This first criterion means that the choice is really between SVG and Canvas. The former is a true vector format, expressed in XML, and with nodes which can be manipulated via the DOM. The latter is essentially a scriptable bitmap image, which is often more efficient, but manipulation and interactivity aren’t what it’s good at. So which should we use?

When assessing the two technologies we need to consider three classes of browsers. The first are the modern, standards-compliant desktop browsers (Firefox, Chrome, Safari, Opera and hopefully soon IE9). The second include all shipping versions of Internet Explorer. The third group are new-fangled touch-based devices like Apple’s iPhone and iPad.

The first group is the easiest and least interesting: both SVG and canvas just work.

Visitors using IE make up around 38% of our userbase, yet they don’t support either of these technologies. Luckily, they do support Microsoft’s version of SVG, called VML. The excanvas library can be used to emulate the canvas element, and the actual drawing is done in VML. Alternatively, the excellent Raphaël library provides a vector drawing API which will output SVG to browsers that support it, and VML to IE. Take-home message: both can be made to work, but since either will result in VML nodes in IE’s DOM, it’s better to use the method that’ll take full advantage of that, namely SVG.

The touch-based devices provided the knock-out blow I was looking for. When you’re using the browser on your iPhone or iPad you’re generally looking at the page at an arbitrary zoom level. Bitmaps look pretty bad at arbitrary zoom levels, while vector graphics look clean and sharp.

Vector vs. bitmap graphics at arbitrary zoom level

That was rather a long-winded way of saying that SVG is better at drawing vector-y things like graphs.

Charting Frameworks

Having made the decision to use SVG via Raphaël, I took a look at gRaphaël. Sadly, the project seems rather immature, isn’t that well supported (only one commit in the last 9 months), and has no documentation. Since we’re heavy users of jQuery, what I really wanted was a version of the awesome Flot plotting library which could output SVG instead of Canvas.

Nothing like that seemed to exist, so I set about replacing Flot’s drawing code with Raphaël calls. The result I’m calling Raphlot, and it’s available on GitHub under the same MIT license as Flot. Not all of Flot’s functionality has been translated, but enough for it to be useful on Timetric.

I’ve made a start on the interactivity, which you can try on Timetric, but I’ll write about that in a future blog post.

What do you think? Leave your comments below!

It wasn’t a good night for the Lib Dems. However, with the current electoral system, it’s hard to see how it could be a good one:

(thanks to Michael Dales for the analysis).

Each Lib Dem vote is about a quarter as effective as a Labour or Conservative vote at getting an MP elected. Oxford East and Oxford West and Abingdon elected one Conservative and one Labour MP.

Oxford West and Abingdon

Oxford East

One strong Labour first place, one weak third; same for the Conservatives. The Lib Dems actually outpolled the other parties across the two seats — and not by a little, by roughly 20% — but get nothing. Oxford is an extreme example, but that story played itself out all across the UK.

Something else which I hadn’t realised, but which charting the election made clear: Labour seats declare earlier than Conservative ones. There are a few causes behind that: smaller, more urban seats, so it’s easier to collect the ballot boxes — the Sunderland operation being the most spectacular example! — and, in general, lower turnout. There’s been quite a bit of research on that, but loosely-speaking Labour supporters stay at home in safe seats. You can actually see this effect on Michael’s graph above: the number of votes per Labour MP rises from around 20,000 in the early stages of the election to its final value of 40,000 as the evening, and morning, wear on.

This really shapes how the election story gets reported. For the first few hours of a general election, all of the early seats declaring are Labour-held, and usually they’re ultra-safe, because they’re the ones which get counted quickly. The really safe Tory seats are rural and the driving alone takes longer, and the marginals get people to turn out and vote. The drama in the election is always going to come right at the end, for purely mechanical reasons if nothing else. Maybe that’s why even the boring bits of election-night TV are so compelling!

One last thing – you wouldn’t believe how hard it is to keep up with the busiest declaration times! By 3am everyone’s a bit bleary-eyed, even on a good day, and when a seat’s declaring every twenty seconds…

Here’s all the data – total votes and total seats for everyone.

Anyway, over to Jon:

METAR weather data

New to timetric.com: temperature and atmospheric pressure for the last year from over 6000 ICAO (International Civil Aviation Organization) monitoring locations; most with hourly data points, some at every half-hour!

Pilots use this information as part of their pre-flight planning. The locations are a mix of airports and weather observation stations, some of which are automated.

We’ve tagged the data automatically to make it easy to search; we did that via a process of reverse geocoding from location, latitude, and longitude. This let us, once we’d worked out the locality, follow the geographical hierarchy for each place, letting us attach further tags like the country, region and country a station is in.

The map below shows a few hundred of the locations we have data for. Click on a location near you to see what we have!