My topic for the night was “Scaling search to a million pages, with Solr, Python and Django”. I’ve put the slides up at SlideShare (direct PDF link) if anyone wants them.

The tl;dr is summarized on the last-but-one-slide. If you want to be able to scale your search across millions of pages, and still get good results from your users, then you need to pay attention to some details at the small scale, and some details at the large scale.

At the small scale, you need to spend time thinking about how to construct your index schema. What queries do you want to be able to run, and what information do you need to present when your search results come back? The shape of your index schema needs to be driven entirely by the answers to these two questions, and that depends heavily on the shape of your data, and the way your users want to interact with it.

On the large scale, each installation will have its own problems, but three things you’ll almost certainly need to pay attention to are:

• Decoupling reading from and writing to the index. They have very different performance characteristics (and writing presents special problems if you’re updating documents as well as adding brand new documents).
• Working out the right balance of adding/commiting/optimizing data. This will be driven by the frequency with which you add data, and how soon you need to be able to serve results from newly-added data. Must it be immediate, or can you wait seconds/minutes/hours?
• Fine-tuning your tokenizers/analyzers. Although small and fiddly, this is an issue which will bite you more and more as a corpus of data grows. You’ll need to tweak your indexing algorithms away from the defaults; extracting relevant results from a pile of a million documents is much harder than from a few thousand.

I also took the opportunity to plug my Python/Solr library, sunburnt. It’s a work in progress, but it’s battle-tested here at Timetric. If you’re trying to use Solr in any interesting Python project, I think its API is worth a look.

Andrew Walkingshaw of Timetric at News Innovation from Martin Belam on Vimeo.

From News Innovation: London. Thank you to Martin Belam for his very sympathetic editing!

Andrew here; Dan and I took part in NewsInnovation:London at NESTA on Friday 10th July (just under a week ago). I spoke alongside Tom Loosemore of 4iP about building tools to help journalists, and members of the public, get more out of public data.

If you’ve wondered what we’re doing with the Guardian, and more importantly why we’re doing it, here are some of the things we’ve been doing and the reasons why we’ve been doing them.

The video’s about four minutes long, and if nothing else, it’s a good opportunity to see (and poke fun at) what one of us on the Timetric team looks like! There is a serious point here, though: to give people better tools to find, view, share and analyse data is to empower people to make better-informed, more considered, smarter judgements — and it occurs to us that the best journalism has the same goal: to help people to understand and to decide. That’s something which really matters.

Thank you to Mark and the other organisers for putting on such an inspirational event.

Well, we’ve changed all that.

Take employment data as an example:

Search results for "Employment" on Timetric

Two big changes here – firstly, there are sparklines, so you can get a feel for all the data in front of you immediately; secondly, you now get all the relevant tags at the top of the page on every search result, so you can immediately start filtering through the search results to find what you’re interested in. Let’s say that’s the utilities sector.

The stars on the right here, if you’re logged in, immediately save series to your dashboard; they’re gold for each series you’re already watching. But the bigger change is on the left; if you check the series you’re interested in…

Selecting series in Timetric search results

and then hit “Overlay” or “Versus”, which you’ll find at the start and end of the search results page:

you can start making plots straight from search results.

Transportation employment's much more seasonal in Alaska than in Alabama.

You can even get the embed code to put a graph into your blog straight from there:

And that gives us a chance to mention another new feature which a few of you’ve been asking for – if you hover your mouse over the points in this graph, you’ll see each measurement in the scatter plot labelled with the time it comes from.

The big changes to graphing here are actually under the covers, though: after this, we’ll be able to make some really exciting improvements in the near future.

That means two things, really; I build ways for you to find the data already in Timetric, and I find and upload new sources of data which we think you might be interested in. I’m going to write about the first of these today, but if there’s data out there which you’d like us to get, leave us a suggestion on our Get Satisfaction site and we’ll see what we can do.

Anyway, about finding data. If you’ve used sites like Flickr or Delicious, then Timetric’s going to feel pretty familiar – like them, we use search and tagging to help you get to the data you’re after.

When you log into Timetric, the first thing you see is your dashboard.

Dashboard

Any nuggets you’re watching appear in here. Up in the top right hand corner, though, there’s a search box; if you type something in there and hit “Search”, then you get back a page of search results.

Search results

I searched for “bond” here, and I’ve got back a page of information about bond prices. Let’s say I’m not interested in data from Moody’s: if I search for “bond -Moody”, then I can exclude them from the search results.

Search results for "bond -Moody"

Under each of these search results, you can see a list of tags (in light grey). When you upload data, you can give a list of tags to remember it by – categories, more or less. A lot of the data here comes from the UK National Statistics dataset from http://www.statistics.gov.uk/, and we’ve tagged that “National Statistics” – so if you click on that tag:

… then you get back a list of everything tagged with that tag – for this dataset, about 2000 pages worth! So you’ll need to be able to filter that. From this page, you can filter by another tag – the most common tags found alongside the ones you’re filtering by are listed at the top of the page. Let’s filter by “monthly”:

Multi-tag filters

We’re based in Cambridge, about an hour away from London, so I’m most interested in what’s going on in the south of England. I can search within these results by typing “south” into the search box there.

Tagging and searching together

So you can use tags and searching together to find the data you’re interested in. If there’s something you’re looking for which you can’t find, though, let us know and we’ll try and help you out.