These are both reflections of one of the biggest changes we’ve made – we’ve entirely replaced the search infrastructure the site runs on. We’re now backed by Apache Solr, and we’ve written a new Python-Solr interface, called sunburnt.

We let users search using both free text search and drill-down tagging — we used to run these on a combination of postgres-backed full-search text and django-tagging, but this combination wasn’t particularly satisfactory. Unsurprisingly, when you’re trying to add search infrastructure to a site, what you really want is a proper search-engine backend.

For a mature, full-featured, well-supported open-source search engine, the choice boils down to Solr or Xapian. We were strongly tempted by the latter — there’s no shortage of Xapian expertise around Cambridge, but we were swayed by the Apache licensing of Solr, rather than Xapian’s GPL.

And although there are a number of existing Python-Solr interfaces, none of them did what we wanted, which was to provide an intelligent and robust Pythonic API, which lets you pass arbitrary objects in and out of Solr. So we built our own, and called it sunburnt.

Sunburnt is most directly comparable to Haystack, but with a couple of major differences. Firstly, it’s not restricted to Django model data, and secondly, it’s schema-driven rather than schema-generating — it lets you construct your own Solr schema, and automatically derives all the type-checking/conversion/coercion code necessary to map your objects to and from the Solr index, when constructing queries and exchanging data.

The only documentation at the moment is the examples below, but the code is all up on Github. Patches and contributions are more than welcome!

Sunburnt in use

To start indexing & querying, you initialize a SolrInterface with your schema. At the moment, you need to do this by passing in the schema xml — sunburnt won’t query the Solr server for its schema.

solr_interface = sunburnt.SolrInterface("http://localhost:8983", "schema.xml")

To index objects, add() them to the interface. sunburnt doesn’t care what form the data comes in, so long as

• if it looks like an object, it has attributes named according to fields defined in the schema
• if it looks like a dictionary, it has keys named according to fields defined in the schema

class Document(object):
    def __init__(self, title, contents):
        self.title = title
        self.contents = contents
 
documents = [
   {"title":"This is a dictionary", "contents":"Lorem ipsum"},
   Document("This is an object", "dolor")
]
 
solr_interface.add(documents)

If you haven’t set your Solr instance up to do autocommit, then you might want to commit your documents to the index:

solr_interface.commit()

after which the documents are searchable. The API is fairly close to that offered by Haystack (and indeed Django’s QuerySet) – unsurprisingly, since they’re solving similar problems.

solr_interface.query("This")

does what you might expect, searching on the default field.

solr_interface.query("This").filter("dictionary")

while chaining with filter() allows you to choose which parts of your queries are cached by Solr.

For fields representing numbers, or dates, then searching by range is useful, for example

solr_interface.query("This", last_modified__gt="2009-01-01")

if you have a last_modified field in your schema. Queries can be faceted – if you had tags on your objects, you might do this:

solr_interface.query("This").facet_by("tags", limit=20, mincount=1)

and if you wanted to search for similar documents, you can do a more-like-this query (in this case, looking for similarity in the tags field)

solr_interface.query("This").mlt("tags")

sunburnt doesn’t support all of the Solr API, but it gives you access to a goodly portion, and all of these operations are chainable.

solr_interface.query("This").filter("dictionary").\
    facet_by("tags", limit=20, mincount=1).\
    query(last_modified__gt="2009-01-01").paginate(rows=10)

Finally, having set up a query, you can get a result object back by execute()ing the query. This bit of the API is still a bit rough around the edges, but

r = solr_interface.query(...).execute()
r.result # has the main query results
r.facet_counts # has the faceting results
r.more_like_these # has any more_like_these results

and if you poke around in that object, it has all the rest of the information that Solr provides.

Sunburnt in practice

The code is now running live on Timetric, and is problem-free for us. We’ve been able to throw away scads of code working around djangosearch/django-tagging shortcomings, and performance is significantly faster all round, especially for anything regarding tagging. Most usefully though, it’s provided us with a platform to start experimenting with new navigation features much more rapidly.

Postscript: djangosearch/django-tagging shortcomings

djangosearch, though easy to set up (if you’re already using postgres), offers very little in the way of control over various parameters and options you might want to tune, and requires filtering/escaping of some queries. (Searching for a string with “£” in it causes interesting errors!)

django-tagging had slightly more serious issues;

• we had to maintain our own fork of the codebase due to a couple of long-standing issues; corner cases which upstream weren’t interested in fixing (One that kept biting us was its lack of support for models with non-integer primary keys. Easily fixed, but support is still not included in upstream django-tagging).
• extending its functionality turned out to involve writing a lot of hand-tuned and often inherently slow SQL. Writing related-tags functionality was particularly painful – it involves inverting the index, which is very time-consuming – we had to do that offline.

At Timetric, we’re on a mission to get the world’s statistics to you in a form which you can use. A lot of those numbers start their lives in local and national governments, so part of the job is talking, and working, with people in government.

Today is the Government 2010 conference in London, and I’m going to be there. What’s more, I’m going to be covering it live all day on Twitter at @walkingshaw and over on Davepress. It’ll be much easier with your help, though! Have a look at the agenda, and if you’ve got anything you’d like me to cover in more detail or to relay to the conference, please get in touch.

As of last Monday, we’ve upgraded timetric.com to cope with the OAuth 1.0a workflow.

If you follow these things, you can hardly have avoided noticing that there was a big fuss in April this year, when a vulnerability in the OAuth protocol was discovered. When it was made public, it turned out to be less a technical than a social vulnerability. The OAuth workflow involves several transactions, and the exchange of multiple tokens. In version 1.0, there was an opportunity for a malicious third party to step into the exchange, and by tricking the end user (essentially, by phishing them into clicking a link) gain their credentials.

Still, social vulnerabilities are as important as technical ones, and the OAuth team rapidly developed version 1.0a of the workflow which avoids the problem. In the interim, and since upgrading existing servers and clients is hard work, and since the issue can be mitigated by anti-phishing provisions, it’s been standard practice to carry on supporting the 1.0 workflow, while attaching big warnings everywhere, and that’s what we’ve been doing.

However, we’ve now finished implementing a 1.0a-compliant server for timetric, so that 1.0a-capable clients can take advantage of the improved workflow. But because most clients don’t yet support 1.0a, our server currently supports both 1.0 and 1.0a transactions. Doing this has involved borrowing from, and extending, both python-oauth & django-oauth. We’ve fed our changes to both the upstream authors of both projects, we’ve tested our codebase, and it’s now running live on timetric.

(As of writing, our changes haven’t yet made it into the upstream version of django-oauth, but you can get a hold of what we’ve done from our fork on github.)

We’ll continue to support the 1.0 workflow for the immediately-foreseeable future, but obviously at some point we’ll want to retire it in favour of the more secure 1.0a. For those of you who’ve written OAuth clients , I can highly recommend this blog post as a very nice overview of the changes in the workflow, and what you need to do to 1.0a-enable a client.

OAuth autodiscovery

While I was poking around with the OAuth code, I also managed to address a niggle I’ve had for a long time with OAuth. Oauth uses three separate URL endpoints to manage the token request/exchange process. These need to be published somewhere, and then any OAuth clients need to know these service-specific URLs. This is annoying; practically, because it makes it hard to write a generic OAuth client framework, and also it offends the RESTian purist in me – resources should be machine-discoverable, dammit.

Anyway, it turns out that there is an experimental OAuth auto-discovery spec, which piggybacks off the XRDS resource discovery scheme. It’s not final, and it seems there’s not a lot of active development on it, but I thought I’d try it out anyway. Having implemented it as an experiment, I’m actually quite happy with it. All timetric OAuth resources are now completely auto-discoverable, knowing nothing but the xrds mimetype.

The workflow goes like this: firstly, ask for the location of the XRDS resource description, by using content-negotiation on whatever OAuth-protected resource you’re trying to gain access to:

Request:
GET /resource-of-interest
Accept: application/xrds+xml

Response:
HTTP/1.1 302 Found
X-XRDS-Location: /xrds.xrds
Location: http://timetric.com/xrds.xrds
[...]

then follow the redirect;

Request:
GET /xrds.xrds

Response:
HTTP/1.1 200 OK
Content-Type: application/xrds+xml
[...]

<?xml version="1.0" encoding="UTF-8"?>
<XRDS xmlns="xri://$xrds">
[...]

The XRDS file has a well-defined XML format, and the client can parse it to pull out the location of the OAuth endpoints. This means that you can write a very generic OAuth client library; all the library needs to be told is the location of any interesting oauth-protected resources, and now it can find out everything it needs to negotiate the OAuth workflow. Helpfully, you can also use the XRDS file to advertise which forms of OAuth negotiation you support – parameters in the URI, or as HTTP headers, different signature schemes, and so on.

I’ve had an immediate benefit, because now it’s made my test framework much simpler – I don’t need to store arbitrary strings denoting my OAuth URLs, nor manipulate them every time I run tests against differently-named test servers. Since the spec isn’t final, this scheme is obviously liable to change – but it’s a nice example of how to make a service machine-discoverable.

Previously our API was basically only let you add and retrieve data. This has been useful to a whole lot of people, but there’s much more that you can do with Timetric.

The new release involves several features. There’s a bit of improvement to existing functionality to make life a bit easier when uploading data; but more excitingly, we’ve opened up access to even more of the capabilities of the timetric platform.

Search endpoints

When building applications on top of Timetric, one of things we’ve been asked for is the ability to retrieve lists of relevant data. This might simply be to get hold of all of your own series, or it might be a list of tagged series, or it might be a complex search query.

For all of these, we’ve exposed search endpoints that let you do powerful queries across our data. You can search through the full text of our titles and descriptions, over tags, and by user. This means you can build much more useful interactive interfaces on top of Timetric. In fact, these are exactly the same endpoints that timetric.com uses internally when you browse our data.

Calculated series

Through the timetric.com website, you’ve always had the ability to build model calculations, and to filter series. We’ve now exposed this at the API level as well, so you can build these models and filters programmatically.

Cross-domain requests

If you’re a web developer, you’ll be all too familiar with the headaches of restrictions on cross-domain requests. In many cases, there are perfectly good security-related reasons for them, but these restrictions make writing some web applications much harder than it ought to be.

Fortunately, the newest generation of browsers (Firefox 3.5, IE8, and Safari 4) let you make secure cross-domain requests directly — so long as the server supports it (see https://developer.mozilla.org/en/HTTP_access_control). Since this is such a useful feature — for us as much as anyone else – we’ve enabled it so you can use it too, and build much more exciting Timetric mashups in modern browsers.

Easier uploading

And finally, we had feedback from several people about ways in which we could make pushing data into the platform through the API a bit easier. The details are probably uninteresting unless you like constructing HTTP messages yourself (which I do, but it’s not everyone’s cup of tea!) so I’ll simply point you at the new documentation. In short, you can POST data directly, rather than having to multipart-encode it.

So …

If you’re a developer, get out there and play! We’re always happy to get any feedback – positive or negative!

More important than the lick of paint, though, is all the new stuff you can do with it! Though it might sound boring, the Dashboard is all about lists. The old Dashboard had only one — a list of series you’ve ’starred’. We’ve added two others: ”My Series” for series you’ve created yourself and “Recently Viewed” which shows you the last few series you’ve looked at. Soon we’ll let you create your own custom lists too.

Now that you have all those series at your fingertips, you’ll want to compare and analyze them, right? Just select the ones you’re interested in, and click ‘Build’, ‘Overlay’ or ‘Versus’ at the top!

Talking about building… If you’re a developer, you can now build models through our API as well — Toby’ll be blogging about that very soon.

]]> http://blog.timetric.com/2009/08/timetrics-new-logo/feed/ 0 http://blog.timetric.com/2009/07/oxford-geek-night-13/ http://blog.timetric.com/2009/07/oxford-geek-night-13/#comments Thu, 23 Jul 2009 13:21:04 +0000 andrew http://blog.timetric.com/?p=180 Andrew here. I was recently invited to be one of the keynote speakers at Oxford Geek Nights #13. Thanks to everyone there! It was a real treat to get a chance to talk about what we’re doing.

I took the opportunity to discuss some of, if it’s not too pretentious a way of phrasing it, the philosophy behind Timetric and some of the work we’ve been doing with the Guardian. The talk was videoed – see it for yourself:

Right-click here to download the talk (41MB).

If you’d rather read than watch, though, my speaker notes are over on my blog.

Once again, thank you to J-P and his co-conspirators for inviting us!

Andrew Walkingshaw of Timetric at News Innovation from Martin Belam on Vimeo.

From News Innovation: London. Thank you to Martin Belam for his very sympathetic editing!

Andrew here; Dan and I took part in NewsInnovation:London at NESTA on Friday 10th July (just under a week ago). I spoke alongside Tom Loosemore of 4iP about building tools to help journalists, and members of the public, get more out of public data.

If you’ve wondered what we’re doing with the Guardian, and more importantly why we’re doing it, here are some of the things we’ve been doing and the reasons why we’ve been doing them.

The video’s about four minutes long, and if nothing else, it’s a good opportunity to see (and poke fun at) what one of us on the Timetric team looks like! There is a serious point here, though: to give people better tools to find, view, share and analyse data is to empower people to make better-informed, more considered, smarter judgements — and it occurs to us that the best journalism has the same goal: to help people to understand and to decide. That’s something which really matters.

Thank you to Mark and the other organisers for putting on such an inspirational event.

To make this easier, we’ve put a big ‘Share this series‘ button on the pages of all public series, which when clicked will reveal:

The immediate benefits are that you now have access to the code which will give you a sparkline (a common request!), and that either bit of code can be quickly copied to your clipboard using the little button at the side (thanks to Tom Preston-Werner’s Clippy). We don’t plan to stop here though — look out for more advanced features over the coming months!

For some high-profile examples of our graphs in the wild, check out the fascinating Guardian Data Blog:

http://www.guardian.co.uk/news/datablog/2009/mar/01/government-borrowing-economy1

http://www.guardian.co.uk/news/datablog/2009/mar/02/unemployment-and-employment-statistics-economics

From the beginning, we’ve been focussed as much on the developer experience as the user experience, and so our API has been the focus of just as much TLC as the website. One unavoidable issue in using any web API is dealing with authentication, and it’s especially hard to deal with it in a way that simultaneously satisfies

• security concerns
• ease-of-use for the developer
• ease-of-use for the end-user of any third-party apps.

At launch, we supported OAuth, which is a relatively new standard, designed for web services. It tries to solve all three of these problems, particularly with respect to letting end-users use third-party applications safely.

However, OAuth can be quite complex for developers to get started with. And in particular, if you’re just trying to write scripts to experiment with, for your own use, there’s an awful lot of hoops to jump through, even in understanding the basic OAuth workflow.

We also allowed developers to fallback to using their username and password over HTTP Basic Auth — but with strong caveats around it. Using your login credentials for this sort of purpose is a very bad idea.

Basic Auth (over http, rather than https) has almost no security – it’s easy to sniff over insecure connections. And when using login credentials for anything except logging in through a web-browser, you have to leave your password lying around embedded in scripts, or give it to applications, either of which makes it prone to leakage, whether by accidental loss, or by deliberate attempts to steal.

Worse, you can’t revoke your login credentials easily; once someone has stolen them, the only way to prevent them being used is to change your password — and worse still even that that, since they are your primary login credentials, any attacker can change your password before you do, thus completely denying you access to your account.

So, we’ve made a new authentication scheme publically available. This is designed to be much easier for developers to get going with. Our API tokens still use Basic Auth — despite its drawbacks, it’s got two major advantages over OAuth

• it’s supported by almost every http library out there. Whatever your favourite programming language, chances are its http libraries do HTTP Basic already. (This even includes shell script – curl supports Basic Auth out of the box, see below!)
• it involves a single set of credentials which are basically stateless; unlike OAuth you don’t have to worry about going through a dance to set up your tokens.

Essentially, our API tokens can be used as replacement username/password combinations for certain, limited, operations. You can leave these embedded in scripts, or pass them to applications without fear of the consequences; they can’t be used to change your login password, and they are easily revoked if they do leak out.

Our help pages have proper documentation, both on how to generate API tokens and how to use them, but here’s a quick overview. To create an API token, you go to your settings page, where you’ll see this:

If you follow the “Create new” link, you’ll be prompted for a name for the API token (here, I’ll call it “shell script”, since I’m going to use it with a shell script).

And after you’ve created it, the API token will appear in the list on the page.

As you can see, your API token consists of a Key and a Secret, to be used like a Username and Password. These can be used for authenticating with your http library.

If you’re using Python, we’d recommend httplib2. (If you want to restrict yourself to using the Python standard library, see this article)

import httplib2
h = httplib2.Http()
h.add_credentials('APITOKEN_KEY', 'APITOKEN_SECRET')
url = "https://timetric.com/series/pk6PiWcpTpimhozf0FTjmA/iso.csv"
resp, content = h.request(url)

 
And in shell script you would do this:

curl http://APITOKEN_KEY:APITOKEN_SECRET@timetric.com/path/

To get you started, there’s a couple of libraries out there already for talking to timetric using the API tokens. You can use these directly, or as examples for your own scripts.

Jacob Kaplan-Moss published his timetric library sometime ago, using OAuth. We’ve adapted it to use API tokens as well; until our patches make their way upstream, you can use our fork.

I’ve been playing around with Clojure recently, and my first project has been a library to retrieve data from timetric. You can see it on github, as another example of how to use the API tokens. (But probably not an example of how to write Clojure – any criticisms of my style are welcome, I’m only learning!)

But the great advantage of API tokens is that you don’t really need to go to the hassle of building a whole library or application to query a web service – sometimes all you want is a script. So here’s a shellscript which will poll timetric every 60 seconds for the latest value of a given timeseries (in this case, the Euro/Rupee exchange rate).

KEY="APITOKEN_KEY"
SECRET="APITOKEN_SECRET"
URL=https://$KEY:$SECRET@timetric.com/series/pk6PiWcpTpimhozf0FTjmA/value/json/
while 1; do
  curl $URL
  echo
  sleep 60
done

We’ve found the ability to explore web-services from the command-line like this is invaluable in developing applications on top of them — we’ve been using these API tokens for some time internally, and we’re happy we can finally make them available to you.

I’ll be following this post up shortly with a couple of exciting things we’ve done in building services on Timetric with API tokens — we’re looking forward to seeing what the rest of the world does with them.