TL;DR
The Service Innovation Lab team work in a cross agency and user centred way to co-design and co-deliver better services for New Zealanders. In the process of doing so, we explore and test the validity of reusable components that can enable multiple life event based services as well as third party innovation.
We recently worked with data.govt.nz (a platform to discover and host open government data) to deliver “Services Near Me” on SmartStart. This work was a collaboration between SmartStart, our Lab, the data.govt.nz team and the Ministry of Social Development's (MSD) Family Services Directory team. You can read more about that collaboration in our blog post about the Family Services Directory reusable component. And we've also built a proof of concept search interface for the Family Services Directory.
This post delves into the technical details of how data.govt.nz saved us time (and money) in delivering this functionality as a reusable and constantly updating component that anyone needing information about social and family services in New Zealand can build upon. This post is written for a technical audience, both techies in the New Zealand government, and anyone wanting to build on the NZ government as a platform for service delivery and innovation. Please read on if such detail is of interest or forward on to your technical friends.
Building on open data infrastructure (For The Win)
data.govt.nz is powered by an open source application called CKAN. It has a growing catalogue of datasets and APIs. Data can be added to the catalogue as a simple link to elsewhere, or as an actual dataset uploaded into data.govt.nz's own database and filesystem.
We helped MSD automate the uploading of a dataset into data.govt.nz for anyone to use, and then we built on top of this data in our own applications. This means the data is stored by data.govt.nz, and there was no additional work needed to present this data as an API. It is updated every day and so stays current.
It is worth noting that being open government data, there is no personal data included in this dataset. The Family Services Directory has lists of services and the relevant providers which is already public information.
There are two nifty features the data.govt.nz API has that we used:
- datastore_search
- datastore_search_sql
data.govt.nz has an index of the data it holds (actual datasets stored, not the offsite links). datastore_search is how you search this data. For example, here is a datastore search for any record with the word "kai" in it:
See the full documentation for how to use the datastore_search action.
Here are the useful parameters for querying data:
- resource_id (string) – id or alias of the resource to be searched against
- filters< (dictionary) – matching conditions to select, e.g {“key1”: “a”, “key2”: “b”} (optional)
- q (string or dictionary) – full-text query. If it’s a string, it’ll search on all fields on each row. If it’s a dictionary as {“key1”: “a”, “key2”: “b”}, it’ll search on each specific field (optional)
- distinct (bool) – return only distinct rows (optional, default: false)
- plain (bool) – treat as plain text query (optional, default: true)
- language (string) – language of the full-text query (optional, default: English)
- limit (int) – maximum number of rows to return (optional, default: 100)
- offset (int) – offset this number of rows (optional)
- fields (list or comma separated string) – fields to return (optional, default: all fields in original order)
- sort (string) – comma separated field names with ordering, e.g., “fieldname1, fieldname2 desc”
- include_total (bool) – True to return total matching record count (optional, default: true)
- records_format (controlled list) – the format for the records return value: ‘objects’ (default) list of {fieldname1: value1, …} dicts, ‘lists’ list of [value1, value2, …] lists, ‘csv’ string containing comma-separated values with no header, ‘tsv’ string containing tab-separated values with no header
You can, for example, combine the distinct with fields parameter and get a list of categories, like so:
There's also a SQL-like interface for querying. This lets you do the same query as above, but I also added the COUNT()
The results look like this:
{"help": "https://catalogue.data.govt.nz/api/3/action/help_show?name=datastore_search_sql", "success": true, "result": {"records": [{"num": "361", "name": "Addiction"}, {"num": "2175", "name": "Basic Needs"}, {"num": "820", "name": "Budgeting / Financial Capability"}, {"num": "1375", "name": "Child Care"}, {"num": "241", "name": "Disaster Recovery"}, {"num": "1244", "name": "Education and Training"}, {"num": "608", "name": "Employment"}, {"num": "397", "name": "Ethnic Services"}, {"num": "757", "name": "Family Violence"}, {"num": "5377", "name": "Family / Whanau Services"}, {"num": "1687", "name": "Health"}, {"num": "305", "name": "Legal / Civil Services"}, {"num": "664", "name": "Mental Health"}, {"num": "2284", "name": "Parents and Caregivers"}, {"num": "1281", "name": "Special Needs / Disability"}, {"num": "2469", "name": "Youth Services"}], "fields": [{"type": "text", "id": "name"}, {"type": "int8", "id": "num"}], "sql": "SELECT \"LEVEL_1_CATEGORY\" as name, COUNT(*) as num FROM \"35de6bf8-b254-4025-89f5-da9eb6adf9a0\" GROUP BY name ORDER BY name"}}
Two applications have so far been built from the data.govt.nz resources in the examples above:
- The Family Services Directory Searchinator. This is a proof of concept application that lets you search and view the full dataset. We've deployed it to GitHub pages - to demonstrate that it's a single file (built in reactjs). There's no server needed, the data and query are all performed on data.govt.nz.
- The SmartStart Services Near Me search, built by Catalyst IT, which presents the Family Services Directory data relevant to having a baby.
Anyone can use the Family Services Directory API on data.govt.nz to innovate and deliver new services for New Zealanders. We look forward to seeing how people use this helpful reusable component! Please leave a comment if you find this reusable component useful and let us know when you build clever things with it.
