We’ve recently upgraded from Solr to Elasticsearch for some of our websites. We’ll share our challenges with our previous search system, our technology selection process, the features and benefits of a modern search system and how we went from discovery to actual implementation.
Kia ora from the Digital Public Service Branch of the Department of Internal Affairs.
The main All of Government websites that we look after are Govt.nz, New Zealand Gazette and Digital.govt.nz.
These websites are built on Silverstripe CMS (content management system) and run on Silverstripe Cloud Public Sector, which comes bundled with an old version of Solr search.
Recently, we upgraded the search system to Elasticsearch, however this was suited to our particular needs. No part of this blog post should be read as an endorsement of any particular product; it describes how we went about improving search in our context.
In this blog post, we’ll share:
- the challenges of our previous search system
- our technology selection process
- the features and benefits of a modern search system
- how we went from discovery to actual implementation.
The need for a fit-for-purpose search system
Having a high-quality search engine for a website is a top priority for all stakeholders. Even when web content is structured into well-organised categories, users are time poor and hesitant to click through multiple pages to get to the content they want.
It’s important for product owners to understand their users’ search behaviour — especially these basic questions:
- What are users searching for?
- Can users easily find their desired information?
Our websites have Google Analytics integration which helps us understand user behaviour, but the old version of Solr wasn’t providing us with the search behaviour information we needed.
We weren’t getting the data we wanted — such as analysis of search terms and which search results were being clicked on.
Also, the old version of Solr we used:
- was locked
- was at End of Life
- had known security issues
- required code changes by software developers to fine tune search results.
We started looking for a new search system.
Note: Current supported versions of Solr do not have all the same issues that we experienced with the locked old version.
Choosing the right search experience provider
We were looking for a provider that was compatible with Silverstripe and could meet our needs of:
- improved user insights
- cost effectiveness
- good security posture
- the ability to support the multiple websites we manage
- adapting the search infrastructure to meet our external and internal user needs.
We assessed a number of search engines and chose Elasticsearch as it better met our needs at the time. However, this blog post is not limited to this particular vendor — the information presented here is beneficial to any agency who wants to improve the search experience on their websites.
Elasticsearch also allowed us to choose a cloud provider (such as those covered by the All of Government Cloud Framework Agreements) and deployment region (for example, Australia, Ireland). This is done through their Elastic Cloud offering.
Opportunities for other agencies
The right search solution for an agency depends on many factors — such as the number of websites, team size, team capabilities, data jurisdiction, budget, and time to market.
Any search system chosen should be modern, robust, and capable of handling the current and future needs of users.
While going for a turnkey (ready-to-use) solution with a vendor may offer short term cost advantages, other options may have long term strategic advantages and should be considered.
The choice we made suited our specific needs. We recommend agencies choose a technology that meets their specific needs, and provides good documentation and tooling to develop custom integration.
This empowers agencies to choose between multiple vendors when agencies seek to:
- develop search system integrations to their websites
- work with websites built on any technology.
It also makes it possible to build, operate and maintain the solution partly or entirely in-house and reduces the risk of a single-vendor lock-in.
A modern search system will provide advanced functionalities like the use of artificial intelligence (AI) to provide recommendations and content analysis.
The features your agency chooses to use will depend on where you’re at on your website development journey.
Implementation
After vendor selection, we needed to decide how to get the website content into the search engine. There are two approaches to this.
- Crawler — this is a simple integration for small sites where the search engine fetches the website’s content once or more daily and ingests it. Google or Bing fetches content for their search engines in a similar way.
- Custom integration to CMS — this is done through a plugin on the CMS. On installation, a full transfer of all web content to the search engine is carried out. Any subsequent content changes in the CMS also updates the search engine. This is the option we chose.
We developed our proof of concept internally, in a week. This gave us the confidence to engage our New Zealand support partner to develop a fully functional plugin. The complexity and effort of the development of the plugin was considered low.
After the plugin was developed, we ran it through our internal quality assurance (QA) process — which included getting security certification. The search system was deployed to production in February 2024 for Govt.nz and Digital.govt.nz.
We had the option to run our search infrastructure in many of the All of Government Cloud Framework Agreements on a region of our choice. At present these global Cloud suppliers don’t offer a New Zealand region. We chose Australia as our deployment region.
New capabilities and challenges
The new search system gives us many new capabilities, some of which are listed below.
- Better reporting — increased visibility of users’ search behaviour such as identifying the most popular search terms.
- Click tracking capability — identify the links clicked by users after performing a keyword search. We can also track keyword searches that do not return any search results, which can assist in developing our content strategy.
- Search results fine tuning — return a curated list of results and affect the ranking of the search results for a specific search term. Similarly, we can create synonyms for search keywords and return results accordingly.
- Role Based Access Control (RBAC) — team members can be assigned different privileges and access control. For example, analysts can only view reporting, whereas editors can create synonym lists and curations.
All the above is done through a web-based control panel and does not require any coding. This enables product owners and content editors to work with the search system without requiring assistance from software developers. However, the team members need to dedicate time and effort to upskill themselves.
Another challenge we discovered during implementation is search spam. For many websites, significant search traffic is generated by search bots performing automated requests. The product owners need to work closely with their firewall providers to block these bots. This problem is not related to our search technology selection. We noticed that search spam can lower the quality of reporting.
We considered building a ‘search as you type’ search bar for our websites where search results appear as soon as you start typing in a query. However, we found that each keystroke counts as new search request, which makes the implementation bandwidth heavy and expensive.
Similarly, search as you type relies on JavaScript and scores low in accessibility benchmarks. It’s rapidly becoming important for agencies to modernise their search systems. There’s a high level of interest among public to have chatbots integrated with agency websites and building a modern search system is an integral part of that journey.
Contact us
If you want to discuss our solution or want to see it in action, feel free to reach out. If you have access to software developers, either in-house or through an agency, you’ll be able to test our Silverstripe plugin and see if it meets your needs.
We also want to know:
- about your challenges and opportunities with website search
- what search solutions you have implemented.
Let’s keep the conversation going!
To find out more or share your search solutions, email Shaf and his team at info@digital.govt.nz
More information
Find out more about some of the subjects in this post:
Published
