You can transfer commercial risk … but you can’t transfer reputational risk.
The United Kingdom Public Record Office (“the Public Record Office”) administers the public record system of the United Kingdom and advises government on public record issues and related information matters.
The Public Record Office has a statutory duty to make census information available to the public following the 100 year period during which records are closed to public access.
To improve public access, widen social inclusion, cater to a growing worldwide interest in genealogy and contribute to the modernizing government agenda, the Public Record Office decided to make the 1901 census data available on the internet (http://www.1901censusonline.com/) in addition to making it available in microfiche format.
The 1901 census website was officially launched by the Public Record Office at 9am on 2nd January 2002. The service was designed to provide access to 1 million users, with a peak of 1.2 million users, in a 24 hour period.
However, by midday on 2nd January 2002, 1.2 million users per hour were attempting to access the site from locations across the world.
The level of press interest generated by the launch was unexpectedly high. And the site was introduced in a holiday period increasing the number of people who could access the site from home. These two events were contributory factors to the creation of the high level of demand that exceeded the site’s capacity.
Between 2nd January and 6th January 2002, the site continued to experience 1.2 million users per hour, overwhelming the site.
On 7th January 2002, the Public Record Office and its contractor, QinetiQ, agreed to close the site to general internet access to allow for a technical investigation to be undertaken.
Eight months later the website was released to the public on a limited basis.
It wasn’t until 21st November 2002 – 11 months after launch – that the site was made fully available to the public and since when it has operated effectively.
***
An invitation to tender was published in November 1998.
Two of the short listed firms withdrew in the course of the process: one considered the rate of investment return was not acceptable and the other could not meet the delivery deadline.
One of the remaining two bidders subsequently withdrew, again citing reservations over whether the income generated by the site would given them sufficiently quick return on the development costs they expected to incur.
This left only the bid from QinetiQ.
QinetiQ’s role was to design, develop, implement, commission, operate and maintain an online access service which would also collect the revenues due from the use of the service.
The Public Record Office retained ownership of the census data and associated intellectual property rights.
Under the arrangements, QinetiQ retained ownership of the hardware and software used for the provision of the service and provided all the development investment and bore the operational costs of the service.
The Public Record Office incurred only contract management and administrative costs.
QinetiQ subcontracted the transcription of the census returns to Enterprise and Supply Services, a division of Her Majesty’s Prison Service, which entered into agreements with individual prisons for the transcription to be completed by prisoners. During 2000, the Public Record Office and QinetiQ identified that Enterprise and Supply Services was not making sufficient progress in transcribing the census returns – the Prison Service had other priorities for the prisoners’ time, such as providing them with information technology skills to increase their likelihood of employment and reduce their chance of reoffending.
Enterprise and Supply Services subcontracted and paid for the services of additional transcribers to complete the digitization process.
78% of the transcription work was undertaken in India and Sri Lanka. In order for the transcription to be undertaken offshore, the Public Record Office had to secure the Government’s agreement for the census forms to be transcribed outside of the United Kingdom.
At the peak of the quality assurance work, the Public Record Office employed the equivalent of 68 full-time staff, over a period of approximately four months, on the process. Quality assurers were required to undertake and pass a test on reading late nineteenth century writing. In addition, ten staff from the Public Record Office with expertise in interpreting census enumeration returns spent between them 20 weeks in Sri Lanka and India to train local staff.
In May 2002 – 7 months before launch – testing by 100 users, mainly Public Record Office staff, over a period of two hours, showed that 74% of all users had problems with the site, particularly with image viewing, error messages and printing of images.
The Public Record Office and QinetiQ accepted the transcription task as complete by August 2001 despite being aware that residual inaccuracies in transcription would remain partly due to the volume of data and the interpretation of Victorian handwriting.
Data loading was completed by early December 2001, leaving only one month for QinetiQ to carry out pre-launch testing.
On the day before to the launch, the Public Record Office received a request from the London Daily Telegraph for a press release. In response, a press release was issued and the Daily Telegraph ran an article on the 1901 census on 1st January 2002, the eve of the launch.
The article was the precursor to intensive and unexpected levels of press coverage in the days immediately after the launch: the website featured regularly on news bulletins and in newspapers worldwide.
This should note have been unexpected – the FamilySearch website run for the Genealogical Society of Utah launched in May 1999 with 640 million personal records received four times the expected demand and crashed at launch.
The United Kingdom National Audit Office in a report on the launch of the website to the House of Lords noted that:
The initial crash was triggered by overwhelming demand that caused technical problems for a system that was designed for much lower capacity and which was not able to divert such a large initial surge of traffic.
The resolution of these problems became protracted, due to the parties failing easily to agree on the results from their separate testing programmes, and the relationship between the parties deteriorated.
The excess demand of 24 times specified levels resulted in a number of technical consequences:
- The internet service provider had to deploy a wider than expected bandwidth.
- Firewalls within the internet service provider ceased functioning, affecting other customers’ sites. Within the configuration, the 1901 census site did not have dedicated firewalls. Instead it shared firewalls with other customers of the internet service provider, particularly financial institutions with high volumes of transactions.
- The database engine was working close to capacity.
The 1901 census project raised a number of internal control issues of relevance for managing services on the internet:
- The need to recognize the distinction between commercial risk which may be transferred to third parties and reputational risk that usually remains with the contracting party.
- Maximise the clarity of contractor obligations so that both parties are clear as to the intended completion state.
- Include in contracts the facility to obtain independent assurance on the rigour of pre-launch system testing by the contractor.
- Before the service goes live, develop an agreed post-launch disaster recovery strategy.
- Make a realistic assessment, as far as possible, of usage and ensure that this assessment is supported by a capacity management strategy that will successfully divert unexpected overloads experienced in practice.
- Where appropriate, carefully select both the rate and timing of launch of new services so as to maximize the opportunity to resolve unforeseen problems before peak demand has built up.
***
On the six month anniversary of the site’s failure, the Public Record Office released the following statement:
The PRO apologises for how long the testing has been taking.
We are aware that six months have passed since 1901 Census Online was first launched. Both the PRO and QinetiQ Ltd, who are responsible for the technical aspects of the service, are testing the system. The nature of testing is such that it throws up issues that need to be addressed, and on occasion requires tests to be re-done. The primary objective is to ensure that the Web site is sufficiently robust, so that our customers can have a satisfactory experience when they use it.
This has meant that we cannot give precise dates for the return of the service and we do not want to raise expectations by giving deadlines that are not met.
We are aware that this has caused much frustration to our customers, frustration which we share, and for which we apologise.
Four years later, the 1901 census online is now a successful project which receives between 8,000 and 10,000 visitors per day
