CrowdStrike crash: Government agencies using third-party software need risk assessments, quality assurance measures
After the CrowdStrike crash in July, the Ministry for Digital Development and Information set up an internal task force to assess if further measures should be taken to improve Singapore’s resilience to such disruptions.
Overhead view of the congestion at Changi Airport Terminal 1 for Scoot Airlines check-in counters, amid a global IT outage, on Jul 19, 2024. (Photo: CNA/Marcus Mark Ramos)
This audio is generated by an AI tool.
SINGAPORE: Government agencies that use third-party software are required to undergo thorough risk assessments and put mitigating measures in place, said Minister for Digital Development and Information Josephine Teo on Wednesday (Aug 7).
Responding to parliamentary questions about the CrowdStrike crash in July, Mrs Teo said agencies must also put quality assurance measures in place to ensure that software changes will not introduce errors in critical systems.
For example, this includes testing software updates in controlled settings before they go live and deploying software changes progressively before rolling them out widely.
“This usually allows us to catch and isolate issues early, but I say usually, because it doesn’t happen all the time. There are ways in which the system components interact with each other that are not always possible to map out so clearly,” she added.
Agencies with critical systems need to review the change management processes of their software providers through regular, independent audits, said Mrs Teo.
“This ensures that software changes can be rolled out smoothly and securely.”
Airlines, banks, TV channels and financial institutions around the world were thrown into turmoil on Jul 19 by one of the biggest IT crashes in recent years, caused by an update to CrowdStrike, an antivirus program.
Microsoft said the issue began at 1900GMT on Jul 19, affecting Windows users running the cybersecurity software CrowdStrike Falcon.
In Singapore, more than 100 flights at Changi Airport were delayed due to CrowdStrike outage. Airlines were forced to implement manual check-ins, with self-service machines going down.
Gantry operations at some Housing and Development Board (HDB) carparks were also affected.
The Ministry of Digital Development and Information (MDDI) has set up an internal taskforce to engage relevant partners to gain insights into the IT global outage caused by CrowdStrike and assess if further measures should be taken to improve Singapore’s resilience when such disruptions occur. Digital Development and Information Minister Josephine Teo announced this in reply to MPs’ questions in Parliament on Wednesday (Aug 7) on the global disruption caused by a faulty software update on Jul 19 by CrowdStrike. “Fortuitously, Government services and most essential services in Singapore were unaffected by the outages,” she said. However, some businesses were affected. Most of the affected IT systems had recovered within a day, and services returned to normal. Mrs Teo said one key lesson can already be reinforced. System owners should have plans in place to help them to recover quickly from unexpected disturbances. On its part, the Government adopts a risk-based approach to ensure that its critical systems and essential services are resilient. Businesses must also play their part to improve their resilience when disruptions occur and recognise that it is in their own and their customers’ interests to do so, she said. Mrs Teo said MDDI offers practical resources and financial assistance to encourage robust IT practices. While these efforts may not specifically address IT outages like the one related to CrowdStrike, they can help businesses prevent incidents and recover more quickly should disruptions occur, she said. She urged all businesses to take advantage of the Government’s resource support to strengthen their digital resilience.
Government services and most essential services in Singapore were unaffected by the outages, but some businesses that used CrowdStrike Falcon were affected, said Mrs Teo on Wednesday.
In most cases, the impact was to internal staff, she continued. “In a minority of the cases, customers were impacted due to service disruptions.”
Most of the affected systems recovered and returned to normal within a day, said Mrs Teo.
The Ministry of Digital Development and Information (MDDI) has set up an internal task force to assess if further measures should be taken to improve Singapore’s resilience to such disruptions.
Responding to a supplementary question from Mr Yip Hong Weng (PAP-Yio Chu Kang) about public confidence in government digital services, Mrs Teo likened the reliability of digital systems to that of lifts in HDB blocks.
“I think there is no shortcut to achieving public confidence. You need to be able to put the systems in place, you need to also demonstrate that when disruptions occur, and they inevitably will occur, you are able to recover very quickly,” she added.
For example, lifts will break down from time to time. This happens in every constituency and residents will accept that as long as services recover within a short time, she added.
“There is a difference between the lift system being out of service for two weeks, versus two days, versus two hours. And that is also the approach that we must take,” said Mrs Teo.
“There is no amount of assurance that you can provide except by demonstrating that this is indeed possible, which is why our emphasis has to be on the ability to respond to incidents.”
The minister also responded to a supplementary question from Mr Alex Yam (PAP-Marsiling-Yew Tee) about whether the government would consider making it compulsory for some businesses to adopt contingency plans.
Mrs Teo noted that it would be in the businesses’ own interests to have contingency plans in place, and prescribing the measures might take a sense of agency and ownership away from the IT system’s owners.
There are also many different components that go into a system’s resilience, and to imagine that the government has full understanding of all the different things that could cause major disruptions is “unwise”.
