Oracle Cloud went down in global outage

A few hours ago reports of Oracle Cloud services going down surfaced on the internet.

The outage that appears to have lasted about two hours, impacted a large number of Oracle sites across the world.

“This incident affected: Network Infrastructure (South Korea North (Chuncheon), India South (Hyderabad), Australia Southeast (Melbourne), India West (Mumbai), Japan Central (Osaka), South Korea Central (Seoul), Japan East (Tokyo), Australia East (Sydney), Canada Southeast (Montreal), Canada Southeast (Toronto), Netherlands Northwest (Amsterdam), Germany Central (Frankfurt), Switzerland North (Zurich), UAE East (Dubai), Saudi Arabia West (Jeddah), Chile Central (Santiago), Brazil East (Sao Paulo), UK West (Cardiff), UK South (London), US East (Ashburn), US West (Phoenix), US West (San Jose)) and Console (South Korea North (Chuncheon), India South (Hyderabad), Australia Southeast (Melbourne), ap-mumbai-1, Japan Central (Osaka), ap-seoul-1, Australia East (Sydney), ap-tokyo-1, Canada Southeast (Montreal), ca-toronto-1 region, Netherlands Northwest (Amsterdam), eu-frankfurt-1 region, eu-zurich-1 region, UAE East (Dubai), Saudi Arabia West (Jeddah), Chile Central (Santiago), sa-saopaulo-1, UK West (Cardiff), uk-london-1 region, us-ashburn-1 region, us-phoenix-1 region, US West (San Jose)),” reads oracle’s status page.

Experts have concluded this issue to be related to Oracle’s dynamic DNS provider, DynDNS.

Some users in a thread further discussed the root cause of the issue being DNS-related.

“DNS is a mission-critical service for every organization. But as with any technology, it is susceptible to configuration and software errors or equipment failures that cause outages. When DNS fails, websites, applications, and online services become unavailable, bringing operations, revenue, and brand reputation down with it. In many cases, the monetary loss can be substantial.”

“Companies that rely on digital apps and services should implement the following best practices to minimize the risk of downtime and ensure resiliency,” Kris Beevers, co-founder, and CEO of NS1 told Security Report.

The tips shared by Beevers that digital companies can leverage include:

1. Build-in redundancy at the DNS layer: Deploy a secondary, always-on DNS network that does not share the same infrastructure (servers, networks and data centers) as the first. This ensures that if one DNS network falls under duress, that the other will subsume the queries for the pair ensuring that queries don’t go unanswered.
   
2. Monitor system performance continuously: Monitoring the health and response times of infrastructure and applications can provide early indications that there are problems and allow IT teams to get in front of these obstacles.
   
3. Leverage anycast DNS: A highly-resilient anycast network ensures that DNS requests are dynamically diverted to an available server when there are global connectivity issues.
   
4. Automate DNS management processes: Reduce manual errors and improve resiliency by automating DNS management and embedding intelligent decision-making and traffic steering capabilities within networking infrastructure.

As of March 11th, 2021, 01:55 UTC, Oracle stated that the incident is now resolved.