Multiple Qubole environments intermittently unavailable - cloudflare issue
Incident Report for Qubole
Resolved
Qubole DevOps has monitored the current situation, determined the environments are stable and we are resolving this issue. The impact window was from 2:13 pm until 2:37 pm Pacific time which corresponds to the outage noted by Cloudflare which had major impact to many major properties on the internet.

Any UI access or API commands submitted during the impact timeframe would have failed and the remediation is to re-run those commands. Commands which had been in-flight and interrupted during the impact window which show as failed in the Qubole UI or API response would also need to be re-run if needed.

Please note that while we posted the impact to all Qubole environments, GCP and Azure-based environments were not similarly impacted during this time as they leverage a different kubernetes-based DNS resolution service than Cloudflare.
Posted Jul 17, 2020 - 15:58 PDT
Monitoring
Qubole DevOps has verified access and functionality to the Qubole environments. We have also confirmed with several customers that operations are functional. For more information on the Cloudflare outage of their Resolver component, please refer here: https://www.cloudflarestatus.com/
Posted Jul 17, 2020 - 14:47 PDT
Identified
In the past few minutes, we have noticed connection to our environments, along with access to other well-known internet sites has started restoring. We will keep active watch on this situation and provide updates in a timely fashion.
Posted Jul 17, 2020 - 14:36 PDT
Investigating
Qubole DevOps received alerts that the user interface and API access is currently unavailable. It appears there is a major outage with cloudflare that we are actively following-up with. Underlying Qubole infrastructure is running, but access to our environments is unavailable.
Posted Jul 17, 2020 - 14:23 PDT
This incident affected: api.qubole.com Environment (AWS) (Site Availability, QDS API), us.qubole.com Environment (AWS) (Site Availability, QDS API), wellness.qubole.com Environment (AWS) (Site Availability, QDS API), in.qubole.com Environment (AWS) (Site Availability, QDS API), eu-central-1.qubole.com Environment (AWS) (Site Availability, QDS API), oraclecloud.qubole.com Environment (Oracle) (Site Availability, QDS API), gcp.qubole.com Environment (GCP) (Site Availability, QDS API), gcp-eu.qubole.com Environment (GCP) - BETA (Site Availability, QDS API), and Site Availability, QDS API, Site Availability, QDS API.