Update - Update - Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability.

While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers can immediately refer to the Apache website for mitigation steps and will be able to contact Qubole support to apply new AMIs once these are available.

We are providing new AMIs for AWS currently. We are working on remediation steps for GCP and OCI currently. We will continue to update here when those will be available.


Monitoring - NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility.

Qubole’s investigation of the Log4j vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure.

Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability.

For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. Customers will need to contact support to apply new AMIs

Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.
Dec 31, 10:46 PST
Update - Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The Qubole control plane does not utilize a version of log4j affected by the vulnerability.

While Qubole does not use an affected version of log4j, we do provide AMIs intended for deployment within customer-controlled hardware (i.e. the Data Plane) that include third-party software such as Spark, Hive, and Presto that contain potentially vulnerable versions of the log4j libraries. The Apache foundation has provided general mitigation strategies that you may apply to your Data Plane clusters to ensure you are not impacted by these vulnerabilities.

IMPORTANT NOTE: The following guidance is provided as general guidance that you can apply to your Data Plane clusters based on the Apache Foundation tech bulletin. You will need to customize your individual steps according to your environment. We have tested these strategies with sample Data Plane clusters and they prove effective at eliminating the vulnerabilities identified in Log4J 2.x versions.

1. Customers that have Java 7 and/or Java 8 installed on their clusters will need to add a line similar to the one below to their bootstrap.sh script. The cluster will then need to be restarted. This will remove the offending JndiLookup class from the classpath:

zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class

2. Based on newer CVEs published, customers with Java 8 are also vulnerable if Context Lookups are being used in the property files. Upon investigation, we have determined that there are no Context Lookups being used in the Qubole provided AMIs. There is no action required to respond to these newer CVEs.

Note: Apache has also reported a vulnerability with the 1.x version of Log4j. They have advised to investigate the use of JMSAppender. Log4j 1.x configurations without JMSAppender are not impacted by this vulnerability. Qubole does not use JMSAppender in either the Control nor the Data planes therefore Qubole is not impacted by this issue.
Dec 22, 23:13 PST
Monitoring - NOTE: This incident is no longer considered active, but is being maintained as Monitoring for short-term visibility.

Qubole’s investigation of the CVE-2021-44228 vulnerability in the Apache Log4j library continues to advance, with focus on identifying any exposed instance of a vulnerable Apache Log4j library as per Apache’s public updates. Qubole consists of two parts: (1) the Control Plane, which resides on Qubole-controlled hardware and (2) the Data Plane, which resides on Customer-controlled hardware. The investigation is guided by this structure.

Within the Control Plane (on Qubole-controlled hardware), our investigation confirmed there are no exposed instances of the Apache Log4j library within the version range that contains this vulnerability. Therefore, the investigation confidently concludes the Control Plane is not impacted by the Apache Log4j vulnerability.

For the Data Plane (on Customer-controlled hardware), we understand Qubole customers will want to take immediate action to protect the environments they control. This immediate action can be achieved by following the mitigation instructions as published by Apache on the Apache website (https://logging.apache.org/log4j/2.x/security.html).

Although our initial and thorough investigation has concluded, and Qubole continues to monitor for potential breaches, we will continue actively to monitor this situation and communicate with stakeholders as appropriate.
Dec 16, 06:55 PST

About This Site

This site is the Qubole's home for information on QDS system performance and availability.

Privacy Statement

api.qubole.com Environment (AWS) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
us.qubole.com Environment (AWS) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
Quantum ? Operational
wellness.qubole.com Environment (AWS) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
gcp.qubole.com Environment (GCP) Operational
Site Availability Operational
QDS API Operational
Command Processing Operational
Qubole Scheduler Operational
Cluster Operations Operational
Notebooks Operational
gcp-eu.qubole.com Environment (GCP) - BETA Operational
Site Availability ? Operational
QDS API Operational
Command Processing Operational
Qubole Scheduler Operational
Cluster Operations Operational
Notebooks Operational
azure.qubole.com Environment (Azure) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
azure-eu.qubole.com Environment (Azure) - BETA Operational
Site Availability ? Operational
QDS API Operational
Command Processing Operational
Qubole Scheduler Operational
Cluster Operations Operational
Notebooks Operational
in.qubole.com Environment (AWS) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
eu-central-1.qubole.com Environment (AWS) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
oraclecloud.qubole.com Environment (Oracle) Operational
Site Availability ? Operational
QDS API ? Operational
Command Processing ? Operational
Qubole Scheduler ? Operational
Cluster Operations ? Operational
Notebooks ? Operational
Qubole Community & Support Portal ? Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Jan 22, 2022

No incidents reported today.

Jan 21, 2022

No incidents reported.

Jan 20, 2022

No incidents reported.

Jan 19, 2022
Resolved - DevOps has confirmed that the issue has been resolved and processing commands are working fine in api.qubole.com.
Jan 19, 01:53 PST
Monitoring - Our internal team has resolved the tunnel issue. The team continues to monitor the status.
Jan 19, 00:23 PST
Identified - We have identified that there was tunnel issue in api.qubole environment. Our Devops team is working on it.
Jan 19, 00:19 PST
Investigating - api.qubole.com is currently seeing some degraded performance while processing commands. At this time issue appears to be partial, but DevOps is investigating on it.
Jan 18, 18:23 PST
Jan 18, 2022
Jan 17, 2022

No incidents reported.

Jan 16, 2022

No incidents reported.

Jan 15, 2022

No incidents reported.

Jan 14, 2022

No incidents reported.

Jan 13, 2022

No incidents reported.

Jan 12, 2022

No incidents reported.

Jan 11, 2022

No incidents reported.

Jan 10, 2022

No incidents reported.

Jan 9, 2022

No incidents reported.

Jan 8, 2022

No incidents reported.