Degraded performance issue on api.qubole.com
Incident Report for Qubole
Resolved
The issue with degradation in performance on API has been resolved. Customers should be able to execute their jobs and workloads now.
Posted May 16, 2022 - 16:18 PDT
Update
The issue with degradation in performance on API has been resolved. Customers should be able to execute their jobs and workloads now.
Posted May 16, 2022 - 15:15 PDT
Identified
The available space in the RStore MySQL database is reaching its limit and causing degraded performance. To resolve the issue we are in the progress of:

1. The quickest resolution to the issue is for AWS to convert our filesystem on classic from 32 bit to 64 bit on classic. This will expand the MySQL limit on data file size from 2TB to 64TB.

2. For AWS to complete this we need to create a read replica for them to convert while the current rStore stays running.

3. Once the conversion is complete we can cut over to the new instance on the 64-bit file system.

4. AWS estimates about 8 hrs to do the conversion, the Read Replica is currently being created. Once we have that then we will have an estimate on getting a fix in the performance degradation
Posted May 16, 2022 - 12:04 PDT
Investigating
We are seeing a degradation in performance on API and are looking into it currently, we will have an update in the next hour.
Posted May 16, 2022 - 10:09 PDT
This incident affected: api.qubole.com Environment (AWS) (Command Processing, Cluster Operations).