Increased latency and failures
Incident Report for Qubole
Resolved
Qubole services have been functioning with normal performance levels for the past 24 hours . No new occurrences have been detected.
Posted Feb 09, 2017 - 13:55 PST
Update
Qubole services are now fully restored and all components are back to normal performance levels. We will continue active monitoring for 24 hours.
Posted Feb 08, 2017 - 13:37 PST
Monitoring
The measures being taken to reduce database load have been completed and Qubole should be operating at normal performance. Qubole engineers are continuing to monitor. Thank you for your continued patience.
Posted Feb 08, 2017 - 10:08 PST
Investigating
The temporary steps to reduce database load are still in process. We are utilizing all available resources to bring this performance slowness to resolution. We appreciate your patience. Qubole engineering team is continuing to actively investigate the situation. Next update at 9:00am PST on 8 Feb 2017.
Posted Feb 08, 2017 - 08:22 PST
Monitoring
The steps to reduce database load have been completed and Qubole should be operating at normal performance. Qubole engineers are continuing to monitor.
Posted Feb 08, 2017 - 06:37 PST
Update
Qubole engineers have taken steps to reduce database load. The system is being monitored and the fix will be applied after additional monitoring. ETA: 7AM PST on 8 Feb 2017.
Posted Feb 08, 2017 - 05:20 PST
Update
Deployment of the fix has been delayed due to continued load on the database. Qubole engineers are working on reducing the database load before the production fix is deployed. New ETA will communicated by 5AM PST on 8 Feb 2017.
Posted Feb 08, 2017 - 03:33 PST
Update
The fix for reducing database load and restoring system performance is undergoing additional testing. Deployment of the fix is now planned for 3AM PST on 8 Feb 2017.
Posted Feb 08, 2017 - 01:51 PST
Update
Qubole engineers are working actively on the fix to reduce database load and bring system performance back to normal ranges. The fix is expected to be deployed by 2AM PST on 8 Feb 2017.
Posted Feb 07, 2017 - 22:14 PST
Update
QDS has identified the issue causing high load and is rolling out the fix.
Posted Feb 07, 2017 - 17:49 PST
Update
QDS is still facing degraded performance and failures. We are still working on resolving these issues.
Posted Feb 07, 2017 - 15:03 PST
Identified
We continue to see degraded performance and failures. We have narrowed down the problem to be related to high load on one of our databases. We are investigating options to reduce the load.
Posted Feb 07, 2017 - 10:43 PST
Update
We continue to see latency and failures. The team is actively working on restoring our service back to normal.
Posted Feb 07, 2017 - 09:14 PST
Investigating
We continue to see some latency and failures and we are actively working on restoring our service back to normal.
Posted Feb 07, 2017 - 06:29 PST
Identified
The issue has been identified and a fix is being implemented.
Posted Feb 07, 2017 - 05:16 PST
Investigating
Starting around 2:55am PST on 7 Feb 2017, Qubole has observed increased latency and failures while processing queries. We are actively investigating and taking actions to reduce latency and errors.
Posted Feb 07, 2017 - 04:16 PST