Degraded performance issue on api.qubole.com
Incident Report for Qubole
Resolved
The issue has been resolved.
Posted Mar 13, 2022 - 21:56 PDT
Update
DevOps team is actively working on the issue and checking with the customers and trying to resolve it at the earliest.
Posted Mar 13, 2022 - 19:04 PDT
Update
DevOps team is actively working on the issue and checking with the customers and trying to resolve it at the earliest.
Posted Mar 13, 2022 - 16:30 PDT
Update
DevOps team is actively working on the issue and checking with the customers and trying to resolve it at the earliest.
Posted Mar 13, 2022 - 13:34 PDT
Update
DevOps team is actively working on the issue and checking with the customers and trying to resolve it at the earliest.
Posted Mar 13, 2022 - 10:11 PDT
Update
DevOps team is actively working on the issue with individual customers and checking with the customers and trying to resolve it at the earliest.
Posted Mar 13, 2022 - 05:11 PDT
Update
DevOps team is actively working on the Clusters and Scheduler issue with individual customers and checking with the customers and trying to resolve it soon.
Posted Mar 13, 2022 - 00:08 PST
Update
DevOps team is still working on the Scheduler issue with individual customers and trying to resolve it.
Posted Mar 12, 2022 - 20:53 PST
Update
DevOps team has identified the cause of the issue as scheduler autoscaling that is contributing to the remaining intermittent issues. They are currently working to resolve it.
Posted Mar 12, 2022 - 16:43 PST
Update
DevOps team is actively working on it and they have identified a secondary issue with scheduler autoscaling that is contributing to the remaining intermittent issues. They are currently working to resolve the autoscaling issue.
Posted Mar 12, 2022 - 13:34 PST
Update
DevOps team is actively working on it and they have identified a secondary issue with scheduler autoscaling that is contributing to the remaining intermittent issues. They are currently working to resolve the autoscaling issue.
Posted Mar 12, 2022 - 10:32 PST
Identified
Devops has identified a secondary issue with scheduler autoscaling that is contributing to the remaining intermittent issues. They are currently working to resolve the autoscaling issue.
Posted Mar 12, 2022 - 07:35 PST
Update
DevOps team is continuously trying to resolve this issue as soon as possible as they are working on individual customers to resolve it.
Posted Mar 12, 2022 - 03:58 PST
Update
DevOps team is continuously trying to resolve this issue as soon as possible as they are working on individual customers to resolve it.
Posted Mar 12, 2022 - 00:45 PST
Update
DevOps team is trying to resolve this issue as soon as possible as they are continuing to resolve issues for specific individual customers.
Posted Mar 11, 2022 - 20:59 PST
Update
Overall, the 'api.qubole.com' environment seems to be stabilizing. DevOps team is continuing to resolve issues for specific individual customers.
Posted Mar 11, 2022 - 15:32 PST
Update
Overall, the environment 'api.qubole.com' seems to be stabilizing. DevOps team is continuing to resolve issues for specific individual customers.
Posted Mar 11, 2022 - 12:42 PST
Update
Overall, the api.qubole.com environment seems to be stabilizing. DevOps team is continuing to resolve issues for specific individual customers.
Posted Mar 11, 2022 - 09:23 PST
Monitoring
Overall api.q environment seems to be stabilizing. Devops team is continuously monitoring the environment
Posted Mar 11, 2022 - 05:47 PST
Update
New ASG has been created under classic and added a couple of nodes and those are serving traffic. Now DevOps is working on the Redis connection issue and also, still they are working on the root cause of this issue to resolve it.
Posted Mar 10, 2022 - 19:56 PST
Update
New ASG has been created under classic and added a couple of nodes and those are serving traffic. Scheduler nodes and the connectivity issues between worker and memcache are fixed. Now Devops try to run a sample jobs and observing the stability.
Posted Mar 10, 2022 - 12:31 PST
Update
The scheduler tier on api.q is not allowing to create a new instances. Devops team has created a new non-vpc ASG and trying to add and scale up scheduler nodes. Once it is done we could switch from existing ASG to the new ASG.
Posted Mar 10, 2022 - 08:18 PST
Update
Our internal team has resolved the issue with worker nodes. The team is working on auto scaling of the nodes under scheduler ELB.
Posted Mar 10, 2022 - 03:26 PST
Identified
DevOps had identified that there is an issue with memcache and redis in api.qubole.com. Devops team is investigating further.
Posted Mar 09, 2022 - 17:19 PST
Investigating
api.qubole.com is currently seeing some degraded performance while processing commands and UI. At this time issue appears to be intermittent.
Posted Mar 09, 2022 - 14:05 PST
This incident affected: api.qubole.com Environment (AWS) (Site Availability, Command Processing, Qubole Scheduler, Cluster Operations).