CockroachDB Outage History

We have deployed and verified a fix. Advanced clusters should now create within normal time frames.

We are continuing to work on the fix for this issue. Next update will be at 15:00 US/Eastern.

We are preparing a fix for deployment, which should restore AWS cluster creation times to normal levels. We will update the status page when the fix is deployed, or by 14:30 US/Eastern with more information.

We have identified an issue that is causing Advanced cluster creation on AWS to take longer than usual. We are preparing a fix, and will update this status page with more information when we have an ETA for resolution.

Cluster creation is operating normally, and all delayed clusters have finished creation.

We have applied a mitigation and observe that cluster creations are proceeding again. We will monitor all impacted cluster creations to ensure they succeed.

We are investigating errors that cause creation for Advanced clusters to take longer than usual. Standard and Basic cluster creation is operating normally.

The earlier AWS outage in the us-east-1 region has been fully resolved. All CockroachDB Cloud operations, including cluster creation, scaling, and backups, are functioning normally again. We’ve verified recovery across affected systems and customer clusters, and normal operations have resumed.

AWS has resolved the underlying outage in the us-east-1 region, and we’re seeing recovery across CockroachDB Cloud operations. Cluster creation, scaling, and backups are now succeeding again. We’re continuing to monitor for full stability before marking this incident as resolved.

We’re currently impacted by an ongoing AWS outage in the us-east-1 region. Customers may experience failures or delays when creating new clusters, adding nodes, or running backups in this region. Existing clusters remain operational, but backup operations may be delayed or fail intermittently. We’re continuing to monitor AWS recovery efforts and will provide updates as more information becomes available.

Google has reported the issue has been resolved and we've verified cluster operations are functional.

We are currently experiencing an incident impacting certain cluster operations on Google Cloud Platform due to an known ongoing Google Kubernetes-related issue. Customers may encounter errors when creating or editing clusters. Our engineering teams are actively monitoring the situation and communicating with Google regarding resolution.

We have confirmed resolution of the incident impacting SQL availability for the affected cluster. SQL availability is restored and the associated cluster is now healthy.

Correction from the original status update. This issue appears to have been isolated to cluster crl-prod-6rw. We have identified the cause, mitigated and confirmed that cluster availability has returned. We are currently completing additional validations.

We are observing an incident impacting SQL availability across multiple Serverless clusters. We are currently investigating the issue.

We have corrected a misconfiguration which was preventing cluster operations from succeeding. We are monitoring the success of cluster operations already in progress and new operations.

We are investigating an issue that may prevent or delay cluster operations from succeeding.

All cluster operations are succeeding normally at this time.

We continue to see cluster operations succeeding, and will actively monitor to ensure the issue does not reoccur.

We have identified the issue preventing cluster operations from proceeding, and we see that operations are moving forward. We will next update this incident by 16:00 EDT.

We are investigating cluster operations, including creating or editing clusters, that are taking longer than usual.

We have identified and rolled out a fix for the issue impacting cluster creation in AWS. This incident is now resolved.

We have identified an issue impacting Basic and Standard tier cluster creation in AWS. We have conducted an initial investigation and actively working to resolve the issue.

Google has reported that mitigations have been applied and services are now recovering.

Google has updated their public status page to indicate the underlying issue is resolved, and individual Google Cloud services are in the process of restoring full service. We are continuing to monitor the issue and impact to CockroachDB Cloud.

In addition to cluster operations, backups to Google Cloud Storage, including managed backups from CockroachDB Cloud clusters on GCP, may be impacted.

We are aware of errors in the Google Cloud API which may impact cluster creation and edit operations for CockroachDB Cloud Advanced clusters in GCP. We will update this status page with more information as it becomes available.

The issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters has been fully resolved. Metrics are now loading as expected, and we’ve received confirmation from our service provider that the underlying issue has been addressed. We will continue to monitor for any signs of recurrence, but no further impact is expected at this time. Thank you for your patience.

Metrics visualizations are now loading as expected. We are continuing to monitor system behavior to ensure stability and will mark this incident as resolved once we confirm full recovery.

We’ve identified the source of the issue that was preventing metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters. The underlying service has recovered, and metrics visualizations are now loading as expected. We are continuing to monitor the situation to ensure full stability.

We are currently investigating an issue that is preventing some metrics pages from loading in the Cockroach Cloud console for Basic and Standard tier clusters. Metrics data collection remains unaffected; however, we are seeing unexpected errors when retrieving metrics for display in the console. We are actively working to identify the root cause and restore full functionality. Further updates will be provided as our investigation continues.

Incident Start AWS experienced a brief network blip in the eu-west-1, eu-west-2, and eu-central-1 regions. Clusters in these regions were unavailable between June 1, 19:29–19:35 UTC. Incident End The network issue affecting clusters in the impacted AWS regions has been fully resolved *automatically* as of 19:35 UTC. All affected clusters are now operating normally. Please contact support if you have any concerns.

The Engineering team has isolated and resolved the elevated error rate for all affected clusters.

The root cause of the issue was identified and resolved. Engineering is monitoring the situation.

We have received alerts indicating elevated error rates on isolated Standard and Basic clusters. The Engineering team is currently investigating the issue.

We have corrected a misconfiguration that was causing restore errors, and have applied the fix to all serverless clusters.

We are investigating increased errors in restore operations to serverless clusters.

We have now verified that all Azure clusters are operational and have completed backup jobs.

Latest update from Microsoft status page is all green, with some residual effects. CockroachDB clusters with nodes in Microsoft’s Central US region are all up and running with some background jobs like backup failing. We are continuously monitoring the status of the clusters.

CockroachDB clusters with nodes in Microsoft’s Central US region may be unavailable. We are monitoring the situation and waiting for an update from Microsoft. For more information see Microsoft’s status page at https://azure.status.microsoft/en-us/status/

Cluster operations and Cloud Console pages are restored to normal working order.

We have resolved the underlying cause of the outages, and are monitoring to ensure the system remains operational.

We are continuing to work on a fix for this issue.

We have further identified that this issue may impact pages within the Cloud Console that connect to Serverless and Dedicated clusters. SQL connections to both Serverless and Dedicated clusters are working normally. We are continuing to investigate the cause of this outage and working towards a solution.

We are continuing to work on a fix for this issue.

We are continuing to work on a fix for this issue.

We are investigating an issue that is preventing CockroachDB Cloud Dedicated cluster creation and edit operations from running successfully.

The Google Cloud Platform networking issue has been resolved. If you have any questions or concerns please submit a ticket to our support portal.

We are aware of an ongoing Google Cloud Platform networking issue. This is impacting a select number of GCP Cockroach Cloud clusters. Cockroach Labs engineers are actively monitoring the situation. More information can be found here: https://status.cloud.google.com/incidents/xVSEV3kVaJBmS7SZbnre

All cluster operations should now succeed. We are monitoring for any additional failures.

Vertical scaling of clusters on AWS may take longer than expected.

We've identified a root cause and are working on a solution. Current impact is limited to vertical scaling of AWS clusters only.

We are currently seeing some operations on AWS clusters failing, including cluster creation and modification. We've investigated and found a root cause and are working on a mitigation.

Serverless clusters in GCP europe-west1 were inaccessible.

The upstream provider incident has been resolved, and SSO authentication is working again.

Due to an upstream provider outage, all authentication methods for the Cockroach Cloud console are failing, other than authenticating via username and password. Existing sessions will continue to work until they time out.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are currently looking into connectivity issues in our Serverless environment. The impact is currently isolated to our `Europe-West-1` Serverless cluster in GCP

Our engineers have deployed a fix and the issue is now resolved

Serverless clusters are experiencing intermittent connection failures in the AWS us-east-1 region.

This issue has now been resolved. Users with clusters that were previously throttled will now see their changes to resource limits reflected.

The scope of the impact has been limited to serverless clusters running AWS eu-west-1 and AWS ap-south-1. Clusters that have hit their resource limits will be throttled despite having their limits increased. One of two issues identified has been mitigated.

Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased.

Some Serverless clusters that have hit their resource limits are still being throttled even after limits have been increased. Currently only AWS eu-west-1 and ap-south-1 host clusters are impacted

Issues impacting GCP Multi-Region Serverless Public Preview host cluster connectivity are now resolved.

We have identified the issue impacting connectivity to Serverless clusters on GCP Multi-Region Serverless Public Preview host and are currently working towards resolution.

Serverless customers may experience issues when connecting to serverless clusters running on the GCP Multi-Region Serverless Public Preview host. Other serverless clusters continue to be fully functional.

This incident has been resolved.

Cluster creation is still delayed or failing un AWS us-east-1 regions.

AWS us-east-1 appears to be recovering and we are seeing recovery of impacted systems. Please see https://health.aws.amazon.com/health/status for details

AWS is seeing elevated error rates in us-east-1, clusters in that region will be in a degraded state.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.

We are currently investigating this issue.

Partial outage for Serverless clusters in us-east-1. Some users may be unable to connect to their clusters.