Grafana Cloud Outage History

This incident has been resolved.

A fix has been implemented, and we are monitoring the results.

We are currently investigating an issue impacting dashboards for users in the prod-us-central-3 region. This is preventing impacted dashboards from loading as expected. This is also impacting a very small subset of users in the prod-us-central-0 region as well. We will provide more details regarding the scope as they become available.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

Some users experience their Grafana Cloud instances as unavailable.

Engineering has released a fix and as of 00:13 UTC, customers should no longer experience issues upgrading from Free to Pro subscriptions. At this time, we are considering this issue resolved. No further updates.

Engineering has identified the issue and is currently exploring remediation options. At this time, users will continue to experience the inability to upgrade from Free to Pro subscriptions. We will continue to provide updates as more information is shared.

As of 20:05 UTC, our engineering team became aware of an issue related to subscription plan upgrades. Users experiencing this issue will not be able to upgrade from a Free plan to a Pro subscription. Engineering is actively engaged and assessing the issue. We will provide updates accordingly.

This incident has been resolved. Both read and write 5xx's and increased latency were experienced in the two periods: 23:56:15 to 00:32:45 UTC 00:55:30 to 01:36:15 UTC

Customers should no longer experience issues. We will continue to monitor and provide updates.

We are continuing to investigate this issue.

Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data. We continue to investigate and will update when we have more to share.

As of 00:28 UTC, we have observed improvement with the partial write outage. Customers should no longer experience issues with metrics ingestion. We will continue to monitor and provide updates.

As of 23:57 UTC, our engineers became aware of an issue with prod-us-west-0 resulting in a partial write outage. Users may experience intermittent 5xx errors when writing metrics, though retries may eventually succeed, which can lead to delayed or missing data. We continue to investigate and will update when we have more to share.

The scope of this incident was smaller than originally anticipated. As of 16:27 UTC our engineering team merged a fix for those affected and we are considering this as resolved.

We're experiencing an issue with connectivity loss for Azure PrivateLink endpoints in all available Azure regions. The issue affects users trying to ingest Alloy data or use PDC over Azure PrivateLink. Our team is actively investigating the issue for the root cause.

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Engineering has released a fix and as of 17:01 UTC, customers should no longer experience connectivity issues. We will continue to monitor for recurrence and provide updates accordingly.

Engineering has identified the issue and will be deploying a fix shortly. At this time, users will continue to experience disruptions for queries routed via PDC. We will continue to provide updates as more information is shared.

We are investigating an issue in prod-eu-west-3 where PDC agents are failing to maintain/re-establish connectivity. Some agents are struggling to reconnect, which may cause disruptions or degraded performance for customer queries routed over PDC. We’ll share updates as we learn more.

This incident has been resolved.

We are seeing some recovery in affected products. We are continuing to monitor the progress.

We are currently investigating an issue causing degraded Mimir and Tempo read performance in the prod-us-central-7 region.

This incident has been resolved.

We are continuing to investigate this issue.

We are currently investigating an issue that is causing intermittent failure when trying to start new k6 tests on the k6-app in the cloud. The tests are starting normally from CLI.

This incident has been resolved.

TraceQL queries with "= nil" in Explore Traces and part of Drilldown Traces are failing with 400 Bad Request errors. The issue has been identified, and a fix is currently being rolled out.

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

Synthetic Monitoring has now also recovered. Customers should no longer experience alert rules failing to evaluate. We continue to monitor for recurrence and will provide updates accordingly.

Engineering has released a fix and as of 22:25 UTC, customers should no longer experience ingestion issues. We will continue to monitor for recurrence and provide updates accordingly.

While investigating this issue, we also became aware that Synthetic Monitoring is affected. Some customers may have alert rules failing to evaluate.

As of 21:30 UTC, we are experiencing a partial ingestion outage in Grafana Mimir. This is affecting the write path, where some ingestion requests are failing or timing out. Our engineering team is actively investigating and working to identify the root cause.

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

We are observing a trend in improvement after implementing a fix. We will continue to monitor and update accordingly. During our investigation, we also became aware that some alerts associated with Synthetic Monitoring checks have been failing to evaluate correctly.

Our engineering team has identified a potential root cause, and a fix is being implemented.

Our engineering team has engaged with our Cloud Service Provider and are working together to continue to investigate this issue.

We are continuing to investigate this issue.

We have also identified that trace ingestion may also be affected. Some customers may experience elevated latency and intermittent errors when sending traces. Investigation is ongoing.

We have detected an issue causing some customers to experience failed metric pushes as well as increased latency when sending metrics. The issue was first observed at 18:30 UTC. Our engineering team is actively investigating the root cause and working to restore normal operation as quickly as possible. We will provide further updates as more information becomes available. Thank you for your patience while we work to resolve this issue.

Users experienced failed log pushes as well as increased latency when sending logs to Loki service hosted on prod-eu-west-0 cluster between 18:30 UTC to ~23:00 UTC. Our engineering team has engaged our Cloud Service Provider and the fix was implemented that mitigated the issue.

The issue has been resolved.

The query service is operational again, logs reads should be available on the cluster. Our engineers are monitoring the health status of the service to ensure full recovery.

Since today 9th, around 12:30 UTC time, we are experiencing problems on the Loki read path of cluster eu-west-2. This translates in difficulties to query logs for customers on this cluster, and can also impact alerts and other services based on these logs. Our engineers are actively working in restoring the service.

We continue to observe a continued period of recovery. At this time, we are considering this issue resolved. No further updates.

A fix has been implemented, and we are seeing latency down across clusters. We are continuing to monitor progress.

We are currently investigating reports of long load times on Grafana Cloud. We will update as more information becomes available.

This incident has been resolved.

From approximately 14:10-14:25 UTC, writes to Loki failed for a subset of customers in the gb-south-0 region. Most of these errors have already recovered, and our team continues to monitor the recovery.

This incident has been resolved.

Things have recovered, and we are monitoring to ensure stability.

The previous post mentioned that this was occurring in some cells in prod-us-central-0. This is incorrect and is occurring in some cells in prod-us-east-0.

At approximately 16:40 UTC we noticed an issue causing increased read & write latency in some prod-us-central-0 cells. At this stage we are noticing some recovery and will continue to monitor.

This incident has been resolved.

We’re continuing to work on this issue, and are actively investigating the remaining details. Our team is making progress, and we’ll share another update as soon as we have more information to provide.

We are continuing to work on a fix for this issue.

We are currently observing a read/write path disruption in prod-us-central.cortex-prod-04.

We continue to observe a continued period of recovery. As of 00:08 UTC, we are considering this issue resolved. No further updates.

A fix has been implemented and we are monitoring the results.

Our teams have been alerted that Synthetic Monitoring will also be affected by this outage. Users may see gaps in their Synthetic Monitoring metrics as well as missed alerts as a result of this. We continue to investigate and will provide further updates as they become available.

The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.

The investigation has revealed that metric ingestion is also affected, including log-generated recording rules. We are continuing to investigate the root cause and will provide further updates as more information becomes available.

Span metrics have also been identified as affected. We are continuing to investigate the root cause and will share further updates as more information becomes available.

We are investigating elevated Mimir read errors beginning at approximately 21:57 UTC. The errors are technically retriable, but most retries are unlikely to succeed at this time. This may result in failed or delayed query responses for some users. Engineering is actively investigating the root cause and working to restore normal read performance. We will provide further updates as we learn more.

As of 23:07 UTC we are considering this incident as resolved. Mitigation efforts have restored normal write performance, and error rates have returned to expected levels. We have confirmed stability across the affected areas and continue to monitor, but no further impact is expected. If you continue to experience any issues, please reach out to support.

We are seeing improvement on the metrics side, with write performance recovering. We continue to investigate the remaining impact to Synthetic Monitoring and are working to determine the underlying cause. Monitoring will continue as recovery progresses, and we’ll provide further updates as we learn more.

Our teams have been alerted that Synthetic Monitoring will also be affected by this outage. Users may see gaps in their Synthetic Monitoring metrics as well as missed alerts as a result of this. We continue to investigate and will provide further updates as they become available.

We’ve re-evaluated the situation and this issue is still ongoing. Although we initially observed signs of recovery, write errors continue to occur in the affected cells. Mitigation work is still in progress, and we’re treating the incident as identified again while we work toward a sustained resolution. We’ll provide further updates as we confirm stabilization.

Mitigation has been applied and Mimir write performance is beginning to recover in the affected cells. prod-us-central-0.cortex-prod-10 appears to have recovered as of 19:52 UTC, and prod-us-central-5.cortex-dedicated-06 is showing signs of recovery as of 20:00 UTC. We are continuing to monitor both cells closely to ensure the mitigation is effective and that the systems remain stable.

We are investigating a partial write outage affecting multiple metrics cells, beginning around 19:30 UTC. Some customers may see intermittent write failures or delays, but most requests should succeed after retries and recent metrics may appear late as a result. Querying previously ingested data remains unaffected. Engineering is continuing to investigate and will provide further updates as more information becomes available.