Jira Outage History

We are aware of an issue that was impacting user authentication to Atlassian services between 6:10AM Tuesday 4th March UTC and 6:35AM Tuesday 4th March UTC. Users that were already logged in would not have been impacted by this issue. A deployment suspected of causing this issue was rolled back and the problem was subsequently resolved.

We have identified the root cause of the issue with an internal infrastructure component that has been impacting the provisioning process for the Mumbai Region with several Atlassian Cloud products. All affected products are now back online and no further impact has been observed.

We have identified the root cause of an issue with an internal infrastructure component that has been impacting the provisioning process for the Mumbai Region. This issue has led to a performance impact or slowness in product provisioning. We are working on a fix to resolve the issue and recovery is in progress.

The slowness issue has been resolved, and we are actively monitoring the systems to ensure stability.

Our team is currently investigating the issue. We are working to identify the cause and will provide updates as soon as possible.

Systems have recovered, and the issue has been mitigated. All affected products are fully operational, with no further impact observed.

We are investigating issues with slow loading and long responsive time for all products (Jira, JSM, JSW and Confluence) and will provide updates here soon.

Earlier today, we experienced RDS problems for Jira, Jira Service Management, and Jira Work Management. The issue has been resolved, and the services are operating normally.

We identified the root cause of the RDS problems causing issues for Jira, Jira Service Management, and Jira Work Management. We mitigated the issue and are now monitoring the fix.

We are observing different RDS problems in different environments for Jira, Jira Service Management, and Jira Work Management. We are actively working on this and will provide more updates here soon.

We are observing different RDS problems in different environments for Jira, Jira Service Management, and Jira Work Management. We are actively working on this and will provide more updates here soon.

We’ve taken steps to address the issue by scaling up the RDS instances, and we will continue to monitor the systems to ensure everything remains stable.

We are investigating issues with Latency and 5xx errors for Jira, Jira Service Management, Jira Work Management due to High RDS CPU on several EU-Central environment and will provide updates here soon.

Between 03-07-2024 20:08 UTC to 03-07-2024 20:31 UTC, we experienced downtime for some of the products . JIRA product impacted with issue creation and project creation. While Confluence was impacted with public link access by anonymous user and the mission control page. Confluence issue was fixed at 04-07-2024 02:30 UTC. The issue has been resolved and the service is operating normally.

We have mitigated the problem and continue looking into the root cause. The outage was between 8:08pm 03/07 UTC - 08:31pm 03/07 UTC We are now monitoring closely.

We are investigating an issue with that is impacting Atlassian, Atlassian Partners, Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira, Opsgenie, Atlassian Developer, Atlassian (deprecated), Trello, Atlassian Bitbucket, Guard, Jira Align, Jira Product Discovery, Atlas, Atlassian Analytics, and Rovo Cloud customers. We will provide more details within the next hour.

Between 05 June 23:30 UTC and 06 June 04:34 UTC, we experienced issues with some of the pages on www.atlassian.com. The issue has been resolved, and the pages are operating normally.

We continue to work on resolving the issue with www.atlassian.com. We have identified the root cause and expect recovery shortly.

We are investigating reports of intermittent errors on some Atlassian, Opsgenie, Atlassian Access, and Compass-related www.atlassian.com web pages. Once we identify the root cause, we will provide more details.

### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. ‌ Thanks, Atlassian Customer Support

We experienced increased errors on Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Trello, Atlassian Bitbucket, Atlassian Access, Jira Align, Jira Product Discovery, Atlas, Compass, and Atlassian Analytics. The issue has been resolved and the services are operating normally.

We have identified the root cause of the Service Disruptions affecting all Atlassian products and have mitigated the problem. We are now monitoring this closely.

We have identified the root cause of the increased errors and have mitigated the problem. We continue to work on resolving the issue and monitoring this closely.

We are investigating reports of intermittent errors for all Cloud Customers across all Atlassian products. We will provide more details once we identify the root cause.

Between 15:40 UTC to 15:57 UTC customers experienced intermittent failures when searching for users in Atlassian cloud services: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. The issue has been resolved and the service is operating normally.

We have mitigated the problem. We are now monitoring closely.

We are investigating an issue with our user search service that is impacting Atlassian cloud service: Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Product Discovery, and Compass. We will provide more details within the next hour.

We experienced degraded SCIM provisioning from external Identity Providers for Confluence, Jira Work Management, Jira Service Management, Jira Software, and Atlassian Access. The issue has been resolved and the service is operating normally.

A fix for the bottleneck identified in the Group synchronization process for SCIM Provisioning has been made. We are seeing processing start to return to normal levels and will be monitoring over the next few hours. The team is all hands on deck to keep improving the situation.

The bottleneck in the Group synchronization process for SCIM Provisioning has seen considerable improvement due to recent changes, but it is still under work. The team is all hands on deck to keep improving the situation.

The bottleneck in the Group synchronization process for SCIM Provisioning is still under work. The team is all hands on deck to improve the situation.

The bottleneck in the Group synchronization process for SCIM Provisioning is still under work. We are closely monitoring the service.

We have identified a bottleneck in the Group synchronisation process for SCIM Provisioning. We have increased the resources allocated to the process in order to mitigate the issue. We are now monitoring the service.

We are investigating cases of degraded performance when SCIM provisioning users/groups for Confluence, Jira Work Management, Jira Service Management, Jira Software, and Atlassian Access Cloud customers. We will provide more details shortly.

### **SUMMARY** On Dec 18, 2023, between 12:29 p.m. and 3:35 p.m. UTC, Dec 18, 2023, Atlassian's cloud customers using Atlas, Bitbucket Cloud, Compass, Confluence Cloud, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery products were unable to search for users or use the "@mention" functionality. Customers' user search results failed or were delayed as Atlassian's service returning user search results was degraded in several regions. The incident originated from a computationally intensive operation that was triggered multiple times in rapid succession, resulting in degraded performance of Atlassian's user search service across several regions. Notably, customers in the EU west region were most affected. The incident was detected within 2 minutes by automated monitoring, and our team promptly took action by recovering unhealthy systems and scaling up the service's infrastructure temporarily. The resolution process concluded in 3 hours and 06 minutes. ### **IMPACT** The overall impact was between Dec 18, 2023, between 12:29 p.m. UTC and Dec 18, 2023, 3:35 p.m. UTC. The Incident caused service disruption to cloud customers worldwide. Customers experienced delayed or failed user searches when using the following Atlassian cloud products: * Atlas * Bitbucket Cloud * Compass * Confluence Cloud * Jira Service Management * Jira Software * Jira Work Management * Jira Product Discovery ### **ROOT CAUSE** The incident stemmed from Atlassian's user search service receiving commands to process multiple computationally intensive operations in rapid succession. These operations were directed at the same customer data set, and therefore overloaded resources within a clustered database system, leading to memory exhaustion and subsequent unresponsiveness to user search requests.  ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** To prevent a recurrence of such incidents, we are implementing the following measures: * Implement a mechanism to queue computationally intensive operations in order to avoid overloading the resources within the systems and process them without impact on customer experience. * Fine-tune our clustered database settings to mitigate the impact of resource exhaustion on the overall system.  We apologize to customers whose services were affected during this incident; we are taking immediate steps to improve the service’s resiliency. Thanks, Atlassian Customer Support

It has been resolved. Atlassian's cross product user search is working.

Atlassian's cross product user search service is currently healthy. Searches for users within Atlassian products are working as expected. We are in the process of investigating the root cause of this incident.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

Atlassian's cross product user search service is recovering. Searches for users within Atlassian products are returning to normal.

We are investigating reports of intermittent errors for Atlassian, Confluence, Jira Work Management, Jira Service Management, Jira Software, Atlassian Bitbucket, Jira Align, Jira Product Discovery, Atlas, and Compass Cloud customers. We will provide more details once we identify the root cause.

Between 01:18 AM UTC to 03:31 AM UTC, we encountered a problem where some organisation admins were not able to access User Management on the Admin hub. We have identified the root cause of this issue and fixed it. Admin hub is functioning as expected now.

100% resolved and services running normally

99% of issues fixed and services are back to normal. Last 1% are manually begin fixed

We've solved 52% of the issues and are still on track to recover 100% within the original timeframe

We've solved 15% of the issues and have a plan to recover 100% exponentially within the hour

We started fixing the affected tenants, we will post an update on the estimated time to resolve all tenants in the next hour

We identified an issue with the product provisioning service. We identified the issue and we are working on a fix. We will post an update in the next hour with estimated fix time

Between 04:36 UTC to 17:50 UTC, we experienced intermittent issues on Atlassian.com and Bitbucket Marketing pages. The issue has been resolved and the service is operating normally.

We are still investigating intermittent issues with these two domains. We will provide more details within the next hour.

We are still investigating intermittent issues with these two domains. We will provide more details within the next hour.

We are investigating cases of degraded performance for Atlassian Cloud customers. We will provide more details within the next hour.

Between 04:00 UTC to 05:00 UTC, we experienced a partial outage for Bitbucket Pipelines, issues viewing in-product attachments; and degraded performance across all products. The issue has been resolved and all services are operating normally.

We have identified the root cause, initiated mitigation actions, and are seeing a reduction in errors when scaling up in response to customer traffic. We are still monitoring closely and will continue to provide updates.

A major infrastructure outage is affecting our ability to scale up in response to customer traffic. We are pre-emptively disabling any downscaling to minimize future impact. This is primarily affecting Bitbucket Pipelines, and attachment storage for all products. All regions are affected. We will provide more details in the next hour.

A major infrastructure outage is affecting our ability to scale up in response to customer traffic. We are pre-emptively disabling any downscaling to minimize future impact. This is primarily affecting Bitbucket Pipelines, and attachment storage for all products. All regions are affected. We will provide more details in the next hour.

We mitigated the issue with Sign-ups, Product Activation, and Billing, and the systems are back to BAU, and all functionality is restored.

We have identified the root cause of the Sign-ups, Product Activation, and Billing not working and have mitigated the problem. We are now monitoring closely.

### **SUMMARY** We understand the importance of providing reliable and consistent service to our valued customers. On July 6, 2023, from 03:52 to 15:11 UTC, we experienced an issue with an upgraded version of a third-party tool that functions as our internal artifact management system. Despite our monitoring system identifying the incident within two minutes, this issue led to the degradation of the scaling capabilities of our internal hosting platform, resulting in service degradation or outages for customers of Atlassian cloud. In response to this situation, we are taking immediate measures to enhance the stability of our system and prevent similar issues from re-occurring. ### **IMPACT** This incident affected multiple regions and products due to the diminished scaling capabilities of our internal hosting platform. In most products and offerings, customers faced reduced functionality, slower response times, and limited access to specific features. ### **ROOT CAUSE** The root cause of the incident was the introduction of new functionality in a third-party tool that functions as our internal artifact management system. It led to an unexpected increase in the load on the primary database of the artifact system. Upon identifying and localizing the problem, we promptly adjusted the system configuration to regain stability. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** Over the next months, we will enact a temporary freeze on non-critical upgrades of the artifact management system, and we will focus our efforts on three high-priority initiatives: 1. **Enhancing system scaling:** We prioritized work ensuring that downtime in a critical infrastructure component does not affect the scaling of other components. We expect to complete this initiative within the next two months. 2. **Reducing interdependencies:** We are working to mitigate the risk of potential cascading failures by ensuring that significant system components are able to operate independently in the case of issues. Initiatives 1 and 2 are already in progress but have been given priority to be completed as soon as possible. 3. **Strengthening testing procedures:** Alongside these initiatives, we are addressing the need for even more stringent testing procedures than we already have in place to prevent potential issues in future updates. We are committed to collaborating closely with our technology partners to ensure the most optimal experience for our customers. We apologize for any inconvenience caused by this incident and appreciate your understanding. Our team is dedicated to continually improving our systems and processes to provide you with the exceptional service you deserve. Thank you for your continued support and trust in us. Sincerely, Atlassian Customer Support

We experienced performance issues and outages for several Atlassian Cloud Products. The issue has been resolved and the service is operating normally.

We have identified the root cause of an issue with an internal infrastructure component that has been impacting multiple Cloud products, including Jira Software, Jira Service Management and Confluence, and customers. This issue had lead to a performance impact and, in some cases, outages. We have implemented a fix to resolve the issue and recovery is in progress.

We are investigating an issue with an internal infrastructure component that is impacting multiple Cloud products, including Jira Software, Bitbucket, Jira Service Management and Confluence, and customers. These issues include performance impact and, in some cases, outages. Users may experience slow loading and uploading of attachments, login issues or inability for new customers to sign up. We have identified the root cause and are actively working on the service recovery.

Between 05:15 UTC to 05:55 UTC, we experienced user registration failing. The issue has been resolved and the service is operating normally.

Between 2:05 UTC and 2:42 UTC, we experienced failures to load some marketplace apps for customers tenanted in the apse, eu-west and eu-central regions. The issue has been resolved and the service is operating normally.

We are investigating reports of intermittent errors for some marketplace apps across the following regions: apse, eu west and eu central. Service has now been restored to each of the affected regions. We are continuing to monitor, but believe there is no further impact to customers and Atlassian developers.

Between 15/Apr/21 17:20 AM PDT to 27/Apr/21 10:00 PM PDT, we experienced some cloud customers of Atlassian Support, Confluence, Jira Work Management, Jira Service Management, Jira Software, Opsgenie, Atlassian Developer, Trello, Atlassian Bitbucket, Atlassian Access, and Jira Align were logged out of there account. The issue has been resolved and the service is operating normally.

We are investigating an incident impacting Jira Cloud, Confluence Cloud, Bitbucket Cloud, and Statuspage. During our investigation, users may be logged out of their accounts as we work towards a resolution. We are continuing to investigate and will update this incident with more details as they are available.

Impact on User Management Syncs has been mitigated.

We continue investigating issues with syncing User Management changes to products. We will provide more details in an hour.

Between 20:20 UTC to 22:00 UTC, we experienced failures in user searches for Confluence, Jira Core, Jira Service Desk, Jira Software, and Bitbucket due to significantly elevated latencies, action has been taken and latencies have returned to an acceptable level.

We are investigating cases of degraded performance for the user search component affecting Confluence, Jira Core, Jira Service Desk, Jira Software, and Bitbucket Cloud customers. Some customers may experience delays and elevated latencies when making user searches.

Between 23:45 UTC to 02:19 UTC, some customers experienced intermittent failure connecting to Atlassian Cloud. The root cause was an increased DNS error rate from our infrastructure supplier. The supplier fixed the upstream issue and we have verified that the services have recovered. The conditions that caused the issue have been addressed and we are actively working on a permanent fix. The issue has been resolved and the service is operating normally.

We have identified the root cause of the intermittent failure connecting to Atlassian Cloud for some cloud customers and have mitigated the problem. We are now monitoring closely.

We have identified the root cause of the intermittent failure connecting to Atlassian Cloud for some cloud customers and have mitigated the problem. We are now monitoring closely.

We continue to work on resolving intermittent failure connecting to Atlassian Cloud for some cloud customers. We will provide hourly updates.

We continue to work on resolving intermittent failure connecting to Atlassian Cloud for some cloud customers. We will provide hourly updates.

We are aware that some customers may be experiencing intermittent failure connecting to Atlassian Cloud. We are working with our cloud hosting provider (AWS) to resolve the DNS related errors. We will provide hourly updates.

This incident has been resolved

Service has been restored, we are continuing to monitor the situation for any regression.

We are currently investigating an incident involving all purchasing systems. Currently all customers will be unable to: - sign up for new products. - receive quotes on or purchase market place add-ons. - provision new products or in some cases deactivate users.

This incident has been resolved.

We are investigating an issue with our avatar storage system that is impacting visibility of avatars across multiple services. We will provide more details as they're discovered.

Between 14:12 UTC to 16:13 UTC, we experienced increased errors rates for Access, Confluence and Jira customers connecting to our US east API endpoints. The issue has been resolved and the service is operating normally.

We are investigating an issue with Atlassians API Gateway in US East that is impacting some Access, Jira and Confluence Cloud customers. We will provide more details within the next hour.

We have identified the root cause of the custom sidebar colour issue with Jira Cloud and have mitigated the problem. The issue has been resolved and the service is operating normally.