On 2023-08-02, Atlassian customers were unable to perform a range of billing and cloud provisioning actions. It affected the Jira family of products, Confluence, Marketplace apps, and Bitbucket. The event started when in the normal order of operation, traffic to Atlassian billing systems increased and new nodes were supposed to be created. However, due to a prior rotation of passwords, new nodes were unable to connect to the Atlassian billing systems database. This meant that no new nodes were spun up to handle the increased traffic and the existing nodes became overloaded.
The incident was mitigated by resetting the passwords and new nodes were able to be created.
The overall impact was on 2023-08-02, on www.atlassian.com, my.atlassian.com, Billing Admin UI, Marketplace, and Bitbucket. The incident caused customers to be unable to use the aforementioned Atlassian billing systems in all regions. The outage lasted for 3 hours and 10 minutes and impacted the following products and services:
In-app billing admin UI for cloud accounts
Marketplace
Bitbucket
For reasons related to keeping our systems secure, passwords used for some of the Atlassian billing systems are rotated. However, the same passwords were used in other parts of the Atlassian billing infrastructure. This had the unintended side effect that it became impossible to create new nodes to handle increased traffic. Nodes that already were up and running kept functioning, but weren't able to handle the increased load.
We know that any outage impacts your productivity. We make every effort to keep all our systems up and running in a reliable and secure manner. Unfortunately, in this case, rotating passwords for security reasons had the unintended side effect of making our billing systems unavailable.
As a high-priority action, we have separated the passwords being used in different services across the billing systems. Furthermore, we are undertaking a comprehensive program to modernize the security infrastructure for our billing systems, significantly reducing the need to use passwords.
We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.
Thanks,
Atlassian Customer Support