SUMMARY

On 2023-08-02, Atlassian customers were unable to perform a range of billing and cloud provisioning actions. It affected the Jira family of products, Confluence, Marketplace apps, and Bitbucket. The event started when in the normal order of operation, traffic to Atlassian billing systems increased and new nodes were supposed to be created. However, due to a prior rotation of passwords, new nodes were unable to connect to the Atlassian billing systems database. This meant that no new nodes were spun up to handle the increased traffic and the existing nodes became overloaded.

The incident was mitigated by resetting the passwords and new nodes were able to be created.

IMPACT

The overall impact was on 2023-08-02, on www.atlassian.com, my.atlassian.com, Billing Admin UI, Marketplace, and Bitbucket. The incident caused customers to be unable to use the aforementioned Atlassian billing systems in all regions. The outage lasted for 3 hours and 10 minutes and impacted the following products and services:

www.atlassian.com

• Could not signup for new cloud accounts
• Could not activate new products on existing cloud accounts
• Could not create, view or pay for quotes 

my.atlassian.com

• Offline

In-app billing admin UI for cloud accounts

• Could not view/update billing details
• Could not provide bill estimate
• Could not activate new products or apps

Marketplace

• Could not activate or deactivate apps on existing cloud accounts
• Vendors could not update their configuration or the configuration of their apps
• Some delay in updating vendor sales reports during the time of the incident

Bitbucket

• Could not update billing details
• Could not provide bill estimate

ROOT CAUSE

For reasons related to keeping our systems secure, passwords used for some of the Atlassian billing systems are rotated. However, the same passwords were used in other parts of the Atlassian billing infrastructure. This had the unintended side effect that it became impossible to create new nodes to handle increased traffic. Nodes that already were up and running kept functioning, but weren't able to handle the increased load.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that any outage impacts your productivity. We make every effort to keep all our systems up and running in a reliable and secure manner. Unfortunately, in this case, rotating passwords for security reasons had the unintended side effect of making our billing systems unavailable. 

As a high-priority action, we have separated the passwords being used in different services across the billing systems. Furthermore, we are undertaking a comprehensive program to modernize the security infrastructure for our billing systems, significantly reducing the need to use passwords.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support