Bitbucket has an increased error rate

Incident Report for Atlassian Bitbucket

Postmortem

SUMMARY

On September 25, 2023, between 10:07 UTC and 10:31 UTC, Atlassian customers using Bitbucket Cloud faced significant degradation to its website, APIs, and Git functionality. This event was triggered when our storage vendor experienced an outage at their data center. The incident was detected immediately by our internal monitoring, escalated to the vendor, and fully resolved after 24 minutes.

IMPACT

Customers who were impacted were unable to access the bitbucket.org website, APIs, Pipelines, and Git over HTTPS and SSH during the duration of the incident.

ROOT CAUSE

The incident was caused by an outage at our storage vendor’s data center. This affected our ability to communicate with our storage volumes, backing up our internal services and preventing them from continuing to serve requests.

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. We are prioritizing the following improvement actions to reduce recovery time, limit impact, and avoid repeating these types of incidents in the future:

  • Working with our storage vendor on preventative measures as well as improved detection on their end.
  • Improve resilience to reduce widespread customer impact when a subset of infrastructure is impacted.

We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support

Posted 1 year ago. Oct 04, 2023 - 00:50 UTC

Resolved

The issue has been resolved and all services are operating normally.
Posted 1 year ago. Sep 25, 2023 - 11:36 UTC

Update

We've identified the cause of the issue and remediated the impact, the Bitbucket Cloud team will continue to monitor the issue now the impact has been resolved.
Posted 1 year ago. Sep 25, 2023 - 10:42 UTC

Update

We are continuing to investigate this issue.
Posted 1 year ago. Sep 25, 2023 - 10:39 UTC

Investigating

The bitbucket team has detected an issue with multiple bitbucket cloud services, we're currently investigating the issue and will provide an update shortly.
Posted 1 year ago. Sep 25, 2023 - 10:27 UTC
This incident affected: Website, API, and Pipelines.