Bitbucket Cloud web, api, and Pipelines service outage

Incident Report for Atlassian Bitbucket

Postmortem

Summary

On January 21, 2025, between 14:02 and 17:49 UTC, Atlassian customers using Bitbucket Cloud were unable to use the website, API, or Pipelines. The event was triggered by write contention in a high traffic database table. The incident was detected within eight minutes. We then worked to both throttle traffic and improve query performance, which allowed services to recover. The total time to resolution was about three hours and 47 minutes.

IMPACT

The overall impact was between 14:02 and 17:49 UTC, affecting Bitbucket Cloud. This impacted customers globally, and they were unable to use the website, APIs, or Pipelines services. Git hosting (SSH) was unaffected.

ROOT CAUSE

The issue was caused by an increase in API traffic triggering write contention on a high-traffic table, resulting in increased CPU usage and degraded database performance. This ultimately impacted the availability of core services (web, API, and Pipelines).

REMEDIAL ACTIONS PLAN & NEXT STEPS

We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified because the code path being triggered does not commonly experience this type of traffic.

We are prioritizing the following improvement actions to avoid repeating this type of incident:

Running additional maintenance on core database tables
Added throttling on write-heavy operations

To improve service resilience and recovery time for our environments, we will implement additional preventative measures such as:

Improving database observability to isolate failures
Continuing to shard data to better distribute traffic load

We apologize to customers whose services were impacted by this incident and are taking immediate steps to improve the platform’s performance and availability.

Thanks,

Atlassian Customer Support

Posted Jan 28, 2025 - 18:13 UTC

Resolved

Earlier we experienced database contention on high traffic tables, which resulted in website, API, and Pipelines outages. All Bitbucket services are now operational. A full post mortem will be published.

Posted Jan 22, 2025 - 00:08 UTC

Update

All Git, Web, API and Pipelines services are now operational. We are continuing to monitor database and Pipelines reliability.

Posted Jan 21, 2025 - 20:08 UTC

Update

We have identified the root cause of the database issue that impacted Bitbucket website and Git operations; this has been mitigated now. We are experiencing Pipelines degradation that we are working to resolve.

Posted Jan 21, 2025 - 19:28 UTC

Monitoring

We have identified the root cause of the database issue and have mitigated the problem. We are now monitoring closely.

Posted Jan 21, 2025 - 18:02 UTC

Update

We are investigating an issue with saturated Bitbucket database that impacts all Bitbucket operations. We will provide more details within the next 30 minutes.

Posted Jan 21, 2025 - 17:49 UTC

Update

We are investigating an issue with saturated Bitbucket database that impacts all Bitbucket operations. We will provide more details within the next hour.

Posted Jan 21, 2025 - 17:07 UTC

Update

We are still investigating an issue with Bitbucket Web and Git operations that is impacting Atlassian Bitbucket Cloud customers. We will provide more details within the next hour.

Posted Jan 21, 2025 - 15:58 UTC

Investigating

We are investigating an issue with Bitbucket Web and Git operations that is impacting Atlassian Bitbucket Cloud customers. We will provide more details within the next hour.

Posted Jan 21, 2025 - 15:30 UTC

This incident affected: Website, API, Git via SSH, Authentication and user management, Git via HTTPS, Webhooks, Source downloads, Pipelines, Git LFS, Email delivery, Purchasing & Licensing, and Signup.