Extra webhooks are being delivered for pull request updates
Incident Report for Atlassian Bitbucket
Postmortem

On June 17, 2021, the Bitbucket Cloud support team received reports from a few customers that an excessive number of webhooks were being sent when they merged pull requests. These webhook events were for all open pull requests targeting the same branch (e.g. the default branch).

As part of ongoing efforts to improve reliability and address scaling limitations, one project that Bitbucket Cloud engineers are working on currently is to dramatically reduce the usage of locking to synchronize concurrent repository access for larger teams. Architecturally, locking served Bitbucket well earlier in the product’s maturity when most teams were smaller; but changing usage patterns over time have highlighted the downsides to this approach and illuminated the need for new, lock-free implementations of functionality that previously relied heavily on synchronization.

One example of this is a new lock-free implementation of the code responsible for updating affected pull requests whenever changes are committed to a repository. This new implementation is behind a feature flag and has been tested on a small number of internal repositories over the past few weeks. On June 17 the team deployed a change to address a minor bug in this implementation where the necessary webhooks were not delivered to trigger pull request Pipelines builds when pushing to the source branch of an open pull request. While the lock-free implementation is behind a feature flag, the change intended to fix the bug affected a shared code path that is not exclusive to the new implementation.

As an unintended consequence of this change, Bitbucket’s services started delivering webhooks for all pull requests updated as the result of a merge. Previously, in this scenario open pull requests' diffs and inline comments would be updated transparently as “internal” updates without notifying webhook listeners; so this was a breaking change.

The patch for this bug was implemented and deployed to all Bitbucket services on June 18, 2021.

Posted Jun 20, 2021 - 18:48 UTC

Resolved
Starting June 17, a bug was introduced that caused extra webhooks to be delivered for all open pull requests when a pull request was merged. This issue was resolved on June 18.

(This incident is being created retroactively in order to publish a postmortem.)
Posted Jun 17, 2021 - 17:33 UTC