A GitLab Engineer Deleted the Production Database in 2017. A Copy Made for Load Testing Brought It Back.

Late on January 31, 2017, a GitLab engineer trying to repair database replication wiped the data directory on GitLab.com’s primary PostgreSQL database server. He thought he was working on the secondary. Within seconds he noticed and stopped the command. By then, in GitLab’s words, “Of around 300 GB only about 4.5 GB is left.”

When the team looked for a copy to restore, the database backups turned out to be missing, disk snapshots had not been enabled for the database servers, and the regular daily snapshot was almost a day old. On the first night, GitLab’s engineers wrote that “out of five backup/replication techniques deployed none are working reliably or set up in the first place.”

The copy that brought GitLab.com back had not been made as a backup. About six hours before the outage, the same engineer had taken a snapshot of the database by hand, because he was working on load balancing and wanted an up-to-date copy in the staging environment for testing.

How the wrong server got wiped

The trouble started that evening with a spike in database load. GitLab’s postmortem traced it to two things happening at once: spam, and a process deleting a GitLab employee’s account and data. The account had been flagged for abuse and scheduled for removal by mistake. The first incident report says spammers were “hammering the database by creating snippets.” Writes to the database locked up for a while. Then the secondary — a replica of the database kept ready to take over if the primary failed — fell so far behind the spike in writes that replication effectively stopped.

To rebuild replication, the team wiped the secondary’s data directory so that it could copy a fresh one from the primary. The secondary refused to connect. After configuration changes on the primary, the secondary simply hung. “At this point frustration begins to kick in,” the first incident report says.

Around 11 p.m. UTC (Coordinated Universal Time), the engineer suspected that pg_basebackup — the PostgreSQL tool that copies a running database to start a replica — was refusing to work because the empty data directory was still there. He decided to remove the directory on db2.cluster.gitlab.com, the secondary. He ran the command on db1.cluster.gitlab.com, the primary.

What each backup actually did

GitLab’s postmortem, published ten days later, goes through four of those procedures one by one.

Database dumps. A scheduled job was supposed to export the database with pg_dump, PostgreSQL’s export tool, and upload it to Amazon’s S3 storage. It used pg_dump 9.2 against a database running PostgreSQL 9.6, and the backup failed. “The S3 bucket was empty, and there was no recent backup to be found anywhere.” The job’s failure emails were rejected by the receiving server because DMARC, an email authentication standard, “was not enabled for the cronjob emails,” so the warnings never reached the team.
Disk snapshots. GitLab used Azure disk snapshots for its file servers, but “these snapshots were not enabled for any of the database servers,” because the team had assumed that its other backup procedures were enough.
Replication. The secondary’s data had already been wiped during the repair attempt. Replication was “primarily used for failover purposes and not for disaster recovery.”
Daily LVM snapshots. GitLab used LVM, the Linux tool that manages disk volumes, to take a snapshot once a day and copy production data to staging. The process worked as intended, but the snapshots were “not really meant to be used for disaster recovery,” and the regular one was almost 24 hours old.

The most recent copy was the engineer’s manual LVM snapshot from about six hours earlier, sitting in staging. Copying it back to production “took around 18 hours.” The bottleneck was the staging disks, which ran on Azure classic storage without Premium Storage and were throttled to “around 60Mbps.” In all, GitLab.com was down for about 18 hours.

What was lost, and what wasn’t

The restore brought back the database as it stood when the snapshot was taken. Changes made in the roughly six hours after that — new projects, comments, user accounts, issues and snippets — were lost. GitLab’s “best estimate is that it affected roughly 5,000 projects, 5,000 comments and 700 new user accounts.” Git repositories and wikis, stored separately, were not affected by the data loss.

Some retellings make the damage sound worse. A 2023 post on the developer site DEV Community says the deletion left “GitLab.com with no data at all.” GitLab’s own record shows about six hours of lost database changes, about 18 hours of downtime and Git repositories that were not affected.

How GitLab handled the outage in public

GitLab kept its recovery notes in a public Google Doc and streamed the restore on YouTube. The stream was “the #2 live stream on YouTube for several hours,” with “a peak viewer count of around 5000.”

The postmortem does not name the engineer. The live document did, and GitLab wrote that it would “redact names in future cases as other engineers may not be comfortable with their name being published.” Its CEO wrote: “I apologize personally, as GitLab’s CEO, and on behalf of everyone at GitLab.”

What GitLab found and changed

The postmortem’s root-cause analysis points at the process, not the person: “Because there was no ownership, as a result nobody was responsible for testing this procedure.” After the outage, LVM snapshots were taken every hour instead of once a day. The postmortem lists 15 follow-up issues, including automated restore tests, an owner for data durability and a change to the shell prompt on every host to make hosts and environments easier to tell apart. GitLab said its main focus was “to improve disaster recovery, and making it more obvious as to what host you’re using; instead of preventing production engineers from running certain commands.”

Why the broken backups went unnoticed

Each safety net fell short in its own way, but the gaps had one thing in common. The dump job produced nothing, and its warnings bounced. The disk snapshots had not been enabled for the database servers. Nobody owned the backups, so nobody was responsible for testing them, and the gaps stayed hidden until the night a restore was needed.

A backup nobody owns is a backup nobody tests.

In the end, GitLab.com came back from a copy the same engineer had made for load testing, about six hours before the command he stopped within seconds. The S3 bucket that should have held the database backups was empty.

Explore

Stay connected