As you may have noticed, our wiki and issue tracker were unavailable from Thursday to Sunday last week. What happened?
We host parts of our infrastructure at the Open Source Lab at Oregon State (OSUOSL), including the databases for these two services. So far, there’s no post mortem by OSUOSL (they expect to post one later this week), so we need to piece together what we know.
The databases for the wiki and issue tracker became inaccessible around midnight/12 AM Thu/Fri night (all times UTC). Due to the large number and size of databases on that server, pulling from backups, restoring from backup and replaying the binlogs took them quite a while. During that time, we put up a maintenance screen on the wiki (and messed up the one for Jira, so there were connection timeouts instead).
The databases were back around 3 AM on Sunday. We disabled the maintenance screens around 6 PM later that day.
While this was a rather lengthy outage, it could have been much worse. We lost none of the data, after all. We thank the OSUOSL team for their efforts getting everything back up over the weekend!
|
Daniel Beck
Daniel is a Jenkins core maintainer and member of the Jenkins security team. He was the inaugural Jenkins security officer from 2015 to 2021. He sometimes contributes to developer documentation and project infrastructure in his spare time. |