Categories
Uncategorized

WordPress Auto-Update System Misfires, Updating Live Sites to an Alpha Release

WordPress’ Core systems team had an eventful Friday when an error in the auto-update system caused sites to update to WordPress 5.5.3-alpha-49449, including live production sites with no auto-update constants defined.
I just had a random production site auto-update itself to #WordPress 5.5.3-alpha 😨 Anyone else seeing this?
Those who received an email about the update logged into their sites to see the message: “BETA TESTERS: This site is set up to install updates of future beta versions automatically.” 
Shaun Rieman logged the first ticket about sites being updated to 5.5.3-alpha-49449, which was also incidentally his first WordPress trac ticket. More users and developers confirmed the issue.
“It’s worth noting that there’s no functional difference between 5.5.2 and 5.5.3-alpha, so there’s no need to worry in that regard,” core committer John Blackbourn said.
Sites that were accidentally updated also installed all the default themes released over the last decade, as well as Akismet. Developers will need to manually delete the bundled themes that they don’t need.
In under an hour, all affected sites were automatically returned to 5.5.2, but the incident has eroded trust and damaged confidence in the auto-update system. Several commenting on the ticket asked how they can explicitly disable development updates.
“The worrying thing is that a single developer can do this, seemingly without any checking or confirmation by other developers,” UK-based developer Paul Stenning said. “This is a serious security concern as a rogue developer could push out malicious code in an update that nobody else checks.”
WordPress agency owner Rob Migchels, who had approximately 50 websites affected, tracked 18 minutes between the the trac ticket (#51679) and receiving the fix.
“The 5.5.3-alpha issue is a side effect of another issue that occurred on 5.5.2,” WordPress engineer and 5.6 Triage release lead Tonya Mork said. Jake Spurlock published an official statement regarding the incident as part of the 5.5.3 release notes:
“Earlier today — between approximately 15:30 and 16:00 UTC — the auto-update system for WordPress updated some sites from version 5.5.2 to version 5.5.3-alpha. This auto-update was due to an error in the Updates API caused by the 5.5.3 release preparations.”
Spurlock elaborated on the technical details in a separate post on the WordPress development blog:
While work was being done to prepare for WordPress 5.5.3, the release team attempted to make 5.5.2 unavailable for download on WordPress.org to limit the spread of the issue noted in the section above, as the error only affected fresh installations. This action resulted in some installations being updated to a pre-release “5.5.3-alpha” version.
In a situation like this, where users who haven’t elected to run their live sites on beta releases are getting a forced update, site owners might wonder whether this update is actually arriving from WordPress, or if the system has been hijacked.
Security researcher Slavco Mihajloski, who commented last week on the lack of transparency regarding how automatic updates are tested and performed, said this incident highlights the need for more openness surrounding the process.
“Why is transparency important? Because procedure will become public and when public, the community will be able to contribute in order to improve it,” Mihajloski said. “At the moment it is more than obvious that this process and the whole security at WP.dot org lacks QA and QC. Each task is left to an individual or closed group. Imagine the following: what if an automatic security update could be pushed only if: – X out of Y (where X < Y) authorities agree that update is fine – have a pilot update on let’s say 100 different servers (I hope .org could afford this) where regression tests will be fired against each one. The current problems would not occur.”
Automatic background updates for minor releases have saved developers thousands of hours in updating sites. A UI for allowing users to opt into automatic updates for major releases is on the roadmap for WordPress 5.6, expected in early December.
This particular accidental update has betrayed for many developers what was already a somewhat fragile trust in the auto-update system. It doesn’t shore up more confidence for selling the idea of core updates when 5.6 is released, but it doesn’t mean that auto-updates are not a good idea. WordPress.org will need to put better processes in place in order to win back users’ trust.
The incident affected more than 100 sites for WordPress agency owner Robert Staddon. He reports that they all displayed the “Update now” button with the confusing and incorrect text seen below. Staddon said the incident has not yet caused him to change his approach to allowing clients to receive auto-updates.
“I was very grateful for the extraordinarily fast response time to get the problem fixed,” Staddon said. “However, it did shake my confidence in the WordPress auto-update process. Considering the number of websites using WordPress, a mistake of this magnitude could end up having a rather catastrophic effect around the web. I would hope that the core team would be able to evaluate how this happened and consider putting some checks in place to make sure it doesn’t happen again.”
Perfect example to turn off ALL automatic updates.
Do manual updates, in case something goes wrong and breaks your site…you can fix it if you are in front of your computer.
Report
Unless of course you’re responsible for managing hundreds or thousands of websites for your customers. Then you need something else.
Report
If an automatic update screws up hundreds or even thousands of your clients sites………..if you are there doing the updates, you can start right away to fix the issue(s).
That is what your clients pay you to (the ones that pay for maintenance), for YOU to maintain them, they don’t pay WordPress to update itself. This is a general you, not you specifically.
I have had to update hundreds of sites at once. It took me a while but I did it in under 24 hours.
My clients pay ME to update not WordPress.
I think it’s lazy to charge clients for maintenance and then let the automatic updates do it for you.
Report
Hey Sarah, thanks for writing up a synopsis about the eventful day that we had today! I just wanted to offer a few corrections, as user trust is incredibly important, and we went into all changes today with user trust at the front of our minds.
WordPress’ Core systems team had an eventful Friday when an error in the auto-update system caused sites to update to WordPress 5.5.3-alpha-49449, including live production sites with no auto-update constants defined.
Auto-updates do not need constants defined, or special settings to receive auto-updates on minor releases. This is a core feature of WordPress that makes it possible to dole out features to millions of sites over the hundreds of releases over the last few years. With the 5.5.2 release on Thursday, we auto-updated code to 19 versions of WordPress. All of the way back to 3.7!
The primary issue that led to today’s release stems from the way that WordPress handles database connections. We added a change to the 5.5.2 release, and then after that release, an issue was reported that installations were failing. With failing installations in mind, the release team jumped into action knowing that if installs are prevented, that’s obviously a huge issue, and we aimed to resolve this as soon as possible.
To slow the spread of this problematic update that was already rolling out, we wanted to change the API to prevent new downloads while we fixed the problem and rolled out a new minor release. The update API was rolled back by the systems team to 5.5.1 as a stop-gap.
Unexpectedly, it caused this 5.5.3 alpha state for a small subset of sites that were connected to the update server during this window of about 24 minutes. After realizing the problem, a change was made to pin the version and stop the updates.
It was after this change that the reports started coming into Trac and the WordPress.org slack. Obviously, this was an unintended result of the change. Triaging the issues, both in the forums and in Trac, was carried out quickly.
The minor release team returned focus to the pressing issue around installations and they were able to pull the release together. The change was backported from WordPress 5.5 to WordPress 5.1, fixing all sites that would be affected, both with the alpha status, and the installation issue.
Ideally in testing, both user and automated, we would have caught this issue ahead of the release. That’s not what happened, but I am incredibly proud of how the WordPress core team was able to roll out a fix successfully to the millions of sites that connect for updates.
Report
Thank you for the clarifications!
Report
So now we have to go in and delete themes that were updated that weren’t even on the site anymore?
Is there going to be an update? I have like 15 sites that auto updated which I need to turn off auto updates on.
Report
I always hated havinga cluttered Themes folder and having to delete each one again after each update, even a minor one. I don’t know why WP continues to enforce default themes when most people don’t use them. I don’t need that clutter, along with the Hello plugin & Akismet.
Report
The update is already out. 5.5.3 fixes the issue.
If your site happened to update within that 24 minute window, then you may have gotten some extra themes installed. You can remove them if you don’t want or use them. They’re not active and don’t do anything.
Report
… except that Site Health and similar tools will point out that having unactivated themes (and plugins) is a security risk and that you ought to delete them. My sites weren’t affected as I find no additional themes – which makes sense, you can’t auto-update one-third of the world’s websites in 24 minutes. Probably.
Report
This is generally true and decent advice, however, these are the core themes, and have no known security issues in that respect. Simply having them there is not a risk. Direct access to files is not usually a concern for themes, as the various templates need the underlying WP engine to be running for them to do anything.
Report
I wonder how this will affect the development and rollout of the auto-update functionality on themes/plugins front moving forwards, as that was already a fairly divisive feature as-is. Plugins I can kind of agree with (again, for minors not majors), but if WP core can’t be securely maintained… does not bode well?
Furthermore, how does this affect the even more divisive “major auto-updates” feature that was in discussion a short while ago? If minors can’t even be securely managed, surely handing over majors is a no-go?
Fuel for the “don’t do it” camp (of which, I admit I was a part of)?
Report
Plugins and themes use a separate prices than core, and so are not affected by this at all, really. For plugins, you can turn on auto updates yourself as of 5.5 and that’s pretty much where that will remain for a while. There are no plans to start auto updating those outside of that switch functionality.
For themes, auto updates are different and not planned yet at all, because of the inherent differences there. One would need to detect changes to themes first, since not everybody uses child themes, and that needs a lot more thought. Themes are far more prone to modification.
Report
Agency owner here – 50+ websites affected. I appreciate Jake providing more details on what happened but it makes me nervous of how these updates could go wrong when auto update is on.
Report
Thanks team for great response and diligence!
Report
Nice to see more people beginning to really think about the “Automatic Updates”. I warned against them before 3.7 was released. Back then they used the term “Bozo’d” for shadow banning and that’s what happened to me for disputing with Nacin.
Told ya so!
All it takes is for one rogue developer or someone not paying attention for something to happen to a whole lot of websites. Turn off automatic updates.
Report
It’s concerning that there wasn’t more oversight about this. With one site on WordPress.com and the other on WordPress.org, I keep thinking about moving the .com site to a self-hosted .org for all the advantages, then I remember it’s such a headache when things glitch.
Report
Your email address will not be published. Required fields are marked *









This site uses Akismet to reduce spam. Learn how your comment data is processed.
Enter your email address to subscribe to this blog and receive notifications of new posts by email.


WordPress Tavern is a website about all things WordPress. We cover news and events, write plugin and theme reviews, and talk about key issues within the WordPress ecosystem…read more →
Proudly powered by WordPress.

source

Leave a Reply

Your email address will not be published. Required fields are marked *