Our web server is currently reverted to the software state it was in on October 24th.
This is because a reboot was required after a software update this morning and upon rebooting it would not come back up. It didn’t even provide the normal grub menu.
I attempted to fix this with boot-repair but was unsuccessful. So I saved the database, so at least we wouldn’t lose work you’ve done on your sites, and restored from backup.
That too had problems although less severe and I was able to get the site operational.
I am now in the process of upgrading Ubuntu, after which I’ll upgrade the website and then fix various things that broke.
Taking the web server down to image prior to upgrading Linux in case something goes wrong. It will be back online in approximately 1/2 hour.
Nobody has used MxLinux this month at all. They are preparing a new release but it’s based upon an antique 4.19 kernel and an antique 1.20 version of Mate, and consequently I’ve decided rather than upgrading the machine I am just going to retire it and load some other version of Linux on it. If anyone WANTS to use MxLinux say something now before it goes away. If anyone has a favorite they’d like to recommend this is a good time to make your suggestions to email@example.com.
We had a brief outage around 7:35PM caused by one of our physical host servers spontaneously rebooting. I do not know what caused the reboot as there was nothing logged of the event.
We had a brief problem with webmail which resulted from my failure to add our own IP space to the ignoreip line of the fail2ban configuration in the new mail server. This has been fixed.
Because an upgrade exploded and left Ubuntu in a bad state, I need to restore it from backup and start over. It will be down for about an hour.
I’m going to attempt an upgrade of the web server to Ubuntu 19.10 tonight. This will include an upgrade to PHP 7.3, however, this is likely to break a few things initially. If web mail is not working late tonight, be patient. Typically PHP makes syntactical changes that requires the upgrading of some applications but it is impossible to know which in advance.
So what I am going to do is take it down for about half an hour this evening to make a current backup in case I can not get all the applications to work with 7.3 and we need to back out. Like most PHP upgrades, newer versions are faster and therefore preferable if we can get all the applications to work.
Machine is again running with full RAID.
The failed drive has been replaced. The RAID array is copying to the new drive, it will take about ten hours to complete.
I will be taking most services down around midnight for approximately two hours to replace a failed hard drive on the server that provides NFS service for everyone’s home directories.