We will be upgrading the mail server this weekend by applying the latest security patch. For the most part these upgrades are extremely smooth, only requiring about 20-30 minutes of downtime.
The upgrade will be performed Saturday or Sunday. If you have any questions about this please contact us.
This upgrade will have *no* effect on web sites or e-commerce.
Ever wondered the difference between a hash, mesh, thud and an octothorpe? There is none. It’s this: #
Did you know that an _underscore_ is also called a flatworm? It’s also called a skid.
Coding Horror has released an excellent article on ASCII Pronunciation Rules for Programmers.
Now you can sound smart about unpronounceable characters. Neat!
In the wee hours of Sunday morning April 17th something went wrong. But first, a little history and a quick lesson.
First a little Computers 101: Each file and folder have users and permissions. Users “own” the files/folders and permissions allow other users to take action such as reading, writing and executing on the files/folders. This is important for later.
Earlier this week, as we do every week, we applied security patches to our hosting servers. Usually this process goes so smoothly no one ever knows it’s happening or happened. The process was flawless and we went about our business.
Fast forward to Sunday morning at 4AM. That’s when our servers run through their weekly maintenance routines (cleaning up logs, clearing out caches, rotating logs). It usually lasts about 5 minutes and is a most unspectacular event. That is, unless there is a problem.
With the system security patch earlier in the week a tiny little bug was introduced when patching the module that handles all of the security routines of the web site. In human speak, it’s the thing that makes web addresses that begin with “https” secure.
When the upgrade was applied the “aliases” folder in the web root had permissions and ownership changed. During the maintenance routines the system tried to restart the Apache web server. With the permissions and ownership different on the “aliases” the server could not be restarted.
You may think this is kind of stupid that a silly permissions issue would prevent the server from restarting but it’s a good thing. We don’t want to grant access to everything on the server. That would be bad.
So Sunday morning we spent our time changing permissions and restarting the services. We apologize for the downtime and appreciate your understanding.
If you recently ran a yum update on CentOS 5+ and can no longer start Apache it’s because the mod_nss package changed the permissions and ownership of /etc/hosts/alias folder and it’s files.
You will see entries such as this in the error log (/var/log/httpd/error_log):
[error] NSS_Initialize failed. Certificate database: /etc/httpd/alias.
[error] SSL Library Error: -8038 SEC_ERROR_NOT_INITIALIZED
The fix is quick and easy. Run these commands as root (or sudo):
# chown root.apache /etc/httpd/alias/*.db
# chmod 0640 /etc/httpd/alias/*.db
Then run a /sbin/service httpd restart and you’ll be back up and running.
Edit: An updated version of mod_nss.x86_64 0:1.0.8-4.el5_6.1 has been added to the CentOS yum repositories. This apparently fixes the issue.