SSH startup problem

Recently I had a developer tell me that they could not access a development server through ssh, I had done the system patches, and then rebooted as normal. I then went to update other systems, and I should have checked the reboot for success. Luckily this was a VM so I had console access through the console.

I logged into the console to find sshd was not running, and would not start up. The error messages from the sshd daemon were very cryptic. The first clue I got from googling was that since one of the file systems was a btrfs file system, it may cause regular system boot to fail and not complete properly. So, I added another disk, make it ext4, and copied the btrfs file system contents to the ext4 file system, removed the btrfs disk, then rebooted. After console login again, the problem was still not resolved.

It turns out the symlink in /var/run was not owned by root:root as it should have been, it was owned by www-data:www-data...

root@tst-web ~ # ls -alrt /var/run
lrwxrwxrwx 1 www-data www-data 4 Jul 23 09:16 /var/run -> /run
root@tst-web ~ #

this is what it should have looked like..

root@tst-web ~ # ls -alrt /var/run
lrwxrwxrwx 1 root root 4 Jul 23 10:16 /var/run -> /run
root@tst-web ~ #

So, the answer was to remove the symlink /var/run

rm /var/run

And then recreate it as root:

ln -s /run /var/run

Then I restarted the system, and the sshd daemon was running and I could login again without using the console.