Wednesday, June 11, 2014

Ubuntu 14.04 init scripts fail and throw errors



I recently built out my first couple Ubuntu 14.04 servers at work and when my chef scripts tried to run they blew up all over the place. Chef was getting errors when trying to start or restart services like ssh and rsyslog. Looking a little deeper at the errors, Chef was executing init scripts directly and getting back an exit status of 1. For example when Chef tried to restart ssh it was running '/etc/init.d/ssh restart'

That script on Ubuntu 14.04 has no output and exits with a status of 1. I attempted to run the same thing manually from the command line on one of the servers and had the same result, no output and an exit status of 1. I did some searching and found other people are running into this same issue with various other services. I did find the command 'service ssh restart' would do the right thing and not throw an error. Since it seemed like this is an Ubuntu or Debian bug with the start scripts I decided to just modify my Chef scripts to use the service command instead.

By default Chef attempts run the scripts in /etc/init.d when starting and stopping services. The service resource in Chef has some attributes that let you modify how services are started. The attributes start_command, stop_command, restart_command and reload_command let you define an alternate command for these actions. Here are the changes I made to get my Chef scripts working again on Ubuntu 14.04.

Before
service "rsyslog" do
    supports :restart => true
    action [:enable,:start]
end

After
service "rsyslog" do
    restart_command "service rsyslog restart"
    start_command "service rsyslog start"
    supports :restart => true
    action [:enable,:start]
end

This change is backwards compatible with older versions of Ubuntu so I don't have to worry about special casing this just for 14.04 boxes.


[Update 1]
After reading a bit more I'm starting to suspect Ubuntu and/or Debian has purposely deprecated running the scripts in /etc/init.d to force people to use Upstart. Apparently these init scripts have been broken since Ubuntu 13.10.


[Update 2]
@retr0h gave me a cleaner way of accomplishing this:

service "rsyslog" do
  provider Chef::Provider::Service::Upstart
  supports :restart => true
  action [:enable,:start]
end

This does the same thing without having to define each command individually.


[Update 3]
@jtimberman informed me that this problem will be fixed in Chef 11.14. In that version Chef will automatically use Upstart for Ubuntu 13.10 and higher. (Chef support ticket) (Git commit)


4 comments:

  1. I had this exact same problem and I fixed it this way:

    https://gist.github.com/darron/f5559a399346671bb91f

    Not sure if that's better or worse than your solution - but I had to do that a few times with a few different daemons.

    ReplyDelete
    Replies
    1. Yep that is a cleaner way of doing it. @retr0h recommended the same thing on Twitter. I have added an update [Update 2] above with this info. Thanks!

      Delete
  2. I had this same problem while trying to restart the rsyslog service with chef 11.16.2.

    ReplyDelete
    Replies
    1. Looks like they implemented the change and then reverted it because of some other problems (https://github.com/opscode/chef/issues/1587). So this is still an issue and you will have to handle with one of the above methods for now.

      Delete

Please note all comments are moderated by me before they appear on the site. It may take a day or so for me to get to them. Thanks for your feedback.