Monitoring CoreOS with Rackspace Cloud Monitoring
Updated 8/5 to use systemd unit file.
Recently decided to move my blog to containers and CoreOS for learning and fun. While setting up an HAProxy container on one of my CoreOS hosts I thought about how I would monitor the host. Luckily, smart people have already thought about this. :)
I use Rackpace's Cloud Monitoring and agent which can be setup on any server in any datacenter or cloud provider. Rackspace hosted servers get a nice GUI and awesome dashboard for the checks and monitoring data, but you could still set it all up on a non Rackspace hosted server using the Cloud Monitoring API and raxmon.
The goods: https://github.com/jayofdoom/rackspace-monitoring-agent-coreos
git clone https://github.com/jayofdoom/rackspace-monitoring-agent-coreos
cd rackspace-monitoring-agent-coreos ./build_agent_container.bash mkdir -p /opt/rackspace-monitoring-agent tar -x -C /opt/rackspace-monitoring-agent -f rackspace-monitoring-agent-container.tar.xz
Next you'll register the agent so that it can interact with the control panel.
cd /opt/rackspace-monitoring-agent usr/bin/rackspace-monitoring-agent --setup
Enter your username and API key. If this is the first time setting up Cloud Monitoring on this server choose Create New Token. Next choose the entitiy, this is usually 1., but choose the option which matches your servers hostname and IP. If it's a non Rackspace hosted server you'll have to create a new entity and access the data from the API.
Please select the Entity that corresponds to this server: 1. core1.host - enhAOHzL6U private0_v4: xxxx public1_v6: xxxx access_ip1_v4: xxxx access_ip0_v6: xxxx public0_v4: xxxx 2. Create a new Entity for this server (not supported by Rackspace Cloud Control Panel) 3. Do not associate with an Entity
We can use the systemd unit file included in the repo to start the monitoring agent. When you registered the agent it should have created /etc/rackspace-monitoring-agent.cfg with some data Cloud Monitoring needs. We'll move this file to a location in the directory structure systemd-nspawn creates.
mv /etc/rackspace-monitoring-agent.cfg /opt/rackspace-monitoring-agent/etc/rackspace-monitoring-agent.conf.d/
I had to edit the systemd unit file to start the agent with the config file we just moved. Open the rackspace-monitoring-agent.service to edit. Notice I added the -c /etc/rackspace-monitoring-agent.conf.d/rackspace-monitoring-agent.cfg to starting the agent.
[Service] ExecStart=/usr/bin/systemd-nspawn -D /opt/rackspace-monitoring-agent --share-system --capability=all --bind=/sys:/sys --bind=/dev:/dev --bind=/dev/pts:/dev/pts --bind=/usr/share/oem:/mnt --user=root --keep-unit /usr/bin/rackspace-monitoring-agent -c /etc/rackspace-monitoring-agent.conf.d/rackspace-monitoring-agent.cfg -l /var/log/rackspace-monitoring-agent.log --exit-on-upgrade Restart=always RestartSec=30s [Install] WantedBy=multi-user.target
Now we'll copy the unit file and start it up.
cp rackspace-monitoring-agent.service /etc/systemd/system/ systemctl daemon-reload systemctl enable rackspace-monitoring-agent.service systemctl start rackspace-monitoring-agent.service
From your control panel you should see your real time values, and monitoring data at the dashboard. A few examples.
core1 ~ # free -m total used free shared buffers cached Mem: 7984 285 7699 0 4 144 -/+ buffers/cache: 136 7847 Swap: 0 0 0
This is just an example, as you probably know CoreOS is changing frequently so this solution could change at any time.