There were lots of good questions (which I wish I had written down), including how to use it with iTunes, how to use it as a replacement for logrotate, how I was using Vagrant for testing, how to automate more than set of rules, etc.
Takeaway: Debugging rsyslog and logrotate is easier when using Vagrant and logrotate‘s debug switch (-d).
I’ve been getting my hands greasy with more involvement in Ubuntu server maintenance lately, so I thought I’d share some of my experiences with debugging rsyslog and logrotate.
We’re using an EC2 instance running Ubuntu 12.04 (Precise Pangolin) to receive syslog packets from Heroku and elsewhere. They’re received with rsyslog, and we’re using logrotate to rotate and compress the log files. We’re being a little lazy right now and just letting everything go into /var/log/syslog. So far, that’s been a good choice as the machine only handles log aggregation. Most of the setup is similar to what Heroku recommends.
My first bump in the road was that which Ubuntu 12.04 doesn’t give rsyslog port 514 (the standard syslog port), unlike Ubuntu 10.04. This is because 514 requires root privileges. As a result, the port has to be above 1024, so I chose 1514 to make it easier to remember and know its purpose. It’s not completely clear why Ubuntu changed rsyslog to run under its own user (I would guess security reasons), but there is a bug report which helped me figure this out.
Following that I ran into an issue with getting logrotate to rotate the log files in the way we wanted. We decided that storing 30 days worth of logs, marking the rotation with the date, and compressing older logs would be best for our needs. It sounded easy enough to configure after looking at man logrotate and /etc/logrotate.conf. But it didn’t work! No matter what I tried, only 7 days of logs were retained, although compressing and adding a date extension worked fine.
It turned out to be a simple problem, but it’s more important to know how to debug these problems, I think. Below is what I did.
You probably don’t want to debug configuration problems directly on your production instance, or even on a staging instance. To develop the changes locally, I found that the easiest solution was to use Vagrant. They provide a precise64 box which was perfect for my needs:
From there, you can test logrotate with the -d switch. Point it to your configuration file and then see what it says it will do:
logrotate -d /etc/logrotate.conf
The problem behavior was clearly visible in the output; /var/log/syslog was only being kept for 7 days. Changes to /etc/logrotate.conf did not make a difference for the rotation count (but I could change to doing dateext). Around that time, started poking around in /etc/logrotate.d/, where I found /etc/logrotate.d/rsyslog.
This is the original configuration for /var/log/syslog:
After that point, it was simple to try my changes and retest using logrotate as above:
# NOTE: `dateext` means you can rotate at **most** once a day. Be cafeful setting it globally.
reload rsyslog >/dev/null 2>&1 || true
The debug output then showed it would retain the logs for 30 days. Great! It was then a simple matter of installing the same configuration in production.
While the result is important, I think the bigger takeaway is a process I can use in the future. With Vagrant and logrotate -d /path/to/conf in hand, future configuration changes are easier to develop, test, and deploy.
I’m what you might call an “Iowa City enthusiast”. As residents, we are extremely fortunate to be surrounded by an incredible number of people who work every day to make this a better place to live.
Even so, Iowa City has its blemishes. When I first relocated to study at the University of Iowa, I was taken aback by how hard it is to use a car here. Where I used to live, I was used to driving everywhere and not having to worry much about parking. Iowa City has a famously pedestrian downtown and it can sometimes be at odds with car ownership in general.
But we all have places to be, don’t we? Cars give an amazing sense of freedom and independence, but as many have found, they also can be amazingly expensive and often inconvenient.
I just moved back to Iowa City, after living in New Haven, CT because of a job at the Yale School of Medicine. While I was there, I was happy to find a great car-sharing service called Zipcar that included a Yale-subsidized membership. I even lived there for a year without owning my own car. When we decided to move back, I was happy to learn that Zipcar was coming to Iowa City soon after we arrived. If only it had been here when I was a student!
I still work for a company based out of New Haven, CT (they’ve been gracious enough to let me work remotely from Iowa), and I often travel for work and open source conferences. It’s great to know that I have a car waiting for me downtown Iowa City if I need it, in New Haven two blocks from my office, in Philadelphia and New York if I get stuck, and in many cities where I might visit. That’s not bad for a small yearly membership fee and then $8/hr. That price includes gas, maintenance, and auto insurance too.
We do still own a car, but with Zipcar and everything else Iowa City has to offer, we’ve been able to delay the purchase of another. That helps reduce traffic congestion in Iowa City, saves us money, reduces our environmental footprint, and reduces the number of car related headaches we have. To me, that’s a great example of a win-win scenario.
I’ve seen some of the experimenting that people have done with bringing legal code (i.e., laws) to GitHub. As an open source developer, I think Clay has it right when he says that transparency is only one side of openness and that citizen created bills are (hopefully) inevitable. After all, our government is too important to leave exclusively to politicians.
Or, you know, the Internet will just be a place we go to see funny pictures of cats. It’s what we make of it.
Update: Just as an experiment, I made a repository for the laws of Iowa because I couldn’t find an existing one. I’m experimenting to see if any others are interesed as well, based on star count or contributions.