Tag Archives: programming

Python script to randomise an m3u playlist

While I’m blogging scripts for playlist manipulation here is one I use in a nightly cron job to shuffle our playlists so that various devices playing from them have some daily variety. All disclaimers apply, it’s rough and ready but WorksForMe (TM).

I have an entry in my crontab like this

which takes a static playlist and produces a nightly shuffled version.

Python script to add a file to a playlist

I have a number of playlists on Gondolin, which is a headless machine. I wanted to be able to easily add a given mp3 file to the playlists which are in m3u format. That means that each entry has both the filename and an extended line with some basic metadata, in particular the track length in seconds, the track artist and name. I wanted a script that could extract this information from the mp3 file and make adding the entry easy. So I wrote this in Python. It’s rough and ready and it is probably not very Pythonic but it’s working for me. The script should create a playlist if it doesn’t currently exist, and check for a newline at the end of the file so that the appended lines are really on a new line. ItWorksForMe (TM).

This uses the eyeD3 Python library, which on Debian is provided in python-eyed3.

My basic usage is

the last parameter is the path relative to which the mp3 filename should be written to. This is useful for me because I rsync the whole tree between machines, as you will see there are options for writing an absolute pathname if you prefer. I should probably rewrite the script to do it relative to the playlist, but that’s another day.

Migration from Savane to Redmine

I am admin for a server at work foss.ulster.ac.uk to host our open source development work. It used to run on GNU Savane, but despite several efforts, that project is clearly dead in the ditch.

So having to change the underlying system, I decided to move to Redmine (you can see some previous discussion here). I’m recording aspects of the migration here mostly for my own sake.

This install was on Debian Squeeze. I first of all installed the relevant package

and followed the prompts for the configuration. The documentation for the Debian install is a little unhelpful about how to actually configure the web server, and while I have good experience with Apache, I have very little with Ruby on Rails.

I installed the Apache Passenger module.

and copied the example config

I then edited the newly created redmine file to look like this:

In my case I wanted Redmine on the web root, so you can see the changes I made.

I then disabled the default config and enabled this:

and restarted Apache

Now you can login, with the default username and password (admin and admin) and change them and start some configuration.

Garbage collecting sessions in PHP

In PHP, sessions are by default stored in a file in a directory. Sessions can be specifically destroyed from within the code, for example when users logout explicitly, but frequently they do not. As a result session files tend to hang around, and cause the problem of how to clean them up. The standard way is to use PHP’s own garbage collection which is normally enabled out of the box. In this, we define constants that specify the maximum lifetime for the session, and essentially the probability of clean up.

To make things more interesting, Debian, out of the box doesn’t do garbage collection in this way. It has a cron job that regularly erases session files in the defined session directory. But, if like me, and many others, you put your session files in a different directory for each application to avoid clashes on namespaces for two applications running under the same browser from the same server, you have a problem. If you forget Debian’s behaviour the session files will just grow indefinitely. I had forgotten this issue and found over a year’s worth of session files in a directory recently.

Solving this problem is actually quite difficult to do optimally. I mean, I could create a cron job to mirror Debian’s own, but then I’d have to put the maximum lifetime in a cron job somewhere, out of the way, and difficult for the average sys admin I’m working with to find and deal with. (That is, away from the main configuration of the project). Or I could parse this value out of the main configuration. But this leads to another problem. For some users, a 30 minute maximum idle time is acceptable (although in my case where actually a suite of applications are being used as a single gestalt entity that can even be a problem), but for many of my administrator users you need huge idle times, since they are used to logging in first thing, and periodically working at the application through the day.

In the end I settled on changing our framework to make it easy to pass through garbage collection values. This makes an interface to the configuration really easy, but it doesn’t solve the problems of long session times that not all users need, and huge delays in garbage collection.

In my last article I talked about a Munin plugin for OPUS, but when you look at it you’ll see these kind of cliff fall drops, which are caused by the garbage collection finally kicking in and removing sessions where users have not explicitly logged out. Currently, every ten minutes, OPUS runs through its user database and finds users who are allegedly online but have no active session file and then marks them offline. Then it updates the file with the online user count that Munin reads.

I suspect eventually, I will write a more sophisticated script that actually kills sessions depending upon idle time and the user class, which would make for a more accurate picture here. Any brighter ideas gratefully accepted.

My first Munin plugin

Munin is a great, really useful project for monitoring all sorts of things on servers over short and long term periods, and can help identify and even warn of undue server loads. It is also appropriately and poetically named for one of Odinn’s crows (so I suppose I should have written this on a Wednesday).

We’ve been running Munin on one of our production servers at work for quite some time, and it gives us a lot of confidence that, to say the least, the server is running in its comfort zone around the clock. Among other bits and pieces, we run OPUS and the PDSystem on this box, two of our home grown projects that are available to the students. For some time now I’ve considered writing a plugin for OPUS to show logged in users, and I finally did this, albeit the counts are not nearly so reliable as I’d like for two reasons, but I’ll probably discuss that in another post. Anyway, I arranged for OPUS to drop a simple text file which simply contains counts of online users with the syntax

and so on, for each of the categories of users. Then I needed a plugin to deal with this. I decided to write it simple shell script, since its portable and I’m not much of a perl fan.

The plugin has now been running for several days, and you can see its output here. There are problems with it, but that’s more to do with PHP, Debian and user choice, and I’ll comment on that another time. However, already it gives me a useful feel for a lot of user behaviour.

Writing Munin plugins is easy, and Munin does so much of the hard work of turning your creation into something useful.

Geany and other Development Tools

I’ve tried lots of programming editors and ides over the years, obviously in Unix and Linux this is a Holy War, particularly between the advocates of vi and emacs. It is common for both groups to suggest that the other editor is hopelessly over-complex or clumsy. I think there’s some truth in that, because essentially, they both stink.

I tend to be an emacsen user myself, but I just think emacs is slightly less awful than vi. My first action on a new install is usually to use vi to edit my sources.list in Debian, to help me install emacs. Perhaps thats strange, because I really like sed. So what’s the problem with them? They both share this kind of puritanically awkward interface that works well on a console, but sucks in a GUI. They both use ridiculously arcane sequences of key presses to do anything, and I mean even basic stuff like saving and quitting. Yes, yes, you don’t have to lecture me about old terminals and their limitations, been there done that, got the t-shirt. I tend to do all my systems maintenance in emacs, but when I’m programming, I’ve started to love the softness of a decent editor that actually makes it plain and simple to edit multiple buffers of source code, even though its a pain to use different editors for console and gui work. Continue reading Geany and other Development Tools

Drupal Login Problems

So, in order to post that rant about PHP and SimpleXML I had to fix a problem that seems to have spontaneously arisen with Drupal (this content management system).

For some reason it wasn’t persisting login information, at least from firefox (sorry – iceweasel here on my Debian system). It’s interesting to note, reading about the bug, that it has been around for literally months and doesn’t seem to have been nailed.

So, anyway, I’ve installed some beta of Drupal, and yes, it now seems to be fixed… If I could only solve the problem that I can’t “uncollapse” parts of the content now.

UPDATE: OK, this seems to be a problem with firefox version 2, or probably really the CSS file for it. It works with Galeon, or when I tell firefox to fake being IE .

SimpleXML should be called BloodyAckwardXML

Another night of coding in PHP, and I’ve officially decided that SimpleXML utterly irritates me.

I’d already discovered, much to my irritation, that is virtually impossible to handle SimpleXML objects elegantly with the Smarty template engine – but now I discover I can’t even shove them in a PHP session without trouble – when you next visit the site you get stuff like this:

Warning: session_start() [function.session-start]: Node no longer exists

and then more trouble.

As part of a new Web Application Framework I’m working on I wanted to parse XML configuration files one time only, and then cache the results in the session. It looks like I now have to totally redesign my idea :-(. You can see the work in progress at its home page.