Proving the Obviously Untrue

I'm considering changing the development environment we use to host our projects in work and I'm not alone. The reason why will become more clear further down. To be honest this article is partly to help me work out what my priorities for this are, and to appeal to the lazy web for any other suggestions.

So first of all, here are the things I'm looking for in this project.

These are essential:

1. must be free software;
2. must have flexible issue tracking with user defined fields;
3. must be able to handle multiple projects;
4. must be extremely transparent to non developer users;
5. must support svn and git and graphical front ends to them;
6. must support some kind of announcement system.

These are desireable:

1. should integrate in some way to mailing lists;
2. should support ad hoc tar bar downloads from scm;
3. should be easy to search issues;
4. should graphically depict progress on releases;
5. should have documentation (wiki like) integration;
6. should handles news and download areas;
7. should be able to exchange data (like issues) on projects;
8. should actively maintained;
9. should automate most sysadmin activity, account creation etc.;
10. should be packaged for Debian ideally.

The contenders thus far are:

• Savane, which we are currently using;
• Fusion Forge;
• Redmine;
• Trac.

Let's take these one at a time.

Savane

Savane currently supports all the essential features, with the exception of git support. Being a fork from the original SourceForge software it works in a similar way. A web front end (in PHP), with back end functionality (in Perl) that creates shell users and groups, interfaces with mailman and so on.

But there are some problems. First of all the official project seems to have ground to a complete halt. Fixes that were submitted by users are not applied, there seems to be no forward momentum whatsoever. This is a problem, the offical version needs patched (modestly) to run on PHP 5, and the code base is a mess, with a few pages still requiring register_globals to be on. Bad.

However, another fork was taken some time ago which addresses all these problems including git support, well actually, the code base is still probably in need or work, but most apps of a certain age have this problem. What I'd like to see in Savane is better graphical tools to monitor project progress, better documentation features and incidentally better Debian packaging, but I've signed up to the fork to work on the latter on the first instance (this looks like it will be quite a bit of work, it's non Debian policy compliant in many ways right now). The new fork is going to be forked again by the way, into a Python version. Could be interesting.

Fusion Forge

Previously known, or descended from GForge, Fusion Forge is another fork of the original SourceForge source code. It is well packaged for Debian (naturally, it runs Debian's own code hosting environment, Alioth) and well maintained. It has many of the features of Savane, which probably makes it puzzling why I didn't choose it in the first place. Well, there's one reason why, in my opinion the front pages of FusionForge are rather un-user friendly, I mean what the heck is "Code Snippets" doing there? They feel very aimed at developers, which is great... but I need a nice straightforward interface for less savvy users. Looking today, the navigation still feels it's just too developer centric, but as these sites double as developer sites and user sites (to acquire the software, report bugs and so on), that's not great. I really don't know yet whether Fusion Forge has better graphical tools than Savane.

In a conversation about this last week, Noodles suggested I should look at the ease with which the interface could be changed. A sensible suggestion I shall follow up.

Redmine

Written in Ruby on Rails and advocated by my work colleage Paul Vitty, Redmine is clean and elegant looking. It shows all the signs of benefiting from being a later generation project and has a plugin architecture that seems excellent too. It has easy clean wiki integration, graphical road maps and Gannt charts; and these things make a difference. I don't think it's as remotely scalable as the design of the first two systems, which isn't a huge problem for me, but I mention it in passing.

On the other hand, it seems to have little or no automated integration into the backend, user and group creation, mailing lists and so on. on the other hand it has a very rich set of plug ins, so it may already have such support or it may be possible to implement it.

Trac

Written in Python, Trac feels like Redmine lite, and that's in the wrong direction for me. It explicitly does not have multiple project support and that's a deal breaker for me.

I haven't come to a strong conclusion about this so far. I note John (whom I linked above) notes that Redmine is difficult to install and upgrade because of Ruby, this is a hassle I just don't need, but he also thinks it may be the best of the field. For me, it is probably more trouble free to migrate to the fork of Savane or to FusionForge. More thought needed, and your thoughts are most welcome!

Posted by Colin Turner

New Scientist recently ran an article about steganography. If you don't already know, steganography is essentially encryption with a difference. Specifically, encryption is usually obvious. It may be that the data Alice sends via email to Bob with public key cryptography is entirely secure from eves-dropping by Eve (pun intended, sorry), but Eve will know data is being sent that she might be interested in. Steganography, by contrast, seeks to hide the encrypted data so Eve is not aware of its very existence.

It's a very ancient idea, stretching back to ancient Greece. In modern times a common way to perform the trick is to hide data in an image. One of my more gifted undergraduate students did a final year project on this with me. We used a known password as a seed for a pseudo-random number generator to determine which pixels of the image we would embed the data in. By playing with the least significant bit of one colour in randomly spaced pixels, you can very effectively hide data.

The New Scientist article suggests that if you detect the steganography, and if we obtain the computer of the suspect and if they have carelessly wiped the software, there might be traces that tell you this was done. Now let's remember the whole point of steganography is that the first step is improbable, you most likely won't detect it.

The issue is, in today's geopolitical situation, reasonably serious. It has been suggested (see the wikipedia article I linked above), that such techniques were used to exchange data on site like ebay to plan major terrorist attacks. With lots of analysis software only playing with known algorithms, or relying on comparing modified images with the original (where the original may not be available) what can such a major website do to prevent such abuse? Well, I thought an approach would be to essentially employ the same techniques with random data. That is, randomly poking data into bits in pixels here and there will, up to a certain point, not affect image clarity to the naked eye, but unless the encrypted data is loaded with huge amounts of error correcting code, it will destroy the payload. You could easily automatically run such a filter over uploaded data. I'm sure similar approaches would work for digital sound.

Posted by Colin Turner

In PHP, sessions are by default stored in a file in a directory. Sessions can be specifically destroyed from within the code, for example when users logout explicitly, but frequently they do not. As a result session files tend to hang around, and cause the problem of how to clean them up. The standard way is to use PHP's own garbage collection which is normally enabled out of the box. In this, we define constants that specify the maximum lifetime for the session, and essentially the probability of clean up.

To make things more interesting, Debian, out of the box doesn't do garbage collection in this way. It has a cron job that regularly erases session files in the defined session directory. But, if like me, and many others, you put your session files in a different directory for each application to avoid clashes on namespaces for two applications running under the same browser from the same server, you have a problem. If you forget Debian's behaviour the session files will just grow indefinitely. I had forgotten this issue and found over a year's worth of session files in a directory recently.

Solving this problem is actually quite difficult to do optimally. I mean, I could create a cron job to mirror Debian's own, but then I'd have to put the maximum lifetime in a cron job somewhere, out of the way, and difficult for the average sys admin I'm working with to find and deal with. (That is, away from the main configuration of the project). Or I could parse this value out of the main configuration. But this leads to another problem. For some users, a 30 minute maximum idle time is acceptable (although in my case where actually a suite of applications are being used as a single gestalt entity that can even be a problem), but for many of my administrator users you need huge idle times, since they are used to logging in first thing, and periodically working at the application through the day.

In the end I settled on changing our framework to make it easy to pass through garbage collection values. This makes an interface to the configuration really easy, but it doesn't solve the problems of long session times that not all users need, and huge delays in garbage collection.

In my last article I talked about a Munin plugin for OPUS, but when you look at it you'll see these kind of cliff fall drops, which are caused by the garbage collection finally kicking in and removing sessions where users have not explicitly logged out. Currently, every ten minutes, OPUS runs through its user database and finds users who are allegedly online but have no active session file and then marks them offline. Then it updates the file with the online user count that Munin reads.

I suspect eventually, I will write a more sophisticated script that actually kills sessions depending upon idle time and the user class, which would make for a more accurate picture here. Any brighter ideas gratefully accepted.

Posted by Colin Turner

Munin is a great, really useful project for monitoring all sorts of things on servers over short and long term periods, and can help identify and even warn of undue server loads. It is also appropriately and poetically named for one of Odinn's crows (so I suppose I should have written this on a Wednesday).

We've been running Munin on one of our production servers at work for quite some time, and it gives us a lot of confidence that, to say the least, the server is running in its comfort zone around the clock. Among other bits and pieces, we run OPUS and the PDSystem on this box, two of our home grown projects that are available to the students. For some time now I've considered writing a plugin for OPUS to show logged in users, and I finally did this, albeit the counts are not nearly so reliable as I'd like for two reasons, but I'll probably discuss that in another post. Anyway, I arranged for OPUS to drop a simple text file which simply contains counts of online users with the syntax and so on, for each of the categories of users. Then I needed a plugin to deal with this. I decided to write it simple shell script, since its portable and I'm not much of a perl fan. The plugin has now been running for several days, and you can see its output here. There are problems with it, but that's more to do with PHP, Debian and user choice, and I'll comment on that another time. However, already it gives me a useful feel for a lot of user behaviour.

Writing Munin plugins is easy, and Munin does so much of the hard work of turning your creation into something useful.

Posted by Colin Turner

I've tried lots of programming editors and ides over the years, obviously in Unix and Linux this is a Holy War, particularly between the advocates of vi and emacs. It is common for both groups to suggest that the other editor is hopelessly over-complex or clumsy. I think there's some truth in that, because essentially, they both stink.

I tend to be an emacsen user myself, but I just think emacs is slightly less awful than vi. My first action on a new install is usually to use vi to edit my sources.list in Debian, to help me install emacs. Perhaps thats strange, because I really like sed. So what's the problem with them? They both share this kind of puritanically awkward interface that works well on a console, but sucks in a GUI. They both use ridiculously arcane sequences of key presses to do anything, and I mean even basic stuff like saving and quitting. Yes, yes, you don't have to lecture me about old terminals and their limitations, been there done that, got the t-shirt. I tend to do all my systems maintenance in emacs, but when I'm programming, I've started to love the softness of a decent editor that actually makes it plain and simple to edit multiple buffers of source code, even though its a pain to use different editors for console and gui work.

Continue reading "Geany and other Development Tools"

Posted by Colin Turner

So, in order to post that rant about PHP and SimpleXML I had to fix a problem that seems to have spontaneously arisen with Drupal (this content management system). For some reason it wasn't persisting login information, at least from firefox (sorry - iceweasel here on my Debian system). It's interesting to note, reading about the bug, that it has been around for literally months and doesn't seem to have been nailed. So, anyway, I've installed some beta of Drupal, and yes, it now seems to be fixed... If I could only solve the problem that I can't "uncollapse" parts of the content now. UPDATE: OK, this seems to be a problem with firefox version 2, or probably really the CSS file for it. It works with Galeon, or when I tell firefox to fake being IE .

Posted by ct

Defined tags for this entry: Gondolin, php, programming

Another night of coding in PHP, and I've officially decided that SimpleXML utterly irritates me. I'd already discovered, much to my irritation, that is virtually impossible to handle SimpleXML objects elegantly with the Smarty template engine - but now I discover I can't even shove them in a PHP session without trouble - when you next visit the site you get stuff like this: Warning: session_start() [function.session-start]: Node no longer exists and then more trouble. As part of a new Web Application Framework I'm working on I wanted to parse XML configuration files one time only, and then cache the results in the session. It looks like I now have to totally redesign my idea . You can see the work in progress at its home page.

Posted by ct