the cost of developing software

I had the opportunity (or misfortune!) to visit an organisation recently whose computer systems are somewhat shambolic. They understand that their systems are not in a good shape, but as this is not their core business, spending money on it is really not what they want to be doing. In fact, it wasn’t a business at all, it was a charity.

This is a real dilemma for such an organisation. Vast sums of money tend to be hard to come by for most charities, and explaining to donators that you spent their cash on an IT system probably doesn’t look great either! But boy, do they need a new system.

The current system is essentially an old school Access database and nothing much else. This would not be a problem if it actually worked OK, but it doesn’t. The system was created around 10 years ago and no longer fits the “business” model. Sure, you eventually get what you need from it, but everything takes around 10 clicks, it doesn’t work with the latest MS Office, forms take 30s to 1 minute to load, the output frequently has to be adapted manually and frustration with the system is high.

Understandably, they are frightened of change though. Not change induced by moving away from their old system, but change as in how much change will they get from £10k to get a new system developed. Nothing I would suspect.

This is not good. But hey, we programmers need to eat, so we can’t expect to do stuff like this for nothing, right?. Maybe charities just need to get real with this situation. You either pay up and get the system you need, or you don’t have a system at all. Screw the inefficiencies created by the absence of such systems and also to the people who are affected. This is just the world that we live in. Eat or be eaten. I mean who wants to sit and develop software and not get paid for it?

Mmmmmm, well loads of people. We must be one of the only industries that give away our time for nothing, and for no reason other that we love doing it so much. So maybe instead of wasting our time developing another new open source web framework in whatever language, it might just be a great idea to develop the software for these charities. Sure, it ain’t going to be as exciting, but it may be much more rewarding.

Has this been done before? I dunno. I can’t find much after a quick search. However, if it has been attempted, I certainly don’t know about it and I suspect neither do lots of other developers. If someone has heard of something similar please leave a comment with the details.

The undertaking of such a task would by no means be plain sailing. It would require lots of input from the staff at charities and their volunteers – because as the software developers we ultimately need to know what to develop. This is achievable though, right? And in the era of open communication afforded to us by the internet, there may never have been a better time. If open source projects can work, then I see no reason why this idea should not be possible, it just needs momentum.

it’s easy to live in perfect hell

Just the other night I was watching a video of a live music performance on YouTube of my favourite band of all time Manic Street Preachers. While watching I observed how young they looked in the video. It then struck me how I remember the era when the video was recorded as if it were yesterday. The only problem is that it was nearly 13 years ago!

I’m pretty sure that I had many more ambitions and hopes back then – I would have been around 18-19 at the time. Certainly over the years my hopes and ambitions have changed. For better or for worse? I’m not sure.

There are things that I knew I really wanted. One of them was to attended university, my parents never forced me into this, I simply wanted to go (I was the first person in my family to do so). During my 4 years at uni I became certain that I wanted to do a PhD. However, by the end of my 4th year, the only real post-graduate opportunity was at the University of Bath, and I wasn’t really interested in moving. As a result, I ended up taking a job at OKI. The week after accepting the job, I was contacted by someone at a local children’s hospital about an opportunity to undertake research for a PhD in 3D modelling of the eye for the early detection of certain diseases in children. I turned it down. I thought to accept it would be unfair on my new employer.

After working at OKI for a year I moved on to work with at Cisco Systems. Eventually after 3 years at Cisco (the last year being pretty miserable, but I was kind of holding out for a possible redundancy payment) I got made redundant. As it happens, just before I got made redundant, I had applied for a PhD position in the area of algorithms, and was fortunate enough to have been accepted (I ended up obtaining a really great supervisor and was glad I held off). So not only did I get a pay off as part of my redundancy, I also had been accepted for my PhD prior to being made officially redundant. Sweet. Persistence pays off.

So, why the life story I hear you all say. Well, it was basically to illustrate the point that if you want something, you really need to go and get it. During my time at OKI and Cisco I had become stale – you just settle in and accept what is coming to you. It was rare when I tried to force what I wanted in those jobs – and in actual fact when you did, you were usually met with resistance. When you meet this resistance to ideas it is probably the time to move on. In large companies you have too many people protecting their own jobs.

Now, what exactly is the relationship between what I’m saying and the music video? Well if the band in the music video has aged that much, then I have to – despite the fact that it doesn’t quite feel like I have! And I think the older you get the less likely you are to make bold moves in your life. But you shouldn’t let this happen. You should always have goals, if you don’t believe me, believe Seth Godin.

I often find myself writing/talking about poor management and poor working conditions. However, it’s not just me, with the increase in the number of blogs and social network sites, people are finding it easier than ever to express their opinion. It’s not too difficult to find people saying: “I need two monitors and a Herman Millar chair”, and “My manager doesn’t have a clue how to do his job”. The thing is, many will be like me, that is, they have never been a manager and never owned a company. Both myself and others have a total barefaced cheek to criticize something we have no experience of, but it also shouldn’t stop us.

So, this brings me round to my point. If we are going to complain about, our managers, our co-workers, or the company we work for, then maybe we should get off our arses and do something. Stop sitting about telling people how things should be done or moaning that nobody listens. The cost of doing something for yourself these days in minimal, so if we all really believe that we can do things better then do something about it. This is what moves society and technology forward.

Not everyone is going to read this and think “Yeah let’s do it”. You will have the folk who say “It’ll never work”. You just have to ensure that you don’t surround yourself with these people – I’m not saying ignore advice here, as you may genuinely be exploring something that is never going to work, but ignore advice from those who say a lot but do very little. It will be the same folk that say “He was just lucky with that business”, or “I could have done that”. Their procrastine nature will only dampen your enthusiasm.

Surely there can’t be a person who is happy to live in mediocrity? The only glowing benefit of Capitalism (and it’s the one that allows it to beat all other systems into the ground) is that it promotes the bettering of oneself and in turn society in general. It’s easy to mistake Capitalism for greed, but they are two different things. So if you find yourself hit by the credit crunch and unable to find work, moaning about your job, or moaning about how everyone else appears to be making all the money, stop it and try to do something positive. I’m not the only one saying this. The legendary Paul Graham wrote and essay about it and many people who own their own software companies are saying sales are as strong as ever.

Just remember “Libraries gave us power, and then work came and made us free”.

naïve algorithms will let you down

Over the last few years I have been in a position where I have been party to discussions about software with those who know little to nothing about the art of computer science. Many of these people were in charge of very large governmental budgets for some fairly high-profile public-facing software systems. These systems typically involved a large amount of deep algorithmic work, which on the face of it might seem simple, but was actually very complicated. Given this situation you would have thought people would listen when advice was being given out. Oh how wrong you would be!

For some context to this I will introduce a problem, which is not strictly related to the situation described above (to protect the guilty), but has a similar flavour.

The problem I’m going to describe is that of allocating a set of school children to their (or I suppose their parents) most desirable school. Each parent is asked to rank in strict order of preference 5 schools. A centralised system is then used to assign each child to a school.

The naïve (serial-dictatorship) solution to this problem would be to simply assign each child to the first school on their list that is not already full. So what is the problem with such a solution? Well, firstly we can end up with an assignment of children to schools that does not necessarily have maximum cardinality, i.e. there exists another allocation in which a larger number of children are matched to schools. Also, we may end up with an allocation in which a child c1 ends up at his fifth placed school while another child c2 obtains his first-choice, but if c2 moved to his second-choice then c1 would be able to get his third-choice. If we allow this switch to take place we clearly end up with a “better” allocation.

Despite the poor characteristics obtained from the naïve algorithm, I suspect that many many organisations, both governmental and private, use such an algorithm when assigning people to resources when preference information is available. The problem is that people in charge of building such systems often only see the need to allocate these people to resources in whatever way possible and in the shortest time. So what happens? Well they get a system that looks flashy and kind of works. However a couple of months in when they have a group of people complaining and operating outside the system, the reality finally dawns on them. I’m sure in the UK alone we could count quite a few computer systems that have fallen under the weight of this pressure.

So we have shown that the naïve algorithm does not always work. Can we safely assume that the heart of a software product lies in those complicated algorithms at its core? I think we can say so with Google, but who knows in general. In the example I described I think this is definitely the case. So what would be the “best” way to allocate these children to schools?

There are different meanings of “best” but all would ensure that the allocation has maximum cardinality, without this we are at a lost cause straight away. Following this we could attempt to maximise the number of children assigned to their first-choice school, then subject to this we maximise the number of children assigned to their second choice school, etc. Therefore if we receive an angry parent complaining that a child got their third-choice school you can at least tell them that there was no other possible assignment in which some other child with their first or second place does not get worse off. Hence there is no incentive for the administrator to allow any changes to take place. This is known as a greedy maximum matching (allocation). Lastly we need to make sure that these algorithms are efficient, most algorithms are no use to anybody if they are exponential.

So there we go, just remember that naïve algorithms are not always (and often not) the best.