the internet and i

1 Jun

If you are able to read this please keep it to yourself, don’t tell anyone about it. Your ability to possess information that others do not have access to is crucial to moving only in the forward direction. Yet there is very little information now that is not readily available to everyone. The internet is almost ubiquitous. Rewind a mere 15 years though and observe how the landscape has changed so dramatically.

Back then a PC was the something only the posh kids had. My school was fully laden with BBC micros that you could only use to control some piece-of-shit-circuit-board that you had cobbled together to light up a few LEDs in the form of a traffic light.

I still remember to this day getting my first PC – a (big) Compaq laptop of sorts. It was purchased second-hand from a shop that specialized in guns, knifes and guitars – I still have it somewhere. They probably didn’t even know what it was. When I read stories of people in the 80s having PCs it makes me laugh as, in my world, this stuff was sooo out of reach that you wouldn’t believe.

Still, what would the general public have done with the PC back then anyway (apart from gaming)? The internet was a luxury goods item at this point. Even at university only a few computers had internet access. However, between ’95 and ’97 it exploded. You were faced with a computer lab full of students looking up porn and printing out the pictures (I mean they were not much use unless you printed them out, right)! Those were the days of looking up what you wanted with reckless abandon without feeling that someone or something was watching you – soon to be quashed with the corporate firewall and government legislation. Make no mistake though, we are still in the golden age of the internet. Laws and legislation will likely mean that the freedom we associate with the internet now will not be the same 30 years down the line.

What puzzles me most is how we got by programming without the internet? If tonight I decided that I wanted to learn OCaml to build some super useful web service, then I would start by searching Google. I’d get binaries, sample programs and documentation in minutes. If I had a problem, I could search on Google or ask a question on Stackoverflow. The barrier to entry is just non-existent. I struggle to remember exactly what we had to do all those years ago. It must have been ridiculously hard compared to the present day. But this is good. We got information easily and moved forward. That said, the internet does not have its bad points.

We now struggle to get things done because we can listen to music online, watch videos, read blogs, not to mention the ease with which we can communicate with friends using social networks. What the internet gives with one hand it takes away with the other.

I’m often tempted to unplug the router and see how I get on with work minus the internet. Invariably at the last minute though I find some compelling reason to have it switched on, then the thought of uplugging it fades into nothing and time just slips away again.

on being open to change

15 Apr

How easy is it just to say no? I think you’ll probably find just about as easy as it is to always say yes. In the past I have often found myself entrenched in that no camp. It’s so easy to hate a company, a stance or an ideology without even attempting to think of the why.

I recently decided to try out an approach where instead of following my five second gut instinct assessment, I would instead spend some time trying to overcome my own (possibly) over zealous opinions.

My test subjects for this were the guys at 37signals. I frequently found myself disagreeing with most things these guys were saying for no reason other than the aggressive way they generally put forward a point. This is a real stupid thing to do – even more so when you see it in writing!

Anyway, in this new era of love and enlightenment I picked up the book Rework by Jason Fried and DHH. I was immediately surprised how much I identified with the things these guys were saying about running a business, from focusing on products that can make a profit from day one, to abandoning long range forecasts (or to paraphrase “elaborate guessing”) about your business. So two guys I thought I “hated” actually hold opinions pretty close to my own, i.e. I was being an asshole!

Before anyone gets the impression that I’ve now turned into the “super fanboi”, I should say that I don’t think that Rework described THE way to run a business. It’s simply A way to go about things, and one that happens to be close to what I believe. For example, the books contains ideas that may fall apart when it comes to dealing with large enterprises or when trying to build “uber” companies like Google and Facebook – who can argue with their success. However most of the ideas seem spot on if you are trying to build a small to medium B2B business.

So the moral of this story is try to see past the way information is being presented to you and look at what the information is actually telling you – this is true for both the hater and the fanboi. However, it should also not be overlooked that if something is shit, then say it’s shit.

an afternoon with Don Knuth

17 Feb

It’s not often you get the chance to spend an afternoon with probably the most famous hacker/developer/computer scientist the world has ever seen. However, last week I got the chance to do just that when myself and three colleagues had the great fortune to spend some time with Don Knuth.

For the uninformed IT “professionals” that have never heard of Don Knuth this is the guy that brought to us the idea of analysis of algorithms and asymptotic notation (big-O notation), the Knuth-Morris-Pratt string searching algorithm, the Art of Computer Programming book series, the list goes on. However, he is not only an “algorithm’s guy”, he also developed the Tex typesetting system and the METAFONT language used to define vector fonts. So basically he is the most famous computing guy out there.

Knuth is a surprisingly easy guy to talk to. Sure, he can really lose you pretty quickly in a conversation, but he also has some great insights.

Our conversations tended to centre around stuff to do with algorithms. His next volume of the Art of Computer Programming will likely focus on constraint satisfaction problems and satisfiability problems – the former being something I worked on myself in the not to distant past. I asked what he thought was a good algorithm to teach people and he said he thought the biparite graph matching algorithm was a nice one in terms of beauty (he did mention another which escapes me now). Not everyone will find the algorithms stuff that interesting (you should!) but his view of beauty is maybe something more universal.

He also expressed a love for writing code, he said that when he gets up in the morning he thinks about writing code and misses it on days when he doesn’t get the chance. That is pretty cool by me and sits in stark contrast to many academics. I got the feeling that he wasn’t too keen on the “apps” developers as he called them. My guess is that his thoughts lie with more meaningful problems than fart apps – however people download them so who are we to say. Still, there was definitely some lamenting going on about the fact that people use software without ever trying to understand what the software is actually doing. That is, have at least a high-level view of the data structures and algorithms used that make the said piece of software useful. Having this kind of understanding allows you to select the right tools for the job. In my experience people that tend to have this knowledge and understanding are far better developers and is likely why Google, Microsoft, Facebook et al. try to attract developers with this kind of knowledge.

He was telling us that he watched The Social Network on the way over on the plane. He said he thought it was great how Mark Zuckerberg was also someone who just liked building stuff like him – this was something Zuckerberg said himself at Startup School 2010. What is even cooler is Mark Zuckerberg actually sent him a copy of the latest Art of Computer Programming book and asked him if he would sign it for him.

So Don Knuth himself will have long forgotten who I am but at least I will be able to recollect years down the line this encounter with a computing genius.

is ruby killing your career?

15 Feb

I’m probably at the point with Ruby where I consider it my programming language of choice (I program in both Ruby and C++ in my day job).

Over the last few years I’ve kind of grown to love Ruby but I’m not really one to get passionate over someone else’s choice of programming language – apart from Java, which, I’m sorry, I hate. However, when it comes to employment, there is no doubt in my mind that being competent in a particular programming language can strongly influence A) getting an interview and B) getting the job.

This is why ruby developers, like me, are killing their career. Sure Ruby is cool and Rails is awesome but do a quick check on job boards and see how many people are looking for a ruby developer. Actually, let me save you the time I’ve done some of the work already.

I’m not claiming this to be scientific in anyway what-so-ever but it does warrant some thought. I only searched using the programming language as a keyword, which, I know, may not give the full story but should convince you there is some merit in the point that I’m trying to make. Additionally (and I suppose somewhat importantly) my search area was restricted to Scotland.

First up I carried out a search on s1jobs.com. The table below gives a summary of the results:

Language Number of jobs matching keyword
Ruby 3
Java 18
C# 26
C++ 9
PHP 7

I then tried a cwjobs.co.uk:

Language Number of jobs matching keyword
Ruby 2
Java 35
C# 45
C++ 45
PHP 4

As you can see, the job prospects for Ruby developers here in Scotland are somewhat dire. Sure, people don’t always look for a particular programming language when employing someone (which is a decent policy) but, as I said above, it helps a lot.

I decided to take my crude search a little further as I thought “Hell, there will be waaaaaaay more cool Ruby jobs in London”. Below we have the results, just cwjobs this time:

Language Number of jobs matching keyword
Ruby 57
Java 792
C# 838
C++ 611
PHP 196

That was kind of disappointing! Ruby still doesn’t do that great – even worse when you realise there were over 200 that mentioned Perl and 150 Python. By the looks of it if you want to maximise your chances of getting a job in the UK, and already doing Java or C# in your day job, you’d be better off learning C/C++ in your spare time.

Is all this going to stop me coding in Ruby? Probably not. Is it worth thinking about for a minute? Yes sure. If I was starting my own company and was hoping to get some developers in then I’m likely to be faced with a problem. Yes you can train people up, but that costs time and money. When they leave it may be worse, as the chances of finding replacements at the required skill level will be difficult. Finding a Java/C#/C++ programmer is bound to be far easier.

So is it all bad news for us Ruby developers? Well not if you plan to move to California – yeah yeah I know I’ve went on about it before. I’m not exactly sure of the popular job boards in the US so I went with the only one I knew off the top of my head, careers.stackoverflow.com. The results for the Bay Area are as follows:

Language Number of jobs matching keyword
Ruby 27
Java 33
C# 10
C++ 23
PHP 17

Maybe this was a skewed sample set but impressive all the same. So moral of the story is if you want to be a well paid Ruby hacker make sure you don’t stay in Scotland :-) .

what’s it like in silicon valley?

15 Nov

As a computer geek Silicon Valley can almost be seen as the point around which our world spins. For those that haven’t been there I can’t recommend enough heading out to see what all the fuss is about. I mean forget that the climate is, well, pretty fucking special and that cycling to work (in any season) doesn’t mean making sure everything you buy is windproof and waterproof. What blew me away most of all was the staggering number of tech companies.

Within walking distance of my accommodation there was a Yahoo!, McAfee, Cisco, Intel (this place seemed to stretch for miles), not to mention hundreds of other smaller tech companies scattered around the area. If you can’t get a job as a developer in the Valley then you must really question your hygiene.

So is there any chance of this sort of thing happening in this country (the UK) or in particular Scotland? Ehhhh, no chance. Why?

Well I don’t claim to know exactly why, in fact, I admit it’s somewhat of a guess. One thing that I’m certain matters is the number of people who are studying computing at university. In recent years this number has plummeted. For me this is a key point. Until you have got talent, and lots of it, pouring out of universities there is no chance that the Google, the Microsofts and Facebooks of this world are going to even consider a small office, never mind a campus. It’s just not worth it.

There is also a different mindset amongst the graduates here than there is at, say, Standford. It’s just staggering how many undergraduates are considering coming straight out of collage and starting their own business (Startup School was just full of them). It’s ingrained into their psyche. I’d almost guarantee you that if I went to a lecture theatre full of final year students here in Glasgow tomorrow and asked how many are seriously considering starting their own business I’d be able to count them on the one hand and have four fingers left over.

Most students will be happy getting a job at JP Morgan or some other mundane banking establishment where you are simply a cost centre.

This is a vicious circle as the reason that Google, Yahoo!, Twitter, Facebook (to an extent) all have large campuses in Silicon Valley is because this is where they were born. So without this embryonic growth pattern it’s hard to see how places like Scotland can create and attract these large software companies. The fact is the talent is just not here in sufficient numbers. To get the gist of the situation, just ask yourself how many people in your current job are really really smart and productive, it will be low, now imagine how you get 500+ more of them.

I hate to be knocking us as a nation but something has to change and it has to be BIG for there to be any hope of being part of this technical revolution. Or maybe that boat has already well and truly sailed.

observations from academia

17 Oct

I’m going to sum up this post before I’ve barely started it: only pursue a career in academia if your goal in life is to fill out grant applications forms.

Academia, in the UK at least, is going further and further down the line of basing performance of academics on research grant filling stamina. Teaching, and student satisfaction are easily second and third in the list of contributing factors to the success of your academic career which I think is wrong. In actual fact these points are so insignificant in the sliding scale of your career at a university it’s almost worth not even stating they contribute to your career at all.

I’m lucky, I’m only a default academic in that I’m doing a post-doc but have no plans to pursue a career in this beyond the completion of this work, but looking on in from close up it certainly doesn’t look pretty for the future.

So what can we do about this? Honestly I think that the problem has maybe gone to far down the line for any form of redemption.  However, the only people that have any chance of halting this trend are the actual students.

Students hold the power at universities they just don’t use it effectively. Their power arises from the fact that their fees pay for the running of a university. Economics states that if less students apply for specific courses then that department has less money. Student numbers are *key* to an academic department. The fewer students, less money, and less lecturers, it’s as simple as that.

If a department can correlate a reduction of student numbers to bad teaching practices then maybe something can change. Maybe this is all a bit idealistic, I don’t know.

I’m not sure if their are any social networking tools out there at allow students to rate lecturers in such a way – I expect their is. However, looking out from the inside, there use is largely being ignored if they are. Maybe someone can change this. I’d like to think so.

back to the future with c plus plus

21 Apr

Around 6 years ago I was steadily programming away in C++ in Unix using a Sun Workstation and I quickly followed this up with some Windows programming using MFC. During this time I also experimented with PHP creating a small “MVC” style framework long before Rails was even a twinkle in it’s daddy’s eye. Now since then I’ve managed to have quite a varied experience with languages such as Java, C#, and most recently Ruby. However, I now find myself running back 6 years and working with good old C++ again.

So why am I back here? Well to be honest, performance. The stuff that I’m currently working on (Kidney Exchange Problem) requires a fair bit of it – it’s NP-hard optimization problem that that we really do need to find an optimal solution for using an exponential-time algorithm. Anyway, maybe more details on this another day.

With C++ being decidedly “yesteryear” with the cool kids, I was pleased to find that it still seems to have a large user base, a moderately active community, and with Google being heavily invested, it would appear that it ain’t going to die as quickly as people may have though. Despite this, I wasn’t holding out too much hope for finding a wealth of “modern” programming tools, and first on my list was a JUnit/NUnit clone.

After some searching and experimentation (which resulted in some moderate surprise) I came across Google Test – good old Google! In my opinion this is the best C++ unit test framework out there. I did try others (CppUnit, CppUnitLite, and another which I now can’t remember) but Google Test was the easiest to use, has everything I need, and seemed to be the most active in terms of development.

Also, now very old hat, is Boost. Although this has been around for quite some time, back when I was first invested in C++, this just didn’t exist (or if it did it was in its infancy). I just love things like shared_ptr and the Boost Graph Library. I’m sure there will be much much more I will find to use in the coming months in Boost.

Finally, for just now, I also have to put out an honerable mention to Google Mock. I’ve not had a chance to use this much at the moment but, like Google Test, it looks as if it will come in pretty handy.

It’s nice to see that C++ is still living and breath, even if it ain’t so cool any more. However, with the community and C++0x it looks as if it may well just have some sort of future.

does your employer suck?

23 Feb

So you slog your guts out each day programming away, typing faster and faster, writing tests and more test, refactor after refactor, then after five years you think to yourself: “Screw this! I’m doing all this work for Mr Somebody and I’m getting jack shit in return”. Then before you know it you’ve said those inimitable words “I’m going to start my own business”.

Yeah yeah, we’ve all heard it before. The very next day you go back to typing, you end up getting married, you have kids, and you die.

If my business model was that I got a £1 (or $1 – I’m not fussy) for every developer I’ve heard say this then I’d currently be sitting in my winter home in Sydney Australia – rather than cycling my ass down to work every day in sub-zero temperatures. In fact, I’d have contributed much of my wealth from my own contributions.

So why do we do this? I mean, for sure, we developers certainly think we can run a business better than our employers – remember how you tell yourself that every day – so why do so few developers start their own?

To be honest, this is maybe not so much the case in the USA, or in Silicon Valley at least. The Valley seems to bring out the spirit in people to follow through with these ventures, so why can we not harness the same spirit here in the UK?

Over here people frequently talk about it but never do it, and yes I’m one of those people. We need more people doing it here in the UK. It can work as well. Check out Redgate Software, they not only create great products, but they also help other startups. It’s almost like Silicon Valley in Cambridge – OK, I took that one a little too far.

Realistically, to give our economy a brighter future, we need to improve things. I know everyone can’t do it, but everyone wants to work in a cool place that treats their staff well, right? So those who think they have got it stand-up. I’m sure their must be plenty of you?

So the next time you are moaning about your job, spend the time thinking about a product or a service that would be great, get some balls and create your own “thing”. I know I’m going to – well I just made myself look like a fool with this rant if I don’t try and it’s kind of why I done it.

the future is email

22 Feb

In a recent podcast Jeff Attwood and Joel Spolsky were discussing the virtues of email. As a somewhat brief summary of the discussion I think it’s fair to summarise that Jeff hates email. As it happens I’m inclined to go along with him on this one, but possibly for different reasons.

From what I can gather, the problem Jeff and Joel were describing with email is that we end up with an inbox full of emails that never get read or processed, and to allocate the time to process these mails and respond would be a job onto itself. To be honest, I don’t suffer this problem quite as bad, mainly due to the fact that I do not receive in any shape or form the level of email that both of these guys do. As a result, it’s hard for me to appreciate the hate of email for the same reasons. However, I feel there is a far more toxic fallout to the email culture than that which is described.

It has come to my attention recently that many (large) organisations still use email as a way of providing an API to their service. What do I mean by this? Well, in this digital age it is often the case that a web application uses the API provided by another web service to interact with it. For example, if I want to receive a list of the tweets that I’ve made, the twitter API allows me to do this by accessing a given URL, and returns the data in the format of my choosing. Similarly, I can also publish a tweet by posting data to a given URL. This is all rather nice – everything is communicated over good old HTTP allowing easy integration with other services.

However, many firms that are not technologically aware are using email as a mechanism for inter app communication. I don’t want to bash individual companies here but the ones I have personally dealt with are multi-national companies with profits in the billions. Yet despite such profits margins their software systems look like something seen in War Games. To perform any communication with these systems you send an email, and responses are returned to you via the magic of an automated email. Thus, you have to parse the email body and/or parse an email attachment to get the response data. It’s as if Web 2.0 passed like an amoeba in the night without these companies so much as blinking.

You may say that it must work for these companies to be making the profit margin they do. However, as customers, we are paying thousands on development costs to integrate with these systems – in many cases we have no choose in the matter. At the end of the day these companies are slowly falling behind and if they are choosing not to innovate at this level, I personally don’t hold out too much hope for innovation on a wider scale. This kind of strategy no longer works, Google and other such companies have long since blown this way of doing business out the water. No-ones’s saying the death of the non-innovators is going to be quick!

So what is the message I’m trying to get across? Well first, email for this style of inter app communication isn’t really helping anyone. If you are thinking about doing something like this, please think again. Also, on a more general note, can companies that fail to innovate survive in this technology driven society? It’s obviously hard to answer this question with definite authority, but going by gut feeling and history, it doesn’t look good.

idiot’s guide to linux on amazon ec2 – part 2

13 Jan

In Part 1 I covered how to remove the root login, create a new user, and add this user to the list of sudoers on an linux ec2 instance. In this section I will cover how I got Ruby on Rails, MySQL, Nginx and Thin working together on the Ubuntu instance.

First up, I think it’s worth taking a moment to explain what Nginx and Thin actually are, as they are maybe not as well known as the others.

Nginx is a very fast web/proxy server developed by Igor Sysoev. According to wikipedia it currently runs, amongst others, the WordPress and Github websites.

Thin is a ruby web server that “glues together 3 of the best Ruby libraries in web history”[1]:

  1. the Mongrel parser, the root of Mongrel speed and security
  2. Event Machine, a network I/O library with extremely high scalability, performance and stability
  3. Rack, a minimal interface between webservers and Ruby frameworks

Right on to the job at hand and first up was getting apt-get working!

To my surprise (but probably widely known) the Ubuntu ec2 instance did not come with apt pre-configured – unlike yum on a Fedora instance I had previously used. Instead you first have to run apt-get update to download the list of package locations. Now that we’ve done this we can get to work installing the other bit of software required.

MySQL
The first thing we need to install are the MySQL client and server. To do this run the commands:

sudo apt-get install mysql-server
sudo apt-get install mysql-client

Then you need to make sure that the root password for MySQL is set to something secure. This can be done using:

sudo mysqladmin -u root a_good_secure_password

Ruby
Now it’s time to install Ruby on Rails. First we need to install ruby, rake, rubygems, and a couple of other useful packages. The following commands should add the required binaries to your path:

sudo apt-get install rubygems
sudo apt-get install build-essential
sudo apt-get install rake
sudo apt-get install ruby-full

We can now use gem to install rails:

sudo gem install rails

As we will be using MySQL you probably also want to install the MySQL client development library in order to get the ruby gem to build/install correctly. This can be done by running:

sudo apt-get install libmysqlclient15-dev

Obviously the version of the libmysqlclient will depend on the MySQL version that you are using. Finally we can install the mysql gem by running:

sudo gem install mysql

Nginx and Thin
To install the nginx package we run the command:

sudo apt-get install nginx

Nginx then needs to be started so we run:

sudo /etc/init.d/nginx start

By default the package should also add the entries required to restart nginx if the instance is rebooted – you can always check by looking in the /etc/rcX.d directory (where X is the run-level number).

Now it’s time to install thin:

sudo apt-get install thin

Creating application config files for Thin and Nginx
It is a good idea to create config files that can be used to restart your thin clusters. To do this we use the thin config command. Now, let’s assume the app is called myapp and so we run the following command:

sudo thin config -C /etc/thin/myapp.yaml -c ~user/www/myapp --servers 3 -e production

This creates a thin config file /etc/thin/myapp.yaml that starts 3 instances of the rails application found in ~user/www/myapp using the production environment. By default it will start the first server on port 3000 and the next on 3001, and so on. Should you wish to specify the port you can supply it with the -p option, i.e. -p 6666.

You can now start your thin clients using:

sudo /etc/init.d/thin start -C myapp.yaml

It’s worth noting that if you don’t specify the -C option thin will use the config files found in /etc/thin and start the thin clients for each config file found in this directory.

As we want to use nginx as a proxy to our thin client instances we must create a nginx config file for our application. An example of such a config file is shown below:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
upstream myapp {
    server 127.0.0.1:3000;
    server 127.0.0.1:3001;
    server 127.0.0.1:3002;
}
 
server {
    listen   80 default;
    server_name example.co.uk;
 
    access_log /home/user/www/myapp/log/access.log;
    error_log /home/user/www/myapp/log/error.log;
 
    root   /home/user/www/myapp/public/;
    index  index.html;
 
    location / {
        #auth_basic "Please supply login details";
        #auth_basic_user_file /home/user/www/myapp/public/protect.passwd;
        proxy_set_header  X-Real-IP  $remote_addr;
        proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $http_host;
        proxy_redirect off;
 
        if (-f $request_filename/index.html) {
            rewrite (.*) $1/index.html;
            break;
        }
 
        if (-f $request_filename.html) {
            rewrite (.*) $1.html;
            break;
        }
 
        if (!-f $request_filename) {
            proxy_pass http://myapp;
            break;
        }
    }
}

Lines 1-5 set up the proxy for our thin clients that we started on ports 3000-3002. The values that you include here obviously depend on the number of clients that you started and the ports they are running on. The rest of the file is dedicated to setting up the web server with the majority of settings being pretty self explanatory, so I’ll only highlight the important bits.

First, we see that the server waits for requests on port 80 and the domain used for this site is example.co.uk (lines 8-9). It’s worth noting that hosting a subdomain, say subdomain.example.co.uk, is as easy as replacing example.co.uk in line 9 with subdomain.example.co.uk. Lines 20-23 take care of things like forwarding the real IP address to rails as well as some other set up required for https. Finally the remaining lines in the file check to see if an index.html file is available at the url specified and if so displays displays it (lines 25-28), serve static files straight up (lines 30-33), and finally if the file specified by the url does not exit on the file system it sets headers and proxies for our thin clients and passes it on.

As a side note, lines 18 and 19 that are commented out enable basic http authentication in nginx. You can uncomment out these lines if you require this feature. The password file for http auth can be generated using the apache util htpasswd – you will need to install the package that contains the htpasswd utility.

The config file (let’s call it myapp) is placed in /etc/nginx/sites-available, and finally a sim link is set up between the sites-available directory to the sites-enabled directory to enable the website:

sudo ln -s sites-available/myapp sites-enabled/myapp

That’s it. All we need to do now is restart nginx (/etc/init.d/nginx restart) and assuming your config is ok the site should now be up and running. (If nginx is already running and you want to parse the config without restarting you can always get the pid of the nginx process, ps aux | egrep '(PID|nginx)', and run sudo kill -HUP PID – in fact this is all you actually need to do to get your site up and running)

[1] The Thin homepage – http://code.macournoyer.com/thin/