network timeouts

Every developer knows that heart sinking moment when you find out your website is down or when you inadvertently perform an action in the production db rather than the development one. Unfortunately I was that guy this week – though thankfully nothing quite so catastrophic.

Our digital signage platform www.neonburn.com has a nifty feature which allows you to search a collection of royalty free images, provided by the guys over pixabay.com. This feature gets you nice looking backgrounds for your slides in an instant. This is a killer feature that gets you knocking out your digital signs in no time – when it works.

Anyway, this is a tech blog so where is the tech?? Well, the problem was that on the server we couldn’t download any image from pixabay. The search was working great in the browser but when our server tried to download the image selected by the user it was timing out.

Now the debug. First things first, I tred to use curl -O to download the image from the terminal. Nope. Next I tried to use curl to download the BBC logo from their site. Working. Shit. Right so I’m beginning to think that I’ve been barred from Pixabay – so much so that I sent them a support request! I try ping pixabay.com from the server and it works. Next I try traceroute pixabay.com to see the route packets are taking to the server. This showed that Pixabay servers were form the same provider (Hetzner) as ours. Mmmmmm.

Next I attempted to curl a file from hetzner.de. Didn’t work! Oh but now we’re getting somewhere but I’m sort of shitting myself as I’m getting out my depth.

It’s at this point where, let’s call it magic but some might say luck, I have a hunch and I’m not even sure where the hunch came from. The hunch? Maybe this is something to do with IPv6, I’m still confused as to why I thought it might be, I hardly even knew what IPv6 was. But bang!

So it turns out that both sites I’ve had problems accessing are communicating using IPv6 (I’ll be honest I didn’t even know that my server had an IPv6 address setup). Breakthrough. However, now I’m really really out my depth. I now need to try and figure out how this whole IPv6 thing works in general and with CentOS.

Now here is what I learned: my pings and traceroutes were working because both ping and traceroute commands are for use with IPv4 address. If you need to check these things with IPv6 network you must use ping6 and traceroute6 respectively otherwise you are being misled! Lesson #1. The same goes for the ip command which is used to list the allocated ip addresses or list the kernel routes, you must use the -6 switch. For example you use ip -6 route show, to see the kernel routing for IPv6 comms. Lesson #2.

Even at this point I still could not see what was causing the problem. Next I started looking at the configuration for eth0 interface which is found in /etc/sysconfig/network-scripts/ifcfg-eth0, on Hetzner CentOS anyway. Everything seemed fine, it contained both the static IPv4 and IPv6 addresses as described in the Hetzner docs. Bummer. I’d read about issues if the IPv6 loopback was not included in the config but running ip addr to list the network interfaces confirmed that loopback address was there. Running out of ideas now…..

The final check was to make sure that the kernel was routing the IPv6 packets via the correct default gateway and that somehow the rules had not become messed up. First though I checked that I could indeed ping6 the default gateway which is at fe80::1. Yep, working fine. I next ran ip -6 route to 2a01:4f8:b0:a097::2 which is the IP of pixabay and yept it seemed to be selecting the correct gateway – incidentally you can use nslookup -query=AAAA pixabay.com to get its IPv6 address.

That was me I was out! Completely stumped. At this point I simply commented out the lines in the ifcfg-eth0 file that setup the IPv6 address and restarted the network service service network restart to pick up the changes. This basically disabled IPv6 and in turn forced comms via IPv4 and everything worked as before. Not exactly satisfactory but you need to give up at some point. I bet this sounds like a shit blog post now…..

Final task was to mail Hetzner support, who to their credit got back to me within 10 mins. I suspected that they would tell me to get lost though as it was looking like a problem with my OS install. However to my surprise they told me exactly what the problem was!! Should have emailed earlier.

Turns out that when they moved my server to a new data center 6 weeks ago they removed the IPv6 subnet I was on but it was kept alive for 6 weeks to allow people to request a new one. It said all this in the email they sent me about the server move but since I didn’t even know I was using IPv6 I just glossed over it. Whoops, schoolboy error. Still, I think I’ll just live with my IPv4 setup for the meantime to save wasting any more time!

At the end of the day I wouldn’t have learned half the stuff about the IPv6 if I hadn’t faced this issue. So there are even positives in disasters, even more so when you write them down like this

don’t let your customer service suck

I don’t know if anyone around here has changed their utilities companies recently? I imagine that the vast majority of people, well probably everyone that will ever read this, use some sort of comparison site to find out what the best deal is for them. On these sites I noticed that there is often a customer service feedback score. In the past I had sort of wondered how can one utilities company’s customer service really differ so much from another. God how naive.

phone photo

Let’s take telephone services for example here. So surely at the end of the day it’s just not that difficult to screw things up. Right? After all, in the UK, BT (Openreach) control the actual physical phone lines and there is pretty much jack shit any other company can do to influence this. So if you have a problem with your line, no matter who your provider is, then you are at the mercy of BT. Seems simply and obviously who is to blame!

Enter SSE. They have managed to show me how you can balls up everything and not leave a single balls up left unturned. So if you’re reading this wanting to know how different can it be then these guys are top dogs.

First, they just totally lost my application. I got confirmation in the post of my contract. A few weeks pass and no more word from them, so I call, and they say they have no record. To make matters worse they then imply they don’t believe I had ever had a contract with them and had me sending emails with scanned versions of my confirmation of contract. Dick move. They should have just believed me, I mean at the end of the day I’m still a new customer for them, what the fuck does it matter when I signed up, even if they have different plans on offer by now, my original plan must still have been generating them a profit otherwise they wouldn’t have been offering it in the first place. Regardless, I persist – because they’re cheap and I’m poor!

clock photo

We then have a 1hr 20mins telephone call, during which the call centre operator attempts to sign me up to their brand new tariff (not what I want), this means I have to listen to a script being read to me verbatim twice, each of which lasts 20mins, the second with a single word changed.

The problem here is that the people calling you have no idea about your case, none-what-so-ever. Every time you call it’s like starting with a blank sheet of paper. Painful. How much better would it have been if the person calling me actually was fully aware of my case, knew the shite I’d been through before hand, and understood where in this whole process we were at. You just feel like an insignificant dot on their endless day. The legalese verbatim script thing was also painful, I assume it has to be done, if not, don’t waste the customers time with crap like that.

Ok. We must be nearly there you think. Oh no, fear not. So I ask for my router to be delivered elsewhere, as I won’t be at my house for a few weeks. No problem apparently (when I say no problem, I mean I’m put on hold for 5 mins while we find out it’s not a problem). Guess what? Haha, you’re right, they sent it to my house anyway. It then gets shipped back to the local sorting office 2 hours away. I eventually get it delivered to my local post office at my own cost – seemed easier than asking them to get it right. Clearly they have no processes in either the call centre or dispatch. It was as if each member of staff I ever spoke to had been on the job a day.

Photo by michelhrv

Fuck sake, I know, 650 words so far, this must end soon. Not likely. By now I’ve been given a date for the switch. I get a call two days before saying that it’s now not going to take place for a further 2 weeks, is it ok to still go ahead. Everything in my body is wanting to say no, but I need-their-cheap-service so I welcome the further delay with a smile and unbounded enthusiasm.

The original date rolls round and the switch (sort of) happens, WTF. Alas, no, they couldn’t get that right either as my existing provider informs me that they took over the broadband but left the phone line. Words just can’t describe the frustration. Furthermore, SSE never actually tell me this, they only seem to spot it when I call to ask them what is happening 10 days after the broadband switched! Clueless to the end.

So currently I’m still waiting for the phone line to be switched over (end of the month) over 4 months after I tried to make the switch. Customer service should be proactive. First you should notice if there was a fault and at least make the customer aware of it. They just can’t get anything right.

If any customers of Neonburn every have problems as bad as this please remind me of my post here, and I’ll hang my head in shame. Problems can and will always happen. It’s about how you deal with them that matters. With Neonburn, just about any semi-competent programmer can create digital signage software. In a sense that’s not what we’re selling. It’s digital signage software created by people that actually care about the customer.

And with that, we circle right back to the start, and conclude this is why you can have varying customer feedback scores….

scottish national economic forum

suit and tie photo

So a few weeks ago I attended the Scottish National Economic Forum in Edinburgh. The focus was on building Scotland as a digital nation – with the sub topic of cyber crime.

Now, I’ll hold my hand up and say that I wasn’t expecting much and with the 2.5 hour journey to get there expectations had dipped even lower!

The first thing that struck me was the sheer number of suits and ties! I understand that even making this observation sounds trite and, all in all, me a bit of a dick, but hey! The fact is that formality is not my thing, and indeed it does not chime with my experiences in the digital world. Maybe the atmosphere would have been more laid back, rather ungovernmental, without the formality. The fact is that most of the people attended were not in the tech sector or indeed under 40 years old. Maybe those points are related, maybe not.

Anyway the above is nitpicking so on to other matters.

Photo by Robert Scoble

One comment at the forum stood out on my radar. It was made by John Swinney who stated he thinks that they have the (early stage) startup scene and angel funding sorted in Scotland, he thinks the problem is now venture funding. Really! I don’t think this is true at all. There are nowhere near enough people starting companies. From what I can see people still have “the fear”, the completely-and-absolutely-no-risk-taking-I-must-have-a-mortgage-by-25-and-must-never-fail-or-be-seen-to-fail attitude. I attend or have attended quite a few startup/business meetups around Glasgow and quite frankly the scene is “quiet”. For me to feel this about Scotland’s largest city surely means it’s far from being “cracked”. It’s never going to be Silicon Valley so let’s not try to be. Onwards.

The workshop on digital marketing was good. Not exactly ground breaking but was nice to hear what some local marketers and government agencies were doing on the digital marketing front. What was quite poetic was the gentleman sitting next to me showing examples of new types of marketing material in the form of promotional gatefolds with embedded screens which is exactly what Neonburn (my business) can be used to create. Well I never!

network business photo

What followed was the ominous networking time. I say ominous as quite frankly I suck at it, I just find it completely unnatural, and it probably comes across that way as well! I do try though, fear not. Strange thing is I enjoy talking about my business, who would have guessed.

And finally. I was asked “are you looking to raise money” to which my reply was “no just making some sales would do rather nicely”. Is it just he automatic assumption now that to start a business you need to get money and without that you can do nothing? Surely it wasn’t always this way???

the lows

We’ve all heard the startup “war stories”, usually in the form of how hurdles were overcome to “smash it” or the gallant loser that could do nothing to stop his business falter. Now when reading these things I’ve always found it hard to empathise, because at the end of the day, I’m just not actually experiencing them, even the greatest writer in the world can’t make that happen. There often tends to be some new-found wisdom attached to these stories because time affords you that. I’m not sure if hearing it from the pit makes much difference either, you’ve got to feel it, then really not forget it, maybe it should be written down as it happening, not when it’s finishing, or maybe not. However, things really feel real when you’re in the thick of it, there are no anecdotes, no perspective, no money, only tomorrow, only bills, only lack of sleep, only walls not doors, the lights don’t turn on. It’s not hard, it doesn’t make you a hero, it doesn’t deserve sympathy, it’s not the way it should be, how it should be doesn’t matter because it’s the way it is. The Lows.

managing your enthusiasm

If there is one thing that is likely to unsteady the ship as a sole founder is lack of enthusiasm. No one there to push you on. No one to confide in. No one to help when things are just not working. All these things can lead you down the dark path.

Photo by CarbonNYC [in SF!]

I was reminded just how bad this can get over the last week or so when I got myself into a rut while trying to solve a problem with my code.

To be specific: I use fabric.js to draw shapes on an HTML5 canvas, however, I have pretty much wrote my own way to scale objects up and down as I found that if you have an image as a shape background then you get unacceptable pixelization – I tried the static and dynamic resize filters but they appear too slow. All was working fine but things were going wrong when you have a group shape as resizing wasn’t working correctly – amongst other things.

What I thought should have been an easy task turned into a nightmare. I was hacking away – essentially like a headless horseman – grasping at any straw to solve my problem without ever really thinking about the problem. The real problem was just that I wanted to solve the big problem without really knowing what all the little problems were. You see this so much these days with code, folk expect to find the solution to their problem on StackOverflow, and if they do, they copy-paste in the solution without ever understanding the problem. The day always comes where you can’t do this or it doesn’t really solve your problem and you only realise too late. That’s another story for another day though!

So the more days that went past the more I got frustrated by this problem. As each day ticked by my daily schedule (as I talked about before) was just a worthless piece of paper that I filled out each morning trying to pretend that I was in control. Each day my enthusiasm was dropping exponentially. This is where is really hit home having no one to talk over the problem with – talking it over with friends is likely not much use as no one is likely to understand what you are doing deeply enough to make any technical contribution. I was honestly at the point where I thought it was easier to just abandon Neonburn before I’d ever given it a chance. It’s amazing how the mind works, you’re happy to give up something you’ve given months and months of your life over to just because of one little problem.

Thankfully I started making small wins by edging little by little to a working solution, at which point the enthusiasm gradually startsed increasing. I started to understand the problem (I abstracted it mathematically rather than trying to write code to solve my problem). All of a sudden things were looking better and it’s amazing how good it feels getting these little wins. It’s important to accept that just because you can describe the problem succinctly it means that the said thing is easy to accomplish technically – think hoverboard!

So I’m not entirely sure of the moral of this story is. It’s probably along the lines of give yourself little wins all the time which of course means you need to set yourself smaller tasks. Walking away from something is actually the easy option, but then again so is battling away without really thinking about what you are fighting. However, I in no way believe that this won’t happen to me again. The only thing I remember thinking at the time that got me back on track was: this has happened before, and you always get it fixed even if it seems impossible at the time (this is assuming that you know it’s at least possible). I don’t want to trout out the whole “never give up” mantra as that sounds too much self-improvement-guy, it’s more a “never give up without having a proper logical converstation with yourself about it” – that’s not as snappy though!

getting over the line

Why is getting over the line so difficult? When developing software and someone asks you how are you getting on, you always inevitably hear “it’s nearly done, just this, this and this” to do. Next week “this, this and this” are the exact same things. Your first thought is then to say “do we really need these things done before we launch/release?” Surely if they’re not essentially then you just release? The problem is, this will boil down to how you define “essential”.

Photo by comedy_nose

For example, I have a really hard time with the concept of a MVP (minimum viable product). Maybe I’m just not the customer for someone launching an MVP as I would abandon it in minutes if it was (A) difficult to do what I wanted it to do, and (B) bug ridden. The thing is, I can’t honestly understand why anyone would tolerate an app with these shortcomings. Is it just the case that a product has to be 10% better to convince some to buy/switch? That seems low to me. I’d say in my case it’d need to be at least 50% better to motivate me to switch. (Those we made up numbers!)

In reality, I think I’d be more encouraging of MVP+ – basically MVP that does something but I’m not expected to deal and workaround shitty bugs all day, and you wouldn’t be embarrassed if you asked someone to pay for it. I don’t see a meaningful B2B product as something that you can do throw together in 2 weeks.

Having said all this, I think it’s important as a founder to realise when you need to show what you have created – what is the point in doing it otherwise. I’m one of the guilty ones when it comes to holding off for an MVP++! There comes a day when you need to get off your arse and take a risk – but just minimise it and don’t turn customers off for good by releasing crap.

So in summary, if your product is good enough that you’d feel happy charging money for it (assuming you have a reasonable moral compass) then it’s definitely time to let the world see it, otherwise I’d think twice.

PS. 50% better than others and I’d happily take money for it www.neonburn.com. Haha.

accountability and digital signs

One of the main concerns that you have as a single founder is forcing yourself to meet goals on a week to week (or daily) basis. Obviously the easiest way to achieve your goals (assuming you have taken the right step and set some) is to have customers pushing and pressing you for action. On the other hand, as a customer I wouldn’t be too happy having to micro-manage a startup on a daily basis. With co-founders this tasks becomes easier, as peer pressure kicks in and accountability becomes easier to manage.

Photo by Swipespot

One way to mitigate this problem is by using your blog to force a level of accountability. Given this, it’s with great pleasure (or more importantly as I said I would do in my last blog post!) I’d like to introduce my very-soon-to-be-live product Neonburn.

So what is Neonburn? Well it’s an online web app that allows you to easily create and design an interactive digital sign for display on an Android tablet or mobile device. The main benefits of using Neonburn:

Easy to create and design without any specialist knowledge or design skills.
The signs are interactive so you can control what happens when someone taps certain objects on the screen.
Signs remotely update when changed via the online app – ideal if you have new promotions that you want to advertise and don’t want to have to manually change each individual tablet, or maybe you don’t even have physical access to the devices at all.
No other apps on the device can be accessed once the app is deployed, and the device always boots to your sign, i.e. customers can’t interact with the device in ways that you don’t want them to.

The application is going to be of interest to you if you manage a retail store or public house and want to make your customers aware of your current promotions, while multi-site organisations will find the ability to remotely change signs in multiple locations simultaneously (without the need for a physical presence) invaluable. The interactive aspect of the signs are ideal for improving customer engagement in situations such as trade shows, or to create interactive displays at museums and galleries. It could also be used by retail stores looking to give customers an interactive tour of their products – without the need to directly involve sales staff. Finally it just makes for a very cool looking signs in your workplace – especially if you have ~~propaganda~~ a set of values or metrics that you want to ensure everyone knows about.

If you would like to be one of the first to try this out (or you want this product right now!!) then please sign up for the pre-launch newsletter on the www.neonburn.com website.

By next week the plan is to have the web app open to those looking to create some great looking signs and then by the end of next week to have the app on the beta channel of the Google Play Store. Obviously, anyone who’s keen to be involved in the process at this early stage can count on getting a discount before we open to the big wide world! Go on.

being a solo founder

So it’s been a while now since my last post and also since I decided to go solo and start out on business for myself. I’m not 100% sure that the former is due to the latter but it most definitely has had an impact. Writing posts is actually a good healthy habit that I have somehow managed to remove from my diet, and to be honest, has probably been to the detriment of creating my own business overall. The fall-off was mainly to do with the fear of writing about stuff that either A) Seems obvious or B) that no one wants to read about. I’ve now got myself over this by simply thinking who cares!

Photo by Kevin Dinkel

Given the above it’s with great pleasure that I present “The fuckwits guide to starting your own business“. This will probably turn into a more what-not-to-do set of posts given my success rate but it’s a case of you’ll get what you get.

First up. Going solo. That is, being a solo founder.

If anyone has not read the stuff by Paul Graham and so the idea has not been force fed that this is kind-of seen as a bad idea then let me tell you it is. Never do it if you have the option of getting someone on board with you, and this is as close as I can bring myself to say don’t bother starting up unless you have a co-founder – yes it’s that important. Unless you have the complete and total discipline to avoid getting sucked down the tar pit of either A) working too hard on the product development or B) working too hard to customer development, then it’s important to have the perspective granted by having more than one founder. I’d imagine that most developers with suffer from (A) more so than (B), and I’ve known that I’m suffering from (A) for quite some time and yet I continue to sink in the tar pit – I’m almost hoping that vocalising it jolts me into action (it didn’t with my last blog post about the very same thing though).

It was hard for me to even suggest that starting a business without a co-founder just shouldn’t be done, as invariably it’s never that straightforward and, in my heart (not head), I think it’s better to start something than nothing. Before deciding to go solo I had been down the rabbit hole of attempting to start a business with others but it always failed to materialise. 99% of the time the reason it failed to even get out the starting blocks was that the other folk were often not willing to take the risk that was involved of possibly having no meaningful income (by which I mean closely matched to their current income) for an extended period of time – possibly more a British (Scottish?) problem.

OK negativity aside what can you do to make it work? Obviously I don’t know or I wouldn’t be failing so miserably at it. However, here are the steps that I’m about to take moving forward:

Be more accountable. I’m currently pretty disciplined on the number of hours I work – I use a timer to ensure that I do enough hours in a day. However a fellow startup founder Lee from icmobilelab.com suggested an accountability buddy! This struck me as a good idea but I felt I had to take it a step further and use this blog to make my accountability a bit more public rather than depending on a single person.
Segment time better. Be more specific about how I spend my time during the day. So something like 8am-10am software development, 10:15am-11:15am write blog post, 11:30-13:30 customer development, and so on. I’ve tried this in the past and if memory serves me correctly it works well – this blog post is sort of evidence of that! Obviously there are days where this becomes harder to organise but it should be the exception rather than the rule.
Get out and speak to people. Potential customers here are the obvious choice but it’s worth meeting other founders or investors even just to catch up. I want to prevent myself from going dark for extended periods of time as it’s never a good idea.
Blog. People like hearing about what other folk are doing so I should do it more – many Scottish people have a curious trait of enjoying seeing others failing, so there’s always an audience! Also developers love telling people how their way of doing stuff is WRONG, so again, even with technical articles, always an audience.

Anyway, shit, over my allocated segmented time for this. Don’t worry going to pencil in another hour for Monday! Maybe more details of what I’ve actually created. Over and out.

it’s so easy to do things wrong even when you know they are wrong

Jesus. How could I be so stupid. Have you ever done that thing? You know? That thing! It sort of rolls like this.

You’re creating something, it doesn’t matter what it is, you’ve read all the books, been practising your art for years, you’re ready to create something big. You start building it, so utterly sure that you’re doing all the right things, then someone says are you doing *this*, confident you answer “Yes of course I am,” because it’s so obvious that you would be doing that thing as it’s fundamental. But you’re not.

That happened to me when building my product. I would swear I was doing what you should be doing, building something small first then incrementally expanding it based on user feedback, implementing their feedback, and iterating. Then came the question, “What about the many other users that could potentially use this product, what do they say?”. Ehhhhh, what do you mean all the other users? Oh shit. I knew I should have been talking to as many people as possible, it costs nothing, I almost thought I was, I really did, but I wasn’t expanding my horizon to the people that mattered. It seems so fundamental that you’ll not believe that someone wasn’t doing it. You’re probably laughing at me now.

Maybe everyone else is doing the right things and I’m just that loser. Probably not though. I think my point is to make sure you are doing the things you think you are doing. Just asking yourself if you are doing them may not be enough, have someone challenge you on it.

oh no, I just updated a ruby gem!

Are you a ruby developer? Do you want to manufacturer something to do? Well here is a quick tip. Take your beautiful working Rails 3 application that you started creating 6 months ago and type in “bundle update”.

I don’t think there exists a time when I have issued that command and then 10 seconds after it completed not thought “what the fuck was I thinking”. Just about every gem that updates seems to make several breaking changes. For real comedy value it’s worth doing it between major Rails releases, actually, I take that back, make that minor Rails releases.

This just doesn’t seem to happen with C++ libraries. When coding in C++ (which I do almost every day) I can reliably update to even major version changes and still be pretty much guaranteed that what worked before will still be working now. At most you get a “deprecation” warning. Not in the Ruby world, oh no, here progress is rapid but backwards compatibility is an afterthought. It really is no wonder that enterprises are so cagey about moving to, how can we say it, well in the words of Steve Yegge, hardcore liberal languages. There is something to be said for stability.

So is this a bad thing? As much as I would like to say that I think it is, I’m just not that sure. There is nothing more frustrating than using something that is not willing to change (I’m looking squarely at you Java). It’s the fact that there is no attempt to maintain backwards compatibility for at least a short period of time that doesn’t fill me with confidence.

These days you see many people picking the latest and greatest technology to build a business on. This always makes me squirm just a little. This may be OK for a small business with technical founders. However, it makes little sense for a company with shareholders where employees can leave at any time. I certainly wouldn’t want a company I’d invested my money in being in left open to the vagaries of change quite so much.