Hairydog Status Page

Well, one thing did happen. A change of internet providers put this status page offline for a few weeks. There was nothing to report here, so it wasn't a priority to fix.

Google have decided that this blog is too inactive for their liking, so I have decided to move it to one of our own servers. This runs independently of the main server, so should work if something unfortunate happens. Let's hope we never get to that stage!

Following a server upgrade, things seem to be going really well, but there is a problem with some people's email.

Some email boxes are working fine. Others are not receiving any email. 

More info as soon as I have it.

[Later] Seems that the problem is down to an issue with the management software we are using on the server.

The developer has supplied a workaround and is developing a fix. Nobody had reported this issue before today.

If anyone is still having problems with their email, please let me know.

months go by…

It seems that a blog that only reports when there are issues becomes quite quiet if nothing goes wrong.

It is literally years since our last blog post.

In the next few weeks we are moving everyone to a new server. 

Sadly, the company we lease our existing server from was taken over by a multinational company, and over time the support, performance and facilities they offer have declined, whilst prices have done the opposite.

We are in the planning stages for the move, which should be fairly straightforward. 

months go by…

We seem to have successfully completed the server migration and everyone is back on the refurbished server.
However, due to a fat-finger error by someone who shall remain nameless (OK, it was me) the server is currently recovering from a glitch. Back in about 20 minutes, I think.

Our main server is getting fairly old now. Although it is still working just fine, the operating system it runs on only has a few weeks of official support left, so it really is time to upgrade to the latest version.

In theory, it is possible to upgrade in situ, with maybe an hour's downtime, this time round it feels like a better plan to rebuild from scratch.

As with any major renovation project, some disruption is almost inevitable, but to keep the disruption to a minimum, we are taking a staged approach.

First, we are checking that everyone's websites will work on the latest system. That's done (we hope).

Next is to move each website, mailbox, or whatever to a different server.

We have a couple of other servers, each able to cope with the traffic and the load, but not as powerful as our main one.

We have started moving some sites across: basically the sites where you are very unlikely to detect any difference. That's done.

The next stage is to move the rest: sites where you might need to check your email login details, or maybe the mail server name after the move is done.

If you are having any difficulty accessing your email, please get in touch!

Of course, not long afterwards, we hope to move everyone back into our shiny newly-refurbished server. That shouldn't take very long!

months go by…

This is a rare thing: a status message about an administrative change instead of a technical one.

As part of an ongoing plan to be smaller, greener, neater, lighter, Hairydog Ltd is no more. We are now Hairydog, from March 2019.

Nothing else has changed, except that non VAT-registered customers will pay a bit less for the same services because we are no longer VAT registered.

months go by…

Last week, we had a database server error. Nothing seemed to be visibly wrong, and restarting the server sorted it.

About three minutes of downtime, plus the time it took me to find out about the problem.

I wasn't happy because not having found the problem, I knew I hadn't fixed whatever was wrong. But all seemed to be OK.

But it happened again this morning. And restarting didn't sort it. Eventually I found the problem: it was the database server running out of file handles.

The documentation says the maximum number of file handles is automatically configured. The documentation tells fibs sometimes, and this is one of those times.

There is in fact a default value of 5000. But clearly that isn't enough. I've increased the value and all seems fine now.

It meant that anything that needed a database lookup wasn't working for the best part of an hour early today. Luckily, Sunday mornings are not a specially busy time for the system.

I've now implemented more stringent logging and monitoring, so a similar problem should be picked up before it has any impact.

months go by…

This morning, the mail server certificate seemed to be out of date. It wasn't really: it was due to a misconfiguration by me.
Now fixed, but we'll have to look at it closely in three months time to make sure that the issue is resolved, not just patched up.

Following a change of SSL certificate on our mail server, although computers seem unaffected, mobile phone email settings may need updating.
The mail server address (for both sending and receiving mail) should be hdog.co.uk - that is unchanged.
The server certificate setting used to be STARTTLS (accept all certificates). If you experience difficulties with this, please change it to STARTTLS without the accept all...  bit.
On IOS devices, it may be called SSL instead of STARTTLS.
Please get in touch if you are having difficulty with the settings.

This morning, (from about 2am) everyone was informed that they had entered incorrect email account details. This was nothing you did wrong: it was a global issue.

Truly global, in that many millions of people around the world were affected.

We use Debian as our operating system, which is a very reliable, slightly conservative choice. Debian tends to be rock-solid reliable, one step behind the latest and riskiest. However, last night Debian released an update patch that was defective. This resulted in EVERYONE being locked out of their email.

I've manually rolled back the patch to the previous version so everything should be working OK now, as long as you didn't change your email login settings (they were not wrong)

Please get in touch if you are still having problems. My email is working again!

[Later:]
It seems that this was a "double-whoops" the update that broke the email was to fix a vulnerability that wasn't present in the current version of the software.

Only the testing versions had the problem, so they'd foolishly decided to fix something that wasn't broken - so that broke it!

This means that we don't need a revised fix to cover the problem - there wasn't a problem to cover!