UPDATE: 5.10pm Thurs Apr 6

Old Email System Fully Recovered

We have finally recovered the broken email server. We will now continue to use the previously installed backup, but now all emails that would have been received by the server during the outage should be available to you with the following exceptions.

• The are still are few emails stored in the mail queue for resending.
  • These should be processed over the next few hours.
• There were a couple of brief periods where the server was down totally
  • During these short periods, emails would have either bounced or were stored by the sending server to be resent

If there is mail you did not receive on last Tuesday or if you just want to check, click on one of the links below in the Where can you access.. section and log in using your full email address as the username.

Web Mail Users

Those who use the webmail system as your primary means of accessing and storing emails, you should now be able to recover any emails that went missing from the Monday pm to Tuesday pm period.

It will be a little tricky to get them to your current webmail account because you cannot simply send them to yourself, as the system will act like a closed loop.

You can do one of two things:

1. Forward them on to another email account which you have which has nothing to do with Smartspace  (eg hotmail or gmail etc)
   • Forward them a second time to your email address - they will end up in the right place
2. Forward them to our system email address and we will forward them to you within a couple of hours of receiving them - no need to say anything in the email
   • Forward them to postmaster@smartmail.email and we will send them back to your current account.

Where can you access this recovered server?

You can access the old recovered server at one of two addresses (one is our Roundcube installation and the other is the new SOGo panel)

• http://hqyou.com
• http://hqyou.com/SOGo

Do not hesitate to contact us if you have questions or concerns.

Please note that this server is NOT a secure server like our regular Smartmail system.

Thanks for your understanding!

Major Email System Outage

Yesterday we suffered our first major email system outage on our Smartmail mail server. whch went from just after 10 am until just before 7pm.

First of all, our sincere apologies for the inconvenience that this problem caused. Events of this kind are not good for anybody.

If you are interested here is some of the detail of why it happened and what we did to fix it.

We became aware of a separate problem when we discovered the valid SSL certificates that we had purchased last year were going to be deemed invalid in some browsers in the near future.

So we were taking immediate action to solve the issue, which meant ensuring some of our software was up to date.

Unfortunately I entered the wrong command into the system which meant that instead of just one piece of software checking for updates, the whole system updated. (we normally run this at low traffic times in case there are problems).

Because mail servers are a complex mix of a range of different software, something in the update caused a problem with some parts of the server no longer being compatible with others.

Initially we thought that it was just a webmail issue (the user login database connection was lost)  and were attemting to fix that when we realised that the problem was server wide.

We have had problems with mail server update before, but they are normally solved quickly, but on this occasion we spent around 6 hours attempting to resolve the problem, but without success.

With no real prospect of successfully fixing the issue, we decided to initiate another server using the previous day's backup.

It takes around 1 hour to restore a backup and to swap the ip addresses of the old and new mail servers but when we completed the job, there was a technical hitch at the server end which meant that the ip address would not resolve correctly.

Normally the support from our server provider is lightning fast, but yesterday (of all days) it was slower than normal. After finally identifying the poroblem, we had the backup server running, but then discovered that the backup had occcurred after the problems started so we were back to square one.

We had no choice but to restore from a backup that was over 24 hours old from the previous day. Once that restoration was completed we had the server working within a few minutes.

What this means for clients

Whilst the server was not working properly, it was bouncing emails for only a relatively short time. this meant that most of the emails sent suring the outage were absorbed by the server.

They could not be sent on, but were put in the mail queue so that a further attempt to send could be made, usually an hour or so later. Normally, depending on a few factors, resends will be attempted for up to 24 hours.

Because the original server still has problems, these resends will not be successful unless we can manage to get it restarted.

We have taken a record of the mail queue and will be advising all clients that have mail stuck in that queue of attempts to email them.

Meanwhile, today we will attempt to get the server up again, so that all that mail will be delivered, but it is possible that we do not succeed.

We will keep our twitter feed updated with progress.

Please do not hesitate to contact us id there are particular issue that you have from yesterday's problems.