Corporate Blog

The move of server NC018 to the new data centre has been completed. Due to two failures, the downtime for some websites was longer than the planned 12 hours. These two failures were as follows:

1. For some reason the data centre did not actually configure the server to use the new IP address, even though this was expressly a part of — and indeed a requirement of — the move. This resulted in most websites being down when the server came back online because most websites on the server use the server’s primary IP address. (Websites that have their own or share a secondary IP address had no problems, initially.) We have made a submission to the data centre to have this issue reviewed. However, given that such physical moves are so rare, it’s unlikely we’ll be in a position to test whether or not lessons have been learnt. For ourselves, we’ve learnt that a large part of the problem could have been avoided if we actually hosted most domains on a secondary IP address, rather than the primary. We’ll consider following through on this, but given other plans that will come to pass long before the chance of another physical move comes about, we may not do this at this time.
2. Secondly, a script that we were assured by the provider of the control panel would work to assign domains quickly to the new IP address as soon as the server came back online, had no effect. The lesson here is that nothing can take the place of exhaustive testing.

I mentioned above that websites on their own IP address experienced no problems “initially”. Once trouble tickets were opened with the data centre, we and their technicians were working at cross purposes at one point, and they essentially redid work we had done to bring websites on the primary IP address online, and at the same time taking down those websites (including the NinerNet website) on their own IP addresses. When this was discovered it was quickly fixed.

We had some reports from clients that email was arriving out of order. This is to be expected when a server has been offline for a while. This is what happens: Let’s say an email is sent 5 minutes after the server goes offline. It can’t be delivered, so the sending mail server holds onto it and tries again in 5 minutes. It still can’t be delivered, so it tries again in 10 minutes, then half an hour, then every hour, and so on. So if the server comes back online part way through the hour wait, but a different email is sent a minute after the server comes back online, that newer email will be delivered immediately, as usual, but the older email won’t be delivered until the hour wait has expired.

Clients hosting some or all of their services on a server other than NC018 and using the third nameserver we provided were up for the duration of the server move.

There was a minor issue with some outbound email that was on the server before the move. We’re still investigating that. However, there were no issues with inbound email that we’re aware of.

Unrelated to the move itself was the fact that posts to our Identi.ca and Twitter accounts did not appear. Of course, these services are independent of NinerNet — which is part of the point, actually — so this was beyond our control. Our status website remained online at status.ninernet.net. It will revert to status.niner.net, but will still be available at the former address, now and in the future.

Again, we appreciate your patience and understanding during this necessary move. If you have any questions or concerns, please do not hesitate to let us know.

Craig

On 24 July between 01:00 and 13:00 UTC we will be moving server NC018 to a new, “greener”, state-of-the-art data centre. Because this is a physical move — i.e., the server will be carried from one location to another — the server must be powered down, disconnected, moved to the new data centre, reconnected and powered up again. This means the server — and some or all of the services hosted on it — will be unavailable for up to 12 hours this weekend.

Here are the dates and times in some major time zones:

UTC:     24 July, 01:00-13:00
PDT:  23-24 July, 18:00-06:00
CAT:     24 July, 03:00-15:00
AWST:    24 July, 09:00-21:00

Please visit the World Time Server website to convert this date and time into your own time zone if it’s not listed above. Please also ensure that you pass this information along to employees, colleagues, developers, customers, etc., so that they are aware of the outage in advance.

Many of the redundancies already in place will help ensure that the effect of this maintenance outage will be minimised — apart from the fact that, of course, the server will be offline for all or most of 12 hours. However, the recovery from this downtime will be quick because of these redundancies.

Here is some service-specific information that you should be aware of:

IP address

If your website is currently hosted on IP address 65.61.157.54, it will be hosted on 72.3.245.152 after the move. This will not be of interest to most clients, but there are some for whom this might be important.

The IP address of NinerNet’s primary mail server (mail.niner.net) will also change to 72.3.245.152. Again, this will not be of interest to most NinerNet clients. If it is of interest or concern to you, then you will already know that. Such instances usually apply to configuring firewalls, or other security considerations that are based on IP addresses rather than domain names.

DNS

A few days before the move, we will be lowering the length of time that DNS (domain name system) information is cached for your domain around the world. Immediately after the server comes back online, we will then update the DNS information for your domain and associated services so that, if your domain or an associated service is using the new IP address, the change will propagate within minutes.

Communication

Following this post, before and after the move we will be communicating important information through our status website at status.niner.net. However, because the niner.net domain itself will be offline during this move, the status website will also be available at its alternative address: status.ninernet.net. Please make a note of that address and use it to seek updates during the move.

NinerNet website

As indicated above, the niner.net domain and all sub-domains on the niner.net domain will be offline during this move. This includes the main NinerNet website.

Email

All email accounts and forwarders (redirects) hosted on server NC018 will be unavailable during the move. Incoming email will be held on the sending servers until server NC018 is back online, at which point it will be delivered. While some email may arrive out of order, no email will be lost; it will only be delayed. Webmail will be unavailable.

Special cases

Because we have quite a number of clients with unique configurations, those clients may be less affected by this outage than if all of their services were hosted on server NC018. In these cases — all of which assume that you are using the standard NinerNet nameservers (i.e., ns*.niner.net, where the asterisk is a number) — we will put in place (if it’s not already in place for your domain) an extra nameserver that will ensure that your self-hosted mail server or other service remains online during the move. Here are some examples, some of which may apply to you:

• Self- or other externally-hosted mail server: If you host your own mail server or host your email with a third party (e.g., Google Apps or your ISP), that service will remain online during the move.
• Website not hosted on NC018: If you host your website using our virtual private server (VPS) service, on the PHP 5 server (NC020) or with a third party, your website will remain online.

Both of the above assume that we have access to your domain registration to add the extra nameserver. If your domain is not registered through NinerNet, then you will need to add the extra nameserver yourself. That extra nameserver is ns3.niner.net, and its IP address (in case you need it) is 173.45.228.52. Please contact support before adding ns3.niner.net to your list of nameserver to confirm that you should do so.

• DNS hosted elsewhere: If you use your own or a third party’s nameservers, but point one or more of your services to server NC018, the service hosted on NC018 will be offline during the move. Other services pointing to other servers will remain online. If your website or some other service was hosted on IP address 65.61.157.54, please update your DNS to point it to 72.3.245.152 during the move window.
• Zam.co domains: If you have registered a .zam.co domain (e.g., example.zam.co), your domain will remain online.
• SpamSlip.com: Your rotating anti-spam email addresses will continue to work.

In the case of email, a website or any other service hosted with a third party, please contact support to ensure that we’re aware of your configuration and that we have assigned or will assign you an extra nameserver.

Also, please be aware that although your service hosted on another server will remain online, performance may be slightly degraded during the server move. The degradation will be almost negligible, and performance will return to normal after the move has completed.

Emergency contact information

If your domain or service hosted on server NC018 is not back online within 30 minutes of the scheduled conclusion of this maintenance, please check the status website at either status.niner.net or status.ninernet.net for updates that may explain the situation. If updates there indicate that everything is (or should be) back to normal, please follow these steps, checking to see that your domain or service is still down after each step:

1. Reboot your computer.
2. If that doesn’t fix the problem, reboot your router, modem, and any other connection equipment.
3. If that doesn’t fix the problem, please ask someone else — i.e., someone in another location (not the same building) that you have to phone to talk to them — to see if they can load your website.
4. If that person cannot load your website, use the service at Just-Ping to see if the server is up.
5. If that indicates that the server is down, please send an emergency email through the NinerNet website.
6. If you cannot load the NinerNet website, please send an email to (deleted).
7. If you use Skype, add NinerNet.Support to your list of contacts and talk to someone.
8. As a last resort, please phone one of the following numbers:
   • Vancouver, Canada: 604 715 7263
   • Toll-free in North America: 1 855 NINERNET (1 855 646 3763)
   • Outside of North America: +1 604 715 7262

We appreciate your business and your patience, and most of all your understanding during this maintenance to improve the services that we deliver to you. Please contact NinerNet support if you have any questions or concerns.