Corporate Blog

Upcoming changes to mail servers

20 December 2020 12:52:47 +0000

The email world is constantly evolving. More to the point, spam is a never-ending arms race. We have made some changes to our email system, and in the New Year we will be making more.

So far all we have done is add a second alternative route for outbound emails. This gives us (and our clients) a third possible point from which emails can be delivered to your recipients. This action is the result of our data centres’ IP addresses finding themselves in more blacklists as a result of poor management, and the bad behaviour of their other customers.

Our use of this service will result in some very minor changes to the headers of some of these emails when viewed by the recipients. Almost nobody pays any attention to the headers of emails until there is a problem, but we are telling our clients this in advance so that you are aware of it.

There is nothing you need to change in your email programs or apps. The only thing you need to do is forward errors to us if a bounce message for an email you have sent refers to being blocked, as opposed to the destination address not existing or being full. If your email was blocked we can divert future emails to that domain via an alternative route. This option has always been available, but for the reason stated above we’re getting more reports now than in the past.

That addresses an immediate issue. Plans were already in progress for a scheduled upgrade to our primary mail server, but now they have an additional focus: We will be setting up a new mail server in another data centre where the reputations of their IP addresses are an explicit priority.

This plan will probably protect NinerNet for a couple more years. However, with the way the email world is moving, there are some predictions that all IP addresses will eventually be blocked from sending email except for a select few. I don’t believe I need to explain how this will concentrate power over email in the hands of a few, and how detrimental this will be, so we expect that reputable data centres will oppose this. Those are the kinds of data centres we want to work with, but we will maintain accounts with third-party relay providers just in case.

We will be posting more on the subject of email, specifically details of our migration, and information you need to know to ensure that your email will not be considered spam, either by us, our spam filters or your customers.

Please contact NinerNet support If you have any questions or concerns. Thank-you.

Tags: email, maintenance, reputation, spam
Categories: Notices
Comments Off

Configuring our servers against “POODLE”, SSL/TLS, and email security

24 October 2014 15:52:19 +0000

The maintenance to protect against the “POODLE” exploit has been finished, as we’ve noted on our status blog. While I’d like this to be a short post stating just that, like the maintenance itself, there is more to it than meets the eye.

What was anticipated to take about an hour during a scheduled weekend maintenance window ended up taking much longer as we waded through the pros and cons of configuring some or all services to disable SSL version 3. (Of course, very few people know about and can prepare for these things in advance.) First, there was some debate in information security circles about just how serious this issue was/is, how quickly it needed to be addressed, and by whom. In short, some took it more seriously than others, but there was general agreement that other issues (Heartbleed and Shellshock, for example) were much bigger. Those that didn’t feel it was that serious had their reasons, but we’re not in business to gamble with your security.

While this is a vulnerability in a protocol (SSL version 3) that is (or has been) used to secure different types of connections, the main area of concern was with HTTPS connections — i.e., web browsing. To my knowledge, the only known exploit of this protocol vulnerability uses JavaScript, and only over HTTPS connections. In other words, there is currently no known issue with using SSLv3 to secure non-HTTPS connections — e.g., email.

To that end SSLv3 will still work on some of our mail servers. How this is handled — if your email program can’t use TLS — differs between email programs, with some email clients failing silently and establishing a non-secure connection instead, and some failing completely to connect. We expect that most email programs using our existing suggested configurations will continue to work across all of our servers. However, while we have not had any reports of issues from clients, one of the reasons this took longer than anticipated was the surprising number of current or recent email clients that stopped working when we disabled SSLv3 on the mail servers. Connections by email clients configured to use SSLv3 still work on server NC018, while on NC027 they will fail silently as described above. This is related to the differing behaviour of the software running these two mail servers.

All web servers (including control panels) were configured to deny SSLv3 connections by Monday this week. Web browser developers seem to have kept up with and done a better job implementing TLS in current versions than some email client developers. As we’ve stated several times previously, Outlook 2003 should be relegated to the past, along with Microsoft Internet Explorer version 6. The latter uses only SSL (and has TLS disabled) by default. Microsoft, of all people, have actually had an active campaign to discourage the use of MSIE 6 since 2009 with their ie6countdown.com website; according to that website, only 3.3% of users worldwide are still using MSIE 6, and about three quarters of them are in China. Put it this way, using MSIE 6 today is like trying to drive a Model T Ford on modern roads among modern cars, expecting to go as fast as modern cars and to be serviced by modern mechanics. In short, using certain software today is simply a bad idea, even if it still appears to some people to work.

Another thing I’d like to address here is the difference between SSL (secure sockets layer) and TLS (transport layer security) … or, more correctly, the perceived difference. There is no difference. They are essentially the same thing. For all intents and purposes, the lay person can consider TLS version 1.0 to be SSL version 4.0. That’s not true from a technical standpoint, but as someone who deals every day with clients who just want their computers to work and are more concerned about the intricacies of their trucking business (for example), they do the same thing: encrypt your Internet connections. TLS, as the successor to SSL, is newer and better (as the “SSL version 4.0” comparison above makes clear), and you should use TLS in preference to SSL any time you have a choice.

Finally, a word about email security. It has become more and more clear to me over the years that the trend in software development is to hide things from the average user. There is a point to which this is good; after all, if you had to type in all of the commands that your email program (for example) uses to connect to the mail server to download or send your email, you might as well write a letter with a quill and ink and send it via carrier pigeon. However, if your email program is going to fail silently and send your message in the clear — i.e., over an unencrypted connection — that’s something you probably want to know about if you thought you were using an encrypted connection. But this is not something you will read about in glossy brochures extolling the virtues of this email program or that. The fact is, most people will never be aware of such an issue, and those that have the most to fear — for example, people living in or reporting on dictatorships — will only realise they have a problem when there is that ominous knock at the door that reveals their communications have been compromised.

For this reason it is not enough to rely on your email service provider — not even NinerNet Communications — to secure your communications if you are, for example, an activist in a police state or a reporter with confidential sources. No, you have to take that responsibility on yourself by encrypting the actual messages you send before you send them. How to do this is certainly beyond the scope of this post, and even if you were to do it it may not be necessary for all of your communications. But going to this extent to protect yourself in this way takes extra time and effort and may require additional software on your computer, but at the end of the day you need to determine for yourself the pros and cons in your own cost-benefit analysis.

Tags: encryption, maintenance, security, ssl/tls
Categories: Interest, Notices
1 Comment »

SSL version 3 “POODLE” vulnerability

17 October 2014 05:21:12 +0000

The latest in a series of recent vulnerabilities discovered in software commonly used on servers hosting websites and email (among other services) has reared its head. “POODLE” (conveniently discovered by the clever rhymers at Google) is a catchy name for a vulnerability found in a two-decade-old cryptographic protocol used to encrypt network connections. SSL — the secure sockets layer protocol — has become a household word over the years, and those three letters are still now used by many to refer generically to secure connections, even though SSL version 3.0 (published in 1996) was superseded by TLS (transport layer security) version 1.0 fifteen years ago (in 1999).

All of this introductory information is not intended to trivialise the problem, of course, but to give some background and illustrate how it can take a long time for new standards to be adopted, and old ones to be abandoned. Often, old standards live on simply because “if it ain’t broke, don’t fix it” … and now (well, three days ago) we find that the last version of SSL — version 3.0 — is indeed “broke”.

As such we will be re-configuring all of our servers still configured to allow SSL 3.0 connections to use TLS exclusively. This will require reconfiguring and restarting web servers, FTP servers and various email services. While we anticipate the work on all servers taking about an hour, interruptions in service — if there are any — should be brief and last only a few seconds at a time as services are restarted.

Of particular interest — due to a couple of recent support requests related to our newer mail server on NC027 — is that Microsoft Outlook 2003 users will likely no longer be able to connect securely to the mail servers on NC018 and NC023 (the relay server), as Outlook 2003 does not have support for TLS. Apparently a 2004 “hotfix” available from Microsoft will add TLS support to Outlook 2003, but we cannot vouch for this personally, nor are we aware of any clients who have used this. It should be noted that Microsoft stopped supporting Outlook 2003 earlier this year. It is obsolete software.

It is of interest to me personally that my favourite email program of all time — Eudora — will weather this storm and continue to flourish, as it does support TLS. However, sadly, even Eudora will eventually succumb to the ravages of time and the march of technology. In fact, I strongly suspect it only supports TLS version 1.0, and I have noticed that Google actively discourages connections from old email clients such as Eudora, probably because they likely suggest using an email client that supports at least TLS version 1.1. The latest version of TLS is 1.2, already six years old itself.

So, we will be using our weekend maintenance window to perform this maintenance. However, instead of starting at the usual time, this maintenance will begin at 21:00 UTC on Saturday, 18 October and, as stated above, should take roughly one hour. Please consult our status blog for updates on this maintenance, and please contact support if you have any questions or concerns.

Tags: google, maintenance, security
Categories: Notices
1 Comment »

“Shellshock” software bug

26 September 2014 14:17:06 +0000

You may have heard in the media about the so-called Shellshock security issue that affects a software package present on most Internet servers worldwide called “bash”. All of our servers run bash; it is a very basic building block on almost all UNIX- and Linux-based servers, which run most services on the Internet that you access every day. Bash can be loosely compared to the “command line” available on Windows-based computers.

Upon checking, we determined that the version of bash running on all of our servers was vulnerable to exploits aimed at the bug. All were immediately patched, and are no longer vulnerable. We continue to monitor security bulletins from the vendors of the operating systems we use for possible further patches related to newly-discovered vulnerabilities, should they materialise.

NinerNet takes keeping our servers updated and secure seriously. If you have any questions about this in general or this bug in particular, please contact us. Thank-you.

Tags: maintenance, security
Categories: Notices
Comments Off

Server NC018 move: The aftermath

25 July 2011 07:39:44 +0000

The move of server NC018 to the new data centre has been completed. Due to two failures, the downtime for some websites was longer than the planned 12 hours. These two failures were as follows:

For some reason the data centre did not actually configure the server to use the new IP address, even though this was expressly a part of — and indeed a requirement of — the move. This resulted in most websites being down when the server came back online because most websites on the server use the server’s primary IP address. (Websites that have their own or share a secondary IP address had no problems, initially.) We have made a submission to the data centre to have this issue reviewed. However, given that such physical moves are so rare, it’s unlikely we’ll be in a position to test whether or not lessons have been learnt. For ourselves, we’ve learnt that a large part of the problem could have been avoided if we actually hosted most domains on a secondary IP address, rather than the primary. We’ll consider following through on this, but given other plans that will come to pass long before the chance of another physical move comes about, we may not do this at this time.
Secondly, a script that we were assured by the provider of the control panel would work to assign domains quickly to the new IP address as soon as the server came back online, had no effect. The lesson here is that nothing can take the place of exhaustive testing.

I mentioned above that websites on their own IP address experienced no problems “initially”. Once trouble tickets were opened with the data centre, we and their technicians were working at cross purposes at one point, and they essentially redid work we had done to bring websites on the primary IP address online, and at the same time taking down those websites (including the NinerNet website) on their own IP addresses. When this was discovered it was quickly fixed.

We had some reports from clients that email was arriving out of order. This is to be expected when a server has been offline for a while. This is what happens: Let’s say an email is sent 5 minutes after the server goes offline. It can’t be delivered, so the sending mail server holds onto it and tries again in 5 minutes. It still can’t be delivered, so it tries again in 10 minutes, then half an hour, then every hour, and so on. So if the server comes back online part way through the hour wait, but a different email is sent a minute after the server comes back online, that newer email will be delivered immediately, as usual, but the older email won’t be delivered until the hour wait has expired.

Clients hosting some or all of their services on a server other than NC018 and using the third nameserver we provided were up for the duration of the server move.

There was a minor issue with some outbound email that was on the server before the move. We’re still investigating that. However, there were no issues with inbound email that we’re aware of.

Unrelated to the move itself was the fact that posts to our Identi.ca and Twitter accounts did not appear. Of course, these services are independent of NinerNet — which is part of the point, actually — so this was beyond our control. Our status website remained online at status.ninernet.net. It will revert to status.niner.net, but will still be available at the former address, now and in the future.

Again, we appreciate your patience and understanding during this necessary move. If you have any questions or concerns, please do not hesitate to let us know.

Craig

Tags: down time, maintenance
Categories: Notices
1 Comment »

Physical move of server NC018 this weekend

18 July 2011 05:07:49 +0000

On 24 July between 01:00 and 13:00 UTC we will be moving server NC018 to a new, “greener”, state-of-the-art data centre. Because this is a physical move — i.e., the server will be carried from one location to another — the server must be powered down, disconnected, moved to the new data centre, reconnected and powered up again. This means the server — and some or all of the services hosted on it — will be unavailable for up to 12 hours this weekend.

Here are the dates and times in some major time zones:

UTC:     24 July, 01:00-13:00
PDT:  23-24 July, 18:00-06:00
CAT:     24 July, 03:00-15:00
AWST:    24 July, 09:00-21:00

Please visit the World Time Server website to convert this date and time into your own time zone if it’s not listed above. Please also ensure that you pass this information along to employees, colleagues, developers, customers, etc., so that they are aware of the outage in advance.

Many of the redundancies already in place will help ensure that the effect of this maintenance outage will be minimised — apart from the fact that, of course, the server will be offline for all or most of 12 hours. However, the recovery from this downtime will be quick because of these redundancies.

Here is some service-specific information that you should be aware of:

IP address

If your website is currently hosted on IP address 65.61.157.54, it will be hosted on 72.3.245.152 after the move. This will not be of interest to most clients, but there are some for whom this might be important.

The IP address of NinerNet’s primary mail server (mail.niner.net) will also change to 72.3.245.152. Again, this will not be of interest to most NinerNet clients. If it is of interest or concern to you, then you will already know that. Such instances usually apply to configuring firewalls, or other security considerations that are based on IP addresses rather than domain names.

DNS

A few days before the move, we will be lowering the length of time that DNS (domain name system) information is cached for your domain around the world. Immediately after the server comes back online, we will then update the DNS information for your domain and associated services so that, if your domain or an associated service is using the new IP address, the change will propagate within minutes.

Communication

Following this post, before and after the move we will be communicating important information through our status website at status.niner.net. However, because the niner.net domain itself will be offline during this move, the status website will also be available at its alternative address: status.ninernet.net. Please make a note of that address and use it to seek updates during the move.

NinerNet website

As indicated above, the niner.net domain and all sub-domains on the niner.net domain will be offline during this move. This includes the main NinerNet website.

Email

All email accounts and forwarders (redirects) hosted on server NC018 will be unavailable during the move. Incoming email will be held on the sending servers until server NC018 is back online, at which point it will be delivered. While some email may arrive out of order, no email will be lost; it will only be delayed. Webmail will be unavailable.

Special cases

Because we have quite a number of clients with unique configurations, those clients may be less affected by this outage than if all of their services were hosted on server NC018. In these cases — all of which assume that you are using the standard NinerNet nameservers (i.e., ns*.niner.net, where the asterisk is a number) — we will put in place (if it’s not already in place for your domain) an extra nameserver that will ensure that your self-hosted mail server or other service remains online during the move. Here are some examples, some of which may apply to you:

Self- or other externally-hosted mail server: If you host your own mail server or host your email with a third party (e.g., Google Apps or your ISP), that service will remain online during the move.
Website not hosted on NC018: If you host your website using our virtual private server (VPS) service, on the PHP 5 server (NC020) or with a third party, your website will remain online.

Both of the above assume that we have access to your domain registration to add the extra nameserver. If your domain is not registered through NinerNet, then you will need to add the extra nameserver yourself. That extra nameserver is ns3.niner.net, and its IP address (in case you need it) is 173.45.228.52. Please contact support before adding ns3.niner.net to your list of nameserver to confirm that you should do so.

DNS hosted elsewhere: If you use your own or a third party’s nameservers, but point one or more of your services to server NC018, the service hosted on NC018 will be offline during the move. Other services pointing to other servers will remain online. If your website or some other service was hosted on IP address 65.61.157.54, please update your DNS to point it to 72.3.245.152 during the move window.
Zam.co domains: If you have registered a .zam.co domain (e.g., example.zam.co), your domain will remain online.
SpamSlip.com: Your rotating anti-spam email addresses will continue to work.

In the case of email, a website or any other service hosted with a third party, please contact support to ensure that we’re aware of your configuration and that we have assigned or will assign you an extra nameserver.

Also, please be aware that although your service hosted on another server will remain online, performance may be slightly degraded during the server move. The degradation will be almost negligible, and performance will return to normal after the move has completed.

Emergency contact information

If your domain or service hosted on server NC018 is not back online within 30 minutes of the scheduled conclusion of this maintenance, please check the status website at either status.niner.net or status.ninernet.net for updates that may explain the situation. If updates there indicate that everything is (or should be) back to normal, please follow these steps, checking to see that your domain or service is still down after each step:

Reboot your computer.
If that doesn’t fix the problem, reboot your router, modem, and any other connection equipment.
If that doesn’t fix the problem, please ask someone else — i.e., someone in another location (not the same building) that you have to phone to talk to them — to see if they can load your website.
If that person cannot load your website, use the service at Just-Ping to see if the server is up.
If that indicates that the server is down, please send an emergency email through the NinerNet website.
If you cannot load the NinerNet website, please send an email to (deleted).
If you use Skype, add NinerNet.Support to your list of contacts and talk to someone.
As a last resort, please phone one of the following numbers:
- Vancouver, Canada: 604 715 7263
- Toll-free in North America: 1 855 NINERNET (1 855 646 3763)
- Outside of North America: +1 604 715 7262

We appreciate your business and your patience, and most of all your understanding during this maintenance to improve the services that we deliver to you. Please contact NinerNet support if you have any questions or concerns.

Tags: down time, maintenance
Categories: Notices
1 Comment »