Corporate Blog

Crowdstrike incident: Client update

20 July 2024 13:24:51 +0000

After a very trying day for many customers around the world that use Microsoft Windows or rely on companies that use Microsoft Windows — like Hotmail/Outlook.com, Office 365, Google Cloud / Compute Engine, Amazon Web Services (AWS), Azure, etc., etc., etc. — we would like to take this opportunity to ensure that our clientele know that we were wholly unaffected by the worldwide chaos.

Is this blog post an opportunistic jab at people who rely on an operating system and company that was late to realise the potential of the Internet? Yes, of course. Why? Well, just look at the trouble that Microsoft gave us last month, and are still giving us today. Microsoft are not our favourite people these days, even though Microsoft themselves weren’t responsible for the Crowdstrike failure.

Hey, we get it, shi … stuff happens. Our status blog currently shows 207 posts in the “incidents” category since 2009. Of course, that’s not 207 failures; at the very most it’s 104 failures if you assume a post announcing an incident and a second announcing it’s over, but in reality some incidents had multiple posts and some posts were only to alert clients to issues with other companies. I’d say that there were far fewer than 100 incidents in fifteen years; feel free to do the maths and check our live uptime monitor for yourself. But one does wonder how an update was pushed out by Crowdstrike without it being tested. That’s just unfathomable. On the other hand, NinerNet doesn’t check every single update we apply to our servers, but we have to rely on our operating system vendors to do that for us. As Crowdstrike customers and their customers found out yesterday, the IT world is very interdependent.

Of course, NinerNet will almost certainly have some major incident in the future, and I know that some will then say that this post will come back to bite us in the ass. Not really. I’m always amused when an incident happens and people say or claim, “We will learn and it will never happen again!” That cracks me up. Incidents — whether they are global IT meltdowns or plane crashes — are almost always human-caused. So yeah, it will happen again, and NinerNet will have some issue at some time in the future and we will learn from it and promise that we will take steps to prevent it from happening again. But we have never and will never claim that it will never happen again.

The other purpose of this post is for marketing. The word “marketing” is a four-letter word to me, simply because about the only skill that marketers have is the ability to lie, with a straight face. I certainly wouldn’t accuse Microsoft or Crowdstrike of any kind of over-marketing or marketing subterfuge but, you know, there’s a part of me that looks askance at claims made by companies that over-promise and under-deliver … and over-promising and under-delivering are pretty much the meat and potatoes of marketers! It is far beyond my remit to determine whether or not either Microsoft or Crowdstrike have ever over-promised or under-delivered, but yesterday under-delivery was rampant.

Update, 2024-07-24: I wasn’t planning to drive home any of my points above, but I was cleaning up some open browser tabs and there were a few Crowdstrike-related tabs still open.

At “Helping our customers through the CrowdStrike outage” Microsoft proudly states, “We currently estimate that CrowdStrike’s update affected 8.5 million Windows devices, or less than one percent of all Windows machines.” Umm, so? Your point is? What they fail to state here is that those 8.5 million Windows devices affected many, many more millions (a billion?) of poor saps who rely on companies that rely on Crowdstrike that relies on Microsoft’s crappy operating system. It reminds me of a saying: “Figures don’t lie, but liars sure as hell can figure!”.

George Kurtz, CEO of Crowdstrike, also stated in a tweet, “Today was not a security or cyber incident. Our customers remain fully protected.” This statement is freaking hilarious! If you can’t turn on your “Windows device”, of course it’s “fully protected”! OMG, this is one for the comedy annals!

Browsing through Mr. Kurtz’s Twitter feed you see a lot of the aforementioned “marketing”:

* Wow… another great quarter in the books for $CRWD.
* $CRWD delivered a strong 2Q23 with record $218M net new ARR, $2.14B ending ARR, record net new customers & $136M free cash flow.
* $CRWD delivered record Q4 results.

*yawn*

Tags: down time, reputation, security
Categories: Interest, Warnings
Comments Off

Massive outage at massive hosting company

3 August 2013 13:05:11 +0000

Hardly a week goes by that there isn’t a notable outage of one sort or another in the IT business. But some — like the one experienced by Endurance International Group yesterday — are more notable than others.

Endurance International Group is one of the world’s largest hosting companies, and specifically the parent company of the “stack ’em deep and sell ’em cheap” hosting brands Bluehost, Hostmonster, Hostgator and Just Host (as well as a plethora of others), with only one of them honest enough to put an asterisk beside their “unlimited” claims on their home page explaining how “unlimited” in their dictionary doesn’t mean what it probably means in your dictionary.

Anyway, now is not the time to kick a competitor when they’re down over minor issues like what they say and what they mean. But it is interesting to note that a massive operation claiming to host millions of domains and/or websites apparently had no system in place for communicating with customers during such an outage. We’re not immune to the occasional technical glitch and communication fumble ourselves, but Endurance actually needed to go out and register a new domain and hastily set up a blog to keep their customers updated.

Kudos to them for doing what was needed when it was needed, but one does wonder if they’ll be maintaining this website for future issues.

Update, 2016-02-01: Nope, they didn’t. It’s now being squatted by a Japanese domain squatter.

Tags: bluehost, communication, down time, endurance international group, hostgator, hostmonster, just host, reputation
Categories: Interest, Updated posts
Comments Off

Server NC018 move: The aftermath

25 July 2011 07:39:44 +0000

The move of server NC018 to the new data centre has been completed. Due to two failures, the downtime for some websites was longer than the planned 12 hours. These two failures were as follows:

For some reason the data centre did not actually configure the server to use the new IP address, even though this was expressly a part of — and indeed a requirement of — the move. This resulted in most websites being down when the server came back online because most websites on the server use the server’s primary IP address. (Websites that have their own or share a secondary IP address had no problems, initially.) We have made a submission to the data centre to have this issue reviewed. However, given that such physical moves are so rare, it’s unlikely we’ll be in a position to test whether or not lessons have been learnt. For ourselves, we’ve learnt that a large part of the problem could have been avoided if we actually hosted most domains on a secondary IP address, rather than the primary. We’ll consider following through on this, but given other plans that will come to pass long before the chance of another physical move comes about, we may not do this at this time.
Secondly, a script that we were assured by the provider of the control panel would work to assign domains quickly to the new IP address as soon as the server came back online, had no effect. The lesson here is that nothing can take the place of exhaustive testing.

I mentioned above that websites on their own IP address experienced no problems “initially”. Once trouble tickets were opened with the data centre, we and their technicians were working at cross purposes at one point, and they essentially redid work we had done to bring websites on the primary IP address online, and at the same time taking down those websites (including the NinerNet website) on their own IP addresses. When this was discovered it was quickly fixed.

We had some reports from clients that email was arriving out of order. This is to be expected when a server has been offline for a while. This is what happens: Let’s say an email is sent 5 minutes after the server goes offline. It can’t be delivered, so the sending mail server holds onto it and tries again in 5 minutes. It still can’t be delivered, so it tries again in 10 minutes, then half an hour, then every hour, and so on. So if the server comes back online part way through the hour wait, but a different email is sent a minute after the server comes back online, that newer email will be delivered immediately, as usual, but the older email won’t be delivered until the hour wait has expired.

Clients hosting some or all of their services on a server other than NC018 and using the third nameserver we provided were up for the duration of the server move.

There was a minor issue with some outbound email that was on the server before the move. We’re still investigating that. However, there were no issues with inbound email that we’re aware of.

Unrelated to the move itself was the fact that posts to our Identi.ca and Twitter accounts did not appear. Of course, these services are independent of NinerNet — which is part of the point, actually — so this was beyond our control. Our status website remained online at status.ninernet.net. It will revert to status.niner.net, but will still be available at the former address, now and in the future.

Again, we appreciate your patience and understanding during this necessary move. If you have any questions or concerns, please do not hesitate to let us know.

Craig

Tags: down time, maintenance
Categories: Notices
1 Comment »

Physical move of server NC018 this weekend

18 July 2011 05:07:49 +0000

On 24 July between 01:00 and 13:00 UTC we will be moving server NC018 to a new, “greener”, state-of-the-art data centre. Because this is a physical move — i.e., the server will be carried from one location to another — the server must be powered down, disconnected, moved to the new data centre, reconnected and powered up again. This means the server — and some or all of the services hosted on it — will be unavailable for up to 12 hours this weekend.

Here are the dates and times in some major time zones:

UTC:     24 July, 01:00-13:00
PDT:  23-24 July, 18:00-06:00
CAT:     24 July, 03:00-15:00
AWST:    24 July, 09:00-21:00

Please visit the World Time Server website to convert this date and time into your own time zone if it’s not listed above. Please also ensure that you pass this information along to employees, colleagues, developers, customers, etc., so that they are aware of the outage in advance.

Many of the redundancies already in place will help ensure that the effect of this maintenance outage will be minimised — apart from the fact that, of course, the server will be offline for all or most of 12 hours. However, the recovery from this downtime will be quick because of these redundancies.

Here is some service-specific information that you should be aware of:

IP address

If your website is currently hosted on IP address 65.61.157.54, it will be hosted on 72.3.245.152 after the move. This will not be of interest to most clients, but there are some for whom this might be important.

The IP address of NinerNet’s primary mail server (mail.niner.net) will also change to 72.3.245.152. Again, this will not be of interest to most NinerNet clients. If it is of interest or concern to you, then you will already know that. Such instances usually apply to configuring firewalls, or other security considerations that are based on IP addresses rather than domain names.

DNS

A few days before the move, we will be lowering the length of time that DNS (domain name system) information is cached for your domain around the world. Immediately after the server comes back online, we will then update the DNS information for your domain and associated services so that, if your domain or an associated service is using the new IP address, the change will propagate within minutes.

Communication

Following this post, before and after the move we will be communicating important information through our status website at status.niner.net. However, because the niner.net domain itself will be offline during this move, the status website will also be available at its alternative address: status.ninernet.net. Please make a note of that address and use it to seek updates during the move.

NinerNet website

As indicated above, the niner.net domain and all sub-domains on the niner.net domain will be offline during this move. This includes the main NinerNet website.

Email

All email accounts and forwarders (redirects) hosted on server NC018 will be unavailable during the move. Incoming email will be held on the sending servers until server NC018 is back online, at which point it will be delivered. While some email may arrive out of order, no email will be lost; it will only be delayed. Webmail will be unavailable.

Special cases

Because we have quite a number of clients with unique configurations, those clients may be less affected by this outage than if all of their services were hosted on server NC018. In these cases — all of which assume that you are using the standard NinerNet nameservers (i.e., ns*.niner.net, where the asterisk is a number) — we will put in place (if it’s not already in place for your domain) an extra nameserver that will ensure that your self-hosted mail server or other service remains online during the move. Here are some examples, some of which may apply to you:

Self- or other externally-hosted mail server: If you host your own mail server or host your email with a third party (e.g., Google Apps or your ISP), that service will remain online during the move.
Website not hosted on NC018: If you host your website using our virtual private server (VPS) service, on the PHP 5 server (NC020) or with a third party, your website will remain online.

Both of the above assume that we have access to your domain registration to add the extra nameserver. If your domain is not registered through NinerNet, then you will need to add the extra nameserver yourself. That extra nameserver is ns3.niner.net, and its IP address (in case you need it) is 173.45.228.52. Please contact support before adding ns3.niner.net to your list of nameserver to confirm that you should do so.

DNS hosted elsewhere: If you use your own or a third party’s nameservers, but point one or more of your services to server NC018, the service hosted on NC018 will be offline during the move. Other services pointing to other servers will remain online. If your website or some other service was hosted on IP address 65.61.157.54, please update your DNS to point it to 72.3.245.152 during the move window.
Zam.co domains: If you have registered a .zam.co domain (e.g., example.zam.co), your domain will remain online.
SpamSlip.com: Your rotating anti-spam email addresses will continue to work.

In the case of email, a website or any other service hosted with a third party, please contact support to ensure that we’re aware of your configuration and that we have assigned or will assign you an extra nameserver.

Also, please be aware that although your service hosted on another server will remain online, performance may be slightly degraded during the server move. The degradation will be almost negligible, and performance will return to normal after the move has completed.

Emergency contact information

If your domain or service hosted on server NC018 is not back online within 30 minutes of the scheduled conclusion of this maintenance, please check the status website at either status.niner.net or status.ninernet.net for updates that may explain the situation. If updates there indicate that everything is (or should be) back to normal, please follow these steps, checking to see that your domain or service is still down after each step:

Reboot your computer.
If that doesn’t fix the problem, reboot your router, modem, and any other connection equipment.
If that doesn’t fix the problem, please ask someone else — i.e., someone in another location (not the same building) that you have to phone to talk to them — to see if they can load your website.
If that person cannot load your website, use the service at Just-Ping to see if the server is up.
If that indicates that the server is down, please send an emergency email through the NinerNet website.
If you cannot load the NinerNet website, please send an email to (deleted).
If you use Skype, add NinerNet.Support to your list of contacts and talk to someone.
As a last resort, please phone one of the following numbers:
- Vancouver, Canada: 604 715 7263
- Toll-free in North America: 1 855 NINERNET (1 855 646 3763)
- Outside of North America: +1 604 715 7262

We appreciate your business and your patience, and most of all your understanding during this maintenance to improve the services that we deliver to you. Please contact NinerNet support if you have any questions or concerns.

Tags: down time, maintenance
Categories: Notices
1 Comment »

How NOT to transfer your hosting

19 March 2011 12:52:28 +0000

More often than not, we’re helping new clients transfer their hosting from their former hosting provider to NinerNet. Over the years we’ve become very good at this, and we have a tried-and-true process we follow to make sure there are no problems and that (most importantly) the new client’s email and website do not go down. There is simply no need (or excuse) for even a second of down time when you’re moving your hosting from one place to another.

Transferring your hosting takes time and planning; not that much time, but it’s not something that’s completed in five minutes while you’re on the phone. When a potential client contacts us about transferring in, we ask a number of questions and then send them a detailed plan laying out our step-by-step transfer process, how long each step takes, setting out what needs to be done and who does what, and the checks and balances that happen at each step. The transfer document is a little lengthy (if you’ll excuse the oxymoron) and potentially intimidating, but it has a bullet point summary and the details are there for the purpose of full transparency.

The point of this post though, is how not to transfer your hosting. We do all of the legwork detailed above because — guess what? — we want this new client and we’re willing to do the legwork up front for years of business from a happy client down the road. Sometimes though, I have to admit, we see the occasional client transfer away from us to one of our competitors. This can sometimes be a painful process for us to watch — not just because we’re losing a client, but because we see the amateurish way in which the transfer is handled. Often this is because of one of two reasons:

The client is transferring to a “stack ’em deep and sell ’em cheap” hosting company because … well … they’re cheaper than we are. In this case the client is usually on their own during the transfer.
The client is transferring to hosting resold by a web designer or a “search engine optimisation” company that they’re using. In this case, while these companies might be good at web design or SEO, they’re often clueless when it comes to the technical aspects of hosting. Contrary to what some of these companies believe, hosting is about more than just clicking pretty icons in a web-based control panel provided by the aforementioned “stack ’em deep and sell ’em cheap” hosting companies.

So we recently waved goodbye to a long-time client whose ownership had changed hands since they came on board with us seven years ago. (Almost all of our ex-clients left us because of things beyond our control — e.g., business closure, a sales pitch from the aforementioned web designer or SEO “expert”, recommendations of close friends or trusted advisors, etc. — not because they didn’t like the service they were getting from us.) Experience has taught us that, when the client has made up their mind, we have to let go. We do say that we’re sorry to see them go — and we mean it — and we ask if there’s anything we can do to keep their business, but if they’re committed then we back off. At that point they’re in the hands of and following the advice of third parties, and as bad as that advice might turn out to be, we’d be interfering if we tried to point that out. (That’s not always the case. To be fair, most of the clients we’ve lost over the years had no problems when they transferred away from us, but the exceptions stick out in one’s memory.)

And so it was that this client transferred to an SEO company that resells the hosting services of a well-known “stack ’em deep and sell ’em cheap” hosting company. The sad result? The client’s website and email were down for ten days! TEN DAYS! Not ten minutes, but ten days! During those ten days much of the work that the SEO company had done (and been paid to do!) previously was wiped out. Not only that, but in their panic — evident in the dozen or so emails sent rapid-fire in the span of about forty-five minutes — they issued confusing and conflicting instructions which actually resulted in further damage to our ex-client’s reputation in the search engines — not to mention the damage already caused in the eyes of their customers and potential (but probably lost) customers. As I said, it was a painful process to watch.

The bright side to this? We actually have a number of former clients that have transferred away come back to us a year or two later, most recently one about a week ago. Now that’s definitely what I’d call a vote of confidence!

Contacts us if you have any questions about transferring your hosting. We’re here to help.

Tags: down time, hosting transfer, reputation, search engine optimisation, search engine optimization, seo, transparency
Categories: Interest
Comments Off