Corporate Blog

Back on 16 September a client with a dot-zm domain hosted with us came to us with a problem that had started the day before. The majority of this post documents the details of the monumental effort we had to expend to get the domain registrar (Realtime/Hai) and registry (ZICTA, the Zambia Information & Communications Technology Authority) to do their jobs and provide a working domain.

However, the reason for the timing of this post (which we should have made two months ago) is that we are seeing these same problems again with dot-zm domains. Emails sent to existing dot-zm domains, hosted with different hosting companies, are bouncing because these dot-zm domains seem not to exist on the Internet because the dot-zm nameservers are reporting differing, incomplete or even incorrect information for them.

Unfortunately there is nothing that we can do on our servers when ZICTA’s dot-zm nameservers report — incorrectly! — that a dot-zm domain does not exist or directs traffic to the wrong servers.

Our recommendation for years now, due to the shocking unreliability of dot-zm domains, is that you simply should not register dot-zm domains. This seems awfully unpatriotic — after all, you.co.zm is supposed to be a proud acknowledgement of your association with Zambia — but the sad fact of the matter is that your dot-zm domain is actually an embarrassment and a disservice to you, your business and your country.

We have written about this over and over again over the years. Here’s a recap of some of what we’ve written on our blogs (there’s more lost in the mists of time and in emails sent before we set up our corporate and status blogs):

• 2010-08-16: Problems with .com.zm domains
• 2010-08-16: Issues with some .com.zm domains explained
• 2012-07-27: Dot-zm domains down again
• 2012-07-27: Dot-zm domains back
• 2013-06-11: Massive, unannounced deletion of dot-zm domains by Zamnet
• 2013-06-14: Deletion of domains by Zamnet continues

We also have over 50 MB of raw data (some of it compressed) dating back to 2008 documenting problems with dot-zm domains, including all dot-zm domains going down worldwide. It’s the kind of material with which we could write a thesis about TLD mismanagement, if we had the time … but we don’t.

So what happened in September?

Back on 16 September a client with a dot-zm domain hosted with us came to us with a problem that had started the day before. They were suddenly receiving very little email, and they were being told by some of their correspondents that emails sent to their domain were bouncing. Of course, this galvanised us immediately, and we checked all aspects of their domain’s configuration on our nameservers and mail servers. We could find nothing wrong. However, the problem was undeniable; some mail sent to this client’s domain was indeed bouncing.

So we started digging deeper. We found that, without being asked to do so by the domain registrant (our client), the dot-zm domain registry (ZICTA, whose website is again down as I write this) had changed the nameservers of our client’s domain to those of their domain registrar, Realtime Technologies Ltd., now doing business as Hai Alive Telecommunications — whose websites on both of their domains are also down right now. (Seeing a pattern here?) Both the registrar and the registry denied being responsible for changing the nameservers and refused to explain how it happened. The nameservers were changed back to ours, but the problem persisted.

What we found was that the problem was intermittent, hence the fact that some email was getting through and some was being bounced. An intermittent problem is the worst kind of problem because when things are working there is no problem to find, and when things aren’t working you don’t know when the problem will go away and whether or not you’ll find the problem before it disappears. On top of that mail from multiple different and unrelated sources was being bounced, so we couldn’t blame a particular sender’s improperly configured mail server for the problem.

Seeing as the registrar (Realtime/Hai) blamed NinerNet for the problem, and the registry (ZICTA) refused to deal with us, telling us we had to seek help from the registrar (Realtime/Hai) who was too busy blaming us to investigate, some more investigation by NinerNet revealed that one of the six nameservers that run the dot-zm country-code top-level domain (as of today they’re now down to four) was still broadcasting to the world that the authoritative nameservers for the domain were hosted at Realtime/Hai, which was the result of the unauthorised change to the nameservers a couple of days earlier. (One of the six was failing, and one wasn’t providing the information it was supposed to provide, meaning that three of the six nameservers for dot-zm domains were not working properly!)

There was an easy temporary solution to this problem, seeing as ZICTA was being uncooperative: A few keystrokes by someone at Realtime/Hai could have set up the client’s domain on their nameservers, so that when a mail server was incorrectly directed there it would have received the correct information that we would provide to Realtime/Hai. However, Realtime/Hai refused to help unless the client signed a hosting contract with them and paid money up front, despite the fact that the client/registrant had already paid them for a working domain that they expected to work for the entire length of the contract for that domain. This, of course, was an outrageous attempt at extortion and was rejected.

After hammering away at ZICTA for over a week by email (while trying to get Realtime/Hai to take responsibility for addressing the problem, as the domain registrar is supposed to do) and being ignored (including receiving notifications that emails to them had been deleted unread) — except by one person, the head of “Consumer Protection”, who said that he would help me but then also continued to ignore me, never sending me another email again — I finally picked up the phone and eventually (after six tries and being disconnected twice; this is the “Communications Technology Authority”?!) reached a receptionist. (A month earlier when trying unsuccessfully to deal with ZICTA over another issue, the receptionist couldn’t put me through to the person who could help me with my issue because the whole organisation, except apparently the receptionist, was at a staff meeting!) After explaining the situation to the receptionist she stated that I had to deal with Realtime/Hai. After explaining that Realtime/Hai refused to help, she said that she would only continue to talk to me if I would lodge a complaint against Realtime/Hai, so I reluctantly agreed. After being on hold for over ten minutes while she did who knows what, she came back on the line. This time I literally pleaded with her to give me one minute to fully explain the problem and why the problem was caused by ZICTA and how it was within the power of only ZICTA to fix in less than two minutes.

My pleading must have had an effect, as she agreed to put me through to someone else. That person was apparently a “Cyber Security Analyst” with Zambia CIRT (Computer Incident Response Team, whose website is actually up!). I had to actually give him the computer commands to demonstrate the cause and location of the problem. Miraculously he agreed with my assessment — after eight days of banging my head against brick walls, someone finally understood and agreed! — but could do nothing to fix it right away because (although it was nine o’ clock in the morning) the person who was in charge of running the registry back end was not in yet. However, he did assure me that it would be dealt with that day, although it wasn’t until the next day that I finally received written acknowledgement from ZICTA of the problem they had caused, and it was another two days before they fixed it, twelve days after the problem was created!

So, let’s review …

… how a problem with a dot-zm domain is handled by the registry and its registrar, and how long it takes:

• Day one: Client’s dot-zm domain stops working.
• It is found that the nameservers for the domain have been changed to use the registrar’s (Realtime/Hai) nameservers instead of NinerNet’s, without the authority of the domain registrant.
• Nameservers are changed back; registrar (Realtime/Hai) and registry (ZICTA) deny responsibility and refuse to explain.
• Problem persists.
• Registrar (Realtime/Hai) refuses to help, attempts to extort money from registrant for services (domain registration) already paid for.
• Registry (ZICTA) finally responds, refers problem to registrar (Realtime/Hai). Will only address problem with a formal complaint about registrar (Realtime/Hai).
• After pleading with registry (ZICTA) they relent and refer the matter to a staff member who can help.
• We have to hold the hand of the person at the registry (ZICTA) to show him how to confirm the nature and location of the problem.
• Day thirteen: Problem at ZICTA is finally fixed by ZICTA 3 days later, 12 days after they created the problem in the first place.

It is interesting to note that there is almost no information on the ZICTA website about the management of the dot-zm ccTLD (country-code top-level domain; compare that to, for example, the website of Nominet, the organisation that runs the dot-uk ccTLD), and their Facebook page contains nothing but pages and pages of repetitive so-called FAQs about mobile service, and not a single mention (going back to at least the end of 2014) of dot-zm domains and domain registration. Underneath almost all of the posts by ZICTA with FAQs are reams and reams of complaints about how useless ZICTA is and how they do nothing about consumer complaints. See for yourself.

Is there a solution?

Yes, there is. If someone in Zambia cannot be found to provide competent management and direction to ZICTA — and the wherewithal to whip registrars into shape and to educate them on the need to provide customer service without resorting to blaming hosting providers for issues that are demonstrably not theirs — then the job can and should be outsourced to a foreign company with a good track record.

• Just on the other side of the boerewors curtain is ZADNA, the domain authority of South Africa. I don’t know if they offer their services to other national domain authorities to run their ccTLDs, but they’re doing a decent job of running their own (dot-za), have recently (in the last few years) launched a new competitive domain registration system, and launched the new TLDs dot-capetown, dot-durban and dot-joburg. They’re also the leading contender to run dot-africa, if ICANN ever fixes that mess and launches it.
• In New Zealand is CoCCA Registry Services (NZ) Limited, with whom ZICTA already has a relationship as they’re using CoCCA’s registry software and have apparently provided some financial consideration for same. As you can see, CoCCA is already running the registries of at least ten TLDs, most of which are those of other small and developing countries.
• In Canada there is OpenSRS who offer their services to registrars, but who may also be interested in providing service to a ccTLD registry.

In the more immediate term NinerNet has provided, since 2010, the option of registering a dot-zam.co domain — e.g., ninernet.zam.co. We provide it expressly because of the ongoing, years old problems with the management and administration of the dot-zm ccTLD, not to mention the exorbitant cost of dot-zm domains. Instead of registering companyzambia.com (which is probably available, I should point out) you can instead register company.zam.co. It’s the same number of characters as company.com.zm, and only one more than company.co.zm. But more importantly, it’s reliable!

I should point out that we don’t expose the flaws in the dot-zm ccTLD in order to sell dot-zam.co domains; our setting up dot-zam.co is the result of these flaws. If the dot-zm domain system worked properly, we would never have thought to create the dot-zam.co alternative ccTLD, and we’d certainly have no reason to complain. Besides, having been created in 1994, dot-zm had a sixteen-year head start on dot-zam.co! Dot-zam.co is a long way from being a threat to ZICTA and dot-zm.

Finally, you can switch to a new, non-dot-zm domain. Ideally you would never have registered that dot-zm domain in the first place, but it’s an understandable and forgivable mistake. It’s not ideal to switch to a new domain, but where would you be if your domain went down for two weeks and nobody wanted to help you? Here’s how you do it:

1. Register your new non-dot-zm domain.
2. Have your hosting provider set up the hosting for it, and create all of the existing email address on your old domain (e.g., phiri@example.co.zm) on the new domain (e.g., phiri@example.zam.co).
3. Similarly you would upload your existing website to your new domain, reconfigured to use your new domain instead of your old dot-zm domain. (Doing this properly is a little more involved than what I have laid out here and there are options, but it’s very doable and quite straightforward from a technical point of view for a good host that knows what they’re doing.)
4. Disable your old dot-zm domain and “alias” it to your new domain. This means that all email to your old dot-zm domain is automatically redirected to your new domain, and all traffic to your old dot-zm website is redirected to your website on your new domain, preserving your ranking in the search engines and all existing links to your old dot-zm domain.
5. Start using your new domain for email by reconfiguring your email program to use the new domain. While your customers and other contacts may still email you at the old dot-zm domain, your replies will come from your new domain and so their future replies will go straight to your new domain. Of course, you would also advise your customers and contacts of your new domain by email and on your website, and in any other ways that you advertise your business.
6. Keep your old dot-zm domain for at least a year or two, aliased to your new domain. How long you keep your old dot-zm domain is up to you and would depend on a number of factors (we can advise on that), but the good thing is that when (not if) it goes down again, you will use that opportunity to reinforce with your customers and other contacts the importance of using your new domain.

While some less sympathetic hosting companies may charge you for the time involved in making this change (which is actually quite understandable), we will not charge you even one extra ngwee to make this change away from your old dot-zm domain. We also do not charge any extra hosting fees to alias your old dot-zm domain to your new domain. In other words, changing to a new domain will only cost you the price of the new domain.

Appendix

The email below, sent to the person we spoke to at ZICTA (who also seems to be associated with Zambia CIRT) after we finally managed to talk to him, documents the problem experienced by our client with their dot-zm domain. The client’s domain has been redacted for privacy reasons (changed to example.co.zm), as have the names and email addresses of individuals.

From: Craig Hartnett <hxxxxxxx _AT_ niner.net>
To: cxxxxx _AT_ cirt.zm
CC: exxxx.cxxxxxx _AT_ hai-alive.co.zm, exxxx.mxxxx _AT_ hai-alive.co.zm,
     exxxxxx _AT_ rtaa.com.zm, mxxxxxx _AT_ rtaa.com.zm,
     rxxxxx.zxxxx _AT_ hai-alive.co.zm, servicedesk _AT_ hai-alive.co.zm,
     servicedesk _AT_ rtaa.com.zm, cxxxx _AT_ zicta.zm, info _AT_ zicta.zm,
     kxxxxxx _AT_ zicta.zm, nxxxxx _AT_ example.co.zm, ixxxxxx _AT_ jxxxxxxxxx.com
Subject: Problem with example.co.zm domain
Date: Fri, 25 Sep 2015 09:21:22 +0200
Mailer:	Evolution 3.10.4-0ubuntu2
Organization: NinerNet Communications, www.niner.net

Dear Cxxxxx,

Thank-you for taking my call this morning.

As I explained, the nameservers for the dot-zm TLD are not properly
synchronised, and are therefore reporting different nameservers for the
example.co.zm domain. In particular, pch.nic.zm is reporting incorrect
nameservers.

Of course, this results in disruption of the example.co.zm domain, and
this has been the case since at least Tuesday 15 September. At one point
last week the WHOIS showed only Realtime nameservers for the
example.co.zm domain. Realtime claims that this was the result of some
sort of registry activity -- a reset of some sort if I remember
correctly. These were changed back to NinerNet's nameservers but, as I
explained on the phone, one of the dot-zm nameservers is still responding
with Realtime's nameserver instead of NinerNet's.

Realtime have refused to assist thus far -- even though they are the
registrar of this domain -- instead blaming us (NinerNet Communications,
the company hosting the domain) for the problems. However, this is
clearly refuted by the evidence below, which are fresh queries from a
few minutes ago (with date and time stamps) on all of the dot-zm TLD
nameservers.

I have copied this email to all of the people at Realtime who have been
involved thus far, as well as representatives of the registrant of the
domain.

This should take two minutes for someone with access to the pch.nic.zm
nameserver to fix. Please ensure that this is fixed today so that the
registrant of example.co.zm can get back to running their business.

Thank-you.

Craig Hartnett

[00:13:14 leftseat@wrathall ~]$ date -u
Fri Sep 25 07:13:19 UTC 2015
[00:13:19 leftseat@wrathall ~]$ dig example.co.zm ns @hippo.ru.ac.za

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @hippo.ru.ac.za
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 63660
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 3, ADDITIONAL: 2
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; AUTHORITY SECTION:
co.zm.                  86400   IN      NS      pch.nic.zm.
co.zm.                  86400   IN      NS      ns-zm.afrinic.net.
co.zm.                  86400   IN      NS      ns1.zamnet.zm.

;; ADDITIONAL SECTION:
ns1.zamnet.zm.          86400   IN      A       196.46.192.26

;; Query time: 348 msec
;; SERVER: 146.231.128.1#53(146.231.128.1)
;; WHEN: Fri Sep 25 00:13:28 PDT 2015
;; MSG SIZE  rcvd: 137

[00:13:28 leftseat@wrathall ~]$ dig example.co.zm ns @ns1.coppernet.zm

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @ns1.coppernet.zm
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17502
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; Query time: 586 msec
;; SERVER: 41.222.240.15#53(41.222.240.15)
;; WHEN: Fri Sep 25 00:13:32 PDT 2015
;; MSG SIZE  rcvd: 43

[00:13:32 leftseat@wrathall ~]$ dig example.co.zm ns @ns1.zamnet.zm

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @ns1.zamnet.zm
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 52796
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; AUTHORITY SECTION:
example.co.zm.         86400   IN      NS      ns2.niner.net.
example.co.zm.         86400   IN      NS      ns1.niner.net.

;; Query time: 320 msec
;; SERVER: 196.46.192.26#53(196.46.192.26)
;; WHEN: Fri Sep 25 00:13:38 PDT 2015
;; MSG SIZE  rcvd: 88

[00:13:38 leftseat@wrathall ~]$ dig example.co.zm ns @ns2.zamnet.zm

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @ns2.zamnet.zm
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16638
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; AUTHORITY SECTION:
example.co.zm.         86400   IN      NS      ns1.niner.net.
example.co.zm.         86400   IN      NS      ns2.niner.net.

;; Query time: 338 msec
;; SERVER: 196.46.192.21#53(196.46.192.21)
;; WHEN: Fri Sep 25 00:13:42 PDT 2015
;; MSG SIZE  rcvd: 88

[00:13:42 leftseat@wrathall ~]$ dig example.co.zm ns @ns-zm.afrinic.net

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @ns-zm.afrinic.net
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36928
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 2, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; AUTHORITY SECTION:
example.co.zm.         86400   IN      NS      ns1.niner.net.
example.co.zm.         86400   IN      NS      ns2.niner.net.

;; Query time: 310 msec
;; SERVER: 196.216.168.44#53(196.216.168.44)
;; WHEN: Fri Sep 25 00:13:45 PDT 2015
;; MSG SIZE  rcvd: 88

[00:13:45 leftseat@wrathall ~]$ dig example.co.zm ns @pch.nic.zm

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> example.co.zm ns @pch.nic.zm
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37546
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 8, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;example.co.zm.                        IN      NS

;; AUTHORITY SECTION:
example.co.zm.         86400   IN      NS      dns2.hai-alive.zm.
example.co.zm.         86400   IN      NS      dns2.realtime.zm.
example.co.zm.         86400   IN      NS      dns1.hai-alive.zm.
example.co.zm.         86400   IN      NS      dns3.realtime.zm.
example.co.zm.         86400   IN      NS      dns4.realtime.zm.
example.co.zm.         86400   IN      NS      dns1.realtime.zm.
example.co.zm.         86400   IN      NS      dns4.hai-alive.zm.
example.co.zm.         86400   IN      NS      dns3.hai-alive.zm.

;; Query time: 17 msec
;; SERVER: 204.61.216.73#53(204.61.216.73)
;; WHEN: Fri Sep 25 00:13:49 PDT 2015
;; MSG SIZE  rcvd: 214

[00:13:49 leftseat@wrathall ~]$ date -u
Fri Sep 25 07:13:56 UTC 2015
[00:13:56 leftseat@wrathall ~]$

--
NinerNet Communications | Craig Hartnett
* http://www.niner.net | info _AT_ niner.net
Phone: +1 604 630 1772 | +260 21 1 255568 | 1 855 NINERNET

Update, 2016-12-12: Updated link to CoCCA “patrons” page.