We've been telling them about nameserver timeouts for a good month now. When they finally admitted to us there was a problem (Thursday) we were told they were having to reboot ns2 every half hour. It'd have been so much better if they'd just fixed the problem before everything fell apart.

Bloody ridiculous, as soon as the bell rings to go home on Friday thats it, your business crosses its fingers and hopes that things survive until Monday morning.

We had 30 domains down over the weekend, and no way of finding out what was wrong, or when it would be fixed. A cost to us? Probably about £2.5k. Will we see any of it? Erm no.

When the 123-reg site finally limped back into existence on Sunday, not one mention about the problems........obviously do not want to let new customers to know the shite service they are signing upto.

Well I for one am joining the Northern Rock school of savers, and cutting and running.

Am I worried about them transferring my domains to another host........just a little :o)

The question that MUST be aske of 123-reg is:

*why* did an outage at one site cause their DNS service to fail?

A central philosophy of DNS is that you have multiple NS machines and anyone with an ounce of itelligence puts these machine at different locations. It appears that a single failure caused this outage; if this is true then one can only classify 123-reg as a mickey mouse organisation that doesn't deliver what they should.

I would be majorly pissed off, I was able to rebuild a server 2003 DNS / AD strcture in just under 4 hours by myself over a WAN. Ive only recently been in ICT for about a year so I thought this was quite good. I also run my own nameservers/dns servers at home on unix (freebsd) and it only takes several hours to setup. I dont see what the problem was here? Do they not backup there nameservers? Do they not backup configs? This should not have happened imo.

I'm sure it will be the usual crock of shite from these muppets - they couldn't even keep a status message live or updated for the duration of the fault so I expect fessing up to the truth is the last thing they are going to do. It might be their only chance of keeping the big customers though, those that haven't already seen enough and transferred out anyway...

Even if it was a hardware failure how long does it take to get a new DNS server built, online and a backup restored? - not two days, not if you aren't treating your customers with utter contempt at any rate. And this is completely ignoring the issue that a registry as big as this should surely have hot spares? And what are the odds that a hardware failure occured in both ns1 and ns2 together, at the same time? More likely the old proverbial spanner got thrown in by someone who maybe isn't up to the job. There had been intermittent resolving issues for a good 10 days before this happened - hardware failure doesn't quite cut it for me

...the expression 'you get what you pay for' comes to mind and anything generating £3000 in one weekend maybe deserves a little more investment than a cheap as chips ISP.

Reg-123's dns servers have been intermittent for ages but reliability and performance has worsened considerably over the last 3 months.

When the service came back up neither of the servers was authoritative for the domains I checked suggesting some sort of manual setup to get things going whilst they fixed the problem.

A hardware failure should not take the service down for 48 hours. Perhaps a couple of hours whilst a disk is replaced and a backup restored.

Even then there are 2 dns servers so one should still be working so this explanation does not ring true.

Facts are problems like this have been going on for many months with this service, proven by my own experiences and those of our customers and my friends and as many people have posted on various forums getting in touch with support when things go wrong is a joke.

If the servers are overloaded then they need upgrading and new domains or customers should be allocated to a different set of servers.

They might be updating the pipex status page but this is a reg-123 problem as well and their status page has been saying there are no issues all weekend.

I have been a happy customer of 123-reg for a number of years with no major problems but in the last few years their service has started to decrease. I blame it all on the Hoff and Pipex for breaking everything.

On the plus side, it has pushed me to host my own D/NS server. But still, bunch o' ramptant numpties who have clearly been running around with their wee heads on fire for far too long.

This looked like an exact repeat of the 24hr nameserver outage they suffered about 14 months ago. Throughout the outage the nameservers could all be pinged - they just wern't resolving about 97% of the time so it doesn't look much like a hardware failure. It was the same story for 6 different nameservers (hosteurope/webfusion/123-reg) across 6 diferent IPs and at various points in the 48 hrs they changed the nameservers between all of these in futile attempts to fix the problem. In fact they also force changed the nameservers for many registered domains without letting the domain owners know.

If it was just a hardware failure (which looks unlikely) that makes it worse because that's an entirely predictable event that's easily planned for and easily resolved (no pun intended) by a process known as (and I'll let them in on a top tip here) 'having another bit of hardware standing by that can do the same thing'.

Apart from the 48hr outage of such a critical service the fact they couldn't be bothered to post anything about it or any advice for their customers on their system status page or support pages until sunday when the problem started friday lunchtime is beyond any kind of acceptable excuse or rational explanation.

We changed the nameservers on a large number of domains on saturday to work around the problem and now plan to start the transfer out of all of these today.

When Tiscali bought the ISP core of Pipex and left the domain registration and hosting bits behind this looks like it was a wise choice. Unfortunately the once excellent 123-reg and associated webfusion and hosteurope have been a complete shambles for the last 12-18 months and are now looking very much like discarded leftovers on the side of the plate.

It's stupidly easy to change/add DNS servers with 123.

Even if you don't fancy running your own BIND, there's free/cheap alternatives out there you can use to provide backup DNS.

The "Change nameservers" form gives space for 4 nameservers by default, and lists their nameservers on the page so you can use theirs and a backup service.

I run my own DNS on all my 123 registered domains, and I didn't have a single spot of trouble over the weekend. For sites with any half decent amount of traffic, at least having a backup to the 123 servers is pretty essential.

Services like dyndns will provide the backup servers for $17.50 per year, to lose £3k or more for the sake of a £8-10 per year service is just bad administration.

As of 10.41 on Monday 123-reg's status pages have been updated to show the SAME generic message... it's down again although not affecting me this time as I've moved everything I had off of their servers (along with everyone else I know)

Over the weekend I posted a blog post entitled "123-reg outage" for the first time ever my lowly blog has clocked up over 2,300 hits to that web page - showing that people have obviously found my "statement" before 123-reg bothered to acknowledge the problem.

Even now 123-reg's website (other than that carefully hidden status page) is all business as usual claiming to be the UK's No1 hosting provider.... 80 hours of outage... Number 1 on what list exactly?

Farewell 123-reg.

Of all that "Cow-pie" they are serving at pipex by blaming high server load with mysterious failures straight out of the X-files.

I need to look seriously for a new registrar to hold my domain names. 123-Reg may be cherap, but they have ZERO customer service. Last time we had a problem we lost a domain name for 14 days. During that time 123-reg did not reply to countless emails and telephone messages.

Let me introduce you important and powerful businessmen to the concept of a "Service Level Agreement." If you haven't ever heard of one before maybe you should do some homework before entrusting your sole source of income to a bloody domestic ISP.

Must of been software... I spotted this'd happened to one of our domains on Saturday evening, and I logged into the control panel, it was claiming that it did hold the name servers for the domain. Obviously it did still however...

To fix it, I unlocked the domain, changed the name servers to themselves again to get it to rebuild the zone in their system, and updated the details again in the zone. Luckily it was a very simple zone we had there.

It started working within about 5 minutes of me making those changes, so it must of been a totally software based screw up.

""Nameservers have been down for 48+ hours this weekend, loosing me around £3k across my four more popular domain names"

What's this 'loosing' I keep seeing??? Does the correspondent mean that his £3k is not as tight this weekend than its previously been???

I have been advising people for 2 years to avoid 123-Reg like the plague!

Its not hard to transfer your domain to a decent registrar and even if you've bought hosting from them (NEVER buy your hosting from the domain registrar - there are some fantastic deals out there on hosting) its still not the end of the world. It might take a bit of time and effort to move the site but in the long run it will be worth it. 123-Reg will KEEP doing this shit and you will loose much more than it will cost to transfer.

Leave 123-Reg for good!

Unfortunately 123-reg don't allow you to mix 123-reg and non 123-reg nameservers within the 4 nameserver options provided. If they did we'd have done this a long time ago (we tried and were told we couldn't). It would help them as well if they did (as they are incapable of providing their own proper failover).

'This problem was caused by a combination of excessive loading on the DNS servers and a rare hardware failure'

So is this rare, as in their hardware never normally fails, or is it a rare as in a special bit of kit made in the darkest depths of Peru so there is only 2 in existance.

I use 123reg because they are cheap - as (I would think) do most of their other customers. I also run my own DNS servers, Web servers, and Mail relays - I use them for my registration only. I do this partially because I'm anal and like doing so, but mainly because that way it's only MY fault if anything goes wrong.

When you get something that cheap, you don't expect it to have all the bells and whistles. The price I'm paying doesn't include a premium for a massively redundant system.

If you're not happy with the servce, then go elsewhere. I have 20+ domains registered with 123reg, and didn't see a single problem over the weekend.

Yes, it would be nice if they had additional DNS servers hosted in a totally segregated solution, and there's no reason why multiple companies couldn't do that through a reciprical arrangement - hell, I could probably setup something to manage & configure it for them in a couple of hours - but considering what you're paying for, don't complain if it's not got an uptime of 99.999%

Pipex have recently divested themselves of Homecall (only acquired about a year ago itself). Pipex has been sweeping up smaller ISPs (Bulldog seems to ring a bell too) to enable it to compete with BT, NTHell and Tiscali on an even footing through a large customer base and economies of scale.

Seems it is not going well and they are now going through a period of bad cashflow as a result. Bad cashflow leads to compromised infrastructure through lack of maintenance and lack of essential upgrades/replacements. Something breaks, you are short-staffed to save cash and/or you have reduced your support cover (to save cash too) so IBM has 24 hour response instead of 1 hour.

Anyone with GCSE Economics could have told them this. Hang on. Maybe not. ;-)

It's not just a problem with Pipex and bad infrastructure investment, though it is probably the most blatently obvious example so far. Port throttling is another example.

When USwitch shows broadband packages starting at £7.50 and a broadband connection is sold at about £8 wholesale, how much investment do you really think is going into internet infrastructure?

Ironically, they have just sold Homecall to Tiscali making one of their biggest competitors more competitive.

I remember this happening a while ago as well when they lost all comms to one of their DC's, thus taking out their entire DNS infrastructure.

What amazed me then, and still amazes me now is why they haven't bothered to locate their servers in different physical locations, and on different IP networks. For many small companies I could understand it, but come on, 123-reg is owned by Pipex, and they in turn own several other ISP's, who collectively must surely have more than one datacentre to hold their servers. Why the hell hasn't someone there split their core infrastructure servers across these sites to give them all some redundancy, especially following the first time that had problems like this (that I'm aware of) a couple of years ago.

...so this downtime will cost them.

http://www.webhostingtalk.com/showthread.php?t=649754

Hi

I have 4 domains with reg-123. I use their domain reg services for one main reason --> they offer me a DNS server for nowt such that I can point my domain names at any server I choose.

I have been with them for 5 years and up-to now, I have not had any serious issues. This time however, the outage has been so long that the TTL of the domains in question has expired, thus the copies of my domains' info has expired from the caches of other DNS servers. They then do a look-up on the authorative DNS server (in my case ns0 and ns1,reg-123.co.uk), for the domain requested, and are getting back nothing.

Reg-123 claims to have sorted the issue, but in fact, a lookup on the authorative server (ns0.reg-123.co.uk) is currently returning incorrect IP addresses. Thus all my traffic is being redirected to a reg-123 holding page. Additional, only 'A' records are resolving at all, 'C-NAME' fail to resolve to anything.

Until the authorative server starts to push out the correct IPs, and the C-NAME records start to resolve again, I dont stand a chance of getting things back online. Even once that issue is sorted, up to 48Hrs is required for the correct info to propigate around the planet.

This issue has effected both autorative servers NS1 and NS2 which have both suffered failures. The whole reason for having two servers is so that their is no possibility of both authorative servers going down at the same time. They are supposed to be hosted on different networks...

If you are with reg-123 try 'nslookup ns1.reg-123.co.uk' and enter your (sub)domains --> check that they resolve how they should do rather than resolving to the reg-123 holding page. The holding page IP is: 194.154.164.90 The only time that they should resolve to this IP is if you are using their web forwarding services (I am not!)

I think i'll run A DNS server on my own network and be authorative for my own domains... least I get to control it if it goes pear shaped...!

Pretty poor show really. I guess I will have to find another DNS provider. You get what you pay for I guess.

Nigel

Problems with 123 long and well documented, victim of their own success, this only the most recent outrage, yada yada. But answer me this - why, after 18 months of poor service did I decide to stay with them, thinking it isn't really worth the hassle to migrate and they MUST have fixed their customer service issues by now after ALL the flack they were getting back then...

This is the WORST company I have ever dealt with! I have been waiting for a domain to be transferred to my account since July, 123 have used about every blocking tactic in the book to prevent this from happening, is this just a way of getting you to ring them and waste a fortune waiting for someone to answer? by the way these monkeys do business I am surprised they have any customers left, I will be transferring my business elsewhere and suggest others do the same. I bet Paris Hilton doesnt have this trouble........

Why won't they allow this?

"Unfortunately 123-reg don't allow you to mix 123-reg and non 123-reg nameservers within the 4 nameserver options provided. If they did we'd have done this a long time ago (we tried and were told we couldn't). It would help them as well if they did (as they are incapable of providing their own proper failover)."

I only use 123-reg for email forwarding (should I change ISPs I keep my address) - would this affect my incoming emails?

No real reason not to. Just seems to be their policy. Domain owner would have to take responsibility for ensuring all name servers have same record (unlike the ns0 - 1 - 2 varients of 123-reg which still don't).

Given that anybody who knows what a nameserver does and how to change one is going to be leaving 123-reg anyway it's probably a bit late for them to start allowing this now.

No real reason not to. Just seems to be their policy. Domain owner would have to take responsibility for ensuring all name servers have same record (unlike the ns0 - 1 - 2 varients of 123-reg which still don't).

Given that anybody who knows what a nameserver does and how to change one is going to be leaving 123-reg anyway it's probably a bit late for them to start allowing this now.

123-reg's DNS system is rather more complex than you think - it isn't simply two servers doing master/slave. There's millions (I kid you not) of domains in the system, making most "traditional" DNS server apps impossible to manage. Imagine having a BIND config with that many zones in... it'd take days to load. I know this because I used to manage one, and it eventually crapped out at 750,000 zones because of process memory limitations. After taking more than 24 hours to load the file.

DNS is one of those hideous applications - nobody gives a shite about it until it breaks. And if it breaks badly, the noise is terrible.

All those posters going off to DIY, good luck to you. Just remember you're not doing it for millions of zones. And good luck complaining to yourself when it goes wrong, which it will - eventually :)

I hate 123-reg (not registering my domains, NS downtime, SMTP downtime, etc) but to be fair, if you pay £3 a year for a .co.uk domain and DNS hosting for that domain, what do you expect?

There are plently of places that provide professional DNS hosting, and from what I've seen they cost a lot more than £3 a year.

"Imagine having a BIND config with that many zones in... it'd take days to load"

Would a clustered database driven nameserver set up not negate this?

Yes some emails may have bounced back to sender because the Mail-eXchanger (MX) record for the domain is held by 123-Reg.

"Would a clustered database driven nameserver set up not negate this?"

Yes, it would. That being my point - for all the people saying "this is an abomination, DNS isn't hard", the response is "yes it is when you're dealing with that sort of volume". It's not hard in conceptual terms, but volume scaling can be particularly difficult - even when using databases and clusters, which simply add to the complexity of the system over and above flat files. Throw in load balancing, health checking, failover, complex routing, DDoS protection and multiple locations and you have something which is a completely different beast to that which many posters have familiarity with.

Unfortunately for Pipex (and many others before, and to come) building systems for 100% uptime simply isn't possible - you can get close, but there's always *something* that didn't get thought of.

Doing a traceroute to the two muppet-reg^W 123-reg nameservers appears to show both of them sitting behind the same router. Can it be possible?

Rather than having a huge cluster with a gazillion domains on it, just have a number of smaller servers, when you register a domain the control panel automatically assigns you to the latest dns servers.

That way you don't need to run complicated software and the zones don't take an age to load etc etc. When you do have an issue its isolated to a subset of all the domains registered with you rather than every domain. You can limit how many domains are on any one set of servers and upgrade them if the performance starts to take a hit. If you plan your capacity this should never happen.

The down side, a few more servers to run. Up side common kit, common components easy hardware fixes.

On the subject of using other dns servers as a backup, this would be a good option but reg-123 like many reg-123 won't allow you to do this automatically as they have disabled zone transfers and there is no option to enable it.

I guess you could manually update both but that would lead to a situation where at best your dns looks (even if it isn't) out of synch.

I migrated all my domains from reg-123 a while ago once pipex took over and it started going down hill, there was just one too many cockups so I bit the bullet and moved elsewhere (domainmonster.com as it happens).

As for "Whine Whine, Whinge Whinge" By Graham Wood you say you haven't been affected, of course not as you state you run your own dns servers so it would not affect you.

Unfortunately not everyone is as lucky or perhaps didn't realise the importance of dns servers. Perhaps this is a wake up call for many. Its pointless spending thousands on your web site and advertising and then losing your business for the sake of a few pounds spent on some decent dns servers run by a company who provides reasonable support, monitors its kit correctly and can fix things in a reasonable time frame.

A few pointers for anyone looking for a new dns provider.

Ensure they have at least 2 dns servers, preferably 3. At least one should be in a different location/data center. Ensure the dns servers are in the country you expect, many so called uk providers are actually hosted in the US which means slower lookups and more chance of problems due to network issues.

If you are also using them as a registrar also check if it costs to transfer away in the event they don't work out and make sure this can be done automatically via a control panel without their intervention. If its costs money or you can't do it online yourself then avoid them.

Finally check your dns by using someone like dnsreport.com whenever you change it to make sure its working correctly. We regularly get customers querying alerts we send them and when we check their dns setup we generally find it is not totally valid. The most common mistake is different name servers specified at the root servers to those in their own name server records.

Here is a copy/pasted "technical support" thread. (newest first.. so read from bottom up).

Funnily enough I renewed a domain June and had an identical situation which was due to Eurid.org apparently. Having suffered far worse this time I some how doubt it now.

If they even responded it would be good.... but 17 days for the FIRST response while my whole website was down. When I did get a response I am told that my site will be back "within 48hours" .... well yippee - thats only 19 days late for a mistake THEY made!! Not even a proper apology!

Just don't go there....

Response (Farhan S) 21/10/2007 11:55

Hi Duncan

Thanks for your email, dated: 16/10/07

123-reg would like to apologise for the delay in responding to your support query, this has been caused by an unprecedented amount of contacts to our support team

I do apologise for the inconvenience caused. I have asked my manger to get this domain reactivated will be in your control panel with in 48 hours thanks.

Regards,

Farhan S

Customer (Duncan xxxxxxx) 16/10/2007 14:28

4 days on and still nothing at all.

Suprise suprise...

Customer (Duncan xxxxxxx) 12/10/2007 13:07

Oh and yet again if I try to call you - which frankly is probably be the only way to get this addressed - AGAIN - I have to wait 20 minutes, paying 10p a minute.

So in the space of 6 months I will have spent £4 on phone calls because you are incapable of even REPLYING to your technical support queries.

I mean these are all written? Do you not look at them? Not even a "oh its a bit busy but we will do it xxxx".

I have PAID YOU to renew a domain name. It has now been 8 days without a response.

I've not been able to access the site for 10 days. This is simply NOT ACCEPTABLE.

The fact that this is a CARBON COPY of last time when you assured me to was an issue with EurID and the fact you were bringing new technical support people up to speed.

Why are we here again?

Customer (Duncan xxxxxxx) 12/10/2007 12:57

You think 8 days is acceptable?

AGAIN?

Customer (Duncan xxxxxxx) 04/10/2007 16:28

I renewed a domain name - xxxxxxxx.eu - this morning and as yet it has not been updated in the root DNS servers at eurid.org.

As this is exactly what happened with my last two .EU domains, please ensure that the domain is taken out of Quarrantine ASAP. I do not want the downtime I endured when they were updated.

It has been paid.

It still looks like 123-Reg's name servers (or their web admin) are still having problems. When I update a DNS record it is not actually being saved, after several hours their nameserver is still returning the old IP address when queried directly, which is convenient when clients want to know when their site will be "live".

Fucking joke.

Our website/email went down for a while but luckily I found this great free (with good reviews) DNS server backup. everydns.net enables you to use them as secondary (or 3th and 4th) DNS servers and they copy the zone file from your primary so you dont have to update the records in more than one place.

I'm moving all my domains away from 123reg anyway, a hosting company without proper DNS servers is an absolute joke.

If you ever think your job is pants then cheer yourself up by thinking about the poor bloke in outbound sales at 123-Reg who called me this morning to get us to renew a near to end of contract dedicated server we have with them. I don't like shooting fish in a barrel so didn't.

However - when you finally get through on the phone after 25 minutes at 10p a minute (plus your time which I guess is at least 2-3X that ) think of the fact that 123-Reg are currently allocating resource to outbound sales to drum up new business while the same person/phone/desk could perhaps be better allocated to answering thier phones. Time to start shooting those fish.

Good call...I haven't seen anyone else bring up SLA's.

I'm from the U.S., so I cannot comment on 123-Reg...and therefore shall not.

Sent at least half a dozen e-mails to their help desk and got a standard worded reply that took no account of my issues. Finally moved my hosting and DNS to a US based company on Monday evening and my site came back up after being offline for nearly 3 whole days.

I had only been with 123-reg for 3 weeks too!

Pretty poor effort on their part. I guess I'm not alone in moving any reliance on their infrastructure away to someone a bit more responsive?

Unfortunately they had the last laugh because I paid for 2 years up front and had to repeat the outlay again, albeit to a different hosting company, just a few weeks later.

Interestingly reg-123's dns servers are 'open', eg recursive to the whole world which makes them liable to cache poisioning and dos attacks.

or as dnsreport.com says

ERROR: One or more of your nameservers reports that it is an open DNS server. This usually means that anyone in the world can query it for domains it is not authoritative for (it is possible that the DNS server advertises that it does recursive lookups when it does not, but that shouldn't happen). This can cause an excessive load on your DNS server. Also, it is strongly discouraged to have a DNS server be both authoritative for your domain and be recursive (even if it is not open), due to the potential for cache poisoning (with no recursion, there is no cache, and it is impossible to poison it). Also, the bad guys could use your DNS server as part of an attack, by forging their IP address. Problem record(s) are:

Server 212.67.202.1 reports that it will do recursive lookups. [test] Server 212.67.203.240 reports that it will do recursive lookups. [test] See this page for info on closing open DNS servers.

As far as I know secondary name server services like everydns.net will not work with most isps as they don't allow zone transfers. I just checked reg-123 and it does not allow them. Try a host -axfr yourdomain.co.uk if you have a linux system.

It would be great if such companies allowed you to enable zone transfers on a per domain basis via their control panel so that you could use secondary service but the majority do not.

Nice idea but impossible in most cases unless I am mistaken.

There are some good security reasons not to allow zone transfers on a free for all basis. There are however no good security reasons to prevent domain managers from opting to manually configur secondary or even third and fourth nameservers as they see fit which 123-reg currrently do. We've asked them on several occasions to allow us to mix 123-reg with other nameservers and been told in wasn't allowed (without any good reason - or indeed any reason ever being given).

Domains still not responding.

Ahem!

get outta there guys! we DID HAVE 140 domains there until the weekend shite HIT THE FAN ..and yes most of ours FAILED with the hardware dying at 123 ...and yes the dns has been on the blink for months... my server monitor has been reporting dns outages weekly on a spray of 123 domains on different web servers... Weve left now...taken half already to heart internet (ex 123/webfusion folk) great online interface ...but no tel support :-(

we tried fasthosts but they got hacked recently... and they cant host uk,com/centralnic and they hav ONLY TWO A record options in their dns! eh? so no ftp? no cp? no txt? no spf? why not try zoneedit they say eh?? who??

ah well out of the frying pan ... etc etc BIG doh!

Q: what to do where to go for smb semi-enterprise (ie not TOO expensive but sla'd) mid level domain management with GOOD dns options?

gap i the market? anyone up for tax dodge fund investment... ahem ... maybe not...

I already do that. A few years ago I made the decision to outsource my DNS management to to Afraid.org's free services, and now I switch any new domain over as soon as I can after ordering it through 123-reg. You can do this from about 5/10 minutes after the domain name is configured on you 123-reg control panel, and you can preconfigure your domain on your afraid.org control panel in anticipation of the nameserver updates, and then there's pretty much no downtime.

I've got about 14 domains hosted on afraid.org, and they've had precisely one period of (partial) downtime in the past few years I've hosted with them. I like them so much, I subscribe to their premium services. On the contrary, I've had nothing but delayed DNS record updates, shit service, and crap tech support from 123-reg (they STILL wont' let me lock/unlock my .name domain, saying that that feature is 'coming soon' and requiring you to ring them up to ask them to unlock your domain, even though they advise you to unlock your domain to make modifications in the support pages!) Trust me, the only times I've rung them are to ask them to unlock my domain - so I could make the switch for my DNS hosting to afraid.org. Couldn't be happier since I made the change.

That's my little nugget of advice for today...