We’re Back!

Mokka mit Schlag, The Cafes, www.elharo.com, and beand.com are all restored to service after Wednesday night’s network outage. Please holler if you notice any problems. There may yet be a few glitches.

I still don’t know exactly what went wrong. I do know that at the time the network went down I had Warcraft, VOIP, BitTorrent and this server running simultaneously. However that’s not unusual. Based on what it took to eventually fix this, the mostly likely scenario is that Speakeasy/Covad failed. At one point I thought my internal router from D-Link had failed. I did buy a new Linksys router to replace it (A WRT54G. I wasn’t able to find a 54GL on short notice. :-( ). However installing the new router did not fix the problem. Speakeasy has shipped me a new DSL modem, but that has not yet arrived. What finally fixed the problem was Speakeasy asking Covad to do a “manual rebuild of the circuit”. Previously Speakeasy had done a “rebuild and reprovision” from their end, but apparently Covad can do something Speakeasy can’t. Next time I’ll know to ask for a manual rebuild if the initial rebuild fails. Not that I have any clue what any of this means. However it does seem undiagnosable apart from just doing it. The first two SpeakEasy reps I went through did not suggest the manual rebuild.

Once the circuit was back up yesterday morning (total outage: about 12 hours), I still had to reconfigure the web server with the new router info. I didn’t get around to that till this morning. It’s a pain because the server normally runs headless. That meant swapping out various cables, keyboards, and monitors from other systems just so I could see into the server enough to bring it up. Normally I administer it remotely with ssh, httpd, and VNC but one of that works when the server can’t find the router.

I’m going to have to play with this configuration further, since the current setup may not survive and IP address reassignment. It would be nice if the Linksys router could map a service to a dynamically assigned IP so I could use full DHCP. After all, it doles out the IPs to the local boxes so it should know which one has which address. However it’s not quite that smart.

Furthermore, for reasons I have not yet determined, the HTTPD server did not come up automatically when the server rebooted. I had to start it manually. Memo to self: “sudo apachectl start” launches Apple’s bundled Apache 1.3 server. You want “sudo /usr/httpd/apachectl start”. At least PHP seems to have launched itself automatically with the server. I never was able to make Tomcat do that.

In any case, this is the third outage from Speakeasy in less than a month. In all three cases I had to alert them of the problem. Speakeasy was never perfect, but this is an order of magnitude jump in service problems. I knew the Best Buy acquisition would come to no good. Anyone have a solid ISP they can recommend? 1 Mbps+ bandwidth in both directions, static IPs, network neutrality, and no restrictions on the apps I can run. I can afford up to $200 a month for reliable service. Suggestions?

This entry was posted on Friday, October 5th, 2007 at 5:59 AM and is filed under Networks. You can follow any responses to this entry through the Atom feed. You can make a comment, Digg this story, or trackback from your own site.