|
News
« August 2006 |
Main
| May 2007 »
Power Issues at Costa Mesa Facility
March 31, 2007 @ 3:47PM
At approximately 15:30 PST we lost power to all equipment hosted in our Costa Mesa facility. This power failure is localized to specific cabinets at the facility and is not building wide. We are working with engineers to resolve this issue as soon as possible.
This post will be updated with the latest developments. Please check back here for updates.
UPDATE 17:00 - We have been informed by the building engineers that power should be restored within the hour.
UPDATE 18:45 - Power has been restored to all equipment in the Costa Mesa data center. A faulty A/C unit caused a number of breakers to heat up and fail. Currently all servers are online and operational.
RBL: Blackhole.SecuritySage.com Offline
March 14, 2007 @ 11:24AM
This morning one of the Real-time Blackhole Lists ("RBL") we use to help filter inbound E-mail for SPAM was taken offline. The specific RBL was blackhole.securitysage.com. Apparently this domain name expired and was not renewed. Our gateway server than started to reject some inbound mail because it was unable to reach this RBL.
We have since removed this RBL from our gateway server's configuration (as of 11:17 AM PDT).
If someone attempted to send you an E-mail this morning, and received the following bounced message, then they should resend the message.
554 Service unavailable; Client host [somehost.somedomain.tld] blocked using blackhole.securitysage.com;
Verizon DSL Customers
March 8, 2007 @ 8:54AM
We have been made aware that certain customers are unable to reach our network. So far these customers all have DSL service through Verizon. We are unsure if this new issue is related to the issues from the last few days. This could also be an localized issue with Verizon customers only. We are current researching this issue and will post an update soon.
UPDATE: 03.08.2007 @ 09:28 AM - We have verified that Verizon is having national issues which they expect to resolve before 1PM PST.
UPDATE: 03.08.2007 @ 11:45 AM - Verizon DSL has resolved their issues and their customers should no longer have problems accessing certain sites on the Internet including our network.
Emergency Hardware Upgrade 03.08.2007
March 7, 2007 @ 2:28PM
In order to help prevent a similar issue from happening again, Atlantis International, will be performing a hardware upgrade tomorrow. We will be replacing our current Cisco router with a new model.
We do not know the exact time of the upgrade since we are waiting for an overnight DHL shipment, but expect it to be tomorrow morning Thursday, March 8th, 2007. The swapping of equipment will take no more than 10 minutes. During this time our Santa Ana network will loose all connectivity.
Once we know a more exact time this notice will be updated.
UPDATE: 03.07.2007 @ 11:24 PM - The estimated time will be 10 minutes and will be performed some time between 8AM PST - 12PM PST on Thursday, March 8th, 2007.
UPDATE: 03.08.2007 @ 10:46 AM - The hardware maintenance has been completed and all services are running normal.
Back Online!
At 10:35 this morning the MegaPath routers reestablished their BGP session with us and we have received a download of all Internet routes. Traffic to/from our Santa Ana network has improved over the last four days. However, our servers are still trying to catch up with the previous days delays. Please expect normal speeds to/from our network today with brief periods (~5 min.) of heavy congestion.
Thank you again for your understanding.
Circuit Up! Routes Down.
MegaPath has completed their emergency maintenance and our circuit appears to be behaving properly again. However their routers have yet to download a full set of Internet routes to us. Currently all inbound traffic to us can travel through either circuit, however, all our outbound traffic is still continuing to be processed by our EarthLink connection only. This will continue to cause the same network congestion we have seen the last two days.
We have contacted MegaPath to reestablish their BGP session with us and we expect the routes to be in place within a few hours.
Emergency Maintenace Scheduled
March 6, 2007 @ 11:27AM
We have received confirmation that MegaPath will perform their emergency maintenance. However, it is not scheduled for late Tuesday night but instead very early Wednesday morning. Below is a copy of the E-mail we received.
They have also asked that we perform some additional tests with them this afternoon. Therefore we will be bring up and down our circuits this evening after 5PM PST. Please expect our Santa Ana network to be periodically unreachable this evening.
Customers hosted in our Costa Mesa facility will continue to be unaffected.
Thank your for your understaning in this matter.
-----Original Message-----
From: nocops@megapath.com [mailto:nocops@megapath.com]
Sent: Tuesday, March 06, 2007 5:47 AM
To: Suppressed Client Listing
Subject: MegaPath Emergency Scheduled Maintenance
===================MegaPath Maintenance Notification===================
Dear Customer:
To continue to provide best level of service possible, MegaPath will perform the following emergency maintenance at the specified time.
Date : 03/07/2007 0030 PT - 03/07/2007 0115 PT
Ticket# : AN861522
Detail : Troubleshooting router.
Down time: 15 minutes
Regions : Costa Mesa, CA
Contact our 24x7 post-installation customer support center at 1-888-638-3696 or directly at nocops@megapath.com if you have any questions.
Thank you, for your understanding and cooperation
Cause of Network Outage Identified?
March 5, 2007 @ 6:12PM
One of our attempts to get action on this issue has paid off. We have been contacted, and are working with, an extremely knowledgeable network engineer about our circuit. The engineer has determined that a routing issue with one of their core routers may be causing our issue. Unfortunately this router is used by national customers and they need to properly plan for it's maintenance. It was too late today for them to plan for the maintenance tonight so he has scheduled it for late tomorrow (Tuesday, March 6th, 2007) night.
Therefore, please expect continued network congestion until at least Wednesday morning.
Also note that most Internet services are able to handle the congestion just fine. However, POP3 and IMAP E-mail protocols seem to timeout easily. Therefore please try to download your E-mail at off peak times (before 8AM PST or after 5PM PST). During peak times, please do not let your E-mail client constantly attempt to download mail. This causes additional congestion on our already slow network. Between attempts of receiving E-mail, please shut down your E-mail client. We ask that you try to check your E-mail no more than once every 30 minutes during peak traffic times.
Network Congestion
As expected, all services are online however traffic flowing through our EarthLink connection is at capacity. Therefore please expect slower service from our Santa Ana facility until this issue is resolved. Customers in our Costa Mesa facility are no longer affected by this issue and are running at 100%.
We are still constantly trying to get this issue resolved with MegaPath. They are not being very helpful nor empathic of our situation nor of the effects this is having on our customers and yours. Be assured that additional people are being contacted in hopes that this issue can be resolved as soon as possible.
Please bear with us during this time of increased congestion.
All Services Restored
March 4, 2007 @ 9:48PM
All servers hosted in our Santa Ana and Costa Mesa facility are now accessible on the Internet.
We have been communicating constantly with MegaPath concerning our connectivity issue. They have assure us that our issue has been escalated to management, a senior engineer has been paged and that our ticket has been escalated to the highest priority possible. However, it seems almost certain that they will not be able to resolve this issue until tomorrow (Monday, March 6th, 2007). Therefore, we have decided to completely pull our connection with MegaPath. This has forced all visitors to use our EarthLink connection. While all servers are now online and accessible to 100% of our visitors, traffic may be slower due to our Internet connectivity being cut in half.
Please expect additional delays tomorrow until this issue is completely resolved.
New DNS Server
Late last night we installed a new DNS server in our Costa Mesa facility and all users hosted in that facility should have full service within a couple of hours. Also, customers who host their DNS with us and use external servers should be able to access these external servers within a couple of hours.
We are still working with MegaPath in trying to resolve the loss of connectivity in our Santa Ana facility.
Partial Outage
March 3, 2007 @ 8:33AM
Early this morning we experienced a partial Internet outage. Our upstream provider MegaPath Networks, stopped forwarding traffic to us via one of our Internet circuits. Our other circuit supplied to us by Earthlink, Inc. was unaffected. This lead to approximately 50% of our visitors to not be able to access our network (visitors coming in through our MegaPath circuit) while the other 50% were able to access our network just fine (visitors coming in through our Earthlink circuit).
Currently we know that local Time Warner (Road Runner) and Cox cable customers come into our network through MegaPath and therefore are unable to access any of our servers. We also know that local SBC and Yahoo DSL customers are able to access our servers since they come into our network through Earthlink.
We are working with MegaPath in trying to resolve this issue as soon as possible.
Although this outage has occurred in our Santa Ana facility, customers in our Costa Mesa facility may be experiencing a loss of connectivity as well. This is due to hosts not resolving from our DNS servers. Our DNS servers are on 2 independent networks in our Santa Ana facility. One is hosted on the MegaPath network and therefore is currently inaccessible. The other is hosted on our multi-homed network and therefore only 50% of visitors are able to see it. This causes visitors to the Costa Mesa network to sometimes not be able to resolve host names. We are currently working on moving our DNS server, which is hosted on the MegaPath network, to our Costa Mesa facility. This will resolve any DNS issues for our Costa Mesa customers. Please note that if you know the IP address of your server, you can access it that way if DNS is not resolving for you.
| 
