4:30pm Eastern: We haven't received any more messages from our ISP - no more slowness "episodes" either. We suspect that they will get the 2 broken backbones fixed and switch things back to normal around midnight tonight. There may be a bump or two when that happens. It is possible that some residual slowness will remain for a small number of users over the next day or so as the Internet "re-balances" the traffic routes for our website. Let our support team know if you see significant issues with speed after tomorrow.
Unfortunately, we may never know the real culprit behind the outage but I'm guessing it was either a bad router, a guy with a backhoe, or a system administrator deploying a buggy patch. This is the first time in many years that our ISP has caused a noticable outage like this. We will follow up and make sure that they will take steps to prevent these kind of problems in the future.
This will probably (hopefully) be our last update on this incident. Thanks again for your patience and understanding.
2:00pm Eastern: Message from our ISP - "At 10:30 PDT our our third level engineer shut down our BGP session with our upstream provider XO. Customer would have noticed routing reconvergence and sub-optimal routing as traffic was re-routed over our other available providers at the SEF PNAP. The XO BGP session will remain down until we notify customer prior to the restoring of the BGP session."
Translation: Our ISP uses multiple backbones to connect to the Internet. They've identified two of those backbones (Sprint and XO) as having problems this morning for unknown reasons. They have moved our traffic off of those two problem backbones and onto other backbones. That means that for many people, the extreme slowness should be gone, however some people will see slower that normal charts until they get those two misbehaving backbones working again.
Again, thanks for your patience as they sort through all these issues.
1:25pm Eastern: Message from our ISP - "We have shut down our BGP session with our Sprint provider as of 10:10 PDT. We will send out a notification to customers prior to restoring the BGP session with Sprint. Our third level engineers are still investigating the over utilization issue with our Sprint provider and we will provide additional information once it is available."
We - StockCharts - noticed a significant improvement in response times around the time that they turned off Sprint. Hopefully things are better for many of you now. We will continue to keep you updated.
1:07pm Eastern: Message from our ISP - "Our third level engineers are continuing to investigate the issue for resolution. We have vendor tickets open as well to assist in the investigation. At this time we do not have an ETTR however we are working as quickly as possible for the issue to be resolved We will provide additional information as soon as it is available."
(ETTR - "Estimated Time To Resolution")
12:40pm Eastern: We are experiencing another "episode" at the moment. Thank you for your patience. We will pass along any more news as we receive it from our ISP.
12:00pm Eastern: This morning our ISP is reporting problems with one of their backbone connections to the Internet. InterNAP has told us that they have experienced 2 big "packet loss" events so far - one around 10:40am Eastern and a second one around 11:50am Eastern. They are re-routing traffic and continuing to investigate.
The issues appear to have affected a large number of our users with slow response times and missing charts. For that we apologize. At this point things are working fine but we can't be sure that it won't happen again because the issue is essentially "out of our hands." We expect and believe that InterNAP will have things fixed shortly.
Check back here for updates over the course of the day.
