StockCharts.com - Blogs

UPDATE @ 3:30PM:
From our ISP:

"We have found and addressed a DDOS attack on our network at approximately 10:52 PST.  Due to the extreme size of the attack providers experienced periods of increased utilization and caused some packet loss and latency within the PNAP.  We have been monitoring all provider links for stability and have seen no further issues since the initial attack was resolved.  We will continue to monitor going forward and update if there are any further issues."

We, StockCharts.com, apologize for the interuption that this event caused.  We will be following up with InterNAP to see if they can update their procedures to prevent this kind of thing from happening again.  A DDOS attack on some other website should NOT effect us like this.

Again, thank you for your understanding and support.
  - Chip

ORIGINAL POST:
We just received the following notice from InterNAP, our ISP:

"Our routing team is investigating an issue that is affecting multiple circuits in our Seattle data center locations. We are currently investigating this issue and will send updates as we receive them."

This notice coincides with multiple reports from our users of trouble accessing our website.  We don't believe that the problem affects everyone equally.  Check back here for updates throughout the day.

4:30pm Eastern:  We haven't received any more messages from our ISP - no more slowness "episodes" either.  We suspect that they will get the 2 broken backbones fixed and switch things back to normal around midnight tonight.  There may be a bump or two when that happens.  It is possible that some residual slowness will remain for a small number of users over the next day or so as the Internet "re-balances" the traffic routes for our website.  Let our support team know if you see significant issues with speed after tomorrow.

Unfortunately, we may never know the real culprit behind the outage but I'm guessing it was either a bad router, a guy with a backhoe, or a system administrator deploying a buggy patch.  This is the first time in many years that our ISP has caused a noticable outage like this.  We will follow up and make sure that they will take steps to prevent these kind of problems in the future.

This will probably (hopefully) be our last update on this incident.  Thanks again for your patience and understanding.

2:00pm Eastern: Message from our ISP - "At 10:30 PDT our our third level engineer shut down our BGP session with our upstream provider XO. Customer would have noticed routing reconvergence and sub-optimal routing as traffic was re-routed over our other available providers at the SEF PNAP.  The XO BGP session will remain down until we notify customer prior to the restoring of the BGP session."

Translation:  Our ISP uses multiple backbones to connect to the Internet.  They've identified two of those backbones (Sprint and XO) as having problems this morning for unknown reasons.  They have moved our traffic off of those two problem backbones and onto other backbones.  That means that for many people, the extreme slowness should be gone, however some people will see slower that normal charts until they get those two misbehaving backbones working again.

Again, thanks for your patience as they sort through all these issues.

1:25pm Eastern: Message from our ISP - "We have shut down our BGP session with our Sprint provider as of 10:10 PDT. We will send out a notification to customers prior to restoring the BGP session with Sprint. Our third level engineers are still investigating the over utilization issue with our Sprint provider and we will provide additional information once it is available."

We - StockCharts - noticed a significant improvement in response times around the time that they turned off Sprint.  Hopefully things are better for many of you now.  We will continue to keep you updated.

1:07pm Eastern:  Message from our ISP - "Our third level engineers are continuing to investigate the issue for resolution. We have vendor tickets open as well to assist in the investigation. At this time we do not have an ETTR however we are working as quickly as possible for the issue to be resolved We will provide additional information as soon as it is available." 

(ETTR - "Estimated Time To Resolution")

12:40pm Eastern: We are experiencing another "episode" at the moment.  Thank you for your patience.  We will pass along any more news as we receive it from our ISP.

12:00pm Eastern: This morning our ISP is reporting problems with one of their backbone connections to the Internet.  InterNAP has told us that they have experienced 2 big "packet loss" events so far - one around 10:40am Eastern and a second one around 11:50am Eastern.  They are re-routing traffic and continuing to investigate.

The issues appear to have affected a large number of our users with slow response times and missing charts.  For that we apologize.  At this point things are working fine but we can't be sure that it won't happen again because the issue is essentially "out of our hands."  We expect and believe that  InterNAP will have things fixed shortly.

Check back here for updates over the course of the day.

12:00pm - The problem has been fixed.  All annotations should be appearing again reliably.  Again, sorry for whatever problems this issue has caused.

10:45am - Annotations are not always being displayed on their charts this morning.  The annotations themselves are fine - they are still stored in our database.  It's just that our charting program is not always including them on the charts.  It seems to be happening about once every 10 times a chart is drawn (although that is an average and it could be more or less frequent at any one time).

We are working on tracking down the problem and expect to have it fixed well before the end of the day.

In the mean time, if you see a chart that doesn't have the annotations you expect, just refresh you page and the annotations should appear.

We apologize for whatever inconvenience this has caused.

We have been investigating reports from users that our data for the end trading on Thursday, October 27th did not match other sources for several stocks and ETFs.  On our own, we discovered that while our data matched some public sources such as Bloomberg.com and Yahoo.com's Historical data page, it did not match other sources including Google Finance and Yahoo.com's main quote page.

While we were in the middle of getting to the bottom of this conundrum, we received the following message from our data vendor:

"Please be advised that due to late trade activity, there may be inconsistancies in Composite Close from Pacific Exchange for an unknown number of securities. We are currently investigating this with the NYSE and will provide further information as it becomes available.  Please see the attached Trader Update link posted by NYSE.
http://www.nyse.com/pdfs/NOTICE_Duplicated_Extended_Session_Prints_on_October_27_2011.pdf "

The bottom line is that something screwed up the closing data for an "unknown number" of NYSE stocks yesterday and that screw up affected pretty much everyone.  At this point we do not know if our closing data for yesterday is correct or not.  We expect that at some point later today, we will get an update with the new "correct" numbers from the NYSE.  At that point, we will update our databases and the charts may change as a result.

Sorry for the problem, but the issue is "upstream" from us, apparently at the NYSE itself.

Our programming team made a mistake last month when they updated our servers to support London stocks.  The mistake was assuming that London time was the same a Greenwich Mean Time (GMT) which is what our datafeed uses for its time fields.  Up until Sunday, London time actually was the same as GMT, but that was just a coincidence.  On Sunday, London time changed to be one hour different from GMT and that is what caused our system to display incorrect intraday charts this morning.

At this point, most of the problems have been fixed (LSE stocks will be corrected shortly).  The programmers have been re-educated about time zones.  And, because this was due to a mistake on our part, all current members have had a free day of service added to their account.

From approximately 9:05am Eastern until 10:50am this morning, members were unable to log into their accounts.  The problem was traced to a full hard disk on one of our database servers.  The hard disk has been cleared and things are working normally again.  We apologize for whatever problems this problem caused.

Important Side Note:  We posted a message about the problem on our website at 9:45am.  The message appeared at the top of the Problem Report page.  We received over 400 problem reports after we posted that message.  While we normally love hearing from our users, in this case those extra problem reports delayed our ability to find and fix the problem and hindered our ability to communicate the status of the issue to everyone.  We are unsure how we can make our site messages more visible so we are including this reminder to everyone - please check for site messages on our homepage and on the Problem Report page before sending in a problem report.  Thanks!

UPDATE @ 10:00pm - We've moved the problem database onto the new servers.  We expect the open tomorrow to go much smoother.

===============

===============

UPDATE @ 12:15pm - Things continue to improve but are not 100%.  We have disabled some housekeeping activities on the slow server which has reduced the frequency and severity of the slowdowns however they are still occurring from time to time.  We are continuing to work on the ultimate solution which is to migrate the database onto some bigger, newer servers.

=================

UPDATE @ 11:00am - We are still experiencing occasional slowdowns and/or loss of chart style information.  Things are better than they were earlier but they are still not "normal."  We are in the process of moving our style database to a new server but that will take some time.  Until that is complete, you may experience temporary slowness and your charts may temporarily change into a plain candlestick chart.  Thank you for your patience as we sort things out.

==================

ORIGINAL POST @ 10:40am - Sometime over the weekend, one of our key databases became extremely slow for reasons that aren't fully clear.  Unfortunately, that slowness didn't become apparent until the market opened on Monday.  As soon as the market opened, that slow database prevented some users from seeing their saved charts and it affected the overall performance on the website.

At 10:10am we restarted the slow database server.  The restart process took 10 minutes to complete during which time our site was not working correctly.

We are now monitoring things to see if they recover correctly.  So far, we are looking "better" but we are not positive that things are back to normal yet.

Our apologies for this outage.  Please check back here periodically for more updates.

(Edit: Changes times to Eastern timezone)

Grrrrr.  We hit the wrong button today during our London update process and accidentally lost some Intraday data for some North American stocks.  Users may sporadically see flat lines on part of their intraday charts.  We will have the problem fixed shortly.  Our apologies for any problems this has caused.

Update @ 1:30pm - Things are now fixed and the charts should look correct.

Around 10am this morning, our main ChartList database slowed to a crawl for reasons that we do not yet understand.  That caused pretty much everything in our system to clog up and eventually things stopped working altogether.  It took us a while to get the database restarted and everything else unclogged but things appear to be back up and running now.

Our apologies for the outage.  We know how frustrating it can be when our website doesn't work properly.  We appreciate everyone who didn't write in to our support team with hateful comments and we forgive those that did.  Because of the outage, today is now a "free day" for our members.  All members have had one additional day of service added to their account.

This is the second time this database has bogged on us unexpectedly and so we are going to upgrade the server hardware it is running on with the latest-and-greatest from Dell in the next week or so.  We are also continuing to investigate the root cause of the problem in order to prevent it from happening again.

Until today, our reliability track record had been looking really good.  We're redoubling our efforts to try an push our rating as close to 100% as possible which is always our goal.

Around 12:20pm today our connectivity to much of the internet was interrupted for about 2 minutes.  InterNAP, our service provider, is currently blaming the problem on Sprint's connection to the InterNAP data center in downtown Seattle.  They have re-routed traffic to bypass the problem for now.  That re-routing should have happened almost instantly but didn't for some reason.  

Things appear to be back to normal for now however it is possible that some people are seeing slower response times from our website because of the "re-routing" that is currently in place.  It is also possible that there will be additional disruptions while InterNAP and Sprint sort things out.

We are pressing InterNAP for better answers.  We selected them as our ISP specifically to avoid this kind of problem.