StockCharts Blogs - Status

Late Trade Activity on Oct. 27th Caused Inconsistent Closing Values for Some NYSE Stocks

We have been investigating reports from users that our data for the end trading on Thursday, October 27th did not match other sources for several stocks and ETFs.  On our own, we discovered that while our data matched some public sources such as Bloomberg.com and Yahoo.com's Historical data page, it did not match other sources including Google Finance and Yahoo.com's main quote page.

While we were in the middle of getting to the bottom of this conundrum, we received the following message from our data vendor:

"Please be advised that due to late trade activity, there may be inconsistancies in Composite Close from Pacific Exchange for an unknown number of securities. We are currently investigating this with the NYSE and will provide further information as it becomes available.  Please see the attached Trader Update link posted by NYSE.
http://www.nyse.com/pdfs/NOTICE_Duplicated_Extended_Session_Prints_on_October_27_2011.pdf "

The bottom line is that something screwed up the closing data for an "unknown number" of NYSE stocks yesterday and that screw up affected pretty much everyone.  At this point we do not know if our closing data for yesterday is correct or not.  We expect that at some point later today, we will get an update with the new "correct" numbers from the NYSE.  At that point, we will update our databases and the charts may change as a result.

Sorry for the problem, but the issue is "upstream" from us, apparently at the NYSE itself.

March 28, 2011 - Intraday Data Offset by One Hour - Now Fixed

Our programming team made a mistake last month when they updated our servers to support London stocks.  The mistake was assuming that London time was the same a Greenwich Mean Time (GMT) which is what our datafeed uses for its time fields.  Up until Sunday, London time actually was the same as GMT, but that was just a coincidence.  On Sunday, London time changed to be one hour different from GMT and that is what caused our system to display incorrect intraday charts this morning.

At this point, most of the problems have been fixed (LSE stocks will be corrected shortly).  The programmers have been re-educated about time zones.  And, because this was due to a mistake on our part, all current members have had a free day of service added to their account.

March 27, 2011 - Members Unable to Log In for 2 Hours - Now Fixed

From approximately 9:05am Eastern until 10:50am this morning, members were unable to log into their accounts.  The problem was traced to a full hard disk on one of our database servers.  The hard disk has been cleared and things are working normally again.  We apologize for whatever problems this problem caused.

Important Side Note:  We posted a message about the problem on our website at 9:45am.  The message appeared at the top of the Problem Report page.  We received over 400 problem reports after we posted that message.  While we normally love hearing from our users, in this case those extra problem reports delayed our ability to find and fix the problem and hindered our ability to communicate the status of the issue to everyone.  We are unsure how we can make our site messages more visible so we are including this reminder to everyone - please check for site messages on our homepage and on the Problem Report page before sending in a problem report.  Thanks!

March 7th - Saved ChartStyles Missing, Site Slow - 1pm Update

UPDATE @ 10:00pm - We've moved the problem database onto the new servers.  We expect the open tomorrow to go much smoother.

===============

UPDATE @ 1:15pm - No changes at this point.  There are still little bursts of slowness from time to time.  Chip posted a blog article about it also and announced that all active members are getting 2 free days of service as a result of this problem.

===============

UPDATE @ 12:15pm - Things continue to improve but are not 100%.  We have disabled some housekeeping activities on the slow server which has reduced the frequency and severity of the slowdowns however they are still occurring from time to time.  We are continuing to work on the ultimate solution which is to migrate the database onto some bigger, newer servers.

=================

UPDATE @ 11:00am - We are still experiencing occasional slowdowns and/or loss of chart style information.  Things are better than they were earlier but they are still not "normal."  We are in the process of moving our style database to a new server but that will take some time.  Until that is complete, you may experience temporary slowness and your charts may temporarily change into a plain candlestick chart.  Thank you for your patience as we sort things out.

==================

ORIGINAL POST @ 10:40am - Sometime over the weekend, one of our key databases became extremely slow for reasons that aren't fully clear.  Unfortunately, that slowness didn't become apparent until the market opened on Monday.  As soon as the market opened, that slow database prevented some users from seeing their saved charts and it affected the overall performance on the website.

At 10:10am we restarted the slow database server.  The restart process took 10 minutes to complete during which time our site was not working correctly.

We are now monitoring things to see if they recover correctly.  So far, we are looking "better" but we are not positive that things are back to normal yet.

Our apologies for this outage.  Please check back here periodically for more updates.

(Edit: Changes times to Eastern timezone)

Feb. 24th @ 1:15pm - Some Intraday Data Missing Sporadically, Now Fixed

Grrrrr.  We hit the wrong button today during our London update process and accidentally lost some Intraday data for some North American stocks.  Users may sporadically see flat lines on part of their intraday charts.  We will have the problem fixed shortly.  Our apologies for any problems this has caused.

Update @ 1:30pm - Things are now fixed and the charts should look correct.

Dec. 7th @ 10am - Main Database Failure Causes 30 Minute Outage, Today's on Us

Around 10am this morning, our main ChartList database slowed to a crawl for reasons that we do not yet understand.  That caused pretty much everything in our system to clog up and eventually things stopped working altogether.  It took us a while to get the database restarted and everything else unclogged but things appear to be back up and running now.

Our apologies for the outage.  We know how frustrating it can be when our website doesn't work properly.  We appreciate everyone who didn't write in to our support team with hateful comments and we forgive those that did.  Because of the outage, today is now a "free day" for our members.  All members have had one additional day of service added to their account.

This is the second time this database has bogged on us unexpectedly and so we are going to upgrade the server hardware it is running on with the latest-and-greatest from Dell in the next week or so.  We are also continuing to investigate the root cause of the problem in order to prevent it from happening again.

Until today, our reliability track record had been looking really good.  We're redoubling our efforts to try an push our rating as close to 100% as possible which is always our goal.

Oct. 20th @ 12:20pm - 2 Minute Network Disruption, Traffic Re-Routed

Around 12:20pm today our connectivity to much of the internet was interrupted for about 2 minutes.  InterNAP, our service provider, is currently blaming the problem on Sprint's connection to the InterNAP data center in downtown Seattle.  They have re-routed traffic to bypass the problem for now.  That re-routing should have happened almost instantly but didn't for some reason.  

Things appear to be back to normal for now however it is possible that some people are seeing slower response times from our website because of the "re-routing" that is currently in place.  It is also possible that there will be additional disruptions while InterNAP and Sprint sort things out.

We are pressing InterNAP for better answers.  We selected them as our ISP specifically to avoid this kind of problem.

 

Oct. 15th @ 12:00pm - Auto-Refresh Feature Stopping Frequently, Now Fixed

We are aware of a problem with our Auto-Refreshing charts feature that is causing the auto-refresh to stop after a short period of time.  We are investigating that problem now and hope to have things fixed soon.  Our apologies for this inconvenience.  If your charts stop auto-updating, you can click the "Update" button to restart the auto-update process.

UPDATE @ 4:30pm - We have found and fixed the problem that was causing this issue.  One of our 40 servers was occasionally sending out incomplete charts in an unusual way that fooled our system into thinking that they were complete charts.  We've updated our system so that they will catch the problem correctly if it ever happens again.

Oct. 5th, 2010 @ 10:45am - Russell Index Update

Yesterday, Russell decided to discontinue several of their index products. That caused problems for our data vendor which resulted in all of the Russell indexes missing data on Monday.

Russell has discontinued the following indexes:

$RMC Russell MidCap Index 
$RDG Russell MidCap Growth Index 
$RMV Russell MidCap Value Index 
$RIENG Russell 1000 Energy RGS Index 
$RIFIN Russell 1000 Financial Services RGS Index 
$RITEC Russell 1000 Technology RGS Index

We have had no choice but to remove those indexes from our database. Please see our Data Updates blog for more details: 
http://blogs.stockcharts.com/data/

For the remaining Russell Indexes - including $RUT - Monday's missing data has been restored and things appear to be updating normally again now.

Oct. 4th, 2010 - Russell Indexes Not Updating

12:00pm EST - The various indexes from the Russell Company are not updating this morning.  Our data vendor is looking into the issue now and hopes to have it fixed soon.