StockCharts Blogs - Status

Nov. 16th, 2009 @ 2:00pm - ChartStyles Database Goes Offline Unexpectedly

The database that keeps our ChartStyles data - i.e. all the stored settings for the stored charts that our members use - stopped responding for some reason.  We have restarted that server and things have returned to normal. 

If you are still seeing incorrect charts, please try clearing your browser's temporary file cache and restarting your computer.  If that doesn't work, please check back here for any updates.

We are investigating why that server went offline unexpectedly.  We will post a follow-up message here when we learn more.  Thank you for your patience.

UPDATE: 3:00pm - The ChartStyle database continues to work fine now.  We have changed how it uses its memory in order to avoid this problem in the future.  Again, our apologies for the earlier problem.

Oct. 15th, 2009 @ 11am - Thomson Servers Not Serving US Data, Index Data and Delayed Canadian Data, Now Fixed

Three of our four Thomson data servers have stopped providing updates for US indexes and US stocks.  In addition, while those servers are still providing us with real-time Canadian data, they are not providing us with delayed Canadian data.  Thomson is aware of the problem and is investigating.

Currently, this is not a huge problem because the US data for our charts comes from our other data provider, IDC/Comstock.  We have moved all of the index requests onto the one remaining good Thomson server and (knock on wood) that seems to be working fine so far.

At this point, we think the only ongoing problem is for Canadian charts using delayed data.  Our apologies for that.  Hopefully Thomson will have a fix for the problem soon.

UPDATE @ 1:50pm: Thomson has fixed the problem and we are back to running on all four servers.  Please let our support team know if you are still seeing data discrepancies.

Oct. 12th, 2009 @ 6:00pm - Moving Back onto the High-Speed Fiber Link, Complete!

After much testing and configuration work, we are moving back onto our Gigabit Eithernet Fiber link right now.  There may be a slight interruption of service during the move.  It should not last longer that 1 or 2 minutes at the most (hopefully just a couple of seconds).  Check back here for updates if you are unable to reconnect to the site after the switchover.

UPDATE: 6:30pm - We have successfully switched back onto the Fiber circuit.  We will continue to test things throughout the rest of the evening however we don't anticipate that those tests will cause any noticeable disruptions.

12:15am - OK.  That last part of the last update may not have been entirely true.  We've decided to conduct a live test of the failover capabilty tonight at 1am.  There may be a short outage as a result of these test - that's why we're testing it late at night.  Apologies in advance but once the failover capability is 100% tested, we will have a much more reliable, very high-speed Internet connection.

1:15am - The failover testing is complete for tonight.  Failover works however some more tuning work is needed.  That work should be completed tomorrow at which point we will have full failover protection.

12-Oct-2009 @ 9:45am - CBOE Indexes not updating correctly, Now Fixed

CBOE indexes like $SPX, $OEX, $VIX, and others are not updating correctly this morning on Intraday charts.  The problem is happening on both of our data feeds.  The problem is also appearing on other websites.  We've alerted our data vendors and have been told by them that "it's a system problem with the exchange."  Hopefully the CBOE will correct the situation soon.

Updates:

10:20am - Even the chart on CBOE.com has the problem.

10:25am - Just received the following email from CBOE:

Subject:     Alert #44 2009 - S&P 500 and S&P 100 Indices Not Updating
From:     Market Data Operations
Date:     October 12, 2009 10:20:05 AM EDT

Please be advised that the S&P 500 (INX) and S&P 100 (OEX) indices stopped updating as of 09:31 ET.  The issue has been escalated and is being investigated.

The sender provided the following contact information.
  Sender's Name: Market Data Operations
  Sender's Email: Ops@cmegroup.com

(Note: Edited to make all times Eastern.)

10:45am - The CBOE-based indexes have started updating again.  A large gap remains on the charts however.  We are not optimistic that the gap will be fixed.

11:00am - We've sent an email to CBOE formally complaining about their 50-minute delay in notifying us of the problem.

Sept. 28th, 2009 @ 4:38am - Fiber Internet Circuit Failed Again

Our Fiber Internet circuit is down again this morning.  The problem started around 3am Pacific time. We are checking with InterNAP and AboveNet to find out why.  We will post messages here as we learn more.

9:30am Eastern - We have opened a trouble ticket with InterNAP.  They are suspecting a problem with the cable and are contacting the cable owner, AboveNet.  If there isn't a simple resolution shortly, we will be moving back onto the older T3 circuits again within the hour.

9:40am Eastern - Looks like the circuit actually went down exactly at midnight local time.  That is very suspicious and hopefully means that there was some administrative error instead of a physical problem.  I say "hopefully" because those kind of problems can be fixed much more easily.

Correction:  We see now that it went down at 1:38am local time.

10:00am - We're trying to switch back to the old circuits now.

10:25am - We are back up on the old T3 circuits now.  We'll probably spend the rest of the day playing "blame ping-pong" with the people involved.  Look for more information on Chip's blog later today.

Sept. 22, 2009 @ 7:45pm - Back in the Fiber Fastlane

Finally some good news!  We've successfully transitioned the website back onto the high-speed Fiber Gigabit connection that was severed two week ago.  Hopefully, you notice a slight speed increase with the website especially when the markets are busy at the open and close of trading.

Sept. 22, 2009 @ 9:30am - Daily Quotes Not Working

Daily quotes that appear at the top of our charts are not working right now.  Daily, weekly and monthly bars on our charts are not being updated.  Intraday data (1-minute, 5-minute, etc.) IS appearing correctly on Intraday charts however.

We believe that this problem is the result of a change to our systems that we were required to make yesterday afternoon.  ThomsonReuters is requiring us to make that change because of an update they made to their servers.  Unfortunately, we were unable to test our change under live market conditions before making it.  Obviously, there are problems with the change that we made.

We apologize for this problem.  We are working hard to fix things as quickly as possible.  The good news is that the problem is (most likely) in parts of the system that we control and we should have a fix available very soon.

UPDATES -

11:10am Eastern - Things should be working again now.

11:50am Eastern - We are still "cleaning up" some things associated with this problem.  The time on the charts, the Scan Engine updates, and a couple of other smaller issues.  Please bear with us as we work through the problems.

12:30pm Eastern - We have fixed most of the outstanding issues resulting from this morning's problems.  At this point we are continuing to test all aspects of our site for any lingering issues.

Sept. 17th, 2009 @ 9:50am - AMEX Indexes Not Updating

AMEX-based indexes like $DRG, $HUI, $RIFIN and others are not updating this morning for some reason.  We are checking with ThomsonReuters now to see what the problem is.  We hope this isn't a repeat of the CBOE fiasco.

10:20am - The problem appears to have been fixed and those indexes are updating again however the data from the first 45 minutes of trading is missing.  We are working with Thomson to get that data back.

Sept 14th, 2009 @ 9:50am - CBOE-related Indexes Not Updating

Indexes that we get from the CBOE via our Thomson-Reuters datafeed are not updating this morning.  This includes $SPX, $OEX, $VIX and several others.  The problem has been traced to a change that Thomson made over the weekend.  They changed their internal symbols for those indexes without notifying us.  Their system now thinks we are not authorized to use those new symbols.  That, of course, is nuts.

We are currently in the process of getting Thomson to verify the new symbols for us and authorize them so we can access them again.  We apologize for this problem.  We are making our displeasure with Thomson's actions known to them very clearly.

(For those of you who have followed our struggles with Thomson in the past, we are continuing our slow migration away from them exactly because of issues like this.  At this point, all US stock data comes from our IDC-Comstock feed but currently we are still dependent on Thomson for index data.  This screw-up will hasten that change-over.)

12:50pm - Thomson is promising to have this fixed "within an hour of now."  Not sure what is taking so long.

2:00pm - Well, an hour has come and gone and we still do not have access to the missing data symbols.  We are continuing to voice our displeasure with this situation to Thomson.

3:00pm - Big conference call with Thomson.  Some screaming was involved.  Bottom line: They'll call us back.

4:00pm - ThomsonReuters failed us today.  They still have not given us access to our CBOE data.

After further investigation we are now convinced that Thomson should not have changed those symbols last weekend, that Thomson did not notify us of the change, and that Thomson should have been able to fix the problem quickly but didn't.  We're left with the conclusion that Thomson is unable to solve our critical issues in a timely manner.

Currently, we have no choice but to continue trying to get them to fix things before tomorrow's open.  In addition however, we are now starting the process of enabling index data from our alternate data vendor, IDC-Comstock.

4:45pm - With the markets now closed, we are able to download data from an alternate data source and post it.  That means that all CBOE-related Index charts are now up-to-date.  Again, our apologies for the delay in getting this data to you.

6:45pm - Looks like the problem has finally been fixed.  We are now able to get real-time, intraday data for $SPX from the Thomson data feed.

Sept. 10th, 2009 @ 6:45pm - Network Routing Issue Prevented Some Connections, Now Fixed

Starting around 6:30pm Eastern and extending until around 7:30pm Eastern, about 25% of the people trying to connect to our website may have experienced problems.  We tracked the issue down to an router at our ISP that was still trying to send network traffic through the Gigabit Fiber link that was disabled last night.  That router has now been updated and things should be working again for everyone.  We apologize if you were impacted by this issue.

With the exception of the problem mentioned above, things went fairly well today.  We continue to appreciate everyone's patience as we keep working on fixing the problem with the faster Fiber link.  Currently, there is no ETA for when that link will be restored.