StockCharts.com - Blogs

Today we removed the Open, High and Low data from many of our commodity based indexes like $SILVER and $USD.  Those charts now appear as line charts connecting individual closing points instead of the bar or candlestick charts people saw before.

Some people are not happy with us for making this change.  I understand that frustration.  If there were some other way to fix the issue, we would have done that instead.  Here are the reasons for removing that data:

The number one reason for removing that data is because after a long investigation, we have lost faith in the accuracy of those data points.  As we completed our transition off of the ThomsonReuters datafeed and onto the IDC datafeed, we were forced to take a hard look at the "continuous contract" data we've been receiving from Thomson for years.  In the course of comparing that data to other sources, we noticed large numbers of discrepancies with the Open, High and Low data points.  (The Close data points were reliable however.)

Drilling down into the raw data, we saw that many of the problems occurred because the data was inconsistent with itself.  By that I mean that the closing value was lower than the low or the closing value was higher than the high.  Our system automatically adjusts for those inconsistencies so that crazy looking candles don't show up on the charts but the bar and candlesticks on those charts were often inaccurate.

For example, if we saw an End-of-Day quote where the Open was 10, the High was 11, the Low was 9, and the Close was 7 (i.e. lower than the low) what should we do?  Do we move the Low down to 7?  Do we move the Close up to 9?  Do we throw it all out?  It's pretty much a lose-lose situation regardless of what we choose.

Unfortunately, what we discovered this week is that lots of the Open, High and Low values for our EOD commodity based indexes were having that exact problem occur frequently and the problem had been going on for some time.

As long-term subscribers know, accuracy is a big deal with us here at StockCharts.  We work very hard to provide data and calculations that are as accurate as possible.  Faced with the reality that large amounts of our EOD commodity data was incorrect, I decided to have it removed rather than continue to potentially mislead people.

That difficult decision was made easier by the fact that we have always been up-front with people about the nature of our commodity based indexes.  Specifically, our support area has contained the following warning for years:

"We do provide end-of-day charts of certain commodity indexes.  Those charts are based on the non-tradeable "continuous" contract for each commodity.  They are designed to help equity traders understand the mid- and long-term directions of various commodities for help with making decisions about trading equities.  They are not designed for actual futures trading."

We have been very careful over the years to never bill ourselves as a commodities web site.  We are an equities web site.  We are not FuturesCharts.com; we are StockCharts.com.  We do not have access to a full commodities data feed and we have never provided charts or data for actual commodities contracts.

Given those warnings and the end-of-day nature of our commodity charts, I was hopeful that this issue would not affect many people.  Given the proper mid-to-long-term perspective, a close-only chart and a OHLC chart provide you with the same general information about the direction and performance of an index.

For example, many people do what John Murphy suggests and use Intermarket Performance Charts and Ratio charts to compare indexes.  Both of those charting styles focus on closing values, not Highs and Lows and so that kind of analysis is not affected by this change at all.

The problems occur when people made incorrect assumptions about what our commodity charts were based on and began using them to make commodity trading decisions.  To those people I say the same thing now that I've always said about this matter:  Don't use our EOD commodity charts for making futures trading decisions!

In many ways I believe that removing the Open, High, and Low from those charts is a good thing because it re-reinforces the purpose of those charts and minimizes the chances that they will be misused in the future.

Speaking of the future, what comes next?

It has always been StockCharts.com's intention to add legitimate, tradable futures data to our website at some point.  It is still our intention to do that.  The economics of acquiring and redistributing that data legally is still very challenging however and so we are still unable to provide real futures charts at this time.

That said, we are continuing to look for possible "middle-ground" solutions which would allow us to provide, for example, delayed spot prices for popular commodities.  (The great irony here is that the search for better commodity data is what led us to discover the bad OHL data in our current indexes.)  At this point, we still have not found a solution for commodities data beyond what we currently provide but we are continuing to look.

In summary:

1. The old commodity data for Open, High and Low was incorrect and has been removed.
2. Our commodity indexes shouldn't be used for trading.
3. You can still do trend, performance and ratio analysis of our current commodity charts which is always how we intended them to be used.
4. Better commodity data will be available at some point in the future but we do not know when.

- Chip

Here's great news for ExtraRT members that want to save some money.  ExtraRT members can now choose to downgrade their account from ExtraRT to regular Extra.

Why would they want to do that?

Now that we provide "free" realtime charts using data from the BATS exchange, some of our current ExtraRT members may decide that BATS data is good enough for them and they want to save the extra $9.95 per month that ExtraRT's "real" realtime costs.

For example, consider an ExtraRT member that only looks at popular US stocks and has 10 months remaining on their ExtraRT subscription.  If that member chooses to downgrade to Extra, they will get 4 additional months of service added to their account(!) and their renewal next year will cost much less.

You need to make absolutely sure that you understand the differences between BATS realtime data and "real" realtime data before deciding to downgrade.  While you can always choose to upgrade back to ExtraRT, you'll have to pay for upgrading those additional months also and that could get expensive.

Here are some links to my previous articles about BATS data and how it differs from NYSE/Nasdaq/TSX realtime data:

STOCKCHARTS IS GOING TO "BATS" FOR YOU
and
"Real" real-time vs. "Free" real-time

If those differences don't matter to you, click here and enter your User ID and password, then select "Downgrade to Extra" on the page that appears and follow the instructions.  Soon, you'll be enjoying free realtime data and saving money!

Over the past year, more and more stock sites are offering "Free real-time" data and charts.  Here at StockCharts, we too have been working hard to provide our users with more options when it comes to up-to-date data.  The really exciting news is that we are now really close to being able to provide a "Free real-time" option also.

Before I talk about that however, I wanted to talk a little bit about what "Free real-time" actually is and how it differs from the "Real real-time" that we currently offer.  It is important to understand that there are several significant limitations that most of those other sites don't tell you about when it comes to "Free real-time."

First off, where does "Free real-time" data come from?  It's important to understand that it does NOT come from the major stock exchanges.  Those exchanges only provide trading data to companies (like StockCharts) that pay licensing fees and follow their strict rules for distributing exchange data.  For the sake of this discussion, I'm going to call non-delayed price and volume information that comes from the major stock exchanges "Real real-time" data.  "Real real-time" data is what we've been offering on StockCharts for years and we will continue to offer it into the foreseeable future.

"Real real-time" data is a very accurate reflection of the price and volume for a particular stock.  Because the vast majority of investors buy and sell stocks via the major exchanges "Real real-time" data from those exchanges most closely reflects the actual value and demand for those stocks.  If you trade a stock via a normal stock brokerage without any special instructions, they will execute the trade on one of the major exchanges using the "Real real-time" pricing.

"Free real-time" data comes from one of the minor exchanges.  There are lots of smaller exchanges around which trade stocks but historically the volume of these exchanges is very, very low.  Historically, the smaller exchanges also had the same legal restrictions about distributing data as the major exchanges did.  Given those limitations / restrictions (and the fact that we already had data from the major exchanges), we chose not to provide data from the smaller exchanges on our site.

But then something changed...

In June of 2005 a new player appeared on the scene, the BATS Exchange, and it decided to use different rules.  Specifically, it didn't implement all of the heavy restrictions on trading data that all other exchanges had.  BATS data is free to redistribute(!).

When BATS first came out, it was an interesting idea but it still needed to prove itself - it needed to become popular enough that its data would closely match the data coming from the major exchanges.  If very few people traded on the BATS Exchange, its prices would be very different.  In 2009, BATS eclipsed the other minor exchanges to become the third largest exchange in the US, a good indication that BATS price data is very close to the prices on the major exchanges.

Given that accomplishment, we felt it was time to start adding BATS data to our website.  That project is just about complete.  When finished, it means that all of our charts that currently display delayed data (with the yellow highlight) will display "Free real-time" data from the BATS exchange instead.  Here's an example of what that kind of chart will look like:

The dashed yellow vertical line shows the dividing point between the delayed data from the NYSE and the non-delayed data from BATS.  All of the yellow highlighted bars are from the BATS exchange.

As time goes on, the yellow bars will get replaced with "official" data from the NYSE (on the left side of the dashed line) and new BATS bars will appear on the right edge of the chart.

Now, there is one other aspect of this chart that is very important to notice and understand.  There is no volume displayed for the BATS bars.  While the BATS exchange is more popular than the other small exchanges, it is still much smaller than the major exchanges.  Thus, the volume data that BATS provides is much smaller.  If we added BATS volume data to our charts, it would significantly distort any volume-based indicators and signals.  Because of those issues, we have decided to leave BATS volume data off of our charts.

(Side Note: Some other services will multiply BATS volume values by some arbitrary factor to make its bars look similar to the "official" volume bars.  The problem with that approach is that it overemphasizes small changes in BATS volume.  It also doesn't help if BATS volume is zero for a given period.)

So in summary:

"Free real-time" == BATS Exchange data
Pros:  Free
Cons: Price data may be slightly off,  Volume data is waaaaay off.  US stocks only.

"Real real-time" == Major Exchanges' data
Pros: "Official" price and volume data.
Cons: Not Free

We hope to offer "Free real-time" on our site by the end of January (although that could change - don't shoot me if it does!).  When we do, you will have a choice of using either the BATS real-time data or the "Real real-time" data that we offer now.  Delayed data will become a thing of the past.

And yes, if you are a "Real real-time" subscriber now and you want to downgrade to "Free real-time" when it becomes available, we will give you a way to do that.  Stay tuned for all the details.

The world just shifted underneath your feet.  Did you feel it?

Today, after 18 months of preparations, we have finally started to serve charts using data from our new IDC/Comstock datafeed.  At this point, intraday data for most US stocks is coming from our new IDC/Comstock datafeed instead of our Thomson/Reuters datafeed.

I've documented our past problems with the Thomson datafeed on many occasions.  We were all reminded of those problems recently.  Unfortunately, at this point, I can't guarantee that the IDC/Comstock feed will be any more reliabled, but at least there is reason for hope.  And, in the spirit of verify-then-trust, we plan on keeping the Thomson datafeed around as a backup in case the IDC/Comstock feed has trouble.

Over the next couple of days/weeks/months, we will be tuning and tweaking our use of these two feeds and making sure that things are accurate, that the new feed performs quickly, and that it remains stable.  There's a good chance that we will hit a bump or two as we go down that road.  But the goal of having a better, more reliable data service is closer than ever and we are not going to stop until that goal is achieved.

Today's problems with the Thomson data feed were extremely disappointing on a number of levels.  The biggest frustration from my perspective is that promises made directly to me by Thomson were not kept.  Thomson - for reasons that I cannot fathom - continues to rely on us to notify them of problems with their equipment rather than install effective monitoring systems.  It is a situation I consider unacceptable and I have told them so in no uncertain terms.

This is not the first time a vendor has let us down.  This is not the first time that Thomson has seriously let us down.  As long as we only have one data feed here at StockCharts.com however we are stuck with Thomson and their unreliable servers.  Sigh.

Ahhh, but notice how I phrased that last sentence?  As long as we only have one data feed here...

The good news (and the ironic news) is that we are very, very close to having two data feeds here at StockCharts.  Over the past month, we've been working out the remaining kinks with IDC/Comstock and the new data equipment that they've installed here.  The process has taken waaaaaay too long to complete (mostly due to delays over unnecessary exchange agreements with the stock exchanges), but the end is very close now.

I expect that we will start using data from the IDC/Comstock feed for some of our charts starting next week.  We will start with just a couple of stocks and then add more and more as we become more comfortable with the feed.  Our goal is to have the data for all stocks coming off the IDC/Comstock feed by the end of April.

By having the majority of our charts using data from the IDC/Comstock feed, our reliance on Thomson will be dramatically reduced.  And by having two feeds here, our ability to continue serving charts if one of the feeds has problems will be dramatically increased.  Having two feeds is expensive, but today's problems underscored how important it is.

Time Warner (TWX) is splitting off their cable business.  That change will have a big impact on the stock starting on Monday (March 30th).  Basically it will result in a big reverse split and a distribution-like adjustment to TWX.

As you hopefully know, we adjust our historical data to eliminate the "artificial" effects of things like splits and distributions.  Unfortunately, right now (Sunday PM) we do not know the exact details of the adjustments that are being made.

In order to be sure, we've decided to delay the adjusting of TWX's historical data until Monday afternoon.  By then we will know the exact details of the adjustment(s) that we need to make.   This means that for Monday morning, there will be a disconcerting gap in the TWX chart.  The gap will disappear Monday afternoon.

We apologize in advance for any inconvenience the gap in the TWX chart causes.

UPDATE: 2:33pm Eastern - TWX has been adjusted and the gap has been removed.

$CPC is our symbol for the CBOE Put/Call ratio.  It is a market sentiment indicator that shows the total number of "Put" options (bearish) divided by the total number of "Call" options (bullish).  That means that it is an "inverse" indicator - i.e., readings below 1.0 are bullish while readings above 1.0 are bearish.  Here's a daily $CPC chart for the past couple of years: