StockCharts Blogs - Chip Anderson

Calling All "Stocks and Commodities" Subscribers!

It's that time of year again.  Time for the annual "TASC Readers' Choice Awards" voting.  If you haven't done so already, please fill out your online ballot and submit it before the deadline. 

This year we've been nominated in the "Online Analysis Platform" category which is quite flattering however we aren't holding our breath for that one.

The one we really hope to do well in is the "Technical Analysis Website" category which is, once again, a write-in only category.  That's why we need your help.  Please make sure to fill in our name - "StockCharts.com" - in that category and vote for us.  Without lots of participation by our users, there's no telling what will happen when the votes are counted.

Once again, thanks for taking the time to fill out your ballot - these awards mean a lot to everyone that receives one.

(BTW, if you don't subscribe to "Stocks and Commodities" magazine, do yourself a favor and sign up for it.  It's a never-ending source of inspiration for technical traders.  Highly recommended.)

Fiber Circuit Outage #2 Update #2 - 10/7/2009

I wanted to let everyone know where things stand with our Internet connections after last month's dual whammy.

The fiber cable that was damaged is fully repaired and working well.  We've had our vendor add additional physical protection to the cable itself down in the manhole where the problem occurred.  We don't expect a repeat of that particular problem as a result of those changes.

The Cisco router that also failed on us has been replaced.  The new router is installed and is connected to the fiber cable using completely new hardware.  As of late yesterday afternoon, the entire Fiber Circuit (i.e. cable + router) was working.

As I mentioned before however, we are not going to switch the site traffic back onto the Fiber Circuit for a couple more days.  We do not want a repeat of last time where we quickly switched back only to have it fail again.

Instead, we are going to be testing the circuit and installing an automatic fail-over system.  That work should be done by the middle of next week.  When completed we will be able to switch back on to the Fiber Circuit with confidence that we won't have another site outage.  If the Fiber Circuit fails again for any reason, the automatic fail-over will immediately move our site traffic onto the older T3 circuits that we are using now.

We are also making progress on getting a second Fiber Circuit installed via a completely different physical path.  That will give us fail-over capabilities without the limitations of the older T3 circuits.  That new cable should be installed sometime next month.

As a reminder, we are currently still using those older, slower T3 circuits for the site.  Those 4 circuits have a combined bandwidth of 180 megabits per second.  (The Fiber Circuit can handle 1024 megabits per second.)  Right now, during market hours, those T3 circuits are completely full.  That means that you may experience some slowness during the day - especially around the opening and closing bells.  Please help us out by trying to reduce the number of charts you use during the day whenever possible.

Looking back at our Status blog, so far this year we've endured the following "once in a lifetime" events:

• A major fire and power outage at our ISP's main Seattle facility
• The failure of a critical Dell server
• An idiot in our manhole stomping and breaking our fiber cable
• Our data vendor changing the S&P 500 symbol without warning
• A complete failure of a top-over-the-line Cisco router

Let's hope that our luck improves.  I, on the other hand, will continue to work on improving our infrastructure so that we can better survive these (supposedly) rare kind of issues.

Fiber Circuit Outage #2 Update - 10/1/2009

Just a quick update for people who have been following our struggles with the fiber circuit. 

At this point, all site traffic is still going over the slower T3 circuits.  In general, those circuits are working well but you will probably notice slowness around market open and market closing times.

Based on what we've learned so far, it appears that physical fiber connection is fine - it was not damaged like what occurred two weeks ago.  The current culprit appears to be the brand new Cisco router that is used to connect our network to the fiber circuit.  For some reason, that router is no longer responding to commands via any of its interfaces.  For all intents and purposes, it is currently a very large, very expensive brick.

We are working with our ISP to get a replacement router from Cisco out here ASAP.  Currently there is no ETA on when that will happen but we expect to have something here before the end of the week.

We have already come up with a plan for making things automatically failover to the T3 circuits once the fiber circuit is fixed.  As soon as the new router arrives, we will install it and wait one full week to ensure that it works well.  After that week is up, we will implement the automatic failover changes and move the site back onto the faster fiber circuit.  I will be posting updates here as that process progresses.

UPDATE: The new Cisco router is expected to arrive late today (Thursday, Oct. 1st).  As I mentioned above, we will need a week to install and test it.

10/03 UPDATE: The new router is here and installed.  Our ISP will be working this week to configure it to support automatic failover.  Once that work is done and tested, we will switch back to the faster fiber connection.  The will probably not happen until Thursday at the earliest; a week from Monday may be a better estimate.

La Fiber est Morte - We've Been Fooled Twice

At the end of my last post, I exclaimed "Vive la Fiber!"  In light of this morning's second major fiber failure, that comment is now extremely ironic.  The fiber connection did not "vive" (live long).  It did not "vive" much at all.

Compounding the problem, our cell phone alerts didn't go off for various reasons - thus we didn't discover the problem until after 9:00am Eastern (6:00am local time).

After spending 30 minutes ensuring that it was indeed a problem with the fiber connection, we had our ISP manually move our web traffic back onto our four older T3 circuits.  That process finished around 10:20am Eastern.

We are now working on four different issues to try and get back to the level of reliability that you expect from us.

1.) We are waiting for the cable operator - AboveNet - to find the problem and fix it.  We assume there was another break in the cable somewhere, but we are waiting for confirmation.

2.) We will never put all of our traffic on just one AboveNet fiber link again.  Despite the obvious benefits of increased speed and bandwidth, the reliability is not there and reliability trumps everything.  This is a case of "Fool me twice."   Once the problem is found and fixed, we will work with our ISP to configure things so that if the Fiber connection fails again, things will automatically switch over to the T3 circuits immediately.  The older T3 circuits are very expensive, but we are now going to keep them around to ensure reliability.

3.) We are expanding the number of people that get cell phone alerts whenever there's a problem with our website.  We will also be testing those cell phone alerts more frequently.

4.) We are making changes to ensure that our Status blog is always available even when the rest of the website is offline.  You should be able to visit http://blogs.stockcharts.com/status at any time and see our site status.

As you can see from our Pingdom Monitoring report, we've been down 14 hours 38 minutes this month.  This is our worst month by far for outages.  Despite all evidence to the contrary, we are working hard regain your trust on these issues.  I will post additional information as it becomes available.

Why our Gigabit Fiber Connection Matters

In case you missed the announcement, we are now sending our site traffic back across our high-speed gigabit fiber internet connection.  (That connection was damaged 3 weeks ago - more details here.)  We made the switch back onto the fiber connection on Tuesday night.  Up until then, we had been limping along on four older T3 connections.

Below is a graph showing the number of retransmitted data packets that we sent out during this week.

Each peak represents a different day of the week - Monday, Tuesday, etc.  As you can see, the number of retransmits send on Monday and Tuesday was much higher than during the rest of the week.  Vive la Fiber!

Fiber Link Outage Forces Us Back onto Older, Slower Circuits

Blech.  We were offline for nine hours today.  That was really, really painful.  I know it was painful for our users also.

From our perspective, this outage was as sudden as a bolt of lightning.  One moment things were working; the next moment, nada.  The problem was with our new fiber Gigabit ethernet link that runs from our offices in Redmond, Washington to our ISP's data center in downtown Seattle.  If you haven't read the play-by-play account of the problem and how it was diagnosed, feel free to check out our Status Blog entry.

At this point, there are two things that we want everyone to know:

1.) We have given all of our subscribers an additional week of service for free because of this outage.  Your account's expiration date has already been updated to reflect that additional time.

2.) For the next couple of days, our site might be somewhat slower than it has been.  The older T3 circuits that we are now using can only handle 180 megabits of traffic at a time whereas the new fiber link which broke was able to handle 1024 megabits of traffic.  Until that fiber link is fixed, the site will have to slow down some.

Again, our apologies for this prolonged outage and for any slowness that you see over the next couple of days.  I'll be hard at work pressing our network vendors for answers and better solutions to prevent this kind of thing from happening again.

- Chip

What's the Difference between Price Channels and Donchian Bands?

Here's a hint:  Nothing.

If there is anyone out there that wants to use Donchian Bands on their charts, they can just use the "Price Channel" overlay - it is the same thing.

Donchian Bands consist of three lines that overlay a price chart.  The upper line represents the highest high during the specified period of time.  The lower line represents to lowest low during that same time.  The middle line is the vertical mid-point between the upper and lower lines.

That description is identical to the "Price Channels" overlay that you'll find on our SharpCharts.  Here's an example:

Yahoo Hates Us - Currently Rejecting our Mail to Yahoo Users

Starting last Friday, for unexplained reasons, Yahoo.com's mail servers started rejecting all mail messages that we send to Yahoo email users.  I am trying everything I know to get the people at Yahoo to fix this problem but so far, no luck.  Unlike the great support people here at StockCharts, the Yahoo mail support folks are not responding to my pleas for assistance.  It is a very frustrating situation.  We are doing the same things we've always done with respect to sending mail message - this issue has popped up because of changes that Yahoo made.

If you use a Yahoo.com email address, you might want to get an email address somewhere else.  If they starting blocking our emails out of the blue, who knows what other important messages may not be getting through?

This problem means that Yahoo.com mail users are currently not getting Market Message update notifications, emailed SharpCharts, or subscription update notifications.

My apologizes for whatever problems this has caused (or will cause in the future).  I will continue to try and reach someone at Yahoo that can fix this problem but currently things don't look promising.  If anything changes, I will post an update on this blog.

26-Aug:  Update:  The Yahoo mail gods appear to have fixed the problem.  Currently we are able to send mail messages to yahoo.com users.  Hopefully it will stay fixed this time.

Support Response Email Problem Fixed - Please Resend Recent Requests

Today, to our shock and dismay, we discovered that because of a technical screw-up, for the past 10 days many of our responses to customer service requests have not been going out in email like they were supposed to.  Our customer service team was working hard (as always) answering everyone's support requests, but those answers weren't being sent out by the email server!

Our sincere apologies to everyone who has sent us a support request anytime after August 9th.  We weren't deliberately ignoring you!

As soon as we realized that emails weren't going out, we were able to find and fix the problem.  Support responses are now being emailed out again like that are supposed to be.  We are now working hard to re-send out our responses to requests we got during the past 10 days however, if you sent us a question and we did not get back to you, please re-send us your question so we can help.  You can use the form on this page to send us your question.

Big Bandwidth Boost Coming Friday Afternoon

Bandwidth is defined as the amount of data that one can "push" through an Internet connection at the same time.  The commercials on TV for home service equate bandwidth with "speed" and, while that is somewhat true, in the case of a high performance web site like StockCharts.com, bandwidth is really all about capacity.

Over the years, our bandwidth has grown tremendously and each time it increased our customers benefited.  Initially, we had a 1.5 megabit T1 connection to the Internet.  We managed to fill that up pretty quickly.  So we added another (3.0 megabits), and then another (4.5 megabits) and then another (6.0 megabits).

At that point (2002) adding bandwidth by adding T1 circuits didn't really make sense.  So next we stepped up to a huge (at the time) T3 circuit - 45 megabits!  Over 7 times more capacity than we currently had.  We'd never run out of bandwidth again!  Right?

We added our second T3 (90 megabits) in 2004.  And our third (135 megabits) in 2006.  And our fourth (180 megabits) in 2008.  That's where things stand today.

Right now, on a busy day, we use about 160 megabits for serving our charts.  Since we never want to use up all of our bandwidth, we've been looking into adding even more bandwidth.  But, as with the T1s, adding yet another T3 doesn't seem to be the right solution.

Fortunately, there is now another option: Fiber Gigabit!  That's 1024 megabits in one connection.  That's over 5 times more capacity than we currently have!  We'll never run out of... bandwidth... again... oh, wait.

Well, even if (when?) history repeats itself, we'll be ready.  In the mean time, by moving to Fiber Gigabit, we will be enabling much more growth in the future and we are ensuring that you will always have full access to our charts without any delays.

We are going to be switching over to our new Fiber Gigabit connection on Friday afternoon after the market closes.  IN THEORY, the move will be invisible to you.  IN THEORY, our site will run uninterrupted throughout the process.  IN THEORY, no one will notice the change.  That said, I wouldn't be surprized if there was a short interruption in service just because I like to err on the side of caution.

I'll make another announcement once the change over is complete.  If we are offline for more than a couple of minutes tomorrow, we'll post messages up on our Status blog.