Today we had problems with our website that were caused by our "Content Distribution Network (CDN)" provider, a company called MaxCDN. Last weekend we started using MaxCDN as a way to increase the speed and (in theory) the reliability of our website. A CDN is a network of computers that copies parts of our website and puts those copies on servers located all over the world so that the content is quicker to download.
After testing MaxCDN's services extensively, we rolled it out on our site earlier this week. We did the roll-out in stages being careful to test things at each stage. Everything went great until this morning when we rolled out the final piece of the puzzle. As soon as that part was added, all heck broke loose and MaxCDN's servers suddenly stopped sending out ANY of our files. We have been scrambling to recover ever since.
The part we added this morning was really very innocent - we just checked a checkbox telling MaxCDN to add a server in Singapore to our service so that people in Asia would get our pages faster. Evidently, that was a bad idea.
The first problem happened at 9am Eastern. We immediately took steps to disable (but not remove) our use of MaxCDN. We then started waiting for an explanation from them as to what exactly went wrong and why. At that point we hoped that they would tell us that this was a one-time unusual circumstance and that we could re-enable the performance improvements that MaxCDN provided.
Unfortunately, at 1PM Eastern, the problem suddenly reappeared. At that point, we threw in the towel and removed MaxCDN from our system completely.
If you are still seeing problems with our website, you may need to clear your browser's cache and restart your computer to get rid of any lasting vestiges of the old code.
Again, we are always looking for ways to make our service faster and more reliable. At some point, we will add a reliable CDN to our system. We thought we had found a good one. We will keep looking. Our apologies for whatever issues today's problems have caused.
UPDATE 3/21 - The people at MaxCDN are working very hard to try and win us back and we appreciate their efforts. They have explained that we somehow hit a very rare problem in their system and they have fixed that problem. They are making sure that our account is properly configured by inspecting it with a fine toothed comb. Once that process is complete, we will start testing things again _very_ slowly and carefully. Hopefully, things will go more smoothly this time. We'll keep you up-to-date.
UPDATE 3/25 - After MaxCDN's technical boffins gave us the all clear, we have gradually re-added our static content back onto their network and so far so good. People throughout the world are now getting those file much faster than before and that in term means our datacenter is able to send out our charts more efficiently. Knock on wood, but it appears that our 2nd try at rolling things out has worked.