Monday, June 15, 2015

Oh Equinix and your cross connects, you vex me.

vex [veks] verb (used with object) to irritate; annoy; provoke; to torment

Things have been quite busy at work over the last year, and that has prevented me from posting for a while. But one thorn in my side that is continually inflamed is Equinix and their dreaded cross connects (maybe it is spelled crossconnects, I'm not sure). Anyway, this is Equinix's cash cow, and they seem to be milking it even harder these days.

Don't get me wrong, I love Equinix for many reasons. Their colocation facilities are top notch, highly redundant, very secure and as obscure as possible (e.g. in remote areas without signage). I love them for that and compared to a plethora of other data center facilities I've been in (like Telx, which has huge signs on their old, decaying and power-starved buildings), they are one of the best. Plus, I love them because every major provider is also there, giving you the ability to interconnect to anyone you want for the low, low price of $325 per month!  Yes, you heard me right, ladies and gentlemen. Only $325/month. (to be fair to Telx, they charge a lot less, and give some volume discounts because we only pay $175 per over there when you order a pack of 3).

Equinix clearly has a corner on the POP market. If your business requires interconnecting to other providers, you have to be at an Equinix facility. Coresite and Telx are trying to catch up, but they aren't even close yet. Evoswitch is trying to make a HUGE DENT in this, however. But it will be a long while for them too. Anyway, I digress. Back to my story.

My most recent experience prompted me to write this little blog post because I think Equinix is trying to milk things more than usual these days.  We recently required a fiber cross connect to a new provider at Equinix in Los Angeles, Dallas and Ashburn. So we placed the order for the $325/month patch which also comes with a hefty $500 installation fee. When you place the order, you can request an "expedite" for $300 more, or just wait for their normal schedule. We opted to wait for their normal schedule, which was estimated to be about 5 days. Yep, 5 days to run a fiber from one part of the facility to another. That's fine, maybe they are quite busy (although i will say, a number of years ago I got a cross connect to another provider who was in the cage next to ours, and it still took 5 days, but perhaps there were different reasons for the delay back then.... <wink>). Anyway, when it was completed, you'd think you're in business right? Nope. Even though the fiber is run from one rack or cage to your rack or cage, it isn't connected.

So after completion, you need to open up a remote-hands ticket (which, as you might guess, is not free - they bill at $200 per hour) to run a fiber patch from the patch panel in your cabinet to the correct port on your device.  Yay, you're finally up and running! Right? Nope!  You now must alert your provider that you're connected. They too must run a fiber patch from the patch panel to the right port on their device. True, both you and your provider can avoid this by going to the facility yourself, but most everyone uses remote hands for two reasons: 1) you are not local to the data center, and; 2) even if you are local, you can't run the fiber between cages/racks to save that money, and the remote hands fee is certainly a drop in the bucket compared to the fiber install fee, so why not have them do it all.

Ok, so you might say that even though all of the above sounds expensive, it is also pretty normal.  And you are right. You make a point, although I would probably suggest that a good and ethical organization would (could and should) include the patch from the panel to your device in the $500 one-time fee... heck, they are in your cage/rack already, aren't they? That might add another 60 seconds to the job and certainly would make the customer feel better. But they don't do that, and they never have in my experience over the years, so I'll leave it alone.

But it gets worse!

With my latest experience and the three cross connects I ordered, 2 of them were still not functional after the entire process I outlined above was completed. Strange, right? Could it be that the provider left their port down? Could it be that we accidentally left our port down? Well, no. You're wrong. Both we and our provider had our ports configured and UP. One of us (it was us) had to engage Equinix remote hands yet again to "roll the fiber" (that is, swap the rx and the tx) because it must have been patched incorrectly during the final panel-to-port remote-hands request. Sure enough, that brought up the connection in both locations right away.

So my question now is, when patching from the panel to the device, why don't you check for light? This adds another 20 seconds to the job. If there is no light coming from the patch panel, fine, patch and be done. But if there is light from the patch panel on one of the fibers, also check the light at the device you're patching to and ensure the rx and tx line up. How hard is that?

So, to summarize, we ordered one single mode fiber cross connect at three locations. Here is the total cost for the month:

$500 install x 3 = $1500.00
$325 monthly x 3 = $975.00
$100 x 3 for patch = $300.00
$100 x 2 for roll = $200.00
=====================
GRAND TOTAL: $2975.00

Then, of course, we pay $975/month for as long as we're connected to that provider.

I wish I dreamed up the idea of Equinix many years ago. That business model is sure paying dividends these days! Stupid me and my useless brain :(

Oh, I'm not sure what their labor costs are, but 50 meters (164 feet) of single-mode fiber is about $36 here at monoprice. Ok, maybe they had to go twice or three times that distance... but they don't buy their fiber retail, do they? I bet they buy it in significant quantities.

Sorry for my rant. Still love Equinix facilities, just not the new milking strategy.

C.J.