Verizon Broadband router with voice BRV - Novatel T1114 problems

I live in a fairly rural area, so Internet connectivity with my primary ISP is occasionally spotty. Because I work at home most of the time, I thought I'd improve the quality of my Internet connection by getting a backup link. I already have a Cisco ASA 5510 in my house, and it supports dual-WAN connections very nicely.  So I procured the Verizon 4G LTE Broadband Router with Voice, the BRV which is actually made by Novatel, their model T1114. I already am a Verizon Wireless customer and have a family plan, so adding the device was a no-brainer at a $0 upfront fee and $20 per month to share my existing data.

I should inform you at this point that I confirmed 4G was good at my house. I use the Wifi hotspot feature on my DROID MAXX all the time to broadcast wireless so I can connect to the Internet with my laptop during primary Internet outages. It works quite well indeed, so a router like this should only be better than my little phone.

I received it just a couple days after ordering and was pretty excited to get going. I placed it on my desk and plugged it in. It quickly connected to Verizon and within a few minutes I could use the handy buttons and display right on the device to see what the WIFI SSID and password were. I saw the SSID on my laptop, connected and boom. I'm online. It couldn't be any easier. Seriously!

The next step involved a little testing. I went to my favorite site, speedtest.net and ran some tests. Impressive! I easily received 20Mbps up and down, right here in the countryside. Who could be disappointed with that? And even though the ping time was in the high 80ms to google's 8.8.8.8 DNS server where my other ISP is 20ms, it's better than my primary Internet connection for speed. Yeah, yeah, I know Verizon charges a rather hefty per GB fee, but that's okay. I have 50GB on my plan which I barely touch, and since I'm only going to route through the device when my primary Internet is down, I should be just fine.

I typed verizonbrv into my browser and was greeted with a login page for the device. I entered the password I received from the device's screen and voila, I was in.  I was quite pleased to see that just about every configuration option is now at my fingertips, so I began to click around and easily changed the LAN settings to a more desirable IP range and specified a DMZ IP which I subsequently attached to my Cisco ASA.

I know Cisco ASAs fairly well, so I connected a CAT6 cable from one of the ports on the back of the Broadband Router into another port on the ASA and configured it as a backup route. I set a ping check on my primary to Google DNS and when it fails, to update the route to the other. I won't go into all the gorey details of setting up the ASA in this post. Since I don't have any published NAT connections since there is nothing at home to reach from the outside (sorry folks, no web or mail servers here for you to hack into :) I was done.  I pulled the cord on my primary link and within 10 seconds I was running on my backup.

I use VOIP here at home, so I made a few phone calls using my Vonage connection and couldn't be more pleased with the audio quality through the Verizon 4G network. I really wasn't sure how it would handle VOIP, so this was a very pleasant surprise. In fact, just to be sure it was solid, I had a good 15 minute conversation with my friend and never did it gripe. Crystal clear the whole time. Brilliant!

So what could be the "problem" you're alluding to in the title, CJ? Well, after about a week my primary Internet died, which usually happens every week or two. Everything switched over nicely to my backup, but I noticed that as I surfed the web I would get frequent blank pages. So I ran a constant ping test using my favorite tool, Pingplotter Pro, to a dozen locations. What I discovered is that the pings all timeout at the same time, for about 3 or 4 tests.

So I logged back into the BRV to see what was going on and what appeared to be happening is that the IP address was disappearing, and then a new one came back precisely every minute or two.  I enabled the logging and it confirmed that it was losing WAN connectivity every couple of minutes. Here is what I saw:

[L_WAN] QmiEvtConnStateHandler: IDLE
[L_WAN] QmiEvtDevStateHandler: WAN_STATE_IDLE
[L_WAN] ModemEventCallback call dropped
[L_WAN] Device state: WAN_STATE_IDLE
[L_WAN] Performing auto-connect base on service
[L_WAN] validate_roaming_for_call: current roaming pref = 0xff
[L_WAN] validate_roaming_for_call: current roam status: ccm=0x00
[L_WAN] validate_roaming_for_call: Roaming check is okay
[L_WAN] Modem_StartCall: Modem passed mininum service time
[L_WAN] Starting Call: prof=3, apn=, user=, pwd=, auth_pref=0, tech_pref=0, ip_family=0
[L_WAN] Data call started
[L_WAN] start_network_interface_callback: call_handle = 0x43d86850
[L_WAN] start_network_interface_callback: end_reason = 0x00000000
[L_WAN] start_network_interface_callback: verbose_end_reason = 0x00000000
[L_WAN] QmiEvtConnStateHandler: CONNECTED
[L_WAN] QmiEvtDevStateHandler: WAN_STATE_CONNECTED
[L_WAN] ModemEventCallback call up
[L_WAN] QmiEvtConnDormHandler: NOT DORMANT
[L_WAN] Device state: WAN_STATE_CONNECTED
[L_WAN] Dormancy: 0

Well... interesting stuff. And if I wait a couple minutes, the entire stream is there again and again. So why WAN_STATE_IDLE? It wasn't idle? I was using the Internet, and my pings were going through too.

So I took a look the diagnostics tab and saw a signal strength of -91db with an SNR of 7. Perhaps that was the problem... but why now and not before? So I Took a look on verizonwireless.com and found the Proband Repeater Antenna. After looking at it and reading the reviews, however, this did not seem to be a good fit. Many users complained that it did nothing to improve things, and in my opinion, it is an indoor antenna and I need a big outdoor one to get the biggest boost possible.

So I went to my favorite store (amazon.com) and ordered a very nice Wilson Yagi Antenna, ensuring that it operated at 700Mhz, the Verizon 4G LTE frequency. This is important, because I almost ordered another Yagi antenna that was 850Mhz and up, which would have surely disappointed. I added to my cart some connectors, 50 feet of ultra low-loss Coax with N-connectors, a surge protector and waited patiently for prime shipping to deliver it within 2 days.

To my delight, Amazon delivered the coax cable the next day, so I began my install project right away. Being a complete geek, when I built my home, I ran conduit from my lower-level office's storage closet to the roof. And because it is a nice flat roof, it's a breeze to get everything in place. So the following day when the antenna arrived, I installed it on a pre-existing pole (I have a CDMA booster in my house which was already up there... and it works great, BTW), tightened up the connections and attached it to the back of the BRV down in my office. Incidentally, the BRV sits on the top of my ASA.

At first when I logged into the device I saw zero signal gain. What!?  But I had a hunch what the problem was. On the back of the BRV are two antenna connectors, and I connected to the top one, which obviously must have been for the voice (1x) network, which I haven't even subscribed to. So I relocated it to the bottom connection and voila. A 11 db gain. Now I was sitting at -80 with the occasional peak to -79. That's pretty darn good, right?

Back to testing! I pulled the connection on my primary link in order to throw everything onto the backup. I did the same speedtest.net tests and constant pings and while the speedtest continued to perform well, my constant pings continued their periodic timeouts, and surfing the web continued to be interrupted by the disconnects.  The BRV was still losing the IP address every few minutes... and to make matters more interesting, every time it regained an IP, it was a different one.

So I called up the support folks at Verizon for some assistance. I didn't have much time since I had a conference call looming in about 30 minutes, so I ran through some basic troubleshooting with the individual I was speaking to who was obviously reading from a guide for this particular router. Everything he suggested was stuff I already tried, but for posterity and to amuse him, I did it all again. He suggested boosting the signal with "any kind of antenna, it didn't have to be a Verizon one" and I said I'd give it a try and call back because by that time I only had a few minutes left.

The following day I called into the Verizon phone system again. Interestingly enough it detected that I had an issue the prior day and asked if I was calling about the same issue again. That's handy! So I pressed "1" for yes and was routed to a much more intelligent individual. Perhaps their phone system knows that if you're calling about the same issue again, they better up the ante, so they did. This fellow seemed to know a lot more about this device and I had an intelligent conversation with him.  We went through some additional troubleshooting including a full system reset and concluded that one of two things needs to happen. 1) We replace the device to see if that will solve the issue, or 2) He opens a ticket to have someone check our area to see if there are 4G issues and/or if something on the tower in my area is slightly off.

Because 4G tests via my phone work very nicely without ping loss, I opted for option 1 and requested a new device which would arrive in 3 to 5 days.

Well, 2 days later the replacement device arrives. Because it does not have a user swappable SIM card, I had to spend about 30 minutes on the phone with a lovely lady at Verizon to get it working. But she was very kind and patient, and so was I. So we got the job done.  Well... you want to know the results, don't you? After getting everything working and configured again, it still disconnects. This time it doesn't disconnect every couple of minutes, it lasts a little longer, but the same thing appears in the logs roughly every 5 minutes or less and continues to annoy me.

What gives? I guess I'm going to call Verizon again and go with option 2 of having someone check this area. I'll keep you updated! In the interim, I sure hope you don't have the same issues with this device.

C.J.

UPDATE: 05/31/2014
I spoke to Verizon this morning for quite some time. They insist that the "way I'm using it" is just abnormal and additionally, the disconnects are quite normal too. In a 24 hour period, they saw that the modem disconnected and reconnected 1558 times but only 8 of those times were "bad" disconnects. Therefore, based on the fact that my usage is abnormal and that the percentage of "bad" disconnects is significantly less than 2% means they cannot escalate this to anyone.

So you're curious what I do on the Internet now, right? Well, I ping things, I surf the web, go to Twitter, Facebook, News websites, remote desktop to different servers to work on them etc. Does that sound abnormal? Apparently Verizon thinks so.

So I asked if he thought 1558 modem disconnects in a 24 hour period was abnormal, because that only confirms how frequently it is cutting me off. He said that was quite normal because it is a wireless network and there is no guarantee of constant connectivity.  Sure, I'll give you that, but every few minutes a disconnect.  But this is the real kicker, in my opinion. When I use the mobile hotspot on my phone to share the 4G Internet with my laptop, I can run a constant ping to something like Google DNS servers all day long. I can also keep an open RDP session to a server all day. Via this device, that does not happen. Every 2 to 3 minutes the pings stop for 4 to 8 consecutive tests and my RDP session disconnects.

Your disconnects are abnormal, Verizon, not my activity.

More soon.
C.J.

UPDATE: 06/15/2015
I'm so sorry it has taken me a year to update this post. Thanks to a comment today that triggered an email to me (and apologies to the prior commenter, I guess the email from blogger got stuck in my spam folder)... I have an update, and it is not very exciting.

I was waiting for the Novatel folks to release a software update for the device to fix this apparent bug, but in over a year, no software update has been released. Trust me, I check often. This is somewhat typical and reminds me entirely of Samsung. Once a device goes to market, they release one or two updates to fix bugs, but never any updates to release desperately needed new features.  They would MUCH rather you purchase the next model instead. But that is the subject for another rant, I suppose : )

So to make a long story short, I have determined that the router detects the double NAT and restarts the 4G connection as a result. I have a hunch this is to stop people from getting around the 10 device limit. It is somewhat annoying that the ASA5510 doesn't appear as a single device to the broadband router. I'm not smart enough to know why that is the case, and I haven't been able to test other firewall devices since I'm married to Cisco (I know, I'm sick) primarily because I have a PTP VPN between my home and the office and it really is a pain to get a Cisco firewall (at the office) to talk to a non-Cisco firewall elsewhere (like my home). Trust me, I fought with many sonicwalls a number of years ago. Ugh.

So with that said, I researched other devices, and Verizon does have a solution for this. They call it their M2M or machine-to-machine router service. It is compatible with a bunch of very popular router manufacturers, including Cisco, but the issue is that you can't get just one of these devices. A business account (which we have) needs to get 5 to start. So that solution is not a solution at all for the guy who works at home and only needs one device. Plus, the bandwidth per GB seems to be a little more expensive too. Makes sense, that is their cash cow at the moment.

So, there you have it. Other than all of this, when I do use the Verizon 4G (which is all the time, I have my VoIP phone plugged into it permanently) it is great. The service is excellent, stable and reliable. I've had one bad call in a year, which is beyond impeccable.

C.J.

Where is Arbor Networks?

Further to my post on May 16th, it seems Arbor Networks is joining the team. The "we don't need business" team, that is.  On April 30th I watched a webinar hosted by the Whir entitled "The Cloud is Under Seige; How Can I Protect it from DDoS Attacks". Ben Fischer, Senior Product Marketing Manager for Arbor Networks presented a lot of interesting statistics about DDoS attacks, and then some information about Arbor's products and how they can help detect and possibly mitigate these attacks.

First of all, Mr. Fischer is in the wrong career. While he knew the product, his "marketing" abilities were extremely lacking. Either he was having a very bad day due to allergies (and I have no idea if he suffers from them), or he just hates life. But his presentation was very cold and somewhat monotone. It sounded as if he simply did not want to be there and that there was no incentive for him to promote their product. He was simply going about "doing his job", and it was obvious he hates doing webinars.

All complaints about Mr. Fischer's presentation aside, I emailed him after the presentation asking for more information about their Peakflow Threat Management System because it appears to be a pretty slick solution that could benefit our organization quite nicely. We have a lot of data centers and DDoS is definitely an issue for our SaaS application. If management bought into the product, it has the potential to be a million dollar contract.

Well, needless to say, as I alluded to in my title, Arbor Networks joins the "we don't need business" team. Come Wednesday, 4 weeks time will have elapsed, and I haven't heard anything from them. This seems to be a growing trend. And yes, I did check my spam folder just in case, but I provided my phone number to Mr. Fischer too, FYI.

C.J.

4 Essential UPS Secrets

Ironically only a few days after I posted my displeasure with the fact that Internap's sales organization is lacking (to put it gently), I found this post at Data Center Knowledge about an outage in their New York location at 111 8th Avenue. This suggests a reason behind the fact that they have yet to return my calls. They are failing. Any organization who does not return sales calls and who cuts corners has me concerned, so quite frankly I'm pleased that I did not connect up with them. I hope I'm wrong, however.

Over the years I've built several data centers and have designed critical power infrastructure. Based on my experience (which should be no different than Internap's) here are the key things everyone should keep in mind about UPS systems:

1. Power is the 2nd most volatile component in the data center, second to the network. Always have redundant power to your racks (that's 2N or better!) Yes Livestream, StackExchange, I'm talking to you. And if Internap said they provided 2N power to your rack(s) at 111 8th, they obviously did not.
   
2. UPS systems must have dual (parallel) battery strings. A single string is a problem waiting to happen, irregardless of how aggressive your maintenance program is.
   
3. Maintain your UPS systems well. Have them inspected at least twice yearly, which must include a complete battery check. Yes, that means disconnecting the batteries, putting the UPS into bypass mode while that electrical Bus is on Generator power and checking every one of them. Batteries are your weakest link.
   
4. Replace the batteries after 4 years. In the datacenters I've designed, we would install 10 year batteries, but replace them after 4 years. Yes, it is expensive, but that is the point where they begin to fail. You'll find a bad one during each inspection at that point. Don't risk it!

If you're not following these UPS guidelines, you're asking for trouble. When datacenters start to cut corners due to budget, critical power systems are not the place to do it. I sure hope budget is not the reason Internap had an issue.

Cheers,

CJ

Rackspace, SoftLayer, Internap, Peer1, Dyn - Where are you?

I'm sensing a trend, and it doesn't make any sense to me. Over the last 8 months I've reached out to Rackspace, SoftLayer, Internap, Peer1, Dyn (and a few others) and the sales process is non-existent. That's right. I fill out a form on their website, email or call and it appears to land in a black hole. Two of the aforementioned organizations eventually reached out to me after several weeks, and after providing details of the services we required, went dark. One of them added my email to a monthly email list (so I know they got it!) but nobody from their sales team reached out to me. The last two are current vendors of ours, so I reached out to our very own account executive to inquire about expanding, but apparently neither of them need any more business either because I have to follow-up with both of them to move the process along.

What gives? This is astonishing! In our organization we pride ourselves on getting back to people within minutes, and it really impresses them. But it shouldn't, really. I would think that any organization, especially those as large as the ones listed here, have a perfected sales process by now, which includes returning phone calls or emails in a timely manner in order to impress the customer, but apparently not. Perhaps they had the process perfected at one point in history which helped catapult them to the size they are now, but lost it along the way, I'm not certain.

Could this be an American problem? Are we too puffed up with our own success and pride to return phone calls? Have we stopped caring for our customers? For fun I filled out the web form for Tata Communications too, because we'd happily consider this Indian headquartered global transit provider for bandwidth too, but it's been about 3 weeks and they haven't emailed or called me either.

What's going on? Does anyone have any ideas?

CJ

Welcome

Welcome to "The Right Data Center", a personal blog where you may find pithy remarks and insight from someone who has been in the IT industry for almost two decades. My experience spans many facets of IT from my early days designing websites to managing hosting facilities to designing data centers. I've got a lot of experience (good and bad) to share.

I have no agenda here. I'm just a guy who loves Information Technology, application availability and data centers. I work for a large software-as-a-service organization that you've most likely heard of, but since this is my personal blog, I will keep it my employer's name off this blog. I will, however, share my experiences, comments, concerns and feelings of injustice right here for all to peruse because I personally enjoy writing at the end of the day when the kids have gone to bed and my wife is watching silly shows on TV that I can't bear to watch with her... you know the ones I'm talking about, right? Desperate Housewives, Say Yes to the Dress, Naked and Afraid, Here Comes Honey Boo Boo... reality TV crap that drives me crazy. Who in their right mind wants to have a window into an idiot's life?

So what will I write about? Well, it could be anything... but most likely related to the day-to-day events that transpire in my regular job. I guess you'll just have to revisit periodically to find out.

Cheers,

CJ