Month: December 2015

Oh God. What did I just break?

Okay, so I’ve been playing around with different tools for network monitoring. It’s a simple setup; I would like to monitor up/down for two core switches, a handful of routers (of which all but one are in remote offices), as well at port stats at WAN links as well as on the server ports. We are a mixed, but heavily HP, shop so flexibility is critical. To the internets! Actually to the NOC, but they…uh…feed and groom the internets. My buddies at the NOC I interned with toss a few ideas my way and I start to play around. A few tools later I’m losing my will to live. The web platforms are glitchy, good tools cost more than higher-ups care to dish out, and I don’t know anything about MIBs…especially on HP gear.

Hey…but ***** just rolled out a new tool. That little networky widgety SNMP  monitoring thing. Hey, let’s try that!

Activate component, build profile, plug in SNMP info and viola!

…It’s not doing anything….Maybe that’s a good sign…let’s just watch it for a few days and see if it generates any hits.

*Later*

Yeah, need to throw big packets around the network to find the weak link….I’ll just ping from a core switch.

putty
##.###.##.###
username
pass
core>ping blah blah
menu

Wait, whats wrong with my menu?

command line

looks like enable mode….does HP have enable mode? I always log straight into exec

core>enable
:invalid entry

Uh, what?

core>en

:invalid entry

core>menu
****telnet – operator mode****

okay…log back out and back in with admin and…..same thing

CRAP

What did I change, what did I do? Am I fired and it dead?

Breathe. Packets are still switching, so it’s not dead. Try the GUI.

Won’t accept my login? This is just weird. BOSS! Hey, this thing is, and when I do this, then this happened, and now I cant exec or config, and I Swear I didn’t Touch Any Admin! All I did was set up read only SNMPv3 and link it to *****.

Good idea, we’ll unplug it and plug it back in. And that fixed it.

Moral: I had permission, the vendor of the monitoring has a great rep with us, but this was a brand new tool. Though I specified the tool was only allowed read only, it still used v3 to do some nasty stuff to the config.

Vet it, get permission, vet it more, test it, dev it, production. Thou shalt not defy this my greatest command.

Almost needed to change myself today…this could have been way worse. What if the tool had VLAN controls and messed up VLAN entries? What if it defaulted to routing all non-192.168.xx.xx to 0.0.0.0? What if I was PuTTyd in right now trying to fix something catastrophic?

That would suck

Advertisements

mtr

How could three letters be so profound? How could a command become more beloved than ping? Who, dear friends, who could displace tracert?

mtr

Thats who.

mtr is a command line tool designed and implemented on Linux systems. I’m not going to tell you how to install it, but I put a link at bottom just for that.

As for how I use it, my department serves the IT needs of 11 companies in 7 locations pocked across the Southeast and Midwest US. We don’t have a lot of budget to throw at fancy tools and I need a way to get real time data about the quality of the  pipe between the net-core here in NW Georgia and the satellite sites. I don’t need an alerting system for drops and outages, I have a swarm of employees each with IT on their cell speed-dial should the connection fail. What I do need is a tool I can use to monitor each site’s connection to home, without using a lot of resource. Solution, I opened 7 mtr terminals on my Ubuntu box and mtr’d each site. I now can see the RTT and jitter at each hop to each location. More specifically, my Order of Fields is “LDSNBAWJMX” so I see loss, number of packets dropped, jitter, and current, worst, and best of both. Best of all, mtr defaults as an extended ping-route. It won’t alert me should a problem arise, but once again, the swarm will tell me about every hiccup on the network. We are very VoIP intensive, so this tool is invaluable. To take it one step further, I use SolarWinds’ bandwidth monitor whenever I notice an anomaly to make sure we aren’t capping out and causing high latency or jitter.

My Linux buddies are over here like “Duh,” but my fellow Microsoft warrior, walk with me for a minute.

Imaging a world in which a lowly network engineer could gather RTT data from each individual hop device on traceroute. Now expand your mind as you assess jitter…at the same time! Now, I know what you are thinking, this is a plot to get you to install Linux and come over to the dark side…No Friend! I say you can have you solution, even on Windows! The link below will direct you to the installation of WinMTR, a program with the same functionality, but uses a GUI and is…For Windows!

That is all, here is the link.

https://www.linode.com/docs/networking/diagnostics/diagnosing-network-issues-with-mtr

 

 

*Update 1/8/2016

So apparently MTR is a stock tool in many Linux distros now…and has been for a while. Don’t shoot me! I didn’t know it came in your Ubuntu. While I may be all knowing, I’m still all learning.