Oh God. What did I just break?

Okay, so I’ve been playing around with different tools for network monitoring. It’s a simple setup; I would like to monitor up/down for two core switches, a handful of routers (of which all but one are in remote offices), as well at port stats at WAN links as well as on the server ports. We are a mixed, but heavily HP, shop so flexibility is critical. To the internets! Actually to the NOC, but they…uh…feed and groom the internets. My buddies at the NOC I interned with toss a few ideas my way and I start to play around. A few tools later I’m losing my will to live. The web platforms are glitchy, good tools cost more than higher-ups care to dish out, and I don’t know anything about MIBs…especially on HP gear.

Hey…but ***** just rolled out a new tool. That little networky widgety SNMP  monitoring thing. Hey, let’s try that!

Activate component, build profile, plug in SNMP info and viola!

…It’s not doing anything….Maybe that’s a good sign…let’s just watch it for a few days and see if it generates any hits.

*Later*

Yeah, need to throw big packets around the network to find the weak link….I’ll just ping from a core switch.

putty
##.###.##.###
username
pass
core>ping blah blah
menu

Wait, whats wrong with my menu?

command line

looks like enable mode….does HP have enable mode? I always log straight into exec

core>enable
:invalid entry

Uh, what?

core>en

:invalid entry

core>menu
****telnet – operator mode****

okay…log back out and back in with admin and…..same thing

CRAP

What did I change, what did I do? Am I fired and it dead?

Breathe. Packets are still switching, so it’s not dead. Try the GUI.

Won’t accept my login? This is just weird. BOSS! Hey, this thing is, and when I do this, then this happened, and now I cant exec or config, and I Swear I didn’t Touch Any Admin! All I did was set up read only SNMPv3 and link it to *****.

Good idea, we’ll unplug it and plug it back in. And that fixed it.

Moral: I had permission, the vendor of the monitoring has a great rep with us, but this was a brand new tool. Though I specified the tool was only allowed read only, it still used v3 to do some nasty stuff to the config.

Vet it, get permission, vet it more, test it, dev it, production. Thou shalt not defy this my greatest command.

Almost needed to change myself today…this could have been way worse. What if the tool had VLAN controls and messed up VLAN entries? What if it defaulted to routing all non-192.168.xx.xx to 0.0.0.0? What if I was PuTTyd in right now trying to fix something catastrophic?

That would suck

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s