Let's play guess what happened?

[draggar]

Or we could call it "What a moronic outsourced tech did"?

One of my stores had an issue with their printer last week so someone from the vendor (yes, a very large company that has the same name as an old piece of artillery that uses gunpowder and large metallic balls) went out to "fix" it.

By "fixing" it I meant he decided to swap the IP address and Gateway address. No, there was no issue with those before.

For example:

The IP was supposed to be 100.100.100.101 (not accurate at all)
The Gateway address was supposed to be 100.100.100.1 (again, not accurate at all, well, except the last .1).

He decided to reverse the two.

Anyone want to guess what happened????

[Pedersen]

Ooooh! Ooooh! Pick me!

I will guess that viruses started printing out all the internet porn on that fixed up printer, right? Mixed in with occasional viagra ads and bank of america certificate issues, right?

[draggar]

Hilarious but no, we have medieocre anti-spamware & virus measures in place.

[Aethian]

Hilarious but no, we have medieocre anti-spamware & virus measures in place.

Awwwww but I like Pedersens....

My guess is....neither works at all anymore and things get lost with the print send jobs?

[Sheldonrs]

I'm going to guess that when one place sends something to print, it goes to the other IP and vice-versa. :-)

[SG15Z]

I'm going to guess that when one place sends something to print, it goes to the other IP and vice-versa. :-)

I second that one!!

[Pedersen]

I'll go ahead and post my real reply now.

By switching the two fields like that, the printer tech gave the printer the same IP address as the gateway/router. This translates into an IP address conflict. Some machines would be trying to use the printer as the router. Others, though, if they got their ARP info earlier and did not update the cache when the printer announced itself, would continue to use the correct machine as the router.

Alternate possibility is that the machines would attempt to find an address every time, ignoring their ARP cache, and whichever machine (the router or the printer) got the reply to the requesting machine first would be used for that request only, which would result in intermittent access to the router.

In either of those cases, though, the absolute best condition to hope for is sporadic extranet/WAN access. More likely, all extranet/WAN access was cut off.

Furthermore, any access by the outside world coming in would stop at the router. Leaving the connection up between the remote and other places. This, of course, results in extremely odd connection issues, since trying to examine the network from outside results in your being able to access the router, but not anything else inside (at least, not reliably).

The upshot of all this is that, very likely, somebody had to go to the store to diagnose the issue, and wasted quite a bit of time doing so.

Oh, and during this whole time, since all computers were expecting the printer to be available on one address, but it was actually on another address, nobody could use the printer either. Which, at the time, would have seemed like a separate (and, by comparison, insignificant) issue, and likely ignored until the root cause of the WAN/extranet failures was found.

How close am I?

[Bandit]

Short version of Pedersen's reply - Network go boom, and Dragger had to drag his arse to the site to try to repair things between all the "Is it fixed yet?" and "When will it be fixed?" calls.

How close am I?

B

[draggar]

Penderson got it.

As I noted in the ticket:

The printer is suffering from an identity crisis - it was thinking it was the store's router (due to the IP address and gateway address being switched). I was able to rehabilitate the printer into thinking it was a printer by switching them back into the correct order.

ARP cache was ignored so the PCs always looked for the gateway at the correct gateway address so each PC was experiencing continual issues with connectivity, including their applications.

The outages were quite random, too, and I'm sure it was causing many other issues which we didn't even know about.