Announcement

Collapse
No announcement yet.

We Were Power-less

Collapse
This topic is closed.
X
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • We Were Power-less

    Ugh, this day just didn't get off to a great start...

    So it was my "on" day with Rear Door LP at the wholesale club. Every other Saturday, I come in at 5am, scan in some vendors' stock at the DSD door, break down a pallet (or two) of some mixed-stock, checking which ones need EAS tags attached, which ones go in the lock-up cage, etc.

    As per usual, I didn't get quite as much sleep as I needed. I've learned to get around this by drinking some Monster energy drink, and periodically munching on a piece of jerky for some protein. So I managed to stumble through the morning. I did briefly screw up as our first vendors arrived. One of the major deliveries is a truckload of 'Lito-Fray' brand chips, which can take a while to scan in, depending on how much there is.

    As it happened, one of our bread vendors arrived as the Lito-Fray guys were unloading their truck, and asked if I could scan him in quickly before I started on them. I said sure, and Bread Guy goes to unload his truck. Unfortunately, because I was only semi-functional, I went on auto-pilot as the first part of the Lito-Fray order was brought up to scan, and started doing so.

    The way the DSD system is set up at the store, once I start on a ticket, I can't back out. (It's possible I could, but I don't know the procedure for it, and knew enough to not try to fudge around with it and possibly screw up the computer.) When Bread Guy came in to find I'd already started on the L-F order, he was pissed. He cursed a bit, snapped, "I don't have time for this!" and threw his load back in his truck (not literally) and drove off.

    I felt bad, but just went back to work. I knew that Bread Guy would come back, and I could apologize then. That's exactly what happened, too, as about an hour or so later, he comes back, and immediately accepts my apology and explanation (my being tired) and we go about our routine without mentioning it again.

    The real suck of the day would come later.

    Nine-a.m., we open up the club. I'm up at Front Door LP now. The day proceeds more or less normally. Then, about half an hour after we open, I notice the lights flicker slightly. I wasn't sure if it was real, or I was seeing things (not enough sleep, remember), and then I hear the FLS on the radio: "(MOD), we just lost power."

    Uh oh.

    There's another flicker, and then the air-curtain at the front door (used for keeping birds and flying bugs out of the store) shuts off. So do about half of our demo TVs in electronics, and the TV hooked up to the demo Blu-Ray player loses picture. (Sound still works fine.) But many parts of the store have lost power:

    * Food court. No power to the soda fountain, pizza oven, hot dog cooker, popcorn machine, warming lights, register, nothing.
    * Gas station. No power to pump the gas, or ring up the transaction.
    * Tire bay. No lights, no power to their tools or propane pump.
    * Refrigeration. Our dairy cooler and freezer are no longer producing more cold. Management quickly acts and uses rope to tie the doors shut (to keep what cold air was still in the freezers and coolers in there), and prints up some signs (the GM office computer and printer are still working) to hang in that area to explain the situation. Because of this, we didn't lose any of our perishable stock. (Except seafood, which was quickly pulled.)
    * Air conditioning. This was the worst for us employees, since the club gets pretty warm during the day in the summertime.

    Now, I was fortunate, in that the outlet by the front door into which a portable fan is plugged was still working, so HA! I had a fan! That was better than nothing. I shut off the Blu-Ray demo player, since audio with no picture isn't much fun.

    I had to field a lot of questions from customers ("How come ya'll's gas station is closed?" "We lost power out there. Pumps aren't working."), but thankfully every single one of them accepted that answer. I offered no timetable on a resolution to this ("We'll have power back when we get power back") nor a definitive answer on the cause of the power failure.

    We didn't have an answer at that time, either. We knew that it was only affecting us. Our neighbors (Wally World and Tarjhey) had power. That ruled out a brownout. I guessed some kind of power surge, while the GM suspected a blown transformer.

    Right around 11am, suddenly the rest of the lights in the club go out. What illumination there is is provided by the emergency lights and the sun from the skylights. A quick radio check confirms that the registers are still up, albeit on back-up power. As a result, I start informing everyone that enters the club "We've just lost power. Our registers are working for the moment, but we can't guarantee they'll stay that way." Most customers chose to continue shopping anyway, and I hadn't been given the word to turn people away.

    Fortunately, that's when my coworker Little Miss Panicky took over for FDLP duty so I could take my long-awaited break. Despite my moniker for her, she handled the situation well, while I went next door to get food from the Way of Subs, buy some bottled water, and (after eating my lunch) use their bathroom.

    By the time I went back to the wholesale club around 12pm noon, the power had been restored! ...to the club, at least. Gas station was still not up and running. The rest of the shift went without incident (the occasional "why's ya'll's gas station not open?" notwithstanding) until Hoss came in to relieve me.

    Ugh. A pox on the deity of technological gremlins!
    PWNADE(TM) - Serve up a glass today! | PWNZER - An act of pwnage so awesome, it's like the victim got hit by a tank.

    There are only Four Horsemen of the Apocalypse because I choose to walk!

  • #2
    Sounds more like you lost a phase.

    Most large commercial buildings run on 3 phase 480 volt power - it's made up of three 120 volt legs, each a bit out of sync with the other (AC = alternating current). Tie them together properly, you get 480 volts. It's broken down to 480, 277, 208, and 120 at the breaker panels.

    Lose a phase, all of your 3 phase equipment craps out (like your refrigeration and climate control), and any single phase stuff on that phase dies too (gas island, some lights, completely random stuff). Sounds like the power company dropped a phase, and eventually had to drop the other 2 to fix whatever caused the first phase to fail (probably transformer since they dropped all 3 eventually). You can have 2 outlets side by side and they wind up being on different phases, maybe wired to completely different panels, depending how much crack the electricians were smoking that day - hence why your demo TV went off, but the player and audio system stayed on.

    Had this happen at my store a few years ago, except it was equipment failure on our side. My store has its own step down transformers, and a large surge caused the main breaker for 1 phase to explode (it disconnected under full load, which is 1000 amps... helllloooo huge arc flash). Knocked out the server for the registers, all of the refrigeration on that side of the store (we have 2 power feeds to the store), random lights, our phones, baler, one of the compactors, dish washer, etc etc etc. Happened overnight, and wasn't discovered until 8am by the opening manager. Seafood, meat, and cheese lost everything refrigerated and most of their frozen stuff too. Grocery lost most of their dairy stock. Deli and bakery's refrigeration was powered by the second power feed. Needless to say, we didn't open until we had everything back up, which didn't happen until about 1pm.
    Last edited by bean; 07-19-2010, 08:40 PM.

    Comment


    • #3
      Had this happen at a machine shop where I used to work. Some of my co-workers were complaining that of the milling machines that would run, they ran slowly (duh - 3-phase motor on only 2 phases). Naturally I told them that they should shut off the machine to avoid damaging the motor. As for why some wouldn't run at all, milling machines have magnetic starters - and it was a random matter which phase the starter used (if it was on the dead phase, it wouldn't let you turn the machine on).

      Found a poor design decision that way - the main and emergency lights were on different phases (with the emergency lights being on the phase that died). What this meant was that even though the lights were on, the emergency lighting came on, and by the time the power company shut down the whole installation where the trouble was, the batteries in the emergency lights had run down.
      Any fool can piss on the floor. It takes a talented SC to shit on the ceiling.

      Comment


      • #4
        Quoth bean View Post
        Sounds more like you lost a phase.
        Had that happen once overnight. SM got the call we lot power, as the computer reported it, came in to find that we had lost 2 of the 3 phases. The generator was running powering the emergency panels 120/208V (lights, phones, servers, alarms, and some outlets). By the time SM arrived and figured out what happened, some of the HVAC and Refrigeration equipment, tried, and tried to run, but only burnt the motors out. Cue very expensive 2 AM service call.

        Comment


        • #5
          Quoth mattm04 View Post
          Had that happen once overnight. SM got the call we lot power, as the computer reported it, came in to find that we had lost 2 of the 3 phases. The generator was running powering the emergency panels 120/208V (lights, phones, servers, alarms, and some outlets). By the time SM arrived and figured out what happened, some of the HVAC and Refrigeration equipment, tried, and tried to run, but only burnt the motors out. Cue very expensive 2 AM service call.
          Oh god I would not want to be the one footing that bill.

          Thankfully, all of our expensive stuff has phase loss monitors built in. Drop a phase, and our excruciatingly loud refrigeration equipment room goes completely silent, all the lights and displays on the equipment go out... sans for a "PHASE LOSS" light coming on on the control panel. You normally need hearing protection just to walk in that room. So when we last lost a phase at work, the only damage we had was to the breaker panel that decided that an arc flash and exploding 1000 amp breaker would make our day. We actually have a company that remotely monitors our compressor rooms - we get a handful of calls a day about temps being too high/low and occasionally about "Dirty oil filter on compressor <random #>" (even our head of maintenance has no idea what compressors actually power what coolers anymore either... he's been with the store since it opened in 1994), but it doesn't help when the phase that drops... also powers the mid 90s 14.4k modem that reports to them. There's also a panel at the customer service desk that shows the same alarms, but nobody knows what the hell each compressor does anymore, so it's largely ignored except to hit "RESET" when it starts making noise. We treat our fire alarm system the same way, every now and then it starts squawking (at the panel) about a wiring fault for a day or two, the alarm company will constantly call us telling us "hey you have a wiring fault, we're putting you on 'test' until your panel stops bitching", and we just hit silence on it. *sigh*

          Our generator never kicked in with the phase loss - even if it did, it can only power 1 phase and there's only a handful of stuff connected to the backup power panel (a handful of lights, the electric doors, exit signs, and oddly enough, the PC in the cash office along with the scales in every dept... but not the registers, they're on a UPS that's good for about 15 minutes). I don't know what capacity the thing has, all I know is it's a Kohler natural gas powered unit and it's on our roof. Can't be too big since I'd guess that we put about 40-50 amps on it (@ 120 volts), though there's probably 10-15 or so 20 amp breakers in the backup power breaker panel.
          Last edited by bean; 07-21-2010, 07:00 AM.

          Comment


          • #6
            Quoth bean View Post
            Oh god I would not want to be the one footing that bill.

            Thankfully, all of our expensive stuff has phase loss monitors built in. Drop a phase, and our excruciatingly loud refrigeration equipment room goes completely silent, all the lights and displays on the equipment go out... sans for a "PHASE LOSS" light coming on on the control panel. You normally need hearing protection just to walk in that room. So when we last lost a phase at work, the only damage we had was to the breaker panel that decided that an arc flash and exploding 1000 amp breaker would make our day. We actually have a company that remotely monitors our compressor rooms - we get a handful of calls a day about temps being too high/low and occasionally about "Dirty oil filter on compressor <random #>" (even our head of maintenance has no idea what compressors actually power what coolers anymore either... he's been with the store since it opened in 1994), but it doesn't help when the phase that drops... also powers the mid 90s 14.4k modem that reports to them. There's also a panel at the customer service desk that shows the same alarms, but nobody knows what the hell each compressor does anymore, so it's largely ignored except to hit "RESET" when it starts making noise. We treat our fire alarm system the same way, every now and then it starts squawking (at the panel) about a wiring fault for a day or two, the alarm company will constantly call us telling us "hey you have a wiring fault, we're putting you on 'test' until your panel stops bitching", and we just hit silence on it. *sigh*

            Our generator never kicked in with the phase loss - even if it did, it can only power 1 phase and there's only a handful of stuff connected to the backup power panel (a handful of lights, the electric doors, exit signs, and oddly enough, the PC in the cash office along with the scales in every dept... but not the registers, they're on a UPS that's good for about 15 minutes). I don't know what capacity the thing has, all I know is it's a Kohler natural gas powered unit and it's on our roof. Can't be too big since I'd guess that we put about 40-50 amps on it (@ 120 volts), though there's probably 10-15 or so 20 amp breakers in the backup power breaker panel.
            Well, after that little incident, and panicked calls to get dry ice they did put a phase loss monitor in. I do know the initial tech arrived around 2:30 AM, by 6 AM were from 3-5 truck form the Refrigeration company and 2 from the electrician from 5-6 AM until almost noon. The last ref. tech left around 2 PM.

            I believe we have a 60 or 80 KW natural gas generator @120/208 (If I remember correctly our UPS for the servers requires 208V). The registers have 2 power feeds for each lane. One is for the PC, monitor, printer and scanner which is on the generator the other is for the belts and lights which is not. For some reason the PC is not on a UPS so if the power dumps we loos the transaction and have to start again, unless they fixed that in the last few years since I stopped begin a cashier.

            Some of our stores have a monster 150KW or larger generator to power the essentials and the walk in coolers and freezers and some of the cases, though most store only have a 50-80 KW, depending on the total SF.

            After many years of making changes to the system not not documenting it, they finally sent a few techs out to figure out what whet where, etc after a walking failed and they spent 2 hours trying to figure out just what where so they could begin to fix it.
            Last edited by mattm04; 07-22-2010, 04:23 AM.

            Comment


            • #7
              Quoth mattm04 View Post
              After many years of making changes to the system not not documenting it, they finally sent a few techs out to figure out what whet where, etc after a walking failed and they spent 2 hours trying to figure out just what where so they could begin to fix it.
              Hah yeah, I finally opened up the breaker panel that the generator powers. Along with our "main lighting panel".

              Both are a huge mess. Neither have been updated during remodels, I'm pretty sure a good chunk of those breakers feed absolutely nothing at this point. Though it's very amusing to kill 75% of the lighting with 2 throws of remote breakers at the service desk (which actually turns off about 90% of the lighting panel).

              Also, 2 massive 40+ breaker panels dedicated to only lighting. Much of it mislabeled due to remodels over the years.

              Comment


              • #8
                Quoth bean View Post
                Hah yeah, I finally opened up the breaker panel that the generator powers. Along with our "main lighting panel".

                Both are a huge mess. Neither have been updated during remodels, I'm pretty sure a good chunk of those breakers feed absolutely nothing at this point. Though it's very amusing to kill 75% of the lighting with 2 throws of remote breakers at the service desk (which actually turns off about 90% of the lighting panel).

                Also, 2 massive 40+ breaker panels dedicated to only lighting. Much of it mislabeled due to remodels over the years.
                Wen we had a light ballast start smoking a yer or two ago, the fun part was finding what breaker to shut off the bad lights was. The non-emergency panles were either labeled "SLS FLR GEN LTS" for the general lighting, "SLS FLR DECSGN LTS" for any decorative or signage lights and "SLS FLR REFFRZDRY CASE LTS" for any case lights. That all we had to go on. We ended up killing all the sales floor lights and turning them on one by one until the damages lights flicked on, then we shut that one off.

                Though the emergency panels are labeled, "SALES FLR EM LTS ROW 1", BKRM HALL EM LTS< RECEIVING EM LTS, DAIRY/FRZN COOLER EM LTS, etc. Go figure.
                Last edited by mattm04; 08-01-2010, 05:57 AM.

                Comment


                • #9
                  Quoth mattm04 View Post
                  Had that happen once overnight. SM got the call we lot power, as the computer reported it, came in to find that we had lost 2 of the 3 phases. The generator was running powering the emergency panels 120/208V (lights, phones, servers, alarms, and some outlets). By the time SM arrived and figured out what happened, some of the HVAC and Refrigeration equipment, tried, and tried to run, but only burnt the motors out. Cue very expensive 2 AM service call.
                  Well, the lower the voltage, the higher the current drawn, per Ohm's Law. 230 (or 208) volt equipment really doesn't like being forced to run on 120, as the current that it draws will go way above safe levels for that equipment. Might not be enough to trip a breaker, but certainly enough to damage the components.

                  I had something similar happen in my old apartment years back that led to the loss of an air conditioner.

                  The A/C is supposed to run on 230 volts, but New York and environs don't get 230 volt lines, they just get 117 volts on three different phases. Each phase to ground is 117, phase to phase is 208, which is what actually comes out of the socket.

                  So one night, one phase ground-faults... 230 volt compressors aren't particularly happy on 208, but they won't burn out until it drops below 197 (per the data sticker). Of course 117 is Right Out, as they say. I came home from work that morning to find the fan running at half speed and the compressor smoked. If that phase had just opened, there wouldn't have been a complete circuit and the unit would have just stopped, but it had to go to ground, which ruined it.

                  Comment


                  • #10
                    One reason three-phase equipment doesn't like running on two or single phases is because they may depend on a rotating magnetic field to run the motors. In that case the dropped phase is not just a reduction in voltage or an increase in current, but there is no longer a rotating field to keep the motors running.

                    So the motors wil stop and will behave as though they are permanently stalled, which is what causes the excessive current and the burn-out.

                    Comment

                    Working...
                    X