[Apcupsd-users] Online 30kVA UPSs ignore killpower and related smart commands on serial!

[Soumen Chakrabarti]

I have been using apcupsd for many years with 750, 800, 1000, 2200 and
3300 VA SmartUPSs, but I have seen nothing like this so far. This is
about the online 30kVA UPS model SUVTP30KH firmware 909.0501.I. This has
a DB9 port and a SmartUPS cable provided by APC, as well as a AP9619
network management card.

After several years of trying various versions of apcupsd and different
communication protocols, wondering why a power outage never results in
the UPS pulling down the load (although all servers shut down and need
to be rebooted via IPMI), I hooked up a laptop to it via serial, ran
apctest, and went into direct serial communication with the UPS. Then I
turned off utility power, so the load was on battery.

Request-response pairs are listed below.

Request: Y
Response: SM
(UPS enters smart mode)

Request: ^A
Response: Smart-UPS VT 30 kVA

Request: A
Response: Beep from front panel and all LEDs light up momentarily

So far so good. Next, something bold.

Request: S
Response: NA
Expected response: OK or * (and UPS going to sleep until utility power
returns)

Request: @000 (delay of 2 seconds between characters)
Response: NANANA
Expected response: OK or * (and UPS turning down load after grace delay)

Request: KK (delay of 2 seconds between characters)
Response: NANA
Expected response: OK or * (and UPS turning down load after grace delay)

Now getting really desperate ---

Request: ZZ (delay of 2 seconds between characters)
Response: NANA
Expected response: UPS should power off load immediately

On a hunch, I removed the network management card. Sadly, results did
not change for the better.

So, no wonder apcupsd killpower at the end of the NIS master's shutdown
does nothing to this UPS. Meanwhile I have been chasing down false leads
one after another (SNMP access control, PCNet pass phrase, keeping the
network up late in the halt script, etc.). The lesson being, reverse
engineering by Linux people without any cooperation from the gadget
maker can only go so far. Clearly the SmartUPS serial protocol coded
into apcupsd will also not work with this UPS. We have two such 30kVA
UPSs, and both show this behavior, so it is very unlikely to be a
hardware or firmware failure.

I have been trying for six weeks to get APC to send a technician to
study the situation; the UPSs are under warranty. They'll probably come
and just say my version of Linux is unsupported and go away. So I need
to try out an approved Linux with some form of PowerChute.

Anyone has experience with these UPSs? I posted the problem here in
different stages of my understanding on this group, but there has been
zero response, which has been disappointing.

[Kris Jordan]

Post by Soumen Chakrabarti
I have been using apcupsd for many years with 750, 800, 1000, 2200 and
3300 VA SmartUPSs, but I have seen nothing like this so far. This is
about the online 30kVA UPS model SUVTP30KH firmware 909.0501.I. This has
a DB9 port and a SmartUPS cable provided by APC, as well as a AP9619
network management card.

The AP9619 should be all you need. I have this card and it works great
with Apcupsd. It has been a while since I've done complete tests, so
it's possible things have changed.

Post by Soumen Chakrabarti
So, no wonder apcupsd killpower at the end of the NIS master's shutdown
does nothing to this UPS. Meanwhile I have been chasing down false leads
one after another (SNMP access control, PCNet pass phrase, keeping the
network up late in the halt script, etc.). The lesson being, reverse
engineering by Linux people without any cooperation from the gadget
maker can only go so far. Clearly the SmartUPS serial protocol coded
into apcupsd will also not work with this UPS. We have two such 30kVA
UPSs, and both show this behavior, so it is very unlikely to be a
hardware or firmware failure.

Shouldn't need killpower. The NMC should shut down the UPS on its own.
With a NMC, I don't know if it would even listen to killpower (eg. to be
sure every system is done shutting down).

See http://nam-en.apc.com/ci/fattach/get/915516/ for a timeline. The
first time range is the maximum required delay.

[Soumen Chakrabarti]

Post by Kris Jordan
The AP9619 should be all you need. I have this card and it works great
with Apcupsd. It has been a while since I've done complete tests, so
it's possible things have changed.

Kris, many thanks for chiming in. I am pretty sure I am doing something
wrong in setting up the AP9619. I would very greatly appreciate some
guidance.

Post by Kris Jordan
Shouldn't need killpower. The NMC should shut down the UPS on its own.
With a NMC, I don't know if it would even listen to killpower (eg. to be
sure every system is done shutting down).

I need some help starting with a factory-reset UPS with a AP9619, to
implement the following:

* After 5 minutes (or 50% if sooner, very unlikely) on battery,
computer shuts down.
* UPS cuts power to computer after shutdown, and powers up computer
on utility power return.
* Once the computer begins the shutdown sequence, the UPS is
committed to cut power to the computer, even if utility power returns.

I know perfectly well how to implement the above using serial and other
APC UPSs. Currently, with a serial connection, the computer shuts down
and properly issues a killpower, but the 30kVA UPS does not kill power.
S, @000, KK and ZZ issued by hand are also ignored (response=NA).

I don't care if I use an Ethernet or serial connection to implement the
above spec. In the past I have only used the serial cable. That protocol
is clear to me, and implements the above spec without fail, 100% of the
time. Please see my earlier post to see how I failed on SNMP and PCnet:

http://sourceforge.net/mailarchive/forum.php?thread_name=4F83020F.6070906%40cse.iitb.ac.in&forum_name=apcupsd-users

With Ethernet communication, there are many potential pitfalls:

* Is eth0 and routing up long enough after shutdown starts to send
the killpower over SNMP? You say this is not needed, and the NMC
has an internal delay to cut power after a given time on battery.
How do I set that? This basically means that unlike serial, there
is no handshake, correct?
* The PCnet part of the NMC setup --- is that needed for apcupsd? I
guess not, because apcupsd is using SNMP. v1 or v3?
* Beyond the well-kept secrets write+ and private (see thread above,
last post), do I need to tell the UPS more about which computer is
allowed to shut it down, if there is any such notion (per your
email, no)?
* However, PCnet clearly has a handshake protocol. The UPS tells
PCnet clients to shut down, and verifies that before pulling down
power. The PCnet passphrase 15-char lower bound is a well kept
secret as well.

One more ominous observation: If I log on to the NMC using a browse and
navigate to UPS -> Configuration -> Shutdown, the first line on that
frame is "Shutdown Mode: Never". Also, if I navigate to Control, the
last line in the frame is "Note: UPS is configured to never shutdown. No
shutdown control commands will initiate." No wonder S, @000, KK and ZZ
are ignored with response=NA.

I just want the same behavior on the 30kVA unit as on my other 2200VA
units. I am going nuts trying to figure out why the UPS does not obey.

Many thanks.

[Kris Jordan]

Post by Soumen Chakrabarti
* Is eth0 and routing up long enough after shutdown starts to send
the killpower over SNMP? You say this is not needed, and the NMC
has an internal delay to cut power after a given time on battery.
How do I set that? This basically means that unlike serial, there
is no handshake, correct?
* The PCnet part of the NMC setup --- is that needed for apcupsd? I
guess not, because apcupsd is using SNMP. v1 or v3?

Use UPSTYPE pcnet.

Post by Soumen Chakrabarti
* Beyond the well-kept secrets write+ and private (see thread above,
last post), do I need to tell the UPS more about which computer is
allowed to shut it down, if there is any such notion (per your
email, no)?
* However, PCnet clearly has a handshake protocol. The UPS tells
PCnet clients to shut down, and verifies that before pulling down
power. The PCnet passphrase 15-char lower bound is a well kept
secret as well.
One more ominous observation: If I log on to the NMC using a browse
and navigate to UPS -> Configuration -> Shutdown, the first line on
that frame is "Shutdown Mode: Never". Also, if I navigate to Control,
the last line in the frame is "Note: UPS is configured to never
shutdown. No shutdown control commands will initiate." No wonder S,
@000, KK and ZZ are ignored with response=NA.

I don't have a shutdown mode, so that is either in a new firmware update
or a difference with your UPS hardware. Can you not change it?

On mine, PowerChute --> configuration, on-battery shutdown behavior is
set to "restart when power is restored". And I have specified the IP
addresses of each system on the clients page, IIRC only needed if there
are clients on a subnet other than the NMC, but I programmed them in
anyway to be sure.

I don't think Apcupsd says anything back to the NMC, it only listens.

Like most (all?) APC UPS's, it makes assumptions on the time needed for
the OS to shut down, this is probably why killpower is not the most
consistent feature. Not much use on Windows after all.

If you're using APC's pcnet software, it will apparently communicate
back the command-file execution time so it can make adjustments to the
UPS power off delay. I think APC's pcnet software also had a shut UPS
down option, but I have no clue on what it really does.

Looking at the PCNET section of Apcupsd's manual, it does mention
killpower, but I think that had more to do with the older NMC before the
9619. Since that is the last time I looked at APC's pcnet software, that
one shutdown option could also be gone.

Perhaps the NMC needs to get a killpower command or similar from at
least one client to activate the UPS power off behavior. Mine does shut
off during a power outage and I think I've seen it properly reboot (when
the power came back during the shutdown operation). I didn't make an
attempt to leave networking up during shutdown, so even though I left
the killpower in the halt script, it shouldn't be doing anything. I
wouldn't want the first system to shut down to cause the UPS to power
off before the last system anyway (if the 9619 even listens to the command).

Post by Soumen Chakrabarti
I just want the same behavior on the 30kVA unit as on my other 2200VA
units. I am going nuts trying to figure out why the UPS does not obey.

Mine's an old SU2200NET. The firmware version on the NMC is 3.7.1,

Post by Soumen Chakrabarti
2years old.

[Soumen Chakrabarti]

Some more info.

Using the Web login into the AP9619, and navigating to UPS ->
Configuration -> Shutdown, the first line I see on that screen is
"Shutdown Mode: Never". Also, if I navigate to Control, the last line in
the frame is "Note: UPS is configured to never shutdown. No shutdown
control commands will initiate."

Also, to prevent APC from blaming apcupsd, I installed their PowerChute
Network Shutdown (PCNS, latest version 3.0.1). While running the
configuration wizard, there was a one screen that asked me to check a
box "Turn off the UPS after shutdown finishes." --- which I obviously
checked. But after that checkbox there was this text: "This feature may
not be available depending on the UPS model or shutdown mode enabled for
the UPS."

The firmware revision is 909.0501.I. I just need a few simple answers
from APC, but they are very uninvested in supporting their product.

* Does this ton of lead allow shutdown?
* If yes, where is my configuration wrong?
* If no, is there any way to change the firmware to enable shutdown?

Some more detail:

Model Number: AP9619
Hardware Revision: A10
Manufacture Date: 12/21/2010

Application Module
Name: sumx
Version: v3.7.2
Date: 02/02/2010
Time: 19:08:16

APC OS (AOS)
Name: aos
Version: v3.7.3
Date: 02/02/2010
Time: 17:05:14

[Kris Jordan]

Do you have Force Negotiation?

http://nam-en.apc.com/app/answers/detail/a_id/10563

I should have provided that link when I gave the timeline.

AP9619 has been discontinued for a while. I had to google just to find
the support/firmware page. I had the same issue finding the support page
for the SUVTP30KH.

I don't know what to think of APC since Schneider took over.

Post by Soumen Chakrabarti
Some more info.
Using the Web login into the AP9619, and navigating to UPS ->
Configuration -> Shutdown, the first line I see on that screen is
"Shutdown Mode: Never". Also, if I navigate to Control, the last line
in the frame is "Note: UPS is configured to never shutdown. No
shutdown control commands will initiate."
Also, to prevent APC from blaming apcupsd, I installed their
PowerChute Network Shutdown (PCNS, latest version 3.0.1). While
running the configuration wizard, there was a one screen that asked me
to check a box "Turn off the UPS after shutdown finishes." --- which I
obviously checked. But after that checkbox there was this text: "This
feature may not be available depending on the UPS model or shutdown
mode enabled for the UPS."
The firmware revision is 909.0501.I. I just need a few simple answers
from APC, but they are very uninvested in supporting their product.
* Does this ton of lead allow shutdown?
* If yes, where is my configuration wrong?
* If no, is there any way to change the firmware to enable shutdown?
Model Number: AP9619
Hardware Revision: A10
Manufacture Date: 12/21/2010
Application Module
Name: sumx
Version: v3.7.2
Date: 02/02/2010
Time: 19:08:16
APC OS (AOS)
Name: aos
Version: v3.7.3
Date: 02/02/2010
Time: 17:05:14

[Soumen Chakrabarti]

Once you know how, it's a piece of cake. Thanks to unknown benefactors ;-)

The SUVTP30KH has two DB9 ports. One is for the Smart signaling cable.
The other one is called the "service port (for APC maintenance personnel
only)". Brr. I'd happily leave it to "APC maintenance personnel only" if
there existed one around here that can fog a mirror. Since there isn't
any, I was forced to intervene.

Connect using a Smart signaling cable to the serial port of a computer.
Configure 115200 8N1 and no flow control (this is important). If you are
using minicom, be careful not to try to initialize the port. You might
need to fool around with CRLF a little. Type APC and hit return. You
should see a simple ASCII menu.