acct_mgmt failed no jobs

[zhouzhuoCSUFT]

Hello, I have compiled pam_lsf.so， but cannot ssh compute node when it has jobs ，the secure log is :

Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: argv[0] = debug
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: argv[1] =  LSF_ENVDIR=/opt/ibm/lsfsuite/lsf/conf
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: argv[2] = LSF_SERVERDIR=/opt/ibm/lsfsuite/lsf/10.1/linux2.6-glibc2.3-x86_64/etc
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: comparing cluster admin lsfadmin
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: comparing cluster admin root
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: initializing LSF
Apr 15 12:26:03 c001 sshd[15509]: pam_lsf.so: checking LSF user=test hostname=c001
Apr 15 12:26:15 c001 sshd[15509]: pam_lsf.so: acct_mgmt failed no jobs for (test)

I added a syslog code

There secure log is

Apr 15 12:19:56 c001 sshd[14499]: pam_lsf.so: comparing cluster admin lsfadmin
Apr 15 12:19:56 c001 sshd[14499]: pam_lsf.so: comparing cluster admin root
Apr 15 12:19:56 c001 sshd[14499]: pam_lsf.so: initializing LSF
Apr 15 12:19:56 c001 sshd[14499]: pam_lsf.so: checking LSF user=test hostname=c001
Apr 15 12:19:56 c001 sshd[14499]: jobs = -1
Apr 15 12:19:58 c001 sshd[14499]: jobs = -1
Apr 15 12:20:00 c001 sshd[14499]: jobs = -1
Apr 15 12:20:02 c001 sshd[14499]: jobs = -1
Apr 15 12:20:04 c001 sshd[14499]: jobs = -1
Apr 15 12:20:06 c001 sshd[14499]: jobs = -1
Apr 15 12:20:28 c001 sshd[14499]: pam_lsf.so: acct_mgmt failed no jobs for (test)
Apr 15 12:20:28 c001 sshd[14499]: fatal: Access denied for user test by PAM account configuration [preauth]

1. I suspect the lsb_openjobinfo() function didn't read jobs
2. I compared the lsbatch.h file with the other cluster and found a significant difference

Thanks for your help!

[adamsla]

What is the output of "bjobs -utest -a" when you run at the time you attempt to login?

[adamsla]

My guess is that there is an issue with the hostname as detected by pam vs. the hostname as recognized by LSF.

[zhouzhuoCSUFT]

What is the output of "bjobs -utest -a" when you run at the time you attempt to login?
output is

$ bjobs -utest -a
JOBID   USER    STAT  QUEUE      FROM_HOST   EXEC_HOST   JOB_NAME   SUBMIT_TIME
4106    test    RUN   short      mn01        48*c001     test       Apr 16 10:41

That seems to be no problem。

The implementation engineer told me that he said this version of LSF is actually 10.2 yesterday,and we have another set of clusters, LSF version is 10.1, not have this problem。
thanks!

[adamsla]

I think the problem was with an change in LSF that prevent's the root user from running job commands. This was starting with LSF 10.1.0.10. So, the pam stack will have to be enhanced to switch users to the LSF Administration account before running the batch command. If you could, as a test only, not for long term production, follow the steps in this article, and then let me know if the pam stack starts to work. If that's the case, a simple change can be made to seteuid() to the LSF admin prior to running the batch commands. Keep me posted. Here is the support link:

https://www.ibm.com/support/pages/root-user-can-not-run-lsf-commands-after-upgrading-lsf-101-sp10

TheWitness

[adamsla]

I've made a simple change to the pam module, please test this as well, using the default LSF configuration.

[TheWitness]

@georgeg000, I think you can close this one...