vCenter Server (6.0 u3 vcsa) failed to start

Sometime over the weekend my lab vCenter Server Appliance stopped working. when attempting to login you see the infamous 503 error

"503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f0ef806c180] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Issue 1: Disk full
Log in to the vCenter Server Appliance through SSH and enable bash shell with the following command

shell.set --enabled true
shell

now you are in shell run df and found the disk was full on couple of the /var/log partitions. after a few du -sh * command I identified the issue to be 3.6GB audit.log file under /var/log/audit
after deleting that and a few other large log files, thought I resolved the issue.

Issue 2: root user password
On reboot the system was still failing, checked the audit.log file again and realized there was some auth issue, based on this VMware KB looks like I need to change the password for user root.

chage -l root

once this is done, reboot again.

Issue 3: vmware-vpxd service still fails, dependent service vmware-invsvc fails becasue dependent server vmware-eam fails.

Stderr =
2018-06-05T22:31:09.816Z   {
    "resolution": null,
    "detail": [
        {
            "args": [
                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
ERROR:root:Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

On more mucking around and looking into /var/log/vmware/eam/eam.log it complained about empty fields so when i looked at /etc/vmware-eam/eam.properties it was empty.
thanks to some kind soul on vmware forums I found a template file and punched in my details
unfortunately there was a error in line 34 should not start with eameam just eam.

the four lines you need to change are

eameam.resourcebundle.filename=eam-resourcebundle.jar --> eam.resourcebundle.filename=eam-resourcebundle.jar
cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5  (from cat /etc/vmware/install-defaults/sca.hostid)
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService  (from cat hostname)
vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService  (same as above)

After this the eam server was working
service-control --start vmware-eam

Issue 4: moving on to vmware-invsvc which was also stuck.
the log file had a following error

[WrapperListener_start_runner  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching adm
in group for the SSO Admin user : [email protected] and
[WrapperListener_start_runner WARN org.springframework.context.support.ClassPathXmlApplicationContext opId=] Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vlsi-server' defined in class path resource [server/config/server-config.xml]: Cannot create inner bean 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' of type [com.vmware.vim.vmomi.server.http.impl.FilterImpl] while setting bean property 'filters' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' defined in class path resource[server/config/server-config.xml]: Cannot resolve reference to bean 'authFilter' while setting bean property 'filter'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authFilter' defined in class path resource [server/config/server-config.xml]: Cannot resolve reference to bean 'authChecker' while setting bean property 'authChecker'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authChecker' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'userSessionManager' while setting bean property 'userSessionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSessionManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authorizationManager' while setting bean property 'authorizationManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizationManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authProvider' while setting bean property 'dataProvider'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authProvider' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'memCache' while setting bean property 'parentChainCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.Invalid
CredentialsLdapException: Invalid credentials LDAP error [code: 49] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:287)

I chased after VMware KB and reset the password with /usr/lib/vmware-vmdir/bin/vdcadmintool and choosing option 3 and then setting that new password in the registry

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit

Final reboot, and login page and logged in, its all working!!, only took 4+ hours. Time to upgrade to 6.5 or 6.7 🙂

2 comments

  1. Great explanation! Please check if there are whitespaces in /var/log/vmware/eam/eam.log. After copy&paste it may happen that two whitespaces are needed to every line. Remove them!

    1. thanks for comment, I don’t remember now if I had to remove whitespaces, but I did cleanup that file. Hopefully your comment will help the next soul who runs into it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.