Sometime over the weekend my lab vCenter Server Appliance stopped working. when attempting to login you see the infamous 503 error

"503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f0ef806c180] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Issue 1: Disk full
Log in to the vCenter Server Appliance through SSH and enable bash shell with the following command

shell.set --enabled true
shell

now you are in shell run df and found the disk was full on couple of the /var/log partitions. after a few du -sh * command I identified the issue to be 3.6GB audit.log file under /var/log/audit
after deleting that and a few other large log files, thought I resolved the issue.

Issue 2: root user password
On reboot the system was still failing, checked the audit.log file again and realized there was some auth issue, based on this VMware KB looks like I need to change the password for user root.

chage -l root

once this is done, reboot again.

Issue 3: vmware-vpxd service still fails, dependent service vmware-invsvc fails becasue dependent server vmware-eam fails.

Stderr =
2018-06-05T22:31:09.816Z   {
    "resolution": null,
    "detail": [
        {
            "args": [
                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
ERROR:root:Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

On more mucking around and looking into /var/log/vmware/eam/eam.log it complained about empty fields so when i looked at /etc/vmware-eam/eam.properties it was empty.
thanks to some kind soul on vmware forums I found a template file and punched in my details
unfortunately there was a error in line 34 should not start with eameam just eam.

the four lines you need to change are

eameam.resourcebundle.filename=eam-resourcebundle.jar --> eam.resourcebundle.filename=eam-resourcebundle.jar
cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5  (from cat /etc/vmware/install-defaults/sca.hostid)
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService  (from cat hostname)
vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService  (same as above)

After this the eam server was working
service-control --start vmware-eam

Issue 4: moving on to vmware-invsvc which was also stuck.
the log file had a following error

[WrapperListener_start_runner  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching adm
in group for the SSO Admin user : [email protected] and
[WrapperListener_start_runner WARN org.springframework.context.support.ClassPathXmlApplicationContext opId=] Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vlsi-server' defined in class path resource [server/config/server-config.xml]: Cannot create inner bean 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' of type [com.vmware.vim.vmomi.server.http.impl.FilterImpl] while setting bean property 'filters' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' defined in class path resource[server/config/server-config.xml]: Cannot resolve reference to bean 'authFilter' while setting bean property 'filter'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authFilter' defined in class path resource [server/config/server-config.xml]: Cannot resolve reference to bean 'authChecker' while setting bean property 'authChecker'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authChecker' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'userSessionManager' while setting bean property 'userSessionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSessionManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authorizationManager' while setting bean property 'authorizationManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizationManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authProvider' while setting bean property 'dataProvider'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authProvider' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'memCache' while setting bean property 'parentChainCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.Invalid
CredentialsLdapException: Invalid credentials LDAP error [code: 49] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:287)

I chased after VMware KB and reset the password with /usr/lib/vmware-vmdir/bin/vdcadmintool and choosing option 3 and then setting that new password in the registry

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit

Final reboot, and login page and logged in, its all working!!, only took 4+ hours. Time to upgrade to 6.5 or 6.7 🙂


5 responses to “vCenter Server (6.0 u3 vcsa) failed to start”

  1. Florian Baumann Avatar
    Florian Baumann

    Great explanation! Please check if there are whitespaces in /var/log/vmware/eam/eam.log. After copy&paste it may happen that two whitespaces are needed to every line. Remove them!

    1. Jay Avatar

      thanks for comment, I don’t remember now if I had to remove whitespaces, but I did cleanup that file. Hopefully your comment will help the next soul who runs into it.

  2. tachytelic Avatar
    tachytelic

    You absolute beauty, such a well written guide and got me out of a mess!

  3. davis wei Avatar
    davis wei

    Issue 4:注意当你重置密码时,新密码是随机生成的,里面包含数个特殊字符,这些在lwregshell中写入注册表时可能被转义,执行set_value命令之后应该用dir命令确认一下。

    Issue 4: Note that when you reset the password, the new pswd is randomly generated and contains several special characters. These may be escaped when you written to the registry in lwregshell. You should use the dir command to confirm after executing the set_value command.

  4. Juan Avatar
    Juan

    You save my life wtih such nice explanation.

    To recreate eam.properties file, there is a template in https://kb.vmware.com/s/article/2148582

Leave a Reply to tachytelicCancel reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.