Sometime over the weekend my lab vCenter Server Appliance stopped working. when attempting to login you see the infamous 503 error

"503 Service Unavailable (Failed to connect to endpoint: [N7Vmacore4Http20NamedPipeServiceSpecE:0x7f0ef806c180] _serverNamespace = / _isRedirect = false _pipeName =/var/run/vmware/vpxd-webserver-pipe)

Issue 1: Disk full
Log in to the vCenter Server Appliance through SSH and enable bash shell with the following command

shell.set --enabled true
shell

now you are in shell run df and found the disk was full on couple of the /var/log partitions. after a few du -sh * command I identified the issue to be 3.6GB audit.log file under /var/log/audit
after deleting that and a few other large log files, thought I resolved the issue.

Issue 2: root user password
On reboot the system was still failing, checked the audit.log file again and realized there was some auth issue, based on this VMware KB looks like I need to change the password for user root.

chage -l root

once this is done, reboot again.

Issue 3: vmware-vpxd service still fails, dependent service vmware-invsvc fails becasue dependent server vmware-eam fails.

Stderr =
2018-06-05T22:31:09.816Z   {
    "resolution": null,
    "detail": [
        {
            "args": [
                "Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: "
            ],
            "id": "install.ciscommon.command.errinvoke",
            "localized": "An error occurred while invoking external command : 'Command: ['/sbin/service', u'vmware-eam', 'start']\nStderr: '",
            "translatable": "An error occurred while invoking external command : '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
ERROR:root:Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}
Unable to start service vmware-eam, Exception: {
    "resolution": null,
    "detail": [
        {
            "args": [
                "vmware-eam"
            ],
            "id": "install.ciscommon.service.failstart",
            "localized": "An error occurred while starting service 'vmware-eam'",
            "translatable": "An error occurred while starting service '%(0)s'"
        }
    ],
    "componentKey": null,
    "problemId": null
}

On more mucking around and looking into /var/log/vmware/eam/eam.log it complained about empty fields so when i looked at /etc/vmware-eam/eam.properties it was empty.
thanks to some kind soul on vmware forums I found a template file and punched in my details
unfortunately there was a error in line 34 should not start with eameam just eam.

the four lines you need to change are

eameam.resourcebundle.filename=eam-resourcebundle.jar --> eam.resourcebundle.filename=eam-resourcebundle.jar
cm.url=http://localhost:18090/cm/sdk/?hostid=fa3bd34a-4b3c-4ff0-ad4f-1c97ad6f93b5  (from cat /etc/vmware/install-defaults/sca.hostid)
vc.tunnelSdkUri.template=https://##{VC_HOST_NAME}##:8089/sdk/vimService  (from cat hostname)
vc.tunnelSdkUri=https://vcenter:8089/sdk/vimService  (same as above)

After this the eam server was working
service-control --start vmware-eam

Issue 4: moving on to vmware-invsvc which was also stuck.
the log file had a following error

[WrapperListener_start_runner  ERROR com.vmware.vim.vcauthenticate.servlets.AuthenticationHelper  opId=] Hit ServiceFaultException while fetching adm
in group for the SSO Admin user : [email protected] and
[WrapperListener_start_runner WARN org.springframework.context.support.ClassPathXmlApplicationContext opId=] Exception encountered during context initialization - cancelling refresh attempt
org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'vlsi-server' defined in class path resource [server/config/server-config.xml]: Cannot create inner bean 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' of type [com.vmware.vim.vmomi.server.http.impl.FilterImpl] while setting bean property 'filters' with key [0]; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'com.vmware.vim.vmomi.server.http.impl.FilterImpl#2ad6d4be' defined in class path resource[server/config/server-config.xml]: Cannot resolve reference to bean 'authFilter' while setting bean property 'filter'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authFilter' defined in class path resource [server/config/server-config.xml]: Cannot resolve reference to bean 'authChecker' while setting bean property 'authChecker'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authChecker' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'userSessionManager' while setting bean property 'userSessionManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'userSessionManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authorizationManager' while setting bean property 'authorizationManager'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authorizationManager' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'authProvider' while setting bean property 'dataProvider'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'authProvider' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'memCache' while setting bean property 'parentChainCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'memCache' defined in class path resource [server/config/security-config.xml]: Cannot resolve reference to bean 'globalAclLotusCache' while setting bean property 'globalAclLotusCache'; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'aclLotusInitializer' defined in class path resource [server/config/authorization-config.xml]: Instantiation of bean failed; nested exception is org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.vmware.vim.query.server.accesscontrol.impl.LotusInitializer]: Constructor threw exception; nested exception is java.lang.RuntimeException: com.vmware.identity.interop.ldap.Invalid
CredentialsLdapException: Invalid credentials LDAP error [code: 49] at org.springframework.beans.factory.support.BeanDefinitionValueResolver.resolveInnerBean(BeanDefinitionValueResolver.java:287)

I chased after VMware KB and reset the password with /usr/lib/vmware-vmdir/bin/vdcadmintool and choosing option 3 and then setting that new password in the registry

/opt/likewise/bin/lwregshell
cd HKEY_THIS_MACHINE\services\vmdir\
set_value dcAccountPassword "new password"
quit

Final reboot, and login page and logged in, its all working!!, only took 4+ hours. Time to upgrade to 6.5 or 6.7 🙂