Web admin interface returns an http 500 error with:
"Cannot parse model definitions file /etc/sipxpbx/commserver/user-settings.xml"
Look at recent sipX log files.
in /var/log/sipxpbx/sipxconfig.log, we see:
"2017-04-03T18:22:55.165000Z":41259:JAVA:ERR:hostname.example.com:P1-18:00000000:SearchManagerImpl:"search by user query error"
java.io.FileNotFoundException: /var/sipxdata/tmp/index/_137b.prx (Too many open files)
[...]
java.net.SocketException: Too many open files
[...]
Caused by: java.io.FileNotFoundException: /etc/sipxpbx/commserver/user-settings.xml (Too many open files)
checking how many file descriptors each process has open:
for i in /proc/*/fd ; do echo -n "$i" ; ls -1 "$i" | wc ; done
shows a very high number for one of the processes:
/proc/5054/fd 16383 16383 87190
associated with the process for sipXconfig:
500 5054 4539 11 11:56 ? 00:01:00 /usr/bin/java -Dcom.ibm.tools.attach.enable=no -XX:MaxPermSize=128M -Xmx1024m -Djava.io.tmpdir=/var/sipxdata/tmp -Djetty.lib.dir=/usr/share/java/sipXecs/sipXconfig
...so we restart sipXconfig:
get the name first, since it's not 'sipXconfig':
sudo -u sipxchange sipxproc -l | grep -i config
{"ConfigServer"=>"Running",
"ConfigAgent"=>"Running",
"CallResolver-Agent"=>"ConfigurationMismatch",
ConfigServer looks like a good bet, restart that:
sudo -u sipxchange sipxproc -r ConfigServer
then wait a few seconds for java to spin up, and things go back to normal.