解决Caused by:java.lang.OutOfMemoryError: unable to create new native thread

[TOC]

业务背景

一台服务器上部署3个tomcat，并且每个tomcat的server.xml配置如下，只是端口不一致

<Executor name="tomcatThreadPool" namePrefix="wkApi-exec-"  maxThreads="800" minSpareThreads="200"/>
<Connector  executor="tomcatThreadPool"  port="9000" 
               protocol="org.apache.coyote.http11.Http11NioProtocol"
               connectionTimeout="60000"
               redirectPort="8443"
               URIEncoding="UTF-8"
               maxPostSize="0"
               maxThreads="1024" 
               minSpareThreads="256"
               acceptCount="300" 
               compression="on" 
               compressionMinSize="2048" 
               noCompressionUserAgents="gozilla,traviata" 
               CompressableMimeType="text/html,text/xml,text/plain,text/css,text/javascript,text/json,application/x-javascript,application/javascript,application/json,image/png,image/jpeg"
               />

出现问题

1）发现日志出现Caused by:java.lang.OutOfMemoryError: unable to create new native thread,但是程序还在继续跑着，
2）当前起服务的用户无法操作任何命令，并且报：-bash: fork: retry: Resource temporarily unavailable

ok，不是服务器配置的错误。是当前用户起的服务创建的进程总和>当前用户的最大值了

通过：ulimit -a 命令

[appuser@123.57.86.172/10.173.41.156 ~]$ ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 7415
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 65535
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 10240
cpu time               (seconds, -t) unlimited
max user processes              (-u) 1024
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

max user processes (-u) 1024

可以看到当前用户最大只能起1024个进程

可以使用pstree -a|wc -l 统计当前进程总数

[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a|wc -l
75
 #将所有的进程树导出来
[appuser@123.57.86.172/10.173.41.156 ~]$ pstree -a > 1.log

可以查看所有的进程

解决办法：

1，限制所有tomcat的进程数总和小于1024,（ps: 最大值在9000左右）
2，通过 /etc/security/limits.d/90-nproc.conf 调整最大进程数，

可以看到其他用户限制在1024

[appuser@123.57.86.172/10.173.41.156 ~]$ cat /etc/security/limits.d/90-nproc.conf
# Default limit for number of user's processes to prevent
# accidental fork bombs.
# See rhbz #432903 for reasoning.

*          soft    nproc     1024
root       soft    nproc     unlimited