Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received

1. Output of the following commands (non root):

ulimit -u
ulimit -a
df -k

2. From any of the JVM that is running, please get it's Process ID and go to the below location and verify the "limits" file:

/proc/<PID>/

########################################################################
2XHOSLEVEL     OS Level         : Linux 2.6.32-358.el6.x86_64
...
1CIUSERLIMITS  User Limits (in bytes except for NOFILE and NPROC)
NULL           ------------------------------------------------------------------------
NULL           type                            soft limit           hard limit
2CIUSERLIMIT   RLIMIT_AS                        unlimited            unlimited
2CIUSERLIMIT   RLIMIT_CORE                      unlimited            unlimited
2CIUSERLIMIT   RLIMIT_CPU                       unlimited            unlimited
2CIUSERLIMIT   RLIMIT_DATA                      unlimited            unlimited
2CIUSERLIMIT   RLIMIT_FSIZE                     unlimited            unlimited
2CIUSERLIMIT   RLIMIT_LOCKS                     unlimited            unlimited
2CIUSERLIMIT   RLIMIT_MEMLOCK                       65536                65536
2CIUSERLIMIT   RLIMIT_NOFILE                       105535               105535
2CIUSERLIMIT   RLIMIT_NPROC                          1024               105676
2CIUSERLIMIT   RLIMIT_RSS                       unlimited            unlimited
2CIUSERLIMIT   RLIMIT_STACK                      10485760            unlimited
2CIUSERLIMIT   RLIMIT_MSGQUEUE                     819200               819200
2CIUSERLIMIT   RLIMIT_NICE                              0                    0
2CIUSERLIMIT   RLIMIT_RTPRIO                            0                    0
2CIUSERLIMIT   RLIMIT_SIGPENDING                   127457               127457
######################################################################

This error is usually caused by an insufficient ulimit setting. Can you confirm if you are using Red Hat Enterprise Linux 6?  If yes, there is a known issue on this version wherein a new configuration file, (/etc/security/limits.d/90-nproc.conf ) was introduced that overrides the nproc setting in the limits.conf. This file contains a line for nproc configured with a soft limit of 1024. Please check on the said file and comment this line out by making the first character on the line a hash/pound symbol similar to below example:

#*          soft    nproc     1024

Guidelines for setting ulimits (WebSphere Application Server)

http://www-01.ibm.com/support/docview.wss?uid=swg21469413

In addition, it is also recommended to set the ulimit -u or nproc to a value of 131072 when running on Linux to safely account for all the forked threads within processes that could be created.  Please change your current setting to 131072 as recommended.  Refer to the below link for the details:

Insufficient ulimit -u (NPROC) Value Contributes to Native OutOfMemory
http://www-01.ibm.com/support/docview.wss?uid=swg21648497