Dump Event "systhrow" (00040000) Detail "java/lang/OutOfMemoryError" "Failed to create a thread: retVal -1073741830, errno 11" received

1. Output of the following commands (non root):

ulimit -u
ulimit -a
df -k

2. From any of the JVM that is running, please get it's Process ID and go to the below location and verify the "limits" file:

/proc/<PID>/

########################################################################
2XHOSLEVEL     OS Level         : Linux 2.6.32-358.el6.x86_64
...
1CIUSERLIMITS  User Limits (in bytes except for NOFILE and NPROC)
NULL           ------------------------------------------------------------------------
NULL           type                            soft limit           hard limit
2CIUSERLIMIT   RLIMIT_AS                        unlimited            unlimited
2CIUSERLIMIT   RLIMIT_CORE                      unlimited            unlimited
2CIUSERLIMIT   RLIMIT_CPU                       unlimited            unlimited
2CIUSERLIMIT   RLIMIT_DATA                      unlimited            unlimited
2CIUSERLIMIT   RLIMIT_FSIZE                     unlimited            unlimited
2CIUSERLIMIT   RLIMIT_LOCKS                     unlimited            unlimited
2CIUSERLIMIT   RLIMIT_MEMLOCK                       65536                65536
2CIUSERLIMIT   RLIMIT_NOFILE                       105535               105535
2CIUSERLIMIT   RLIMIT_NPROC                          1024               105676
2CIUSERLIMIT   RLIMIT_RSS                       unlimited            unlimited
2CIUSERLIMIT   RLIMIT_STACK                      10485760            unlimited
2CIUSERLIMIT   RLIMIT_MSGQUEUE                     819200               819200
2CIUSERLIMIT   RLIMIT_NICE                              0                    0
2CIUSERLIMIT   RLIMIT_RTPRIO                            0                    0
2CIUSERLIMIT   RLIMIT_SIGPENDING                   127457               127457
######################################################################

This error is usually caused by an insufficient ulimit setting. Can you confirm if you are using Red Hat Enterprise Linux 6?  If yes, there is a known issue on this version wherein a new configuration file, (/etc/security/limits.d/90-nproc.conf ) was introduced that overrides the nproc setting in the limits.conf. This file contains a line for nproc configured with a soft limit of 1024. Please check on the said file and comment this line out by making the first character on the line a hash/pound symbol similar to below example:

#*          soft    nproc     1024

Guidelines for setting ulimits (WebSphere Application Server)

http://www-01.ibm.com/support/docview.wss?uid=swg21469413

In addition, it is also recommended to set the ulimit -u or nproc to a value of 131072 when running on Linux to safely account for all the forked threads within processes that could be created.  Please change your current setting to 131072 as recommended.  Refer to the below link for the details:

Insufficient ulimit -u (NPROC) Value Contributes to Native OutOfMemory
http://www-01.ibm.com/support/docview.wss?uid=swg21648497

Optimize WebServer Performance

Optimize WebServer Performance

WebServer performance can be improved by compressing static web files and reducing the network chattiness between webserver and client system.

_Below are the configuration steps for IHS webserver to enable web compression (to compress web static files) and content caching (to cache static files at client system and reduce network chattiness between webserver and client system)

####Files affected: httpd.conf##############################

Enable required libraries by uncommenting or adding lines to httpd.conf

LoadModule headers_module modules/mod_headers.so

LoadModule deflate_module modules/mod_deflate.so

LoadModule expires_module modules/mod_expires.so

_Enable Web Compression (for files with MIME type: text/html text/plain text/xml text/css text/javascript application/x-javascript)

################################## WEB COMPRESSION START *********************************

SetOutputFilter DEFLATE

DeflateCompressionLevel 9

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/x-javascript

BrowserMatch ^Mozilla/4 gzip-only-text/html

BrowserMatch ^Mozilla/4\.[0678] no-gzip

BrowserMatch \bMSIE\s(7|8) !no-gzip !gzip-only-text/html

BrowserMatch \bOpera !no-gzip

Header append Vary User-Agent

################################## WEB COMPRESSION END **********************************

*Note: Compression will be skipped if client uses Mozilla versions 4.0 or 4.6 or 4.7 or 4.8 and IE versions 6 or 9 or 10 as their browser

Enable Content Caching (for files with MIME type: text/javascript application/x-javascript text/css text/html image/png image/gif image/jpeg and files with extentions: gif jpeg jpg png ico css js html htm)

################################## Content Caching START **************************************

<IfModule mod_expires.c>

AllowEncodedSlashes On

ExpiresActive On

ExpiresByType text/javascript "access plus 24 hours"

ExpiresByType application/x-javascript "access plus 24 hours"

ExpiresByType text/css "access plus 24 hours"

ExpiresByType text/html "access plus 24 hours"

ExpiresByType image/png "access plus 24 hours"

ExpiresByType image/gif "access plus 24 hours"

ExpiresByType image/jpeg "access plus 24 hours"

<LocationMatch "\.(gif|jpeg|jpg|png|ico|css|js|html|htm)$">

Header set Cache-Control "public, max-age=86400"

</LocationMatch>

</IfModule>

################################## Content Caching END ****************************************

Restart WebServer

            

configuring the BIG-IP system for SSL offload | WebSphere application server

configuring the BIG-IP system for SSL offload

For certain applications, when SSL offload is used, a special variable must be set on the BIG-IP system to tell the WebSphere Application server that SSL has been offloaded. This allows WebSphere to properly form its URLs and redirects, and reduces the need for iRules or Stream profiles on the BIG-IP system.

If the BIG-IP system is offloading SSL from your WebSphere application, we recommend performing the following tasks on the BIG-IP system and the WebSphere application server as applicable.

On the BIG-IP system

The HTTP profile must have the Request Header Insert enabled. To modify an existing profile to enable this header, use the following procedure.

1.      On the Main tab, expand Local Traffic and then click Profiles.

2.      Click the name of the HTTP profile you created for the WebSphere application.

3.      In the Request Header Insert row, click the Custom button if necessary, and then use the following syntax in the box: <value>:

For example: httpsoffload: Be sure to include the colon. This value must match the value you configure on the WebSphere Application Server in the next section.

1.      Click the Update button.

On the WebSphere application server

The WebSphere application server needs to be configured to detect the header you configured in the preceding procedure. For more specific instructions, consult the WebSphere documentation.

1.      Connect to the administration port for the WebSphere server.

2.      Navigate to Servers > Application Servers and then select the App Server.

3.      Navigate to Web Container Settings > Web Container > Custom Properties.

4.      Add a property named httpsIndicatorHeader and add a value of httpsoffload.

Note that the property value must match the value used in the BIG-IP system, without the trailing colon, and finally, the property and value are case sensitive. Be sure you do not capitalize the "H" in httpsIndicatorHeader for WebSphere versions 7 or greater.

1.      Navigate to Environment >Virtual Hosts and select the host for your application.

2.      Select Host Aliases.

3.      Add a property with the Host Name of * and a port of 443

Log Client IP AND X-Forwarded-For IP in HTTP Server

When placing apache web servers behind a load-balancing proxy like the BigIP or Pound or behind a caching proxy like Squid or a BlueCoat proxy, the client IP address from the browser is replaced with the IP address of the load-balancer/proxy.  A custom HTTP request header was developed by the squid development team, the X-Forwarded-For header, which has evolved into an industry standard.  Systems supporting the X-Forwarded-For header read the IP address, insert it into the X-Forwarded-For header, and pass it along upstream in the http request.  Apache and Tomcat can log this address in the server's access logs but will only do so for those requests that have passed through the proxy.  If you send a request directly to your apache server, for testing purposes or monitoring, the IP address will not show up in the logs.  If you still want to be able to log the client ip address for systems accessing your server's directly, this article provides a mechanism for accomplishing this with the Apache web server.

LogFormat and CustomLog Configuration Changes

Although Apache has a large number of options insofar as what gets logged is concerned, this article is going to focus on the combined log format, which typically involves logging the following items:

• Remote Host (will use hostnames if apache is configured to look them up)
• Remote logname (typically a dash but could contain the rfc1413-compliant remote user name)
• Remote User (typically a dash unless apache is doing some kind of authentication)
• Timestamp of when the request was received.  This is the local time for the server locale.
• The first line of the request (typically the request URI)
• The status code returned by the server (after redirection has taken place)
• The size of the request minus response headers
• The referring website, if present.
• The user-agent (browser, robot, spider, etc) that made the request.

A default logging configuration in your httpd.conf looks like this:

A default logging configuration in your httpd.conf looks like this:

1	LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" combined

2	CustomLog log/acces_log combined

There are several changes you are going to want to make to the default format in order to log the X-Forwarded-For client ip address or the real client ip address if the X-Forwarded-For header does not exist. Those changes are below:

1	LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined

2	LogFormat "%{X-Forwarded-For}i %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" proxy

3	SetEnvIf X-Forwarded-For "^.*\..*\..*\..*" forwarded

4	CustomLog "logs/access_log" combined env=!forwarded

5	CustomLog "logs/access_log" proxy env=forwarded

This format takes advantage of apache's built-in support for conditional logging based upon environment variable.  The first line is the standard combined log formatted string from the default.  The second line replaces the %h (remote host) field with the value(s) pulled from the X-Forwarded-For header and sets the name of this log file pattern to "proxy".  Line 3 is a setting for environment variable "forwarded" that contains a loose regular expression matching an ip address, which is ok in this case since we really care more whether an ip address exists in the X-Forwarded-For header.  Explained another way, line 3 could be read as:  "If there is an X-Forwarded-For value, use it."  Lines 4 and 5 tell apache which log pattern to use.  If and X-Forwarded-For value exists, use the "proxy" pattern, else use the "combined" pattern for that request.  For readability, lines 4 and 5 do not take advantage of Apache's rotatelogs (piped) logging feature but I assume that it is in use by most everyone.

source_reference:http://www.techstacks.com/

JVM arguments

-Djava.net.preferIPv4Stack=true -Xcodecache20m -Xgcpolicy:gencon -Xdisableexplicitgc -verbose:gc -Xverbosegclog:${SERVER_LOG_ROOT}/native_stderr.log.%Y%m%d.%H%M%S.%pid,7,10000 -Dclient.encoding.override=UTF-8

Define Variables in Log4j

we Have WebSphere running multiple JVM servers; each JVM has their log file system directory for log4j logging we can configure path as follows.

Define variables in log4j.properties as below.




1)

log4j.appender.logFileAudit.File=/var/logs/${server.name}/application.log







2)

Set the variable server.name in the JVM arguments. For WebSphere update server.xml




<jvmEntries debugArgs="-Dcom.ibm.ws.classloader.j9enabled=true -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=7777" debugMode="false" disableJIT="false" genericJvmArguments="-Xquickstart -Dserver.name=server1"…




to configure above go to 







Application servers > server1 > Process definition > Java Virtual Machine

add Generic JVM arguments with value .....        

-Dserver.name=server1

X-Forwarded-For

How do we  configure IBM HTTP Server to set a header called "X-Forwarded-For" equal to the IP address of the client?

Note:The X-Forwarded-For header is not required by IBM HTTP Server (IHS) or by IBM WebSphere Application Server (WAS). But it may be required by software from other vendors.

#######set a header called "X-Forwarded-For" equal to the IP address of the client######

LoadModule rewrite_module modules/mod_rewrite.so

LoadModule headers_module modules/mod_headers.so

RewriteEngine on
RewriteRule ^(.*) - [E=CLIENT_IP:%{REMOTE_ADDR},L]
RequestHeader set x-forwarded-for %{CLIENT_IP}e

1. To configure IBM HTTP Server to set the X-Forwarded-For header to the client IP address, in a non-proxy request, you need to enable the mod_rewrite and mod_headers modules, and then add the following lines in the IBM HTTP Server config (httpd.conf):
2. No configuration is necessary when IBM HTTP Server is configured as a reverse proxy, as this header is added automatically by mod_proxy_http
3. more:http://en.wikipedia.org/wiki/X-Forwarded-For

IBM HTTP Server - URL Masking

If we need to mask the URL, we can add below html code at doc root index.html of IBM HTTP Server

[root@testserver htdocs]# cat index.html
<!DOCTYPE html PUBLIC
<HTML><HEAD>
<META NAME="description" CONTENT="domain-name">
</HEAD>
<FRAMESET border=0 rows="100%,*" frameborder="no" marginleft=0 margintop=0 marginright=0 marginbottom=0>
<frame src="https://test.domain.com/contextb/index.jsp" scrolling=auto frameborder="no" border=0 noresize>
<frame topmargin="0" marginwidth=0 scrolling=no marginheight=0 frameborder="no" border=0 noresize>
</FRAMESET>
</HTML>

JSESSIONID OVERWRITTEN

We implement SSO  in WebSphere by using LTPA(default websphere sso) token between WAS(running web application ) and WAS(running filenet content navigator).

SSO is working fine but JSESSIONID gets overwritten ,due to that  user log-out from 

Scenario

JSESSIONID gets overwritten in the following scenario:
1. HTTP Request is sent to "https://test1.domain.com" URL.
2. HTTP session is created for "https://test1.domain.com" and session ID is stored in
JSESSIONID cookie.
3. "https://test.domain.com" stores an object in the session.
4. "https://test1.domain.com" calls "https://test2.domain.com" ("https://test2.domain.com" provides a menu)
5. HTTP session is created for "https://test2.domain.com" and session ID is stored in
JSESSIONID cookie overwriting the previous value ("https://test1.domain.com" session ID).
6. Second HTTP request is sent to "https://test1.domain.com" URL.
7. "https://test1.domain.com" fails to lookup the object in the session stored in step 3.
request.getSession(false) returns null.

Resolution

 https://test1.domain.com and https://test2.domain.com are in two different WAR files and therefore have
different context roots, right?
In that case you could use WebSphere's admin console and change the
cookie path from the default (which is /, meaning the cookie is sent as
long as the URL starts with a /, which is always) to the context root
for each application.
So for application A, set the cookie path to /test1_contextRoot and for
application B, set the cookie path to /test2_contextRoot.
In WebSphere's admin console, go to Enterprise Applications ->
Application A -> Session Management -> {click the Enable Cookies link}
and change the Cookie path.