Page Cannot Be Diplayed OR Hang

[hobaugh]

I am having trouble with users getting "Page cannot be displayed" error messages OR their browser just hangs.  Sometimes they get to the site and sometimes not.  People have had this problem using IE 7 on XP and Vista others are using IE 6

 

I have searched, implemented and verified solutions that I have found for IE problems but still having this trouble.  These people are getting to my server as evidenced below by the packets I have grabbed and I see info messages from mod_ssl about establishing the connection.  The logs below are representative of a client with this problem (no firewall logs are available).  These users are not pressing the stop button in the browser. I was on the phone with a person trying to connect when the log said "Hint: Stop button pressed in browser?"  Their browser just hung.

 

I have a bunch of info below and I think I have everything but let me know if I missed something.  Any help would be greatly appreciated, my client is getting upset and if I don't find a solution soon I may loose this client.

 

Thanks in advance,

Doug

 

 

Setup

------------------------------------------------------------

X86_64 quad Xeon 3GHZ with 2GB memory and 545GB raid and Full T1

Apache/2.0.50

Linux SUSE 9.3

mod_ssl/2.0.50

OpenSSL/0.9.7d

[other modules listed at bottom]

 

 

Apache Log:

------------------------------------------------------------

[Wed Feb 06 14:01:15 2008] [info] Connection to child 1 established (server www.myserver.com:443, client XX.XXX.XXX.X)

[Wed Feb 06 14:01:15 2008] [info] Seeding PRNG with 144 bytes of entropy

[Wed Feb 06 14:01:18 2008] [info] (104)Connection reset by peer: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]

[Wed Feb 06 14:01:18 2008] [info] Connection to child 1 closed with abortive shutdown(server www.myserver.com:443, client XX.XXX.XXX.X)

 

 

Packets

------------------------------------------------------------

14:01:15.094019 IP (tos 0x0, ttl 110, id 57683, offset 0, flags [DF], length: 48) XX.XXX.XXX.X.47540 > XXX.XXX.X.XX.443: S [tcp sum ok] 4067022148:4067022148(0) win 65535 <mss 1380,nop,nop,nop,nop>

14:01:15.121979 IP (tos 0x0, ttl 110, id 57685, offset 0, flags [DF], length: 40) XX.XXX.XXX.X.47540 > XXX.XXX.X.XX.443: . [tcp sum ok] 4067022149:4067022149(0) ack 925029440 win 65535

14:01:15.124066 IP (tos 0x0, ttl 110, id 57686, offset 0, flags [DF], length: 118) XX.XXX.XXX.X.47540 > XXX.XXX.X.XX.443: P [tcp sum ok] 4067022149:4067022227(78) ack 925029440 win 65535

14:01:18.159625 IP (tos 0x0, ttl 110, id 22280, offset 0, flags [DF], length: 40) XX.XXX.XXX.X.47540 > XXX.XXX.X.XX.443: R [tcp sum ok] 4067022227:4067022227(0) win 5840

 

 

IE fixes implemented

------------------------------------------------------------

SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

 

Verified by PHP SERVER vars and response header

[nokeepalive] => 1

[ssl-unclean-shutdown] => 1

[downgrade-1_0] => 1

[force-response-1_0] => 1

 

          Server: Apache/2.0.50 (Linux/SUSE)

    X-Powered-By: PHP/5.2.0

      Set-Cookie: xyz=8bax863a6ff21ec1cb3b04d0e8edf412; path=/

          Expires: Mon, 26 Jul 1997 05:00:00 GMT

    Cache-Control: no-cache, no-store, must-revalidate, proxy-revalidate

          Pragma: no-cache

    Last-Modified: Thu, 07 Feb 2008 15:36:08 GMT

    Cache-Control: post-check=0, pre-check=0

      Connection: close

Transfer-Encoding: chunked

    Content-Type: text/html; charset=iso-8859-1

 

 

BrowserMatch \bMSIE\s7  !no-gzip !gzip-only-text/html

 

Verified by inspecting response headers for CSS & JavaScript requests

IE6

Date: Thu, 07 Feb 2008 15:26:58 GMT

Server: Apache/2.0.50 (Linux/SUSE)

Last-Modified: Wed, 12 Dec 2007 20:32:18 GMT

ETag: "247d6f-b12-4411cb9be7880"

Accept-Ranges: bytes

Content-Length: 2834

Cache-Control: max-age=172801

Expires: Sat, 09 Feb 2008 15:26:59 GMT

Connection: close

Content-Type: text/css

 

FireFox

Date: Thu, 07 Feb 2008 14:46:05 GMT

Server: Apache/2.0.50 (Linux/SUSE)

Last-Modified: Wed, 12 Dec 2007 20:32:18 GMT

Etag: "247d6f-b12-4411cb9be7880"

Accept-Ranges: bytes

Vary: Accept-Encoding

Content-Encoding: gzip

Cache-Control: max-age=172801

Expires: Sat, 09 Feb 2008 14:46:06 GMT

Content-Length: 980

Content-Type: text/css

 

 

SSLSessionCache        shmcb:/var/lib/apache2/ssl_scache(512000)

Verified that its working by using the command below, I do not get any cache information on the server-status page even though I have ExtendedStatus on

 

openssl s_client -connect myserver.com:443 -state  -reconnect

 

Results used same session id for each request:

New, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

Reused, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

Reused, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

Reused, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

Reused, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

Reused, TLSv1/SSLv3, Cipher is DHE-RSA-AES256-SHA

    Session-ID: A160636BAE4C52...TRUNCATED

 

 

Server Tuning

------------------------------------------------------------

<IfModule prefork.c>

  StartServers        5

  MinSpareServers      5

  MaxSpareServers    10

  ServerLimit        150

  MaxClients        150

  MaxRequestsPerChild  10000

</IfModule>

 

<IfModule worker.c>

  StartServers        2

  MinSpareThreads    25

  MaxSpareThreads    75

  MaxClients        150

  ThreadsPerChild    25

  MaxRequestsPerChild  10000

</IfModule>

 

<IfModule leader.c>

  StartServers        2

  MinSpareThreads    25

  MaxSpareThreads    75

  MaxClients        150

  ThreadsPerChild    25

  MaxRequestsPerChild  10000

</IfModule>

 

<IfModule perchild.c>

  NumServers          5

  StartThreads        5

  MinSpareThreads      5

  MaxSpareThreads    10

  MaxThreadsPerChild  20

  MaxRequestsPerChild  10000

  AcceptMutex fcntl

</IfModule>

 

<IfModule metuxmpm.c>

  StartThreads          5

  MinSpareThreads      5

  MaxSpareThreads      10

  MaxRequestsPerChild  0

  Multiplexer  "wwwrun"  "www"

</IfModule>

 

KeepAlive On

MaxKeepAliveRequests 100

KeepAliveTimeout 2

 

BrowserMatch "Mozilla/2" nokeepalive

BrowserMatch "MSIE 4\.0b2;" nokeepalive downgrade-1.0 force-response-1.0

BrowserMatch "RealPlayer 4\.0" force-response-1.0

BrowserMatch "Java/1\.0" force-response-1.0

BrowserMatch "JDK/1\.0" force-response-1.0

BrowserMatch "Microsoft Data Access Internet Publishing Provider" redirect-carefully

BrowserMatch "^WebDrive" redirect-carefully

BrowserMatch "^WebDAVFS/1.[012]" redirect-carefully

BrowserMatch "^gnome-vfs" redirect-carefully

BrowserMatch ^Mozilla/4 gzip-only-text/html

BrowserMatch ^Mozilla/4\.0[678] no-gzip

BrowserMatch \bMSIE !no-gzip !gzip-only-text/html

BrowserMatch \bMSIE\s7  !no-gzip !gzip-only-text/html

SetEnvIf User-Agent ".*MSIE.*" nokeepalive ssl-unclean-shutdown downgrade-1.0 force-response-1.0

AddOutputFilterByType DEFLATE text/html text/plain text/xml text/css text/javascript application/x-javascript

 

 

SSL

------------------------------------------------------------

AddType application/x-x509-ca-cert .crt

AddType application/x-pkcs7-crl    .crl

SSLPassPhraseDialog  builtin

SSLSessionCache        shmcb:/var/lib/apache2/ssl_scache(512000)

SSLSessionCacheTimeout  600

SSLMutex  sem

SSLRandomSeed startup builtin

SSLRandomSeed connect builtin

SSLEngine on

SSLCipherSuite ALL:!ADH:!EXPORT56:RC4+RSA:+HIGH:+MEDIUM:+LOW:+SSLv2:+EXP:+eNULL

SSLCertificateFile /etc/apache2/ssl.crt/server.crt

SSLCertificateKeyFile /etc/apache2/ssl.key/server.key

SSLCACertificatePath /etc/apache2/ssl.crt

SSLCACertificateFile /etc/apache2/ssl.crt/ComodoSecurityServicesCA.crt

SSLOptions +StrictRequire

 

 

GLobal

------------------------------------------------------------

Timeout 300

ServerSignature on

ServerAdmin doug@server.com

ServerName name.server.com

UseCanonicalName off

ServerTokens OS

<IfModule mod_status.c>

  ExtendedStatus on

</IfModule>

LogLevel info

CustomLog /var/log/apache2/access_log combined

 

 

Mod Security

------------------------------------------------------------

<IfModule mod_security.c>

  SecFilterEngine DynamicOnly

  SecFilterDefaultAction "deny,log,status:403"

  SecFilterScanPOST On

  SecFilterCheckURLEncoding On

  SecFilterCheckUnicodeEncoding Off

  SecFilterForceByteRange 1 255

  SecUploadDir /tmp

  SecUploadKeepFiles Off

  SecAuditEngine RelevantOnly

  SecAuditLogRelevantStatus ^5

  SecAuditLog /var/log/apache2/mod_security_log

  SecFilterDebugLevel 0

  SecFilterDebugLog /var/log/apache2/mod_security_debug_log

</IfModule>

 

 

Modules

------------------------------------------------------------

access_module

actions_module

alias_module

auth_module

auth_dbm_module

autoindex_module

cgi_module

dir_module

env_module

expires_module

include_module

log_config_module

mime_module

negotiation_module

setenvif_module

ssl_module

suexec_module

userdir_module

php5_module

rewrite_module

status_module

security_module

deflate_module

auth_shadow_module

 

 

Memory info using free

------------------------------------------------------------

            total      used      free    shared    buffers  cached

Mem:      2055108    1524412    530696          0      69552  1087208

-/+ buffers/cache:    367652    1687456

Swap:      2096440    290524    1805916

 

 

Disk Space

------------------------------------------------------------

Filesystem            Size  Used Avail Use% Mounted on

/dev/sda2            545G  385G  161G  71% /

 

[kwilson]

I'm having this same situation with IE complaining with "Page cannot be displayed" during a POST. Firefox is more informative, it says it's getting error 12237. It's a complaint about an unexpected cipher change, and if I turn on the the logging level to debug I do indeed see a cipher change shortly before OpenSSL reporting an error.

 

This is on a reverse proxy, I had thought it was due to the combination of that and SSL. Is this your situation too?

 

I usually run mod_security, but this server doesn't have it.

[hobaugh]

I'm having this same situation with IE complaining with "Page cannot be displayed" during a POST. Firefox is more informative, it says it's getting error 12237. It's a complaint about an unexpected cipher change, and if I turn on the the logging level to debug I do indeed see a cipher change shortly before OpenSSL reporting an error.

 

This is on a reverse proxy, I had thought it was due to the combination of that and SSL. Is this your situation too?

 

I usually run mod_security, but this server doesn't have it.

 

This is happening on GET requests and I do not have a reverse proxy.

[kwilson]

It's now looking like our problem is with NAT between the browser and the server. This makes sense because I've heard of others trying to reproduce this with stress testing tools and they couldn't do it. We can't reproduce it locally either.

 

Is there a firewall between your browser and server? I've seen situations where the firewall messes with an otherwise working connection.

[hobaugh]

There is not a firewall between my browser and the web server but there is for our customers.  I have talked with my boss and he does not believe the firewall is the problem.  I have done some searching but have not found any firewall problems that describe what I am seeing.

[kwilson]

Are you seeing a cipher change occurring shortly before the abortive close? You'll need to set the logging to debug level before you can see the cipher change messages.

[hobaugh]

I turned on debugging yesterday.

 

Here is the error log for the latest with the "SSL handshake interrupted by system" message.  I do not have any access logs for this session. This person did not contact me about a problem but most don't.

 

[Thu Feb 14 14:51:59 2008] [info] Connection to child 7 established (server www.myserver.com:443, client XX.XX.XXX.XXX)

[Thu Feb 14 14:51:59 2008] [info] Seeding PRNG with 144 bytes of entropy

[Thu Feb 14 14:51:59 2008] [debug] ssl_engine_kernel.c(1775): OpenSSL: Handshake: start

[Thu Feb 14 14:51:59 2008] [debug] ssl_engine_kernel.c(1783): OpenSSL: Loop: before/accept initialization

[Thu Feb 14 14:51:59 2008] [debug] ssl_engine_io.c(1609): OpenSSL: I/O error, 11 bytes expected to read on BIO#980fb0 [mem: 994350]

[Thu Feb 14 14:51:59 2008] [debug] ssl_engine_kernel.c(1812): OpenSSL: Exit: error in SSLv2/v3 read client hello A

[Thu Feb 14 14:51:59 2008] [info] (104)Connection reset by peer: SSL handshake interrupted by system [Hint: Stop button pressed in browser?!]

[Thu Feb 14 14:51:59 2008] [info] Connection to child 7 closed with abortive shutdown(server www.myserver.com:443, client XX.XX.XXX.XXX)

 

Here are the firewall logs for this session.  The firewall time is off from the system time.

 

<166>Feb 14 2008 14:49:18: %PIX-6-302013: Built inbound TCP connection 266843 for outside:XX.XX.XXX.XXX/50270 (XX.XX.XXX.XXX/50270) to inside:192.168.0.9/443 (XX.X.XXX.XXX/443)

<166>Feb 14 2008 14:49:21: %PIX-6-302014: Teardown TCP connection 266843 for outside:XX.XX.XXX.XXX/50270 to inside:192.168.0.9/443 duration 0:00:02 bytes 3866 TCP FINs

<166>Feb 14 2008 14:49:18: %PIX-6-302013: Built inbound TCP connection 266844 for outside:XX.XX.XXX.XXX/50271 (XX.XX.XXX.XXX/50271) to inside:192.168.0.9/443 (XX.X.XXX.XXX/443)

<166>Feb 14 2008 14:49:21: %PIX-6-302014: Teardown TCP connection 266844 for outside:XX.XX.XXX.XXX/50271 to inside:192.168.0.9/443 duration 0:00:02 bytes 5166 TCP FINs

<166>Feb 14 2008 14:49:22: %PIX-6-302013: Built inbound TCP connection 266845 for outside:XX.XX.XXX.XXX/50278 (XX.XX.XXX.XXX/50278) to inside:192.168.0.11/443 (12.4.224.227/443)

<166>Feb 14 2008 14:49:22: %PIX-6-302014: Teardown TCP connection 266845 for outside:XX.XX.XXX.XXX/50278 to inside:192.168.0.11/443 duration 0:00:00 bytes 0 TCP Reset-O

 

[kwilson]

[pre][Thu Feb 14 14:51:59 2008] [debug] ssl_engine_io.c(1609): OpenSSL: I/O error, 11 bytes expected to read on BIO#980fb0 [mem: 994350]

[Thu Feb 14 14:51:59 2008] [debug] ssl_engine_kernel.c(1812): OpenSSL: Exit: error in SSLv2/v3 read client hello A[/pre]

I have to wonder if the client that is connecting is capable of SSLv2 or v3.