Apache and PHP secure websocket stop working on Ubuntu 18

[BrianPez]

0


I have Apache running under Ubuntu 18 on a google cloud instance. It forwards websocket requests to a running PHP process. Everything was working fine for both secure and non-secure HTTP connections.

About two weeks ago the secure websockets stopped working. After about 2 minutes, I get a browser timeout 'Websocket opening handshake timeout'. If I remove my port 80 redirect to 443 and change my PHP to not use secure websockets, non-secure websockets still works.

I do see around the time it stopped (3/19), Apache got upgraded to 2.4.29-1ubuntu4.13 and PHP to 7.2.24-0ubuntu0.18.04.3 via dpkg and unattended upgrades.

My apache config for virtual host :443

  SSLEngine on
  SSLCertificateFile /etc/ssl/certs/XXXX.crt
  SSLCertificateKeyFile /etc/ssl/private/XXXX.key
  SSLCertificateChainFile /etc/ssl/certs/XXXX.crt
  SSLProxyEngine on
  ProxyPass /wss8080 wss://127.0.0.1:8080/
  ProxyPassReverse /wss8080 wss://127.0.0.1:8080/

With the following proxy mods enabled: /etc/apache2/mods-enabled/proxy.conf /etc/apache2/mods-enabled/proxy_http.load /etc/apache2/mods-enabled/proxy_wstunnel.load /etc/apache2/mods-enabled/proxy_connect.load /etc/apache2/mods-enabled/proxy.load

My PHP code

   $loop   = React\EventLoop\Factory::create();
    $context = new React\ZMQ\Context($loop);
    $pull = $context->getSocket(ZMQ::SOCKET_REP);
    $pull->bind('tcp://127.0.0.1:' . $zmqPort); // Binding to 127.0.0.1 means the only client that can connect is itself
    $pull->on('message', function($networkMsg) {
//stuff
    });
$webSock = new React\Socket\Server('0.0.0.0:' . $wsPort, $loop); // Binding to 0.0.0.0 means remotes can connect
$webSock = new React\Socket\SecureServer($webSock, $loop, [
    'local_cert' => $sslCert,
    'local_pk' => $sslPKey,
    'allow_self_signed' => FALSE,
    'verify_peer' => FALSE
]);
$webServer = new Ratchet\Server\IoServer(
                new Ratchet\Http\HttpServer(
                    new Ratchet\WebSocket\WsServer(
                        new Ratchet\Wamp\WampServer($pusher)
                    )
                ),
                $webSock
);
$loop->run();

Trying with curl (which I didn't try before it was broken, so I can't compare)

curl -k -vvv "https://XXXX:8080"
* Rebuilt URL to: https://XXXX:8080/
*   Trying 35.238.154.120...
* TCP_NODELAY set
* Connected to XXXX (XXX.XXX.XXX.XXX) port 8080 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Unknown (8):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Client hello (1):
* TLSv1.3 (OUT), TLS Unknown, Certificate Status (22):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; CN=*.XXXX
*  start date: Sep 30 16:50:20 2019 GMT
*  expire date: Apr 10 18:13:00 2021 GMT
*  issuer: C=US; ST=Arizona; L=Scottsdale; O=GoDaddy.com, Inc.; OU=http://certs.godaddy.com/repository/; CN=Go Daddy Secure Certificate Authority - G2
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* TLSv1.3 (OUT), TLS Unknown, Unknown (23):
> GET / HTTP/1.1
> Host: XXXX:8080
> User-Agent: curl/7.58.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Certificate Status (22):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS Unknown, Unknown (23):
< HTTP/1.1 426 Upgrade header MUST be provided
< Connection: Upgrade
< Upgrade: websocket
< Sec-WebSocket-Version: 13
< Sec-WebSocket-Protocol: wamp
< X-Powered-By: Ratchet/0.4.1
* no chunk, no close, no size. Assume close to signal end
< 
* Closing connection 0
* TLSv1.3 (OUT), TLS Unknown, Unknown (21):
* TLSv1.3 (OUT), TLS alert, Client hello (1):

 

[requinix]

On 3/27/2020 at 7:28 AM, BrianPez said:

I do see around the time it stopped (3/19), Apache got upgraded to 2.4.29-1ubuntu4.13 and PHP to 7.2.24-0ubuntu0.18.04.3 via dpkg and unattended upgrades.

I hope you've learned a lesson from that: don't use unattended or otherwise automatic software upgrades on production systems.

Have you checked if dpkg needed to update configuration files? Hopefully it created backups of the originals - check if the changes are relevant.

Anyway, the curl output looks right. Next step would be writing proper WSS headers and seeing what happens.