Cannot send large TLS packages over socket

NotionCommotion · June 20, 2019

I have a hosted VPS running PHP 7.3.5 and a local RPi running 7.0.33, and will refer to them as VPS and RPi. I also have the following very basic socket client and server.

The server on the VPS and the client on the RPi using TCP works great and so does both the server and the client on the VPS using TLS.

But when I put the server on the VPS and the client on the RPi using TLS, I can only send relatively short packages (approximately 2,600 characters). And there is no hard limit because one moment 2,628 characters works, but then a little later 2,619 will not.

Does it look like something I am doing wrong or maybe upgrading PHP on the RPi will help?

Thank you

<?php
require 'simple_common.php';
$loop = React\EventLoop\Factory::create();
$connector = new React\Socket\Connector($loop);
logger("simple client is running using ".($tls?'tls':'tcp'));
if($tls) {
    $sslContextOptions = [
        'peer_name'=> get('peer_name'),
        //'verify_peer'=>false,
        //'verify_peer_name'=>false,
        'allow_self_signed'=>true,
        'cafile' => get('cafile'),   //Certificate Authority file used with verify_peer.  Use public cert since selfsigned?
        //'verify_depth' not set and defaults to none
        //'ciphers' not used and defaults to DEFAULT
        //'capath'=>null, //Not needed since cafile is given
        //'CN_match' and 'SNI_server_name' are depreciated and peer_name should be used instead.
    ];
    $connector = new React\Socket\SecureConnector($connector, $loop, $sslContextOptions);
}
$connector->connect(get('ipPort'))->then(function (React\Socket\ConnectionInterface $connection) use ($loop) {
    logger("onConnection");
    setEvents($connection, 'client');
    $string=str_repeat(substr(str_repeat('0123456789',10),0,-1), 26).'023456789023456789023456789023456789023456789023456789';
    logger("write strlen: ".strlen($string));
    logData('write', $string);
    $connection->write($string);
});
$loop->run();

<?php
require 'simple_common.php';
$loop = React\EventLoop\Factory::create();
logger("simple server is running using ".($tls?'tls':'tcp'));

$socket = new React\Socket\Server(get('ipPort'), $loop);

if($tls) {
    $sslContextOptions = [
        'local_cert' => get('local_cert'), //local certificate file w/ or w/o private key
        'local_pk' => get('local_pk'),    //Only needed if private key is not in local_cert
        //'passphrase' not set as local_cert is not encoded
        //'capture_peer_cert'=>true, //a peer_certificate context option will be created containing the peer certificate
        //'capture_peer_cert_chain'=>true, //a peer_certificate_chain context option will be created containing the certificate chain.
        //'SNI_enabled'=>true, //server name indication will be enabled. Enabling SNI allows multiple certificates on the same IP address.
        //'disable_compression'=>true,  //disable TLS compression. This can help mitigate the CRIME attack vector.
        //'peer_fingerprint'=>[]    //A hash or array of hashes which the remote certificate digest must match.
    ];
   $socket = new React\Socket\SecureServer($socket, $loop, $sslContextOptions );
}

$socket->on('connection', function (React\Socket\ConnectionInterface $connection) use ($loop) {
    logger('onConnection');
    setEvents($connection, 'server');
    $loop->addPeriodicTimer(1, function ($timer) use ($connection) {
        $data='Hello!';
        logData('write', $data);
        $connection->write($data);
    });
});
$socket->on('error', function (\Exception $e) {
    logger('onServerError: '.$e->getMessage());
});
$loop->run();

<?php
function setEvents($connection, string $type){
    $connection->on('data', function ($data)use($type) {
        static $totalLng=0;
        $curLng=strlen($data);
        $totalLng+=$curLng;
        logger("$type onData: current: $curLng  total: $totalLng");
        logData('read', $data);
    });
    $connection->on('error', function (\Exception $e)use($type) {
        logger("$type onError: ".$e->getMessage());
    });
    $connection->on('close', function()use($type) {
        logger("$type onClose");
    });
    $connection->on('end', function()use($type) {
        logger("$type onEnd");
    });
    $connection->on('drain', function()use($type) {
        logger("$type onDrain");
    });
    $connection->on('pipe', function()use($type) {
        logger("$type onPipe");
    });
}

function emptyLog($type){
    $f = @fopen(get($type), "r+");
    if ($f !== false) {
        ftruncate($f, 0);
        fclose($f);
    }
}

function logData($type, $string){
    file_put_contents(get($type), $string,  FILE_APPEND);
}

function logger($msg){
    echo($msg.PHP_EOL);
    syslog(LOG_INFO, $msg);
}

function get($name){
    $map = [
        'ipPort'=>'184.154.134.91:1338',
        'peer_name'=>'tapmeister.com',
        'cafile'=>'test_ss_crt.pem',   //Use public cert since selfsigned?  Why does even test_ss_csr.pem work?
        'local_pk'=>'test_ss_key.pem',
        'local_cert'=>'test_ss_crt.pem', //Why does even test_ss_csr.pem work?
        'read'=>'raw_read_stream.log',
        'write'=>'raw_write_stream.log',
        'tls'=>true,
    ];
    if(!isset($map[$name])) exit("Invalid name $name");
    return $map[$name];
}

ini_set('display_startup_errors', '1');
ini_set('display_errors', '1');
error_reporting(E_ALL);
require 'vendor/autoload.php';
$tls=$argv[1]??get('tls');
emptyLog('write');
emptyLog('read');

NotionCommotion · June 20, 2019

I upgraded the RPi to PHP 7.3.4-2, however, no change. I was hoping as a read how earlier versions of PHP had bug related to this. Maybe not even PHP related?

NotionCommotion · June 20, 2019

Eureka! It is not the client but the server. Instead of using both my PHP server and client at the same time, I replaced one at a time with openssl s_server and openssl s_client, and found my PHP client can send the data to openssl s_server. Maybe a server side PHP configuration issue? Are there any logs which might pose some clues?

kicken · June 20, 2019

Can you clarify what exactly happens?

If you mean that the larger data segments are split into smaller chunks than that's to be expected. Network packets have a maximum size (1500 bytes, less overhead typically) so large data segments have to be split across several packets before being sent. Due to the way networks work, those individual packets may arrive at different times and out of order. TCP will handle ordering them for you, but if you try and read from the socket and only some of the data has arrived you'll get what's there now then have to read again to get the rest. That means the data you write may be split across multiple data events on the receiving end.

NotionCommotion · June 20, 2019

32 minutes ago, kicken said:

Can you clarify what exactly happens?

Yes, I see the smaller chunks. If the client and server is on the same machine, it appears the chunks can be much larger. Regarding what I am witnessing:

When using the openssl s_client on the RPi to send a long string to the VPN php socket server, the RPi oppenssl s_client complains with write:errno=32 which I believe is a broken pipe.
When using the openssl s_client on the VPN to send a long string to the VPN php socket server, the VPN oppenssl s_client doesn't appear to complain but the string is not received and the VPN server just shows the client as closing.
When using the PHP client on the RPi to send a long string to the VPN php socket server, the RPi PHP client doesn't appear to complain but the string is not received and the VPN server just shows the client as closing.
When using the PHP client on the VPN to send a long string to the VPN php socket server, the VPN PHP client complains with Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 89784032 bytes) in ..WritableResourceStream.php on line 150 and the VPN server just shows the client as closing.

An actually needed string is 17485. When ran on the same machine, all appears good, however, when on different machines, no displayed error but nothing set.

Edited June 20, 2019 by NotionCommotion

NotionCommotion · June 20, 2019

This is a more concise version. Notice how the data received by the server adds up to 22,176 and not my expected 100,000, and the write:errno=32 at the end of the client? If you want, please change is the path in the server to your cert/key and the IP the client is pointing your server and try it out. Thanks

PS. Smaller strings works great. Strings of this size works great if both the client and server is on the same machine. TCP instead of TLS on multiple machines works great. Unfortunately, these are not my requirements.

<?php
ini_set('display_startup_errors', '1');
ini_set('display_errors', '1');
error_reporting(E_ALL);
require 'vendor/autoload.php';
$loop = React\EventLoop\Factory::create();
$socket = new React\Socket\Server('0.0.0.0:1338', $loop);
$socket = new React\Socket\SecureServer($socket, $loop, ['local_cert' => 'cert_with_priv_key.pem'] );

$socket->on('connection', function (React\Socket\ConnectionInterface $connection) {
    $connection->on('data', function ($data) {echo("onData: ".strlen($data).PHP_EOL);});
    $connection->on('error', function (\Exception $e) {echo("onError: ".$e->getMessage().PHP_EOL);});
    $connection->on('close', function() {echo("onClose".PHP_EOL);});
    $connection->on('end', function() {echo("onEnd".PHP_EOL);});
    $connection->on('drain', function() {echo("onDrain".PHP_EOL);});
    $connection->on('pipe', function() {echo("onPipe".PHP_EOL);});
});
$socket->on('error', function (\Exception $e) {echo('onServerError: '.$e->getMessage().PHP_EOL);});
$loop->run();

[michael@example wireshark]$ php server.php
onData: 1008
onData: 1008
onData: 1008
onData: 2016
onData: 2016
onData: 2016
onData: 1008
onData: 3024
onData: 3024
onData: 3024
onData: 3024
onEnd
onClose

michael@rpi:~/test $ printf '=%.0s' {1..100000} | openssl s_client -connect 12.345.67.890:1338
CONNECTED(00000003)
depth=0 C = XX, L = Default City, O = Default Company Ltd, CN = example.net
verify error:num=18:self signed certificate
verify return:1
depth=0 C = XX, L = Default City, O = Default Company Ltd, CN = example.net
verify return:1
---
Certificate chain
 0 s:/C=XX/L=Default City/O=Default Company Ltd/CN=example.net
   i:/C=XX/L=Default City/O=Default Company Ltd/CN=example.net
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIEODCCAqACCQCzx57HUSzlRDANBgkqhkiG9w0BAQsFADBeMQswCQYDVQQGEwJY
WDEVMBMGA1UEBwwMRGVmYXVsdCBDaXR5MRwwGgYDVQQKDBNEZWZhdWx0IENvbXBh
bnkgTHRkMRowGAYDVQQDDBFncmVlbmJlYW50ZWNoLm5ldDAeFw0xOTA2MTIxNjM0
MzZaFw0yMDA2MTExNjM0MzZaMF4xCzAJBgNVBAYTAlhYMRUwEwYDVQQHDAxEZWZh
dWx0IENpdHkxHDAaBgNVBAoME0RlZmF1bHQgQ29tcGFueSBMdGQxGjAYBgNVBAMM
EWdyZWVuYmVhbnRlY2gubmV0MIIBojANBgkqhkiG9w0BAQEFAAOCAY8AMIIBigKC
AYEAxGarBrx3JhDiEq5VVwbEFTY/GHLRnqD9X1Cti8l4s+dbdqHbr0gpyXS0DIF+
xiH1RHAkyw3Nzixf0vEoRRwOaRmkYk9uFTTOFDMNWEv00ZVzhjgCgxYWHBna4KQ+
S3lRpM8wlMPlpeqbjq0LsvfTO1rr/pSQ6Ml34tXVWRvrOjZeaEl4yV0LFzInbHo9
FlsABnmuAuRSD5gCMGIqiVnpChF4Cbu1WrEPi9LlID3zvXh7kED9EjFYDeaSUUwf
X/0AN4LgFuo1qj/iQ4KEDbAMo7L6dTN4AupYWatzTA1fK2K6AGYvYdnjTrNNbomZ
hOSNwlZfb9SD9/S5aQDcGkMyzCzb7svSYKvx5b5+XpxJLI1daDbidstrLg3DvEfp
GcGyOfLAu7ZOfgkZaeOZsFZj3MLQrzSlBFOZ+DYVEf0aq8iTvVt+mrp6MlncU0FL
UHngtmD7ZTJmFEgtbeUbrY/QwHvAQOCd85/0wbUf/5esBJKiiaNstKY7nJJ9gwsl
kF8FAgMBAAEwDQYJKoZIhvcNAQELBQADggGBAK8b7Eqo5GTDV1vgoJCl/SUUjPey
DnakNZ5rVGSmTDntgS7p1N9BaCS5JbRQkwDhRcRaGua6jH68uiAOlp03C+qZHERJ
kZOtHOhK+8Uetn2dD3G80l8OXRmAPLoJ7yEt+wBfohrC8TBScq+e8cjbCkq2lEKd
9BAFFj21dlv8gO/f8QMZjyVsrjLu4Dn1Pjlos3X8jXNNUzVRi2qtA9bLT+ldEkc6
9mcQpYVq2rX+b8uEwFqy26HEvbMjiQ7F8ocC5Kz0RrMMfnJWfELysTXwbF9IvZF1
8d1IKVY4PdhLi7ZLxtAiaUaA3u0zimPDHrlUtuu99v7mbnZ5qVYj0ekMGJ1bRmnb
bhfMqH34L2oSPpQTr9aYpuTOjpTR8juCflvcy0SUO2rinTeJK7BBWLVEhLi5JZ7v
Q2F/Lc2aP+Wot4RtvYpooBi/lB9TqhsfdWOdKEmS3fLFpNLEh4y2bNs8ENvDDtGL
tA1N791c156ih3i0Xdl7hiBV1CTwCoV1GCjyYw==
-----END CERTIFICATE-----
subject=/C=XX/L=Default City/O=Default Company Ltd/CN=example.net
issuer=/C=XX/L=Default City/O=Default Company Ltd/CN=example.net
---
No client certificate CA names sent
Client Certificate Types: RSA sign, DSA sign, ECDSA sign
Requested Signature Algorithms: RSA+SHA512:DSA+SHA512:ECDSA+SHA512:RSA+SHA384:DSA+SHA384:ECDSA+SHA384:RSA+SHA256:DSA+SHA256:ECDSA+SHA256:RSA+SHA224:DSA                                                                                      +SHA224:ECDSA+SHA224:RSA+SHA1:DSA+SHA1:ECDSA+SHA1
Peer signing digest: SHA512
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 1908 bytes and written 431 bytes
Verification error: self signed certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 3072 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: ADECAF1B1EB79A9A8B2E8C4010E7E3DCAE4C22BB8905826F24D0FEDB3CFFB192
    Session-ID-ctx:
    Master-Key: 19C4733CD92428AF59D5A965F8BAC319C60486304365693173364DE2E37FFB6A710B5600A8E4B434FEC79DD0A734EFF8
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 4a 69 06 8e b1 d9 09 da-7a 37 58 74 d6 c3 1a 12   Ji......z7Xt....
    0010 - ee c1 8b 76 ad 75 43 b3-ab 8d 24 cf 85 31 38 60   ...v.uC...$..18`
    0020 - 66 f6 b3 3a 42 78 0d 55-fe 58 dc 75 1a 0e 9d 4e   f..:Bx.U.X.u...N
    0030 - 93 5f cb 71 22 46 03 4a-af a7 43 e5 a5 b8 f5 dc   ._.q"F.J..C.....
    0040 - 88 88 3f 42 54 19 f4 01-64 9d bd 6b 92 20 7d 86   ..?BT...d..k. }.
    0050 - bd 5d aa a6 62 72 b1 74-d7 58 bc 86 ee 13 84 07   .]..br.t.X......
    0060 - a8 3b 00 bc 0f 16 88 4c-98 fd 0a 63 e6 a5 1e f4   .;.....L...c....
    0070 - 8f e9 cf eb 0e c0 c6 a8-25 bf 4f 0c 2c ff 71 e3   ........%.O.,.q.
    0080 - 70 2d f4 2f 30 bd c0 98-ec 29 6c 76 72 e4 fe 45   p-./0....)lvr..E
    0090 - dd 61 94 90 c6 5d 0f 51-ca 4b 66 19 cb 13 27 b4   .a...].Q.Kf...'.

    Start Time: 1561067011
    Timeout   : 7200 (sec)
    Verify return code: 18 (self signed certificate)
    Extended master secret: no
---
write:errno=32
michael@rpi:~/test $

Edited June 20, 2019 by NotionCommotion

kicken · June 21, 2019

Well, I don't have a RPi to test with, but trying between my two desktops I didn't have any issues. Ran the server script on my windows desktop then used a fedora desktop to send data using s_client and everything came through just fine over multiple runs.

There was a bug related to TLS connections that was fixed in PHP 7.3.3. Looks like your on newer versions at this point though.

Edited June 21, 2019 by kicken

NotionCommotion · June 21, 2019

While I am working with PHP clients and a PHP server, I really only care about the PHP server as the clients will not be using PHP but c++. I think, however, the server is at fault and it is just easier for me to use a temporary PHP client. The production server is Centos7 VPN running PHP 7.3.5. RPi's with c++ client which are located on customer's 10.dot network behind their firewall which has outgoing access to the specific server IP only. They periodically error with broken pipe which has been tracked down to when attempting large message.

I did a little testing as shown below. Please confirm you sent a long string from a client to the PHP server script. If so, it seems like my PHP server script as well as ReactPHP's code is all good, but I need to change php.ini to allocate more memory or something similar. I don't really need to receive messages 10 million characters long, but do need to receive messages much longer than 2,000. Any ideas what needs to be changed?

Note sure if relevant, but I think the following might be a relevant clue. Sending a large string from the same machine as the server using the PHP client results in an error even when using openssl s_server. I did not expect this and guess that the lack of a slow network just allowed things to go too fast, but considering that it is an academic test, don't really care. Sending a long message from the same machine using openssl s_client to the php server also worked (even with messages as long as 10000000) which I didn't expect, but again it is not a real requirement. When sending a long message from the same machine using the php client to the php server, an error did result, and while this too is not a real requirement, it is very close to the real requirement of a PHP (actually c++) client sending a long message to a PHP server located on a different machine.

It is frustrating the lack of error logs. I am logging using syslog but nothing is their. Appreciate any suggestions.

Thanks

Legend:

MyServer: Centos7 VPN running PHP 7.3.5
RPiClient1: PHP 7.0.30-0
RPiClient2: PHP 7.3.4-2
OldClient: Centos6 VPN running PHP 5.6.
LaptopClient: Parrot laptop running PHP 7.0.26
YourServer: Windows desktop
YourClient: Fedora desktop

Tests

YourServer running PHP server script
1. Send large string using openssl s_client
  1. YourClient: Success
MyServer running openssl s_server
1. Send large string (100000 characters) using openssl s_client
  1. MyServer: Success
  2. RPiClient1: Success
  3. RPiClient2: Success
  4. LaptopClient: Success
  5. OldClient: Success
2. Send small string (1000 characters) using PHP client script
  1. MyServer: Success
  2. RPiClient1: Success
  3. RPiClient2: Success
  4. LaptopClient: Success
  5. OldClient: Success
3. Send large string (100000000 characters) using PHP client script
  1. MyServer: Failure
    1. Server responds: First downloads much data, but then: ERROR shutting down SSL CONNECTION CLOSED
    2. Client responds: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 96600352 bytes) in ... WritableResourceStream.php on line 150
  2. RPiClient1: Success
  3. RPiClient2: Success
  4. LaptopClient: Success
  5. OldClient: Failure
    1. Server response: Downloads little to no error and shuts down SSL and closes connection.
    2. Client response: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 100000001 bytes) in /home/NotionCommotion/socket_testing/simple_common.php on line 37
MyServer running PHP server script
1. Send small string (100 characters) using openssl s_client
  1. MyServer: Success
  2. RPiClient1: Success
  3. RPiClient2: Success
  4. LaptopClient: Success
  5. OldClient: Success
2. Send large string (100000 characters) using openssl s_client
  1. MyServer: Success
  2. RPiClient1: Failure - write:errno=32
  3. RPiClient1: Failure - write:errno=32
  4. LaptopClient: Failure - read:errno=0.
  5. OldClient: Failure - read:errno=0
3. Send small string (1000 characters) using PHP client script
  1. MyServer: Success
  2. RPiClient1: Success
  3. RPiClient2: Success
  4. LaptopClient: Success
  5. OldClient: Success
4. Send large string (100000000 characters) using PHP client script
  1. MyServer: Failure
    1. Server responds: First downloads 22978560 characters, but then see onEnd and onClose before complete but no error displayed.
    2. Client responds: Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 77021472 bytes) in ... WritableResourceStream.php on line 150
  2. RPiClient1: Failure. Neither server or client appear to have an error, but no data is received.
  3. RPiClient2: Failure. Neither server or client appear to have an error, but no data is received.
  4. LaptopClient: Failure. Neither server or client appear to have an error, but no data is received.
  5. OldClient: Failure. Client responded with: PHP Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 100000001 bytes) in /home/NotionCommotion/socket_testing/simple_common.php on line 37

Notes

When I close the PHP client, openssl s_server responds with ERROR shutting down SSL CONNECTION CLOSED, but this seems normal.
I see "onDrain" echo'd to the terminal when client completes a long message which I think is expected.
openssl s_client displays Verify return code: 18 (self signed certificate) for all requests which seems normal.
openssl s_client displays "Extended master secret: no" for the LaptopClient and both RPiClients, but not the OldClient or the server acting as a client. Doesn't seem a big deal.

kicken · June 21, 2019

Your fatal errors are just due to you trying to generate too large of a string for your current memory_limit to allow. Increasing the memory would resolve that. If you're generating much larger strings than you'll actually need in the field than it doesn't matter too much.

For my testing I was just using the same 100,000 character string you had as an example. I also tried with 300,000 and had no issue.

It appears as your issues only occur across the network, so you might just have a faulty network that is causing connections to be dropped. That could explain the randomness of where the data is interrupted and how it only affects larger data sets as they would take longer to transmit than smaller ones.

If you want to test the network in general, take PHP out of the equation entirely and just use openssl s_client and openssl s_server together. For example, generate a large file with

dd if=/dev/urandom of=/dev/stdout bs=1024 count=1024 | openssl base64 -a >bigdata.txt

Then on one machine start a server

openssl s_server -cert server_combined.pem

And on the other send the data

openssl s_client -connect 192.168.1.10:4433 -ign_eof < bigdata.txt

I did notice in my testing that for some reason without the -ign_eof s_client would sometimes not send the complete file with larger data sets. Not sure why but I suspect it's just an s_client issue.

NotionCommotion · June 21, 2019

Thanks kicken,

My memory_limit is set at 128M which reflects why the errors happen when they do and that makes sense. I am confused why I use so much memory when receiving a relatively short message (even get it for a 22,176 real message). What is your memory_limit set at? Should I increase, or do you think it is a different issue?

I think my test 2.1 did take PHP out of the picture and as seen I had success for all machines. Disagree?

#MyServer running openssl s_server
openssl s_server -port 1338 -cert test_ss_crt_pk.pem
#Send large string (100000 characters) using openssl s_client
printf '=%.0s' {1..100000} | openssl s_client -connect 12.345.67.890:1338

kicken · June 21, 2019

1 hour ago, NotionCommotion said:

My memory_limit is set at 128M which reflects why the errors happen when they do and that makes sense. I am confused why I use so much memory when receiving a relatively short message (even get it for a 22,176 real message). What is your memory_limit set at? Should I increase, or do you think it is a different issue?

Mine is 512MB.

I'd guess there's a memory leak in your code somewhere. Lots of little packets will eventually add up if your storing them forever. I'd say look over the code and make sure you're not holding on to things longer than needed. For example, make sure when a client disconnects you clean up the resources for it instead of just letting it linger in an array or whatever.

If you want to post or PM me your code I can look over it and see if I spot anything.

NotionCommotion · June 22, 2019

Thank you for the offer to review the scripts, but there are no other scripts. I stopped all PHP scripts running on server to make sure there wasn't some other script and am only executing the following server.php on the command line.

I also changed php.ini to use 512MB for memory_limit but no change, and have since set back to 128MB and will change on a per script basis. What is odd is it doesn't change memory_get_usage(true)'s output. Is this to be expected?

I've also tried using another server, and I am able to send large packets. And since when using openssl s_server on the original server, it has to be a PHP configuration issue or memory issue, no?

I've looked over the open_ssl s_client outputs and nothing looks out of line, but I can post them if you think it will provide a clue. One noticeable difference, however, is my script's output on memory. Can it be the cause?

Temp server on start:    MemTotal: 5576800 kB, MemFree: 3474300 kB, MemAvailable: 4835268 kB, phpMemUsage: 365096, phpMemPeak: 403208, phpMemAvail: 2097152
Temp server on complete: MemTotal: 5576800 kB, MemFree: 3473992 kB, MemAvailable: 4834972 kB, phpMemUsage: 913392, phpMemPeak: 984248, phpMemAvail: 2097152

Real server on start:    MemTotal: 2037840 kB, MemFree: 398364 kB, MemAvailable: 1239124 kB, phpMemUsage: 432712, phpMemPeak: 471392, phpMemAvail: 2097152
Real server on complete: MemTotal: 2037840 kB, MemFree: 398116 kB, MemAvailable: 1239128 kB, phpMemUsage: 1078376, phpMemPeak: 1165856, phpMemAvail: 2097152

<?php
function getMemInfo($keys=[]){
    $raw = explode(PHP_EOL, file_get_contents("/proc/meminfo"));
    $meminfo = [];
    foreach ($raw as $line) {
        if($line) {
            $line = explode(":", $line);
            $meminfo[$line[0]] = trim($line[1]);
        }
    }
    return $keys?array_intersect_key($meminfo, array_flip($keys)):$meminfo;
}

function memory(){
    $meminfo = getMemInfo(['MemTotal','MemFree','MemAvailable']);
    $meminfo['phpMemUsage']=memory_get_usage();
    $meminfo['phpMemPeak']=memory_get_peak_usage();
    $meminfo['phpMemAvail'] =memory_get_usage(true);
    $meminfo = implode(', ', array_map(
        function ($v, $k) { return sprintf("%s: %s", $k, $v); },
        $meminfo,
        array_keys($meminfo)
    ));
    echo $meminfo.PHP_EOL;
    syslog(LOG_ERR, $meminfo);
}
ini_set('display_startup_errors', '1');
ini_set('display_errors', '1');
error_reporting(E_ALL);
ini_set ( 'memory_limit' , '-1' );
memory();
require 'vendor/autoload.php';
$loop = React\EventLoop\Factory::create();
$socket = new React\Socket\Server('0.0.0.0:1338', $loop);
$socket = new React\Socket\SecureServer($socket, $loop, ['local_cert' => 'cert_with_priv_key.pem'] );


$socket->on('connection', function (React\Socket\ConnectionInterface $connection) use($loop) {
    $connection->on('data', function ($data) {
        static $totalLng=0;
        $curLng=strlen($data);
        $totalLng+=$curLng;
        echo("onData: current: $curLng  total: $totalLng".PHP_EOL);
    });
    $connection->on('error', function (\Exception $e) {echo("onError: ".$e->getMessage().PHP_EOL);});
    $connection->on('close', function() {echo("onClose".PHP_EOL);});
    $connection->on('end', function() {echo("onEnd".PHP_EOL);
    });
    $connection->on('drain', function() {echo("onDrain".PHP_EOL);});
    $connection->on('pipe', function() {echo("onPipe".PHP_EOL);});
    $loop->addPeriodicTimer(1, function ($timer) {
        $data='Hello!';
        memory();
    });
});
$socket->on('error', function (\Exception $e) {echo('onServerError: '.$e->getMessage().PHP_EOL);});
$loop->run();

kicken · June 22, 2019

1 hour ago, NotionCommotion said:

What is odd is it doesn't change memory_get_usage(true)'s output. Is this to be expected?

The memory_limit setting won't affect how much memory a script uses, just how much it's allowed to use. As such it doesn't have any direct effect on memory_get_usage. The difference between memory_get_usage() and memory_get_usage(true) is whether you're looking at PHP's internal memory pool or the total memory allocated to the process by the OS. When PHP allocates memory from the OS it generally doesn't return it to the OS when it's done, it just marks it free internally and re-uses it in the future.

Are you encountering issues with they very first connection to your server, or is it only after a period of time that the errors occur and/or it crashes due to memory limits? You do have one memory leak in your sample server above with the periodic timer, but it'd take time and a lot of connections for that to build up to a problem. To solve that, you need to cancel your timers when the connection is closed.

/** @var TimerInterface $timer */
$timer = null;
$connection->on('close', function() use (&$timer, $loop){
  echo("onClose" . PHP_EOL);
  $loop->cancelTimer($timer);
});

$timer = $loop->addPeriodicTimer(1, function ($timer) {
  $data='Hello!';
  memory();
});

NotionCommotion · June 22, 2019

Thanks for the info on memory limits as well as memory leaks.

I am not encountering issues when first connecting or after long periods of time. I only encounter them after the client sends a string longer than around 2,500 characters. And the server doesn't crash, but emits the end event and then the close event, and then the client complains of a broken pipe. Any thoughts where applicable information would be logged?

As another test, I made the server send a big string to the client, and did not experience the issue.

The table below shows the memory output of the server.php script I posted earlier where Start is before connecting to the client and Complete is after the clients message is received or the server breaks the pipe. Only the Parrot server can accept a big message from a client. Don't know whether this means anything.

Thanks again for your help on it.

|               |          |       MemTotal |        MemFree |           MemAvailable |             phpMemUsage |            phpMemPeak |          phpMemAvail |
|--------------:|---------:|---------------:|---------------:|-----------------------:|------------------------:|----------------------:|---------------------:|
|   Real Server |    Start |     2,037,840  |       398,364  |             1,239,124  |                432,712  |              471,392  |           2,097,152  |
|   Real Server | Complete |     2,037,840  |       398,116  |             1,239,128  |              1,078,376  |            1,165,856  |           2,097,152  |
|     Pi Server |    Start |       948,304  |       598,784  |               779,740  |                388,960  |              425,728  |           2,097,152  |
|     Pi Server | Complete |       948,304  |       598,784  |               779,744  |                795,752  |              878,672  |           2,097,152  |
| Parrot Server |    Start |     5,576,800  |     3,474,300  |             4,835,268  |                365,096  |              403,208  |           2,097,152  |
| Parrot Server | Complete |     5,576,800  |     3,473,992  |             4,834,972  |                913,392  |              984,248  |           2,097,152  |
|   Real/Parrot |    Start |            37% |            11% |                    26% |                    119% |                  117% |                 100% |
|   Real/Parrot | Complete |            37% |            11% |                    26% |                    118% |                  118% |                 100% |
|   Real/Parrot |    Start |            17% |            17% |                    16% |                    107% |                  106% |                 100% |
|   Real/Parrot | Complete |            17% |            17% |                    16% |                     87% |                   89% |                 100% |

NotionCommotion · June 22, 2019

I am grasping at straws now. Can it be insufficient VPS memory, obsolete openssl, or non-included php module?

On my real server, openssl version shows OpenSSL 1.0.2k-fips 26 Jan 2017, but on the Parrot server that works, openssl version shows OpenSSL 1.1.0g 2 Nov 2017. Was thinking of compiling latest version of openssl, but think there is no way this is the cause. Or maybe?

Regarding hp modules, I only included mod_php php-cli php-mysqlnd php-json php-xml php-common php-gd php-mbstring php-mcrypt php-cli php-xmlrpc php-zip php-pecl-ds. Maybe?

Maybe a more sane approach is to start with when the event is first witnessed and go back? Made the following change:

    $connection->on('end', function() {
        echo("onEnd".PHP_EOL);
        debug_print_backtrace();
    });

which results in:

Use of closure, recursion, decoupling using emitters, and call_user_func makes this hard to follow. Is one able to tell from this what the initiating event which caused the connection to end is?

kicken · June 23, 2019

3 hours ago, NotionCommotion said:

On my real server, openssl version shows OpenSSL 1.0.2k-fips 26 Jan 2017, but on the Parrot server that works, openssl version shows OpenSSL 1.1.0g 2 Nov 2017. Was thinking of compiling latest version of openssl, but think there is no way this is the cause. Or maybe?

The -fips suffix has me wondering if there's a difference that could be causing issues. Seems unlikely, but wouldn't hurt to get updated to the latest version. If that's not possible via the system package manager then compiling your own copy would be necessary. You might have to re-compile PHP, or at least the openssl extension to make it use the updated version. It's been a long time since I compiled my own versions of either so I'm not certain.

3 hours ago, NotionCommotion said:

Can it be insufficient VPS memory, obsolete openssl, or non-included php module?

Seems unlikely to be a memory related issue, and if the php code works then you should have all the modules you need. If you didn't you'd have undefined function errors.

kicken · June 23, 2019

What versions of the various react/* packages are you using exactly? Run composer info to find out.

Based on the backtrace it seems you are probably using react/event-loop v0.4.3, and I'm guessing the other packages you have are from around the same time frame. I downgraded my install to react/socket v0.7.2 and whatever secondary packages match that release and was able to encounter some instances where a client would connect then immediately disconnect causing various errors to be returned by s_client.

I suspect you're hitting this issue: Only close stream when the we reached end of the stream. Try updating your react requirements to their latest versions and see if that resolves your issues.

NotionCommotion · June 23, 2019

12 hours ago, kicken said:

What versions of the various react/* packages are you using exactly? Run composer info to find out.

I owe you big time! Wow, what a rookie mistaken on my part. I just got focused on going down some path and missed what should have been the obvious first check. Thank you!

#Before
2tvenom/cborencode  1.0.0   CBOR encoder for PHP
evenement/evenement v3.0.1  Événement is a very simple event dispatching library for PHP
react/cache         v0.5.0  Async, Promise-based cache interface for ReactPHP
react/dns           v0.4.17 Async DNS resolver for ReactPHP
react/event-loop    v0.4.3  Event loop abstraction layer that libraries can use for evented I/O.
react/promise       v2.7.1  A lightweight implementation of CommonJS Promises/A for PHP
react/promise-timer v1.5.1  A trivial implementation of timeouts for Promises, built on top of ReactPHP.
react/socket        v0.8.12 Async, streaming plaintext TCP/IP and secure TLS socket server and client connections for ReactPHP
react/stream        v0.7.7  Event-driven readable and writable streams for non-blocking I/O in ReactPHP

#After
2tvenom/cborencode  1.0.0   CBOR encoder for PHP
evenement/evenement v3.0.1  Événement is a very simple event dispatching library for PHP
react/cache         v0.5.0  Async, Promise-based cache interface for ReactPHP
react/dns           v0.4.17 Async DNS resolver for ReactPHP
react/event-loop    v1.1.0  ReactPHP's core reactor event loop that libraries can use for evented I/O.
react/promise       v2.7.1  A lightweight implementation of CommonJS Promises/A for PHP
react/promise-timer v1.5.1  A trivial implementation of timeouts for Promises, built on top of ReactPHP.
react/socket        v1.2.1  Async, streaming plaintext TCP/IP and secure TLS socket server and client connections for ReactPHP
react/stream        v1.1.0  Event-driven readable and writable streams for non-blocking I/O in ReactPHP

12 hours ago, kicken said:

I suspect you're hitting this issue: Only close stream when the we reached end of the stream. Try updating your react requirements to their latest versions and see if that resolves your issues.

Your suspect is correct. I too ran it down to #4. Another topic, but I sometime have a difficult time interpreting backtraces and need to get better.

#0  {closure}() called at [/var/www/test/public/test/wireshark/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php:123]
#1  Evenement\EventEmitter->emit() called at [/var/www/test/public/test/wireshark/vendor/react/stream/src/Util.php:71]
#2  React\Stream\Util::React\Stream\{closure}() called at [/var/www/test/public/test/wireshark/vendor/evenement/evenement/src/Evenement/EventEmitterTrait.php:123]
#3  Evenement\EventEmitter->emit() called at [/var/www/test/public/test/wireshark/vendor/react/stream/src/DuplexResourceStream.php:196]
#4  React\Stream\DuplexResourceStream->handleData()
#5  call_user_func() called at [/var/www/test/public/test/wireshark/vendor/react/event-loop/src/StreamSelectLoop.php:236]
#6  React\EventLoop\StreamSelectLoop->waitForStreamActivity() called at [/var/www/test/public/test/wireshark/vendor/react/event-loop/src/StreamSelectLoop.php:205]
#7  React\EventLoop\StreamSelectLoop->run() called at [/var/www/test/public/test/wireshark/server.php:65]

NotionCommotion · June 23, 2019

kicken, why did you pass $timer by reference? Only because you defined $timer after this line of code? Thank you

23 hours ago, kicken said:
Are you encountering issues with they very first connection to your server, or is it only after a period of time that the errors occur and/or it crashes due to memory limits? You do have one memory leak in your sample server above with the periodic timer, but it'd take time and a lot of connections for that to build up to a problem. To solve that, you need to cancel your timers when the connection is closed.
/** @var TimerInterface $timer */
$timer = null;
$connection->on('close', function() use (&$timer, $loop){
  echo("onClose" . PHP_EOL);
  $loop->cancelTimer($timer);
});

$timer = $loop->addPeriodicTimer(1, function ($timer) {
  $data='Hello!';
  memory();
});

kicken · June 23, 2019

7 hours ago, NotionCommotion said:

Only because you defined $timer after this line of code?

Yea, pretty much. If you move the addPeriodicTimer call above the on call then you wouldn't need to pass by reference.

Sign In

Cannot send large TLS packages over socket

Recommended Posts

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Link to comment

Share on other sites

Join the conversation

Important Information