cURL both a file and POST data

[NotionCommotion]

The following script receives an array of files and optional POST data.  The POST data requires http_build_query(), however, this seems to prevent sending the files.  How can this be resolved?  Thank you

PS.  You think it is worthless to delete no longer needed files from the tmp directory?
 

    public function proxyFile(array $files, array $data=[])
    {
        $this->debugDump($data, 'proxyFile data');

        $url=$this->getUrl();

        foreach($files as $name=>$file) {
            $data[$name] = new \CURLFile($file['tmp_name'],$file['type'],$file['name']);
        }

        $data=http_build_query($data);//Doesn't work with files??? http_build_query($data),
        $this->debugDump($data, 'proxyFile data');

        $options=[
            CURLOPT_URL             =>  $url,
            CURLOPT_HTTPHEADER      =>  ['X-Secret-Key: '.$this->datalogger['key']],
            CURLOPT_POST            =>  1,
            CURLOPT_POSTFIELDS      =>  $data,
            CURLOPT_RETURNTRANSFER  =>  true,     // return web page
            CURLOPT_HEADER          =>  false,    // don't return headers
            CURLOPT_FOLLOWLOCATION  =>  true,     // follow redirects
            CURLOPT_ENCODING        =>  "",       // handle all encodings
            CURLOPT_USERAGENT       =>  "unknown",// who am i
            CURLOPT_AUTOREFERER     =>  true,     // set referrer on redirect
            CURLOPT_CONNECTTIMEOUT  =>  120,      // timeout on connect
            CURLOPT_TIMEOUT         =>  120,      // timeout on response
            CURLOPT_MAXREDIRS       =>  10,       // stop after 10 redirects
        ];
        $ch      = curl_init();
        curl_setopt_array( $ch, $options );
        $rsp=['rsp'=>curl_exec( $ch ),'errno'=>curl_errno($ch),'code'=>curl_getinfo($ch, CURLINFO_HTTP_CODE),'error'=>false];
        if($rsp['errno']) {
            $rsp['error']=curl_error($ch);
        }
        curl_close( $ch );
        //syslog(LOG_INFO, json_encode($rsp));
        if($rsp['errno']) {
            $rsp['code']=400;
            $obj=($rsp['errno']==6)
            ?['message'=>'Invalid Datalogger IP','code'=>1]
            :['message'=>"cURL Error: $rsp[error] ($rsp[errno])"];
        }
        elseif(!isset($rsp['rsp']) || !$rsp['rsp']) {
            $obj=null;
        }
        else {
            $obj=json_decode($rsp['rsp']);
            if(json_last_error() != JSON_ERROR_NONE) {
                syslog(LOG_ERR, 'proxyFile(). '.$rsp['rsp']);
                $rsp['code']=400;
                $obj=['message'=>'Invalid JSON response','code'=>1];
            }
        }
        foreach($_FILES as $file) { // Is this necessary?
            unlink($file['tmp_name']);
        }
        return [$obj, $rsp['code']];
    }

 

[requinix]

You can't http_build_query an array of CURLFile objects. Think about it.

POSTFIELDS can be an array too.

[NotionCommotion]

54 minutes ago, requinix said:

You can't http_build_query an array of CURLFile objects. Think about it.

POSTFIELDS can be an array too.

Yes, I realize I can not encode objects.  But when I don't, the post variables come over as a text "array" and not an array of values.  The array in question is a sequential array (i.e. id[], id[], etc).  I was thinking I might need to loop through the array and only encode certain parts, but was not successful.  

[requinix]

Is the method inheriting $data from somewhere? So the problem is you have multidimensional data and you need to add in CURLFile objects. That's a bother. I vaguely remember cURL not supporting that.

Good news: check out this user comment.

[NotionCommotion]

I would rather use http_build_query() if possible.  https://gist.github.com/maxivak/18fcac476a2f4ea02e5f80b303811d5f seems to give a solution.  Think it is a preferred solution?

 

[requinix]

Building the request body manually is worse.

1. You cannot use CURLFile and http_build_query. You have to choose.
2. You know what http_build_query does, right? You know what passing an array for POSTFIELDS does, right?

If you've decided that the hammer for this screw will be http_build_query then you're in for a hard time.

[NotionCommotion]

Thanks requinix,

Even before seeing your reply, I came to the same conclusion.

I thought http_build_query converts an array such that it could be sent in the URL.  Which never made sense to me that I would use to pass to POSTFIELDS.  I mean, is it in the body or the url?  Granted, I need to better understand the http protocol.  And if in the url, why am I having these problems?  I know that a post request could also have $_GET values which are presumably passed not in the body but the url.

Dang, wife is telling me I need to take the dog for a pop.
 

function convert($x) {
    $rs=[];
    if(!is_array($x) && !is_object($x)) {
        throw new Exception('not an array or object');
    }
    foreach($x as $key=>$value) {
        if(is_array($value) || is_object($value)) {
            $rs[$key]=convert($value);
        }
        else {
            $rs[$key]=$value;
        }
    }
    return $rs;
}

<?php
echo('$_POST:<br>');
var_dump($_POST);
echo('<br>$_GET:<br>');
var_dump($_GET);
echo('<br>');

$a=['x'=>123, 'y'=>[5,7,9,0], 'z'=>['a'=>123,'b'=>321,'c'=>111]];
$http_build_query=http_build_query($a);
echo('$http_build_query: '.$http_build_query.'<br>');
$converted=convert($a);
echo('$converted '.var_dump($converted).'<br>');
$x=123;

?>
<form method="post" action="<?php echo $_SERVER['PHP_SELF']; ?>?a=1&b=2&c=3">
   <input type="text" name="name"><br>
   <input type="submit" name="submit" value="Submit Form"><br>
</form>

 

[requinix]

13 minutes ago, NotionCommotion said:

I thought http_build_query converts an array such that it could be sent in the URL.

PHP parses query strings automatically (for $_GET) by using parse_str(). http_build_query() is the opposite of that.

It's great for building query strings that you want to embed in links or forms or wherever that you need to get the query string as a, well, string. But the situation there is that you need it as a string. Like if you wanted to pass $_GET to another script you would

<a href="/path/to/script.php?<?=htmlspecialchars(http_build_query($_GET))?>">Link</a>

Not only is creating that query string manually yourself difficult with something dynamic like $_GET (you don't know the parameters), you'd also have to deal with URL-encoding stuff. http_build_query does all that. To make this even more complicated, consider

<a href="/path/to/script.php?<?=htmlspecialchars(http_build_query(["get" => $_GET]))?>">Link</a>

then in the other script

parse_str($_GET["get"], $get);

Point is, it takes a $_GET-type array and gives you a string, and you use it because you want the string. If you don't want or need a string, like you wouldn't with CURLOPT_POSTFIELDS because cURL will do that for you, then you wouldn't use it.

13 minutes ago, NotionCommotion said:

Which never made sense to me that I would use to pass to POSTFIELDS.  I mean, is it in the body or the url?

Yes.

There are two types of POST bodies, so to speak. The normal one is "application/x-www-form-urlencoded" and sends data that looks and works just like a query string, except it goes in the request body instead of in the URL. The other one, needed by file uploads, is "multipart/form-data" and is much less compact and incidentally much more human readable.

If you're using cURL for an upload then you'll have to get that second format. http_build_query won't help you there.

13 minutes ago, NotionCommotion said:

And if in the url, why am I having these problems?  I know that a post request could also have $_GET values which are presumably passed not in the body but the url.

The problems are indeed because it's not going in the URL. And yes, $_GET values are in the URL - $_GET is always from the URL, $_POST is always the POSTed data (if it's one of the two body types I mentioned above). The REQUEST_METHOD doesn't switch where data goes but rather instructs PHP that it should look for the presence of POSTed data. Kinda.

[kicken]

Looking at that comment on PHP.net linked earlier, it seems like you just need to make sure your $data array is a single dimension.   You can use http_build_query to do some of the work of converting your potentionaly multi-dimensional array into a single dimension, then add your files to the end.

For example:

<?php

$data = [
    'name' => 'Kicken'
    , 'date' => [
        'month' => 10
        , 'day' => 15
        , 'year' => 2018
    ]
];

$files = [
    'photo' => [
        'tmp_name' => '/tmp/blah.jpg'
        , 'type' => 'image/jpeg'
        , 'name' => 'blah.jpg'
    ]
    , 'cv' => [
        'tmp_name' => '/tmp/cv.txt'
        , 'type' => 'text/plain'
        , 'name' => 'cv.txt'
    ]
];


$postData = [];
foreach (explode('&', http_build_query($data)) as $pair){
    list($name, $value) = explode('=', $pair, 2);
    $postData[$name] = $value;
}

foreach ($files as $name=>$file){
    $postData[$name] = new \CURLFile($file['tmp_name'],$file['type'],$file['name']);
}

var_dump($postData);

Output:

array(6) {
  'name' =>
  string(6) "Kicken"
  'date%5Bmonth%5D' =>
  string(2) "10"
  'date%5Bday%5D' =>
  string(2) "15"
  'date%5Byear%5D' =>
  string(4) "2018"
  'photo' =>
  class CURLFile#1 (3) {
    public $name =>
    string(13) "/tmp/blah.jpg"
    public $mime =>
    string(10) "image/jpeg"
    public $postname =>
    string(8) "blah.jpg"
  }
  'cv' =>
  class CURLFile#2 (3) {
    public $name =>
    string(11) "/tmp/cv.txt"
    public $mime =>
    string(10) "text/plain"
    public $postname =>
    string(6) "cv.txt"
  }
}

You could also make your own version of http_build_query that's aware of CURLFile if that'd make things easier.  Then you could just build up your $data array with the file objects where they need to be and then collapse it with your custom function.

edit: Might need to urldecode() the name/value pairs after the http_build_query, not sure on that.

Edited October 16, 2018 by kicken

[requinix]

2 minutes ago, kicken said:

edit: Might need to urldecode() the name/value pairs after the http_build_query, not sure on that.

Probably.

[NotionCommotion]

Thanks kicken,

Why do you think I will need urldecode()?  See below outputs.

Also, back to my original line of thought, any reason (other than being non-typical and thus confusing maybe) that the data array couldn't be transferred in the url and arrive as GET parameters, and CURLOPT_POSTFIELDS used solely for files?

With uldecode():

Array
(
    [name] => Kicken
    [date[month]] => 10
    [date[day]] => 15
    [date[year]] => 2018
    [date[extraSeq][0]] => a 1
    [date[extraSeq][1]] => b 1
    [date[extraSeq][2]] => c 1
    [date[extraAssoc][a 1]] => A 1
    [date[extraAssoc][b 1]] => B 1
    [date[extraAssoc][c 1]] => C 1
)
 

Without uldecode():

Array
(
    [name] => Kicken
    [date%5Bmonth%5D] => 10
    [date%5Bday%5D] => 15
    [date%5Byear%5D] => 2018
    [date%5BextraSeq%5D%5B0%5D] => a+1
    [date%5BextraSeq%5D%5B1%5D] => b+1
    [date%5BextraSeq%5D%5B2%5D] => c+1
    [date%5BextraAssoc%5D%5Ba+1%5D] => A+1
    [date%5BextraAssoc%5D%5Bb+1%5D] => B+1
    [date%5BextraAssoc%5D%5Bc+1%5D] => C+1
)

[NotionCommotion]

2 hours ago, NotionCommotion said:

Why do you think I will need urldecode()?  See below outputs.

Because it won't work if I don't.