Jump to content

How to validate/test a REST API against a spec?

NotionCommotion

By NotionCommotion
in PHP Coding Help

 Share

Recommended Posts

NotionCommotion Prolific Member

NotionCommotion

Posted
Posted

I recently created an OpenAPI https://openapis.org/ (aka Swagger) spec and then built an application to implement that spec.

 

How can I test my application to ensure that it accurately implements the spec?

 

Are there any tools to assist?

 

I looked around and couldn't find anything I liked, and started building my own.  To test, you specify an endpoint and any data to send as well as the expected HTTP status code to be returned, and the class confirms that all required inputs are provided, that no inputs not described by the spec are provided, that all required outputs are returned, and that no outputs not described by the spec are returned, and also that the returned status code was returned.

 

I keep on thinking, however, that there has to be something good already built.  Did I totally waste my time? 

<?php
require('../classes/TestAPI.php');

$test=new TestAPI(file_get_contents ('/var/www/swagger/swagger-ui/dist/swagger.json'));
$test->preloadHeaders(['X-GreenBean-Key'=>'main_key']);

$rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"invalid"}',true),200);
$rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"invalid"}',true),422);
$rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"pie1"}',true),200);
$id=$rsp->id;
$rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200);
$rsp=$test->test('/charts/{id}','put',['id'=>$id],['name'=>'title','value'=>'My Title'],204);
$rsp=$test->test('/charts/{id}','post',['id'=>$id],['points_id'=>98,'legend'=>'Legend for Speed'],204);
$rsp=$test->test('/charts/{id}','post',['id'=>$id],['points_id'=>99,'legend'=>'Legend for Cost'],204);
$rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200);
$rsp=$test->test('/charts/{id}/{legend}','delete',['id'=>$id,'legend'=>'Legend for Speed'],[],204);
$rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200);
$rsp=$test->test('/charts/{id}','delete',['id'=>$id],[],204);
$rsp=$test->test('/charts/{id}','get',['id'=>$id],[],404);
<?php
class TestAPI
{
    private $obj, $repeat, $data=[], $headers=[], $source=[];

    public function __construct($json)
    {
        $this->obj = json_decode($json,true);
        if(json_last_error() !== JSON_ERROR_NONE) {
            throw new Exception('Invalid JSON');
        }
        $this->repeat=true;
        while($this->repeat) {
            $this->repeat=false;
            $this->obj=$this->substituteReferences($this->obj);
        }
        unset($this->obj['definitions']);
        unset($this->obj['parameters']);
        unset($this->obj['responses']);
        $this->obj=$this->substituteAllOf($this->obj);
    }
    public function preloadData(array $arr)
    {
        // Preload data which will be sent with every request
        foreach($arr as $key=>$value) {
            $this->data[$key]=$value;
        }
    }
    public function preloadHeaders(array $arr)
    {
        // Preload headers which will be sent with every request
        foreach($arr as $key=>$value) {
            $this->headers[]=strtolower(str_replace("_","-",$key)).': '.$value;
        }
    }
    public function get()
    {
        return $this->obj;
    }

    private function getValue($name)
    {
        return isset($this->data[$name])?$this->data[$name]:false;
    }

    private function validateResponse($rsp,$config)
    {
        $errors=[];
        if(!$rsp || !is_array($rsp) || count($rsp)!==2 || !isset($rsp['code']) || !isset($rsp['data'])) {
            $errors[]='Invalid response from server.';
        }
        elseif(!isset($config['responses'][$rsp['code']])) {
            $errors[]="HTTP code $rsp[code] is not supported.";
        }
        else {
            $data=json_decode($rsp['data']);
            $type=isset($config['responses'][$rsp['code']]['schema'])
            ?isset($config['responses'][$rsp['code']]['schema']['type'])?$config['responses'][$rsp['code']]['schema']['type']:'object'
            :'NULL';
            if(json_last_error() !== JSON_ERROR_NONE) {
                $errors[]="Server response is not JSON.";
            }
            elseif(gettype($data)!=$type) {
                $errors[]="Response type expected was $type, however, ".gettype($data)." was received";
            }
            else {
                switch($type) {
                    case 'array':
                        /*
                        foreach($x as $s) {

                        }
                        */
                        $status='OKAY';
                        break;
                    case 'object':
                        $data=(array)$data;
                        $required=isset($config['responses'][$rsp['code']]['schema']['required'])?$config['responses'][$rsp['code']]['schema']['required']:[];
                        $properties=isset($config['responses'][$rsp['code']]['schema']['properties'])?$config['responses'][$rsp['code']]['schema']['properties']:[];
                        $propertyNames=array_keys($data);
                        if($missing=array_diff($required,$propertyNames)) {
                            $errors[]='Missing required response properties '.implode(', ',$missing);
                        }
                        if($toMany=array_diff($propertyNames,array_keys($properties))) {
                            $errors[]='Unexpected response properties '.implode(', ',$toMany);
                        }
                        else {
                            //Don't validate response types if too many were given
                            foreach($data as $property=>$value) {
                                $type=gettype($value);
                                if($type!=$properties[$property]['type']) {
                                    $errors[]="Respond property $property is type $type but should have been type $properties[$property][type].";
                                }
                            }
                        }
                        break;
                    case 'string':
                        break;
                    case NULL:
                        break;     // Test for null?
                    default: $status="Non-supported response type $type.  This should never happen!";
                }
            }
        }
        return $errors;
    }

    private function substitute($path, $parameters) {
        foreach ($parameters as $key => $value)
        {
            $path = str_replace('{'.$key.'}', urlencode($value), $path);
        }
        return $path;
    }

    public function testAll()
    {
        foreach($this->obj['paths'] as $path=>$methods) {
            foreach ($methods as $method=>$config) {
                $url=$this->obj['host'].$this->obj['basePath'].$path;
                $rsp=$this->CallAPI($method, $url, $this->data, $this->headers);
                $errors=$this->validateResponse($rsp,$config);
                echo("<p>URL: $url  Method: $method Status: ".(empty($errors)?'OKAY':'ERROR')."</p>");
                if($errors) {echo('<pre>'.print_r($errors,1).'</pre>');}
                //echo('<pre>'.print_r($rsp,1).'</pre>');
            }
            //echo('<pre>'.print_r($path,1).'</pre>');
        }

    }

    public function test($path,$method,array $params, array $data, $expected=null)
    {
        $url=$this->obj['host'].$this->obj['basePath'].($params?$this->substitute($path,$params):$path);
        $rsp=$this->CallAPI($method, $url, array_merge($data,$this->data), $this->headers);
        echo("<h3><u>REQUEST</u></h3><p>Path: $path</p><p>Method: $method</p><p>Parameters: ".json_encode($params)."</p><p>Data: ".json_encode($data));
        echo("<h3><u>RESPONSE</u></h3><p>Code: $rsp[code]</p><p>Data: $rsp[data]");
        echo("<h3><u>VALIDATION</u></h3>");
        if(isset($this->obj['paths'][$path][$method])) {
            $errors=$this->validateResponse($rsp,$this->obj['paths'][$path][$method]);
            if($expected && $expected!=$rsp['code']) {
                $errors[]='HTTP Code does not match expected code "'.$expected.'".';
            }
            if($errors) {
                echo('<p>ERROR</p><pre>'.print_r($errors,1).'</pre>');
            }
            else{
                echo('<p>Valid</p>');
            }
        }
        else {
            echo("INABILITY TO VALIDATE: INVALID PATH ($path) OR METHOD ($method)");
        }
        echo('<hr>');
        return json_decode($rsp['data']);
    }

    private function substituteReferences($o)
    {
        foreach($o as $k=>$v) {
            if(is_array($v)){
                $o[$k]=$this->substituteReferences($v);
            }
            elseif($k==='$ref') {
                if(substr($v,0,2)!='#/') throw new Exception('Invalid reference: Bad prefix.');
                $p=explode('/',substr($v,2));
                if(count($p)!=2) throw new Exception('Invalid reference: Incorrect count.');
                if(empty($this->obj[$p[0]])) throw new Exception('Invalid reference: Property type doesn\'t exist.');
                if(empty($this->obj[$p[0]][$p[1]])) throw new Exception('Invalid reference: Property doesn\'t exist.');
                $o=$this->obj[$p[0]][$p[1]];
                $this->repeat=true;
            }
            elseif($k==='in' && $v==='header'  && isset($o['name'])) {
                //Make compatible
                $o['name']=strtolower(str_replace("_","-",$o['name']));
            }
        }
        return $o;
    }
    private function substituteAllOf($o)
    {
        foreach($o as $k=>$v) {
            if($k==='allOf') {
                $properties=[];
                $required=[];
                foreach ($o['allOf'] as $key=>$item) {
                    $properties=array_merge($properties, $item['properties']);
                    if(isset($item['required'])) {
                        $required=array_merge($required, $item['required']);
                    }
                }
                unset($o['allOf']);
                $o['properties']=$properties;
                $o['required']=$required;
            }
            elseif(is_array($v)){
                $o[$k]=$this->substituteAllOf($v);
            }
        }
        return $o;
    }
    private function CallAPI($method, $url, $data, array $headers=[], $options=[], $debug=false)
    {
        $options=$options+[    //Don't use array_merge since it reorders!
            CURLOPT_RETURNTRANSFER => true,     // return web page
            CURLOPT_HEADER         => false,    // don't return headers
            CURLOPT_FOLLOWLOCATION => true,     // follow redirects
            CURLOPT_ENCODING       => "",       // handle all encodings
            CURLOPT_USERAGENT      => "unknown",// who am i
            CURLOPT_AUTOREFERER    => true,     // set referrer on redirect
            CURLOPT_CONNECTTIMEOUT => 120,      // timeout on connect
            CURLOPT_TIMEOUT        => 120,      // timeout on response
            CURLOPT_MAXREDIRS      => 10,       // stop after 10 redirects
            CURLOPT_SSL_VERIFYPEER => false     // Disabled SSL Cert checks.  FIX!!!!!!!!!
        ];
        //Optional authentication
        if (isset($options[CURLOPT_USERPWD])) {$options[CURLOPT_HTTPAUTH]=CURLAUTH_BASIC;}
        switch (strtolower($method)) {
            case "get":
                if ($data) {$url = sprintf("%s?%s", $url, http_build_query($data));}
                break;
            case "post":
                $options[CURLOPT_POST]=1;
                if ($data) {$options[CURLOPT_POSTFIELDS]=$data;}
                break;
            case "put":
                //$options[CURLOPT_PUT]=1;
                $options[CURLOPT_CUSTOMREQUEST]="PUT";
                if ($data) {$options[CURLOPT_POSTFIELDS]=http_build_query($data);}
                break;
            case "delete":
                //$options[CURLOPT_DELETE]=1;
                $options[CURLOPT_CUSTOMREQUEST]="DELETE";
                if ($data) {$options[CURLOPT_POSTFIELDS]=http_build_query($data);}
                break;
            default:trigger_error("Invalid HTTP method.", E_USER_ERROR);
        }
        //$this->logger->addInfo("$method $url ".json_encode($data));
        $options[CURLOPT_URL]=$url;
        $ch      = curl_init();
        curl_setopt_array( $ch, $options );
        if($headers) {
            curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
        }
        $rsp = curl_exec( $ch );
        if($debug) {
            $results  = array_merge(curl_getinfo( $ch ),['errno'=>curl_errno($ch), 'errmsg'=>curl_error($ch),'data'=>$rsp,'code'=>curl_errno($ch)?null:curl_getinfo($ch, CURLINFO_HTTP_CODE)]);
        }
        else {
            $results=curl_errno($ch)==6
            ?['data'=>json_encode(['message'=>'Invalid Datalogger IP','code'=>1]),'code'=>400]
            :['data'=>$rsp,'code'=>curl_getinfo($ch, CURLINFO_HTTP_CODE)];
        }
        curl_close( $ch );
        return $results;
    }
}
 

REQUEST

Path: /charts/pie

Method: post

Parameters: []

Data: {"name":"My Chart","theme":"invalid"}


RESPONSE

Code: 422

Data: {"code":3,"message":"Invalid Theme 'invalid'."}


VALIDATION

ERROR

Array(    [0] => HTTP Code does not match expected code "200".)

REQUEST

Path: /charts/pie

Method: post

Parameters: []

Data: {"name":"My Chart","theme":"invalid"}


RESPONSE

Code: 422

Data: {"code":3,"message":"Invalid Theme 'invalid'."}


VALIDATION

Valid



REQUEST

Path: /charts/pie

Method: post

Parameters: []

Data: {"name":"My Chart","theme":"pie1"}


RESPONSE

Code: 200

Data: {"name":"My Chart","theme":"pie1","id":"79"}


VALIDATION

ERROR

Array(    [0] => Unexpected response properties theme)

REQUEST

Path: /charts/{id}

Method: get

Parameters: {"id":"79"}

Data: []


RESPONSE

Code: 200

Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":null,"subtitle":null,"points":[]}


VALIDATION

ERROR

Array(    [0] => Unexpected response properties theme, masterType, title, subtitle, points)

REQUEST

Path: /charts/{id}

Method: put

Parameters: {"id":"79"}

Data: {"name":"title","value":"My Title"}


RESPONSE

Code: 204

Data:


VALIDATION

Valid



REQUEST

Path: /charts/{id}

Method: post

Parameters: {"id":"79"}

Data: {"points_id":98,"legend":"Legend for Speed"}


RESPONSE

Code: 204

Data:


VALIDATION

Valid



REQUEST

Path: /charts/{id}

Method: post

Parameters: {"id":"79"}

Data: {"points_id":99,"legend":"Legend for Cost"}


RESPONSE

Code: 204

Data:


VALIDATION

Valid



REQUEST

Path: /charts/{id}

Method: get

Parameters: {"id":"79"}

Data: []


RESPONSE

Code: 200

Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":"My Title","subtitle":null,"points":{"98":{"id":98,"name":"Speed","units":null,"legend":"Legend for Speed"},"99":{"id":99,"name":"Cost","units":null,"legend":"Legend for Cost"}}}


VALIDATION

ERROR

Array(    [0] => Unexpected response properties theme, masterType, title, subtitle, points)

REQUEST

Path: /charts/{id}/{legend}

Method: delete

Parameters: {"id":"79","legend":"Legend for Speed"}

Data: []


RESPONSE

Code: 204

Data:


VALIDATION

Valid



REQUEST

Path: /charts/{id}

Method: get

Parameters: {"id":"79"}

Data: []


RESPONSE

Code: 200

Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":"My Title","subtitle":null,"points":{"99":{"id":99,"name":"Cost","units":null,"legend":"Legend for Cost"}}}


VALIDATION

ERROR

Array(    [0] => Unexpected response properties theme, masterType, title, subtitle, points)

REQUEST

Path: /charts/{id}

Method: delete

Parameters: {"id":"79"}

Data: []


RESPONSE

Code: 204

Data:


VALIDATION

Valid



REQUEST

Path: /charts/{id}

Method: get

Parameters: {"id":"79"}

Data: []


RESPONSE

Code: 404

Data: {"code":3,"message":"A resource does not exist for this ID."}


VALIDATION

Valid

 

 

requinix Prolific Member

requinix

Posted
Posted

You validate by writing and running tests. Like you're doing. Testing as a whole is a complicated subject, but the short version is to have an automated way of checking that successful scenarios are successful while failure scenarios are failures. "Automated" so it encourages you to run them as often as possible; a way to get terse output (eg, only show failures) would be nice so you don't have to parse a lot of output to get your answers.

 

How you go about writing tests varies: there are testing frameworks out there to help, but making your own (provided you don't spend too much time on it) is fine too.

NotionCommotion Prolific Member

NotionCommotion

Posted
  • Author
Posted

How you go about writing tests varies: there are testing frameworks out there to help, but making your own (provided you don't spend too much time on it) is fine too.

 

I haven't spent that much time so far, however, am sure I am not yet done.  For instance, the testing application should test all permutations of data sent (i.e. make a parameter a string, an integer, zero, don't include it, etc) and check for errors.   Likely, I will end up spending too much time.  Do you recommend a framework or other particular tools?  Thanks

requinix Prolific Member

requinix

Posted
Posted

Unless there's something out there that has pre-written code for doing tests like the ones you need (I'd be somewhat surprised), you're going to end up writing those tests by yourself anyways. That TestAPI thing seems decent.

 

You also don't have to necessarily write a test for every single aspect. Testing only for valid situations is a legitimate testing strategy.

 

My advice is to test just for the valid situations now and move on to other more important problems. If you find a bug with your current implementation, write a test, fix it, (rerun all the other tests to be sure nothing broke,) then go back to what you were doing before.

This thread is more than a year old. Please don't revive it unless you have something important to add.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing
×
×
  • Create New...

Important Information

We have placed cookies on your device to help make this website better. You can adjust your cookie settings, otherwise we'll assume you're okay to continue.