NotionCommotion Posted October 4, 2016 Share Posted October 4, 2016 I recently created an OpenAPI https://openapis.org/ (aka Swagger) spec and then built an application to implement that spec. How can I test my application to ensure that it accurately implements the spec? Are there any tools to assist? I looked around and couldn't find anything I liked, and started building my own. To test, you specify an endpoint and any data to send as well as the expected HTTP status code to be returned, and the class confirms that all required inputs are provided, that no inputs not described by the spec are provided, that all required outputs are returned, and that no outputs not described by the spec are returned, and also that the returned status code was returned. I keep on thinking, however, that there has to be something good already built. Did I totally waste my time? <?php require('../classes/TestAPI.php'); $test=new TestAPI(file_get_contents ('/var/www/swagger/swagger-ui/dist/swagger.json')); $test->preloadHeaders(['X-GreenBean-Key'=>'main_key']); $rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"invalid"}',true),200); $rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"invalid"}',true),422); $rsp=$test->test('/charts/pie','post',[],json_decode('{"name": "My Chart","theme":"pie1"}',true),200); $id=$rsp->id; $rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200); $rsp=$test->test('/charts/{id}','put',['id'=>$id],['name'=>'title','value'=>'My Title'],204); $rsp=$test->test('/charts/{id}','post',['id'=>$id],['points_id'=>98,'legend'=>'Legend for Speed'],204); $rsp=$test->test('/charts/{id}','post',['id'=>$id],['points_id'=>99,'legend'=>'Legend for Cost'],204); $rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200); $rsp=$test->test('/charts/{id}/{legend}','delete',['id'=>$id,'legend'=>'Legend for Speed'],[],204); $rsp=$test->test('/charts/{id}','get',['id'=>$id],[],200); $rsp=$test->test('/charts/{id}','delete',['id'=>$id],[],204); $rsp=$test->test('/charts/{id}','get',['id'=>$id],[],404); <?php class TestAPI { private $obj, $repeat, $data=[], $headers=[], $source=[]; public function __construct($json) { $this->obj = json_decode($json,true); if(json_last_error() !== JSON_ERROR_NONE) { throw new Exception('Invalid JSON'); } $this->repeat=true; while($this->repeat) { $this->repeat=false; $this->obj=$this->substituteReferences($this->obj); } unset($this->obj['definitions']); unset($this->obj['parameters']); unset($this->obj['responses']); $this->obj=$this->substituteAllOf($this->obj); } public function preloadData(array $arr) { // Preload data which will be sent with every request foreach($arr as $key=>$value) { $this->data[$key]=$value; } } public function preloadHeaders(array $arr) { // Preload headers which will be sent with every request foreach($arr as $key=>$value) { $this->headers[]=strtolower(str_replace("_","-",$key)).': '.$value; } } public function get() { return $this->obj; } private function getValue($name) { return isset($this->data[$name])?$this->data[$name]:false; } private function validateResponse($rsp,$config) { $errors=[]; if(!$rsp || !is_array($rsp) || count($rsp)!==2 || !isset($rsp['code']) || !isset($rsp['data'])) { $errors[]='Invalid response from server.'; } elseif(!isset($config['responses'][$rsp['code']])) { $errors[]="HTTP code $rsp[code] is not supported."; } else { $data=json_decode($rsp['data']); $type=isset($config['responses'][$rsp['code']]['schema']) ?isset($config['responses'][$rsp['code']]['schema']['type'])?$config['responses'][$rsp['code']]['schema']['type']:'object' :'NULL'; if(json_last_error() !== JSON_ERROR_NONE) { $errors[]="Server response is not JSON."; } elseif(gettype($data)!=$type) { $errors[]="Response type expected was $type, however, ".gettype($data)." was received"; } else { switch($type) { case 'array': /* foreach($x as $s) { } */ $status='OKAY'; break; case 'object': $data=(array)$data; $required=isset($config['responses'][$rsp['code']]['schema']['required'])?$config['responses'][$rsp['code']]['schema']['required']:[]; $properties=isset($config['responses'][$rsp['code']]['schema']['properties'])?$config['responses'][$rsp['code']]['schema']['properties']:[]; $propertyNames=array_keys($data); if($missing=array_diff($required,$propertyNames)) { $errors[]='Missing required response properties '.implode(', ',$missing); } if($toMany=array_diff($propertyNames,array_keys($properties))) { $errors[]='Unexpected response properties '.implode(', ',$toMany); } else { //Don't validate response types if too many were given foreach($data as $property=>$value) { $type=gettype($value); if($type!=$properties[$property]['type']) { $errors[]="Respond property $property is type $type but should have been type $properties[$property][type]."; } } } break; case 'string': break; case NULL: break; // Test for null? default: $status="Non-supported response type $type. This should never happen!"; } } } return $errors; } private function substitute($path, $parameters) { foreach ($parameters as $key => $value) { $path = str_replace('{'.$key.'}', urlencode($value), $path); } return $path; } public function testAll() { foreach($this->obj['paths'] as $path=>$methods) { foreach ($methods as $method=>$config) { $url=$this->obj['host'].$this->obj['basePath'].$path; $rsp=$this->CallAPI($method, $url, $this->data, $this->headers); $errors=$this->validateResponse($rsp,$config); echo("<p>URL: $url Method: $method Status: ".(empty($errors)?'OKAY':'ERROR')."</p>"); if($errors) {echo('<pre>'.print_r($errors,1).'</pre>');} //echo('<pre>'.print_r($rsp,1).'</pre>'); } //echo('<pre>'.print_r($path,1).'</pre>'); } } public function test($path,$method,array $params, array $data, $expected=null) { $url=$this->obj['host'].$this->obj['basePath'].($params?$this->substitute($path,$params):$path); $rsp=$this->CallAPI($method, $url, array_merge($data,$this->data), $this->headers); echo("<h3><u>REQUEST</u></h3><p>Path: $path</p><p>Method: $method</p><p>Parameters: ".json_encode($params)."</p><p>Data: ".json_encode($data)); echo("<h3><u>RESPONSE</u></h3><p>Code: $rsp[code]</p><p>Data: $rsp[data]"); echo("<h3><u>VALIDATION</u></h3>"); if(isset($this->obj['paths'][$path][$method])) { $errors=$this->validateResponse($rsp,$this->obj['paths'][$path][$method]); if($expected && $expected!=$rsp['code']) { $errors[]='HTTP Code does not match expected code "'.$expected.'".'; } if($errors) { echo('<p>ERROR</p><pre>'.print_r($errors,1).'</pre>'); } else{ echo('<p>Valid</p>'); } } else { echo("INABILITY TO VALIDATE: INVALID PATH ($path) OR METHOD ($method)"); } echo('<hr>'); return json_decode($rsp['data']); } private function substituteReferences($o) { foreach($o as $k=>$v) { if(is_array($v)){ $o[$k]=$this->substituteReferences($v); } elseif($k==='$ref') { if(substr($v,0,2)!='#/') throw new Exception('Invalid reference: Bad prefix.'); $p=explode('/',substr($v,2)); if(count($p)!=2) throw new Exception('Invalid reference: Incorrect count.'); if(empty($this->obj[$p[0]])) throw new Exception('Invalid reference: Property type doesn\'t exist.'); if(empty($this->obj[$p[0]][$p[1]])) throw new Exception('Invalid reference: Property doesn\'t exist.'); $o=$this->obj[$p[0]][$p[1]]; $this->repeat=true; } elseif($k==='in' && $v==='header' && isset($o['name'])) { //Make compatible $o['name']=strtolower(str_replace("_","-",$o['name'])); } } return $o; } private function substituteAllOf($o) { foreach($o as $k=>$v) { if($k==='allOf') { $properties=[]; $required=[]; foreach ($o['allOf'] as $key=>$item) { $properties=array_merge($properties, $item['properties']); if(isset($item['required'])) { $required=array_merge($required, $item['required']); } } unset($o['allOf']); $o['properties']=$properties; $o['required']=$required; } elseif(is_array($v)){ $o[$k]=$this->substituteAllOf($v); } } return $o; } private function CallAPI($method, $url, $data, array $headers=[], $options=[], $debug=false) { $options=$options+[ //Don't use array_merge since it reorders! CURLOPT_RETURNTRANSFER => true, // return web page CURLOPT_HEADER => false, // don't return headers CURLOPT_FOLLOWLOCATION => true, // follow redirects CURLOPT_ENCODING => "", // handle all encodings CURLOPT_USERAGENT => "unknown",// who am i CURLOPT_AUTOREFERER => true, // set referrer on redirect CURLOPT_CONNECTTIMEOUT => 120, // timeout on connect CURLOPT_TIMEOUT => 120, // timeout on response CURLOPT_MAXREDIRS => 10, // stop after 10 redirects CURLOPT_SSL_VERIFYPEER => false // Disabled SSL Cert checks. FIX!!!!!!!!! ]; //Optional authentication if (isset($options[CURLOPT_USERPWD])) {$options[CURLOPT_HTTPAUTH]=CURLAUTH_BASIC;} switch (strtolower($method)) { case "get": if ($data) {$url = sprintf("%s?%s", $url, http_build_query($data));} break; case "post": $options[CURLOPT_POST]=1; if ($data) {$options[CURLOPT_POSTFIELDS]=$data;} break; case "put": //$options[CURLOPT_PUT]=1; $options[CURLOPT_CUSTOMREQUEST]="PUT"; if ($data) {$options[CURLOPT_POSTFIELDS]=http_build_query($data);} break; case "delete": //$options[CURLOPT_DELETE]=1; $options[CURLOPT_CUSTOMREQUEST]="DELETE"; if ($data) {$options[CURLOPT_POSTFIELDS]=http_build_query($data);} break; default:trigger_error("Invalid HTTP method.", E_USER_ERROR); } //$this->logger->addInfo("$method $url ".json_encode($data)); $options[CURLOPT_URL]=$url; $ch = curl_init(); curl_setopt_array( $ch, $options ); if($headers) { curl_setopt($ch, CURLOPT_HTTPHEADER, $headers); } $rsp = curl_exec( $ch ); if($debug) { $results = array_merge(curl_getinfo( $ch ),['errno'=>curl_errno($ch), 'errmsg'=>curl_error($ch),'data'=>$rsp,'code'=>curl_errno($ch)?null:curl_getinfo($ch, CURLINFO_HTTP_CODE)]); } else { $results=curl_errno($ch)==6 ?['data'=>json_encode(['message'=>'Invalid Datalogger IP','code'=>1]),'code'=>400] :['data'=>$rsp,'code'=>curl_getinfo($ch, CURLINFO_HTTP_CODE)]; } curl_close( $ch ); return $results; } } REQUESTPath: /charts/pie Method: post Parameters: [] Data: {"name":"My Chart","theme":"invalid"} RESPONSECode: 422 Data: {"code":3,"message":"Invalid Theme 'invalid'."} VALIDATIONERROR Array( [0] => HTTP Code does not match expected code "200".) REQUESTPath: /charts/pie Method: post Parameters: [] Data: {"name":"My Chart","theme":"invalid"} RESPONSECode: 422 Data: {"code":3,"message":"Invalid Theme 'invalid'."} VALIDATIONValid REQUESTPath: /charts/pie Method: post Parameters: [] Data: {"name":"My Chart","theme":"pie1"} RESPONSECode: 200 Data: {"name":"My Chart","theme":"pie1","id":"79"} VALIDATIONERROR Array( [0] => Unexpected response properties theme) REQUESTPath: /charts/{id} Method: get Parameters: {"id":"79"} Data: [] RESPONSECode: 200 Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":null,"subtitle":null,"points":[]} VALIDATIONERROR Array( [0] => Unexpected response properties theme, masterType, title, subtitle, points) REQUESTPath: /charts/{id} Method: put Parameters: {"id":"79"} Data: {"name":"title","value":"My Title"} RESPONSECode: 204 Data: VALIDATIONValid REQUESTPath: /charts/{id} Method: post Parameters: {"id":"79"} Data: {"points_id":98,"legend":"Legend for Speed"} RESPONSECode: 204 Data: VALIDATIONValid REQUESTPath: /charts/{id} Method: post Parameters: {"id":"79"} Data: {"points_id":99,"legend":"Legend for Cost"} RESPONSECode: 204 Data: VALIDATIONValid REQUESTPath: /charts/{id} Method: get Parameters: {"id":"79"} Data: [] RESPONSECode: 200 Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":"My Title","subtitle":null,"points":{"98":{"id":98,"name":"Speed","units":null,"legend":"Legend for Speed"},"99":{"id":99,"name":"Cost","units":null,"legend":"Legend for Cost"}}} VALIDATIONERROR Array( [0] => Unexpected response properties theme, masterType, title, subtitle, points) REQUESTPath: /charts/{id}/{legend} Method: delete Parameters: {"id":"79","legend":"Legend for Speed"} Data: [] RESPONSECode: 204 Data: VALIDATIONValid REQUESTPath: /charts/{id} Method: get Parameters: {"id":"79"} Data: [] RESPONSECode: 200 Data: {"id":79,"name":"My Chart","theme":"pie1","type":"pie","type_name":"Pie chart","masterType":"pie","title":"My Title","subtitle":null,"points":{"99":{"id":99,"name":"Cost","units":null,"legend":"Legend for Cost"}}} VALIDATIONERROR Array( [0] => Unexpected response properties theme, masterType, title, subtitle, points) REQUESTPath: /charts/{id} Method: delete Parameters: {"id":"79"} Data: [] RESPONSECode: 204 Data: VALIDATIONValid REQUESTPath: /charts/{id} Method: get Parameters: {"id":"79"} Data: [] RESPONSECode: 404 Data: {"code":3,"message":"A resource does not exist for this ID."} VALIDATIONValid Quote Link to comment Share on other sites More sharing options...
requinix Posted October 4, 2016 Share Posted October 4, 2016 You validate by writing and running tests. Like you're doing. Testing as a whole is a complicated subject, but the short version is to have an automated way of checking that successful scenarios are successful while failure scenarios are failures. "Automated" so it encourages you to run them as often as possible; a way to get terse output (eg, only show failures) would be nice so you don't have to parse a lot of output to get your answers. How you go about writing tests varies: there are testing frameworks out there to help, but making your own (provided you don't spend too much time on it) is fine too. Quote Link to comment Share on other sites More sharing options...
NotionCommotion Posted October 4, 2016 Author Share Posted October 4, 2016 How you go about writing tests varies: there are testing frameworks out there to help, but making your own (provided you don't spend too much time on it) is fine too. I haven't spent that much time so far, however, am sure I am not yet done. For instance, the testing application should test all permutations of data sent (i.e. make a parameter a string, an integer, zero, don't include it, etc) and check for errors. Likely, I will end up spending too much time. Do you recommend a framework or other particular tools? Thanks Quote Link to comment Share on other sites More sharing options...
requinix Posted October 4, 2016 Share Posted October 4, 2016 Unless there's something out there that has pre-written code for doing tests like the ones you need (I'd be somewhat surprised), you're going to end up writing those tests by yourself anyways. That TestAPI thing seems decent. You also don't have to necessarily write a test for every single aspect. Testing only for valid situations is a legitimate testing strategy. My advice is to test just for the valid situations now and move on to other more important problems. If you find a bug with your current implementation, write a test, fix it, (rerun all the other tests to be sure nothing broke,) then go back to what you were doing before. Quote Link to comment Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.