Uploading a file with JSON to a REST API

[NotionCommotion]

I actually have two parts to this post.  One which describes just my understanding on how a multipart request should look with a REST API and gives a little context, and the second how to actually implement it using Sympony and api-platform.

Part 1

Looking to simultaneously upload a file, some file meta data, and identifier of the uploading user to a REST API, and will do so using a multipart request.

Based on swagger, looks like the request should look something like the following:

POST /upload HTTP/1.1
Content-Length: 428
Content-Type: multipart/form-data; boundary=abcde12345
--abcde12345
Content-Disposition: form-data; name="id"
Content-Type: text/plain
123e4567-e89b-12d3-a456-426655440000
--abcde12345
Content-Disposition: form-data; name="address"
Content-Type: application/json
{
  "street": "3, Garden St",
  "city": "Hillsbery, UT"
}
--abcde12345
Content-Disposition: form-data; name="profileImage "; filename="image1.png"
Content-Type: application/octet-stream
{…file content…}
--abcde12345--

Background noise... The above example shows three parts which include the text ID, JSON address and octet-stream file.  Doing so doesn't make sense to me and I will probably just have two parts for the meta data JSON.  Note that potentially, I will be removing the user from the JSON payload and including it in some JWT header.  Also, showing resources as links such as /api/users/1 in the request and response JSON is new to me, but I guess makes sense.

Regardless, my meta data JSON will look like:

{
    "projectId": 1234,
    "documentDescription": "bla bla bla",
    "specification": "230903.2.4.D",
    "user": "/api/users/1"
}

and the response will look like:

{
    "projectId": 1234,
    "documentDescription": "bla bla bla",
    "specification": "230903.2.4.D",
    "uploadAt": "2021-01-27 08:41:17",
    "fileName": "pressuresensor.pdf".
    "user": "/api/users/1".
    "file": "/api/uplodaed_files/1234"
}

Am I somewhat going down the right path?

Part 2

Wow, should have looked into Sympony before now!  Entities, validation, the API, etc were a breeze.  Still confused with some of the "magic", but understanding what is under the hood more and more.

As stated initially, I am utilizing api-platform and am looking at api-platform's file-upload.

I guess the first question is whether my POST request with both the file and JSON meta data can be performed in a single request.

Per making-a-request-to-the-media_objects-endpoint:

Quote

Your /media_objects endpoint is now ready to receive a POST request with a file. This endpoint accepts standard multipart/form-data-encoded data, but not JSON data.

What?  Am I reading this correct?  Do I first need to make a multipart requests using either a standard HTML form or a XMLHttpRequest which just sends the file, receive the document locator, and then send a second request with the location and my meta data?

If so, I guess I have enough to go forward.  I would rather, however, perform the work in a single request.  My previous mentioned swagger multipart-requests shows both in the same request.  symfonycasts's api-uploads also describes doing so, but not with api-platform.  Is there a way to do so with api-platform?  Does it make sense to use api-platform for all other routes but then make a custom route for document with meta data uploads?  Any recommendations are much appreciated.

Thanks

[requinix]

1. If you want to submit other pieces of information at the same time you upload a file (or other blob) then yeah, multipart is the way to go.

2. I think what they're saying is they support multipart for specifying multiple values but not JSON for specifying multiple values. Any of those values themselves can be (strings that look like) JSON.

[NotionCommotion]

19 hours ago, requinix said:

2. I think what they're saying is they support multipart for specifying multiple values but not JSON for specifying multiple values. Any of those values themselves can be (strings that look like) JSON.

Which would be fine.  Will test, but not certain that is what they are saying.  The reason I think so is it might not reflect a "pure" REST API.  A request to create a file on the server should return the location of the file.  A request to create an entity which includes some meta data as well as a file should return some representation of that entity.  You know, I don't know what I am thinking and will test...

[NotionCommotion]

I've been going down the path to upload both the file and the metadata in the same request, but starting to doubt myself again and want to stop sooner than later if a bad decision.

Evidently, YouTube sends the metadata first, persists an entity, returns the endpoint, and uploads the file using PUT (or maybe PATCH?).

This post says basically there is no option to send the metadata and file in the same request unless the file is Base64 encoded (which evidently is bad).  My interpretation of the general concusses is send the file first, store it in some tmp folder, and then upload the metadata, move the file, and and create the entity.

My originally referenced documentation didn't make judgement but went out to say it isn't JSON which I maybe incorrectly interpret as saying don't do in a single request.

It "seems" like it is possible using multipart to send both in the same request, but I've found nothing confirming this belief.

Am I going down the wrong path trying to send a single request with both the file and metadata?

 

 

 

[requinix]

35 minutes ago, NotionCommotion said:

Evidently, YouTube sends the metadata first, persists an entity, returns the endpoint, and uploads the file using PUT (or maybe PATCH?).

A valid approach.

 

Quote

This post says basically there is no option to send the metadata and file in the same request unless the file is Base64 encoded (which evidently is bad).

Base-64 increases the size of the request body but that doesn't make it bad.

 

Quote

It "seems" like it is possible using multipart to send both in the same request, but I've found nothing confirming this belief.

It is totally possible to do both. After all, haven't you ever used a web form that had you upload a file and submit other data at the same time?

[kicken]

21 minutes ago, NotionCommotion said:

Am I going down the wrong path trying to send a single request with both the file and metadata?

If you do a typical multipart/form-data post then certainly you can send both.  It'd be just like submitting a normal HTML form with a file input.  Most your frameworks and such should probably handle this just fine.

If you want to do a two-part request where part 1 is just a JSON document and part 2 is the file content then you certain can do that, but you may not find built-in support for it in your frameworks because as far as I know it's not something that's usually done.  As such, you'd have to spend time creating your own code to handle such a request in whatever way you need to.

As far as I know the more common way things are done is to use separate requests so you'd first POST your metadata then PUT the file content to s certain URL.  For example if you were adding a document to some project X you might POST /api/project/X/documents to create a new document and the response of that would give you a document ID number Y.  Then you'd PUT /api/project/X/documents/Y/content with the binary content of the file.

So I guess the question mostly is do you want to spend the time writing the code to make your nice-looking single-request or uglify your request to take advantage of existing code.

[NotionCommotion]

11 hours ago, requinix said:

It is totally possible to do both. After all, haven't you ever used a web form that had you upload a file and submit other data at the same time?

If done correctly, a user wouldn't/shouldn't know.  Yes, I knew that it was possible but the lack of others doing so made me feel that it might be a bad idea.  Sorry, should have stated the question not whether it could be done but should be done.

 

11 hours ago, kicken said:

So I guess the question mostly is do you want to spend the time writing the code to make your nice-looking single-request or uglify your request to take advantage of existing code.

The eternal question!  Will also need to consider the added client complexity of making multiple requests.  If it is just me and it will be fairly easy to modify api-platform's upload solution to do both, I expect that would be easiest, but if most API client developers are accustom to making two requests, expect I should implement it that way.

[NotionCommotion]

Thanks for your help requinix and kicken.  I feel good about my plan going forward.  My consensus is:

• While there are some benefits of having the file and associated data in a single request, it is not worth doing so for the following reasons:
  • Limits flexibillity for future changes.
  • Base64 uses increases file size and utilizes more resources.
  • Mixing fields and the file in a multipart form adds complexities and is harder to document and test.
• If the entity with the metadata has one and only one file associated with it, then the file should be uploaded first and the next request should create the entity along with the file location, and a cron process should delete any non-associated files.
• If the entity with the metadata can have multiple files associated with it, then the data should be uploaded first and the entity created, and the next request(s) should upload the files and associate them to the entity.

If I am way off base, let me know, but if only a little, please let me happy in my ignorance

[requinix]

8 minutes ago, NotionCommotion said:

• If the entity with the metadata has one and only one file associated with it, then the file should be uploaded first and the next request should create the entity along with the file location, and a cron process should delete any non-associated files.
• If the entity with the metadata can have multiple files associated with it, then the data should be uploaded first and the entity created, and the next request(s) should upload the files and associate them to the entity.

Big disagree.

Having to throw a cronjob to clean up temporary assets is a sign of problems. You're creating more moving parts this way, and yet you turned down multipart because it "adds complexities"?
But the worst part is that you're creating two completely opposite flows for two similar situations. Whether an entity has one file or more than one file should not result in this much of a difference.

Entity first, file(s) second. No temporary crap, no cronjobs, no inconsistencies.

[NotionCommotion]

So much for bathing in my ignorance   Okay, I see your point about not changing the flow based on whether there can be one or more files.  My reasoning for uploading the file first when associated one-to-one with an entity was that then I could place a not null constraint on entity.file.  Thank you for the update.

 

[NotionCommotion]

Sorry, changing to create the entity first and then the file which resulted in a dilemma.

When previously I sent the file first, I first performed a POST to /files with only the file in the body which returned {"url": "/media/somefile.ext", "foo": "bar"}.  I then performed a POST to /documents with {"file": "/media/somefile.ext", "project": "/projects/321", name": "the name", "etc": "etc"} in the body which returned {"id": 123, "file": "/media/somefile.ext", "project": "/projects/321", name": "the name", "etc": "etc"}.

For the new approach creating the entity first, I will still do the same POST request to /documents but without the file location and will get back  {"id": 123, "file": null, "project": "/projects/321", name": "the name", "etc": "etc"} back.  But now I can't do a POST to /files as it needs the documentID metadata (which obviously I don't want to mix with the file in the body).

Would my two options be to post the file to either /documents/123/file or to /file?documentId=123?  Is one approach considered more proper than the other?

[requinix]

REST would suggest /documents/123/file(s).

[NotionCommotion]

2 hours ago, requinix said:

REST would suggest /documents/123/file(s).

Yeah, I would expect so even though there doesn't appear to be any definitive REST direction regarding sub-resources, the api-platform doesn't do so out of the box, and the previously mentioned youtube google reference doesn't appear to do it this way.  Regardless, I will do it this way.  Thanks