Fetching data from different sources and save to the database best practice?

[asynchronous]

Hello guys, i am developing an application that will manage the APIs of multiple marketplaces from a single point. In order to list the marketplace orders in a single point(page) and to facilitate their management by the customer, I want to periodically pull the orders from the api in the background and save them to the database. Does anyone have any suggestions for multiple users to make individual api calls for multiple api connections and save the orders to the database? I will use symfony messenger for queue but for example assuming 100 users have 200 apis in total, I am confused about pulling orders from each api and saving them in database. Thanks for help.

[requinix]

Process them in batches through cronjobs?

[asynchronous]

21 minutes ago, requinix said:

Process them in batches through cronjobs?

First of all, thanks for your answer.

Symfony messenger component handles background and queue operations, but the point that confuses me is;

There are many api owned by many users.
User 1: 1x amazon, 2x ebay, 3x aliexpress api
User 2: 2x amazon, 1x ebay, 5x aliexpress api
and more

Example DB structure;
integrations Table
user_id, integration_id, type, auth_token

Let's say that for each api registered in the database, I opened a connection in order and pulled the marketplace orders from api and saved them with the id of the relevant user to the orders table. This is the process that comes to mind, but I have concerns. So I'm wondering if anyone has a better idea.

[requinix]

If you want advice about how to design this application then we're going to need a LOT more detail than what you've given so far. Especially about how users "own" APIs.

[asynchronous]

9 hours ago, requinix said:

If you want advice about how to design this application then we're going to need a LOT more detail than what you've given so far. Especially about how users "own" APIs.

Think like that; You are a seller, you own a store on sites such as amazon, ebay, aliexpress and you're selling something on there. You got the api accounts from each of them and you registered in my app and then you integrated each of your api. You will now be able to manage all order transactions through the application. In this way, dozens of users can have dozens of API connections. Periodically, it is necessary to connect all API accounts registered in the database in the background, and to pull new orders from API at certain intervals and transfer them to the database.

[requinix]

Have you investigated whether those services provide push messages or notifications? So that you don't have to poll them yourself. Because if you have to poll them yourself then your data will always be potentially out of date.

[Barand]

is it feasible for your app to store the data as they interact with the various api's?

[asynchronous]

10 hours ago, requinix said:

Have you investigated whether those services provide push messages or notifications? So that you don't have to poll them yourself. Because if you have to poll them yourself then your data will always be potentially out of date.

Unfortunately they do not provide push messages and notifications. I have to solve it in a different way.

[asynchronous]

8 hours ago, Barand said:

is it feasible for your app to store the data as they interact with the various api's?

I did not quite get it. Are you talking about storing when request is sent to the api? This process can slow down the application very much and different problems may occur if an error occurs while inserting a record in the database.