asynchronous Posted December 15, 2021 Share Posted December 15, 2021 Hello guys, i am developing an application that will manage the APIs of multiple marketplaces from a single point. In order to list the marketplace orders in a single point(page) and to facilitate their management by the customer, I want to periodically pull the orders from the api in the background and save them to the database. Does anyone have any suggestions for multiple users to make individual api calls for multiple api connections and save the orders to the database? I will use symfony messenger for queue but for example assuming 100 users have 200 apis in total, I am confused about pulling orders from each api and saving them in database. Thanks for help. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/ Share on other sites More sharing options...
requinix Posted December 15, 2021 Share Posted December 15, 2021 Process them in batches through cronjobs? Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592715 Share on other sites More sharing options...
asynchronous Posted December 15, 2021 Author Share Posted December 15, 2021 21 minutes ago, requinix said: Process them in batches through cronjobs? First of all, thanks for your answer. Symfony messenger component handles background and queue operations, but the point that confuses me is; There are many api owned by many users. User 1: 1x amazon, 2x ebay, 3x aliexpress api User 2: 2x amazon, 1x ebay, 5x aliexpress api and more Example DB structure; integrations Table user_id, integration_id, type, auth_token Let's say that for each api registered in the database, I opened a connection in order and pulled the marketplace orders from api and saved them with the id of the relevant user to the orders table. This is the process that comes to mind, but I have concerns. So I'm wondering if anyone has a better idea. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592716 Share on other sites More sharing options...
requinix Posted December 15, 2021 Share Posted December 15, 2021 If you want advice about how to design this application then we're going to need a LOT more detail than what you've given so far. Especially about how users "own" APIs. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592759 Share on other sites More sharing options...
asynchronous Posted December 16, 2021 Author Share Posted December 16, 2021 9 hours ago, requinix said: If you want advice about how to design this application then we're going to need a LOT more detail than what you've given so far. Especially about how users "own" APIs. Think like that; You are a seller, you own a store on sites such as amazon, ebay, aliexpress and you're selling something on there. You got the api accounts from each of them and you registered in my app and then you integrated each of your api. You will now be able to manage all order transactions through the application. In this way, dozens of users can have dozens of API connections. Periodically, it is necessary to connect all API accounts registered in the database in the background, and to pull new orders from API at certain intervals and transfer them to the database. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592766 Share on other sites More sharing options...
requinix Posted December 17, 2021 Share Posted December 17, 2021 Have you investigated whether those services provide push messages or notifications? So that you don't have to poll them yourself. Because if you have to poll them yourself then your data will always be potentially out of date. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592815 Share on other sites More sharing options...
Barand Posted December 17, 2021 Share Posted December 17, 2021 is it feasible for your app to store the data as they interact with the various api's? Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592819 Share on other sites More sharing options...
asynchronous Posted December 18, 2021 Author Share Posted December 18, 2021 10 hours ago, requinix said: Have you investigated whether those services provide push messages or notifications? So that you don't have to poll them yourself. Because if you have to poll them yourself then your data will always be potentially out of date. Unfortunately they do not provide push messages and notifications. I have to solve it in a different way. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592825 Share on other sites More sharing options...
asynchronous Posted December 18, 2021 Author Share Posted December 18, 2021 8 hours ago, Barand said: is it feasible for your app to store the data as they interact with the various api's? I did not quite get it. Are you talking about storing when request is sent to the api? This process can slow down the application very much and different problems may occur if an error occurs while inserting a record in the database. Quote Link to comment https://forums.phpfreaks.com/topic/314317-fetching-data-from-different-sources-and-save-to-the-database-best-practice/#findComment-1592826 Share on other sites More sharing options...
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.