PHP client to periodically make cURL requests to other servers

[NotionCommotion]

I have a LAMP server which will also act as a client (furthermore referred to as PHP client).  The PHP client's purpose is to periodically make cURL requests to other servers and store the responses in the DB, and should do so indefinitely.  The remote server IPs and poll rates are stored in the PHP client's DB.  Some of the remote server are polled every 15 seconds, others are polled every 1 minute, others are polled every 5 minutes, and others are polled every 15 minutes, and a given remote server's poll rate will never change (at least not without some manual manipulation).  New remote servers may be added in the future.  While poll rates should on average be close to the designed poll rate, they do not need to be exact.  All the remote servers do not need to be polled at the same time, and ideally will not be.

 

Currently, I am polling all remote servers every 15 seconds.  To implement it, I've created a daemon which performs an endless loop.  Each loop, it executes a PHP script in the background, and then sleeps for 15 seconds.  The PHP script retrieves all of the remote server's IP, makes cURL requests to the remote servers in order, and stores the data.

 

I feel there are likely much better ways to do so.  Any recommendations?  Thank you

 

[Jacques1]

Is there any reason why you can't register the clients at the remote server and make it notify the clients of changes? This would let you get rid of polling altogether.

[NotionCommotion]

Is there any reason why you can't register the clients at the remote server and make it notify the clients of changes? This would let you get rid of polling altogether.

 

 

Thanks Jacques1, but either I don't understand your answer or you don't understand my question.  There is only one client and multiple remote servers.  While I "could" add a small client to each of the remote servers, I don't wish to do so.

[Jacques1]

You're not understanding the answer. My point is that servers can send out notifications on data changes, so instead of the client(s) constantly polling the server(s) for new data, the server(s) can notify the client(s) exactly when there actually is new data.

 

This has nothing to do with adding new scripts to the remote servers. All your fetch logic can remain on the client. The difference is that you can get rid of the polling.

 

For example, you can hook into the MySQL replication mechanism to get change events. This actually doesn't require any modifications on the servers at all.

[NotionCommotion]

You're not understanding the answer. My point is that servers can send out notifications on data changes, so instead of the client(s) constantly polling the server(s) for new data, the server(s) can notify the client(s) exactly when there actually is new data.

 

How does the server send out notification on data changes?  Are we talking about long polling, websockets, or something similar?  Or something else all together?  Thanks

[kicken]

What Jacques1 is suggesting is that your client(s) connect to the server then just wait. When something happens on the server that the client needs to know about the server will send a message to the client.

 

Long polling and websockets are one way of handling when your client is a browser. When your client is a PHP script you could just open a socket using stream_socket_client and then wait for data using stream_select. ReactPHP is a framwork you could use to help code such a script.

 

The best way to go tackle your problem will depend a lot of exactly what kind of communication you need between the server and client and how much control you have over both ends.

Edited October 21, 2016 by kicken

[NotionCommotion]

Thank you,

 

Well, I did say I wanted data on regular intervals, but in retrospect, I feel that getting data when it has significantly changed is better.  And gotcha about how websockets are for browsers and plain old sockets for for PHP, etc  I am in complete agreement with you.

 

Can you please elaborate on the criteria which would make one solution more suitable than another?

 

 

The best way to go tackle your problem will depend a lot of exactly what kind of communication you need between the server and client and how much control you have over both ends.

 

[Jacques1]

How does the server send out notification on data changes?

 

I gave you a link in my reply regarding MySQL replication. Read it.

 

Sockets are yet another option, but then you do need extra code on each server to open a socket, periodically query the database for changes and send those changes to the client. You seemed to have a major problem with that idea.

[NotionCommotion]

I gave you a link in my reply regarding MySQL replication. Read it.

 

 

I had skimmed it, but did not read it in detail.  The reasons I did not read it in detail are:

1. It appeared to deal with writing server logs (see below sections), and not meeting my objectives.  Are they referring to "logs" different than https://dev.mysql.com/doc/refman/5.7/en/server-logs.html?
2. I am concerned that MySQL-to-MySQL communication will not give adequate networking, operability, or authentication flexibility.  But considering I didn't read it in detail, I do not know this for sure.

 

 

--stop-never

This option is used with --read-from-remote-server. It tells mysqlbinlog to remain connected to the server. Otherwise mysqlbinlog exits when the last log file has been transferred from the server. --stop-never implies --to-last-log, so only the first log file to transfer need be named on the command line.

 

 

--read-from-remote-server, -R

Read the binary log from a MySQL server rather than reading a local log file. Any connection parameter options are ignored unless this option is given as well. These options are --host, --password, --port, --protocol, --socket, and --user.

This option requires that the remote server be running. It works only for binary log files on the remote server, not relay log files.

This option is like --read-from-remote-master=BINLOG-DUMP-NON-GTIDS.

 

--to-last-log, -t

Do not stop at the end of the requested binary log from a MySQL server, but rather continue printing until the end of the last binary log. If you send the output to the same MySQL server, this may lead to an endless loop. This option requires --read-from-remote-server.

[Jacques1]

If you want to have a discussion, I'm afraid you do have to read the suggestions in detail.

 

There's a variety of approaches, but you seem to be neither interested nor clear about what you want. So maybe you should think about this, do some research on the above suggestions and come back with more solid feedback.

 

Alternatively, just go back to the infinite loop. If that's good enough for you, it's certainly the quickest and most fool-proof solution.

[NotionCommotion]

If you want to have a discussion, I'm afraid you do have to read the suggestions in detail.

 

There's a variety of approaches, but you seem to be neither interested nor clear about what you want. So maybe you should think about this, do some research on the above suggestions and come back with more solid feedback.

 

Alternatively, just go back to the infinite loop. If that's good enough for you, it's certainly the quickest and most fool-proof solution.

 

 

Fair enough.  Let me do some more thinking and research.