Buffering and data tranmission in perl

py_201 has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I seek your help regarding the implementation of transmission of data between two servers(different physical machines).I make my script(main.pl) run on Ist server which calls upon another script on 2nd server through SOAP::Lite and start some processes there(which as a result, get some data and store that in some kind of BUFFER). Meanwhile main.pl needs to keep checking that buffer to see if there is some data to process or not.

I want to know:

1) How can i implement this BUFFER mechanism in perl. can i use File System(store data in form of files), in that case how can i read out files from one server to another.

2) How can CURL help in this transmission on data.

3) Should i look into Apache ActiveMQ for this.

Any help would be appreciated.

Comment on Buffering and data tranmission in perl

Replies are listed 'Best First'.
Re: Buffering and data tranmission in perl by ikegami (Patriarch) on Sep 06, 2009 at 09:35 UTC
Why aren't you returning the output in the SOAP response? Alternatively, a different SOAP request could be used to poll to see if there is output available. It makes no sense to use SOAP for one request and a difference mechanism for the other.	[reply]
Re^2: Buffering and data tranmission in perl by py_201 (Acolyte) on Sep 07, 2009 at 05:34 UTC
The problem with getting the output in SOAP response is that the script running on 2nd server forks our many processes , which return results at different times, and SOAP would return the response only after everything finished. Instead of that i want what main.pl does is , it calls 2nd script on 2nd server, and without waiting for the response from second server it starts looking if the 2nd server script has got some results to display. so my basic question is how can main.pl read out results of many processes running on different server to the 1st server as they arrive. Hope i have demonstrated my problem clearly	[reply]
Re^3: Buffering and data tranmission in perl by ikegami (Patriarch) on Sep 07, 2009 at 05:46 UTC
So you dismissed my first solution. That's why I presented a second solution for exactly the scenario you presented.	[reply]
Re^4: Buffering and data tranmission in perl by py_201 (Acolyte) on Sep 07, 2009 at 07:34 UTC
Re: Buffering and data tranmission in perl by Sewi (Friar) on Sep 06, 2009 at 11:07 UTC
The range of possible solutions heavily depends on your application. Assuming that your "process" takes too long to just return the result in the HTTP reply, there are many solutions. I'll try to show you some of them: If this is a private project, the "process" doesn't need to run very often and finishes in a forseeable amount of time, you could just write your result in a "buffer.txt" within your webservers document root and use curl, wget or others th fetch this file. If you need to be sure that the job has finished before the download runs, create it as "buffer.txt.new" and rename it as soon as you're done. On the other side, curl and wget have parameters which let them download only newer-than-local - files. I'm using wget this way and parse the output to fetch files from an FTP server and process them only when they're updated. This solution is done very fast, but there are many things which could go wrong. The second sample is near by the first, but this time the download isn't triggered by cron or anything else which is timed, but by the server running the process. An abstrace CGI there could be: `Parse_the_parameters(); Run_the_process(); Call_URL(http://server2/trigger_download.cgi);` [download] trigget_download.cgi could be just a bash script which starts curl/wget or a Perl CGI which does the same and then processes the result. (Actually, I'm using curl much more often than LWP, because a simple one-line curl call requires about a dozen lines with LWP :-( ). If you got a commercial environment and both servers could reach each other only via internet, you should add authentication on both sides (htaccess Basic Auth may be enough) and let the called server ack that the request has been fully processed and the calling server must retry if this isn't received. Final note: If you're using SOAP without being absolutly forced to, you should read the first two paragraphs of SOAP::Simple's description on CPAN.	[reply] [d/l]
Re: Buffering and data tranmission in perl by biohisham (Priest) on Sep 06, 2009 at 11:11 UTC
You may need to check out This for more on buffering in Perl too. Excellence is an Endeavor of Persistence. Chance Favors a Prepared Mind.	[reply] [d/l]