Re^2: Reading and writting data in a text file with several scripts simultaneously time synchronization problem

To: boftx

Your approach makes a lot of sense. The Master (Main script) should be aware of how many slave scripts are attempting to extract the data. As a second step he should be aware of how many processes have all ready extra the data and how many how not yet, until the next iteration.

I am afraid that by trying to create a such a process is a bit out of my league. Well I was also thinking the alternative approach that you proposed using a Data Base (e.g. MySQL). I think this approach will speed my process a lot for all the scripts.

Another alternative that I am thinking also, is that instead of reading all the data of the file Main script into an array, I could simply use append. This approach will also significantly improve the process speed.

But now I am facing two new challenges.

Is there a way to measure the process time for the main script to read and write the data on the file? I am mainly curious because I want to compare this time with the MySQL process time and decide which path I should follow.

The second and final challenge that I am facing, is there a way to apply flock on INI files? I will be pushing all files to use one INI file with different locations of reading of course but still there might be produced a conflict.

Sorry for the continuous questions and messages I am brainstorming any possible solution. Since that kind of processes I will encounter several times in the future so I am looking for the best possible solution to implement.

Comment on Re^2: Reading and writting data in a text file with several scripts simultaneously time synchronization problem

Replies are listed 'Best First'.
Re^3: Reading and writting data in a text file with several scripts simultaneously time synchronization problem by boftx (Deacon) on May 19, 2014 at 17:19 UTC
I would go with the database if that is desirable since it might well make future reporting easier to manage. But you could always use the Benchmark module to test out different methods. As for `flock` and INI files, I presume you mean a typical foo.ini file for config variables. I don't see why you would have any conflicts unless the file is remote mounted over a network. Some versions of `flock` might have problems with that. YMMV. The main issue is to devise a decent semaphore scheme that removes any dependence upon `sleep()` to synchronize the various parts other than establishing a timing loop for polling the semaphore. It helps to remember that the primary goal is to drain the swamp even when you are hip-deep in alligators.	[reply] [d/l] [select]
Re^4: Reading and writting data in a text file with several scripts simultaneously time synchronization problem by thanos1983 (Parson) on May 19, 2014 at 22:19 UTC
To: boftx Perfect Benchmark is exactly what I need. I am thinking of applying a few different solutions and where I can get. I am afraid the time grows as the file grows. But the flock comes in handy there. Well the reason that I was avoiding to use DB, is that I was trying to make my code a bit more generic and I am not 100% sure yet if on my final task I will be able to apply DB. The whole story is that I am creating a main test script that I need to produce random data and store them into a file for simplicity. The secondary scripts I want them to retrieve the data process them and also store their output on separate files, for comparison reasons after. I want to compare how many data where able to process and how fast! In conclusion even the secondary scripts are test scripts I will chose only one of them to apply on my real goal. I am testing them to see performance under pressure in order to choose the best one. This is the reason that either I need to store the data to a folder for later retrieving them or into a database. The problem is also that on my real goal several scripts will need to access this folder to retrieve the data. So I am trying to make my experiments as realistic as possible. So enough said, time for coding. I need first of all to compare the times. Initially with the folders in different formations and possible create a short script with MySQL also just to observe if there is a huge difference on process time. P.S. Sorry for the long answer I tried to include everything.	[reply]