in reply to Reading and writting data in a text file with several scripts simultaneously time synchronization problem

What you need is some sort of semaphore system. It would be simple to this with MemCache or a database, but I suspect you want to avoid such as those as well as named pipes or other IPC approaches.

That said, if (and this is a very big if) you can assign individual script 'IDs' to the slaves, then you can take the approach that each slave has a unique ID such that you can use binary operations to extract the relevant ID bit from an int stored in a 'lock' file to determine if data is ready to read or if all slaves have finished reading.

The logic would be something like this: if master, read lock file. If it exists, write new data only if the value of the lock file indicates that all slaves have read old data. If the file does not exist, write new data file and set lock file value to '0' to indicate data is ready. If slave, read lock file. If it does not exist, wait until it does. Use a bit-wise 'AND' (&) operation to determine if the slave bit is already set. If not, read data and then write lock file after bit-wise 'OR'ing (|) in the slaves bit. If the bit is already set, wait until lock file value for the slave ID bit in question is 0, indicating that new data is ready again. You will need to flock the lock file before every read, and be sure to read immediately before any write to get the current value.

If the slaves are also writing the data file, then the above logic would need to be tweaked.

I understand that this is a very crude approach, but the basic logic should be a start.

Edit: I just realized that this presumes that the master knows how many slaves there are in order to determine if all of them have read the data file. Give me another couple of shots of Scotch and I can probably come up with a more general solution. Also, the slave doesn't look for a value of '0' in the lock file, but rather, that the bit corresponding to the slave ID is 0.

It helps to remember that the primary goal is to drain the swamp even when you are hip-deep in alligators.
  • Comment on Re: Reading and writting data in a text file with several scripts simultaneously time synchronization problem

Replies are listed 'Best First'.
Re^2: Reading and writting data in a text file with several scripts simultaneously time synchronization problem
by thanos1983 (Parson) on May 19, 2014 at 10:19 UTC

    To: boftx

    Your approach makes a lot of sense. The Master (Main script) should be aware of how many slave scripts are attempting to extract the data. As a second step he should be aware of how many processes have all ready extra the data and how many how not yet, until the next iteration.

    I am afraid that by trying to create a such a process is a bit out of my league. Well I was also thinking the alternative approach that you proposed using a Data Base (e.g. MySQL). I think this approach will speed my process a lot for all the scripts.

    Another alternative that I am thinking also, is that instead of reading all the data of the file Main script into an array, I could simply use append. This approach will also significantly improve the process speed.

    But now I am facing two new challenges.

    Is there a way to measure the process time for the main script to read and write the data on the file? I am mainly curious because I want to compare this time with the MySQL process time and decide which path I should follow.

    The second and final challenge that I am facing, is there a way to apply flock on INI files? I will be pushing all files to use one INI file with different locations of reading of course but still there might be produced a conflict.

    Sorry for the continuous questions and messages I am brainstorming any possible solution. Since that kind of processes I will encounter several times in the future so I am looking for the best possible solution to implement.

[reply]

      I would go with the database if that is desirable since it might well make future reporting easier to manage. But you could always use the Benchmark module to test out different methods.

      As for flock and INI files, I presume you mean a typical foo.ini file for config variables. I don't see why you would have any conflicts unless the file is remote mounted over a network. Some versions of flock might have problems with that. YMMV.

      The main issue is to devise a decent semaphore scheme that removes any dependence upon sleep() to synchronize the various parts other than establishing a timing loop for polling the semaphore.

      It helps to remember that the primary goal is to drain the swamp even when you are hip-deep in alligators.
[reply]
[d/l]
[select]

        To: boftx

        Perfect Benchmark is exactly what I need. I am thinking of applying a few different solutions and where I can get. I am afraid the time grows as the file grows. But the flock comes in handy there.

        Well the reason that I was avoiding to use DB, is that I was trying to make my code a bit more generic and I am not 100% sure yet if on my final task I will be able to apply DB.

        The whole story is that I am creating a main test script that I need to produce random data and store them into a file for simplicity. The secondary scripts I want them to retrieve the data process them and also store their output on separate files, for comparison reasons after. I want to compare how many data where able to process and how fast!

        In conclusion even the secondary scripts are test scripts I will chose only one of them to apply on my real goal. I am testing them to see performance under pressure in order to choose the best one.

        This is the reason that either I need to store the data to a folder for later retrieving them or into a database. The problem is also that on my real goal several scripts will need to access this folder to retrieve the data.

        So I am trying to make my experiments as realistic as possible.

        So enough said, time for coding. I need first of all to compare the times. Initially with the folders in different formations and possible create a short script with MySQL also just to observe if there is a huge difference on process time.

        P.S. Sorry for the long answer I tried to include everything.

[reply]

In Section Seekers of Perl Wisdom