Perl tk question

ghosh123 has asked for the wisdom of the Perl Monks concerning the following question:

Hi
I have a perl tk gui which can run some 100 jobs. Its inter-process communication is by socket communication. The gui interface has no idea about what job the internal engine is running. Through socket they communicate, and the gui display objects updates the gui pass/fail results as per the info through socket. Basically the gui serves as a server.
It stores the results in MySQL database.
My question is what could be the potential problem to make such a gui to be able to run some 20k jobs . What are the factors to be taken care of. The code has as much as 50 perl module files and each file has more or less 1k lines.
thanks.

Comment on Perl tk question

Replies are listed 'Best First'.
Re: Perl tk question by arkturuz (Curate) on Jul 29, 2013 at 18:05 UTC
the gui serves as a server. This is good. There shouldn't be problems if the number of objects increases as long as any one of them doesn't lock the whole application. It stores the results in MySQL database. The database will be a bottleneck. 100 clients and 20000 is not exactly the same if they want to C, R, U or D every other second or so. Database will have to be optimized for a lot of traffic. You will probably have to pass all queries through small number of server processes dedicated for this. Or you can use some messaging system like RabbitQM or similar although you would probably have to rewrite a lot of code. Other than that, it depends on what kind of jobs these programs are doing. Is it one process forked over 100 times, or a bunch of different process forked many times, or really 100 different processes? Do they respawn frequently on errors? (Like die() on every warn() and similar.) Is there a lot of network activity? I would concentrate on ensuring that the database access is always working properly when increasing the number of clients, that network issues are handled properly with timeouts and that very frequent respawns in case of errors are detected as soon as possible. This is pretty general, but it also depends on what kind of work your software is doing.	[reply]
Re: Perl tk question by locked_user sundialsvc4 (Abbot) on Jul 30, 2013 at 04:12 UTC
Probably the most-important concern will be ... no matter if you have “20,000 jobs” to run ... that you are by some means able to effectively control how many simultaneous jobs run at one time.