Shreyak has asked for the wisdom of the Perl Monks concerning the following question:

i am not able to put more than 10k messages in MQSeries queue without committing them. I've to put all the messages as single unit and therefore could only commit once i put all messages in queue successfully . But i am getting error in putting more than 10k messages . Is there a way to do it?

  • Comment on Not able to put more than 10k uncommited messages in queue

Replies are listed 'Best First'.
Re: Not able to put more than 10k uncommited messages in queue
by AppleFritter (Vicar) on Jul 11, 2014 at 18:54 UTC
      eval{ open(MSGFILE,'/dir') or die("Cannot open dir"); while (<MSGFILE>) { next if ($_ =~ m/header/i); $msg = $_; $counter += 1; my $getmessage = MQSeries::Message->new; $getmessage->Data($msg); $dstqobj->Put(Message => $getmessage , Sync => 1) or $logger- +>logdie("Unable to put message onto Queue at line $counter\nCompCode + = " . $dstqobj->CompCode() . "\nReason = " . $dstqobj->Reason() . "\n"); } close MSGFILE; }; if($@){ $qmgr->Backout() or $logger->logdie("error in backout: $@"); } else{ $qmgr->Commit() or $logger->logdie("error in commiting messages +$!"); }

      I've put auto commit option off

Re: Not able to put more than 10k uncommited messages in queue
by jellisii2 (Hermit) on Jul 11, 2014 at 18:50 UTC
Re: Not able to put more than 10k uncommited messages in queue
by locked_user sundialsvc4 (Abbot) on Jul 12, 2014 at 00:49 UTC

    I suggest that you modify your approach.   You are most likely running into an architectural “big enough” limit, and it is probably going to be necessary for you to commit smaller batches.   The logic on the receiving end would have to understand that the mere receipt of a particular block of messages does not necessarily mean that the transmission is complete.   Instead, you would need to devise a message that corresponds to the radio-operator’s familiar, “over...”   For instance, ten blocks of 1,000 messages each, then, “over ...”   The receiver would have to have a place (aside from the queue) in which to buffer the received stream until it finally received “over...”

      To answer a private-messaged question:

      What you are doing here is eliminating any issues imposed by the architectural limits of the queue.   Send units of work across the wire in chunks of convenient and sensible size.   Accumulate them all.   When the receiver hears “over,” it acts upon them as a single atomic unit-of-work.   (And, say, if it hears “forget that,” instead, then it throws-away what it had heard and ignores it, as instructed.)

      Don’t let the messages be “unduly large,” such that this conversation becomes a “million-pound elephant” for the queueing infrastructure.   Just send reasonably-sized chunks, as many as may be necessary.