in reply to Log Parsing

It just takes a simple regex :)
( ducking )

#!/usr/bin/perl -l # http://perlmonks.org/?node_id=1186713 use strict; use warnings; $_ = do { local $/; <DATA> }; print "topic,start_time,Endtime"; print join ',', $3, $2, $4 while /\b(\d+)-SUCCESSFUL (?= (?:.*\n)* \d+\ (\S+).*Started.*\b(\d+_\1_\d+_IN_0) (?:.*\n)* \d+\ (\S+).*Done.*\b\3 )/gx; __DATA__ 0317 09:53:14.865+0000 {12772} INFO [pm-worker-exec slot-Task:id=8274 +,env=12772,type=11][c.s.w.t.f.s.PostExecutionStage ] Loaded {child r +unId vs completion type}: {8286-SUCCESSFUL}{8287-SUCCESSFUL}{8288-SUC +CESSFUL}{8289-SUCCESSFUL}{8290-SUCCESSFUL}{8291-SUCCESSFUL}{8292-SUCC +ESSFUL}{8293-SUCCESSFUL}{8294-SUCCESSFUL}{8295-SUCCESSFUL}{8296-SUCCE +SSFUL} 0317 09:54:12.498+0000 {12772} INFO [pm-worker-exec slot-Task:id=8273 +,env=12772,type=55][edProcessInputBatchKafkaProducer] Started produci +ng records on topic 12772_8286_20170317_IN_0 0317 09:54:13.428+0000 {12772} INFO [pm-worker-exec slot-Task:id=8273 +,env=12772,type=55][edProcessInputBatchKafkaProducer] Started produci +ng records on topic 12772_8287_20170317_IN_0 0317 09:55:13.027+0000 {12772} INFO [pm-worker-exec slot-Task:id=8273 +,env=12772,type=55][edProcessInputBatchKafkaProducer] Done with produ +cing records on topic 12772_8286_20170317_IN_0 0317 09:55:15.027+0000 {12772} INFO [pm-worker-exec slot-Task:id=8273 +,env=12772,type=55][edProcessInputBatchKafkaProducer] Done with produ +cing records on topic 12772_8287_20170317_IN_0

Produces exactly your desired output.

Replies are listed 'Best First'.
Re^2: Log Parsing
by piyushmnnit06 (Novice) on Apr 03, 2017 at 06:09 UTC
    Thanks it worked fine for this small data set .But I have to do it for complete log ,I mean I have top open a file and the iteratively have to do it .

      How big is your log file? If it's really big, that's a crucial piece of information that should be included with the rest of the problem statement.

        Size is around 30-40 mb.