data67 has asked for the wisdom of the Perl Monks concerning the following question:

Background
I have this problem with a complex sorting report that i have been trying to figure out but it seems that i'll need help from all of you. I want to do is some fancy sorting (with atleast three level depth on some) with data pulled in fom two files. The files are devided in such a way that info. relevant to each is separated.
Here is an example.
Lets say we have two files one with information about the student like so:
student- file: (Note this is and example for this post-Data is ficticous)
<br> 456234567!Bob Smith!Engineering!Graduate 852234567!Ivan Milla!Business!Undergrad 123234567!Sergio Veara!Arts!Graduate 987234567!Richard Thompson!Economics!undergrad 963434567!Patrick Klivert!Economics!Graduate 789234567!Janfranco Vialli!Engineering!Graduate

In the above datafile we have SS#, name, major and status. Now look at the student information file:

<br> <br> 45623456!COMP8769!A!Dr.Smith!MWF!10:00 45623456!COMP7960!B!Dr.Smith!MWF!11:00 45623456!COMP7100!B!Dr.Rose!MWF!1:00 987234567!ECON6900!A!Dr.Jones!W!9:00 45623456!COMP7600!A!Dr.Gillian!MWF!9:00 852234567!BUS4000!C!Dr.Rossa!M!2:00 789234567!BUS4000!A!Dr.Rossa!M!2:00

In the above file, we collected the class info. about the students we have in the student datafile. Class-info file had: SS#,class,grade, Instructor,Days,TIme.

Problem
What i am trying to do is have a report that sorts both of these files in a few different ways automatically going through all the data. Here are examples of what kind of output i am looking for.

Type1 (Sort Students by a particular KEY i.e. By a class):
Students who took BUS4000
SCHOOL
Students
Other Data
Engineering Janfranco Vialli A, Dr.Rosa
Business Ivan Milla ...
etc.    
     
Students who took COMP8769
SCHOOL
Students
Other Data
Engineering Bob Smith A, Dr.Rosa
etc.    
     

 

Type2 (Regular Sort By all students):

SCHOOL
Students
Other Data
Engineering Janfranco Vialli A, Dr.Rosa, BUS4000
  Bob Smith A,Dr.Smith,COMP8769
B,Dr.Smith,COMP7960
etc.
Business Ivan Milla ...
Arts Sergio Veara ...
Economics Richard Thompson A, Dr. Jones,ECON6900
  Patrick Klivert ...
etc.    
     

As you see in the aboove example, all of this output should appear on one page. As you saw in Type1, i need to sort also based on a particular falue in the (student-info) datafile also. I have done soring like this in the past but this is a bit complicated from what i have been use to. As always any input will be great.

 

 

Replies are listed 'Best First'.
Re: Complex Soring/Reporting
by dws (Chancellor) on Apr 17, 2002 at 17:01 UTC
    With the right tool, this becomes a simple problem. Given the data above, the right tool is a relational database with a report writer. Whatever your feelings about Microsoft, MS Access makes quick work of stuff like this.

    Or try MySQL, though you'll have to do the report writing yourself, though having SQL will do half the work.

    Or, depending on what platform you're on, you might be able to use DBI::SQLite for a Perl-only database.

Re: Complex Soring/Reporting
by tachyon (Chancellor) on Apr 17, 2002 at 17:35 UTC

    So what you have is two tables and you want to combine the data. This is the sort of task that a relational database excels at. If you were to store your data in one you could then use SQL to pull the values out in any format you want using syntax like this:

    my $sql = 'SELECT student_id, grade, instructor FROM student_info_table WHERE class = ? ORDER BY student_id'; my $sth = $dbh->prepare_cached($sql); $sth->execute('BUS4000'); while ( @row = $sth->fetchrow_array ) { my ( $id, $grade, $instructor ) = @row; # blah }

    This lets you pull whatever you like out of a single table but the true power comes when you JOIN tables together so that you can pull results out of several tables at once. As you can see databases have inbuilt ORDER BY (sort) functionality as well. You will not regret it in the long run. Converting from your flat files to a database table can be done in around a dozen lines of perl.

    cheers

    tachyon

    s&&rsenoyhcatreve&&&s&n.+t&"$'$`$\"$\&"&ee&&y&srve&&d&&print

Re: Complex Soring/Reporting
by data67 (Monk) on Apr 17, 2002 at 16:45 UTC
    Oh and by the way, i did look at perldsc, perllol, perlref. A have to say they are definately a good resource for datastructures. But i guess i would'nt be posting this if i had my problem fixed.
Re: Complex Sorting/Reporting
by graff (Chancellor) on Apr 20, 2002 at 04:02 UTC
    I wonder if your difficulty stems from trying to anticipate and handle all kinds of reports in a single conception of the problem, envisioning a monolithic data structure that encompasses all the tabulations. Try an incremental approach instead:

    Pick one tabulation to do first, load and manipulate the data to solve that tabulation. Keep it simple, and just think a little about how you'll need to re-use the data in the next tabulation.

    Now pick the next tabulation to do. The data is already loaded and manageable, and the plan for this tabulation will dictate how you need to re-shuffle the data. Do that, then move on to the next tabulation.

    If you keep the primary data in a somewhat minimalist structure, it will be easier to compartmentalize each distinct tabulation, pass the basic data to each compartment for that flavor of output, and add more tabulations / compartments to the code as you need to.

    It's okay for the code to grow incrementally in this fashion. In some (most?) cases, coding the next tabulation will seem easier if you make up additional structures (keeping each one relatively simple) during the initial data read loops. Having three or four distinct HoA and/or HoH to feed four or five tables will get you finished more easily than one or two HoHoA, HoAoHoA or whatever -- especially if you design your simpler data structures the same way you would design a good RDB schema, e.g. one hash keyed by ssn to store "name, major,status", then a HoA keyed by course-id that stores lists of ssn's in each course, and so on.

Re: Complex Sorting/Reporting
by data67 (Monk) on Apr 19, 2002 at 15:22 UTC
    OK guys, i look here:
    foreach $item (sort @items) { @tmp= split (/!/, $item, 16); push (@test, @tmp); push @AoA, [ @tmp ]; } print "\n"; # Print the whole thing, one at a time. print "COMP7650 :\n"; print "_____________________\n"; for $i (0..$#AoA) { print "$class{$AoA[$i][0]} \n\t $student{$AoA[$i][0]}\n"; for $j (0..$#{ $AoA[$i] }) { if ($j eq 1) { print "\t\t\t- $AoA[$i][1]\n"; } } }

    In the above code i made an Array of array which holds the values from the student-information file.
    I also made a a hash for each of the items i want from the student-file. somehing like so:

    open (FILE, "student.data"); while (defined ($line = <FILE>)) { if ($line =~ /^(.*)!(.*)!(.*)!(.*)$/) { $id{$1} = $1; #id's $student{$1} = $2; $major{$1} = $3; $status{$1} = $4; } } close (FILE);

    The structure now looks like this (using Data::Dumper):

    $students = [ [ '45623456', 'COMP8769', 'A', 'Dr.Smith', 'MWF', '10:00', ], [ blah, blah, blah.. ] ];

    THe output for all of this is clear by looking at the for loop . I was able to sort by class but is still need to sort to look somthing like => sort by each attribute in student-info-row->Major->sort by student->info.

    I was wondering if i could do something like HoH->HOH->HoA. Does anyone have suggestions. From the begining i wanted to do this without using a DBMS/SQL but now i am this close to giving up on using complex data sturcture to do this. Any help will be most appreciated.

Re: Complex Soring/Reporting
by data67 (Monk) on Apr 17, 2002 at 18:41 UTC
    First of all, i have used a lot of DBI and RDBMS before. The are many resons i did not choose to use a RDBMS for this, one reason is that there is no need for a DBMS when all i am looking at is at most ten lines of data in each file.

    The other thing is that as i have mentioned before, i am familiar with the documentation on HoH, AoH, HoA etc. but i do not know how to put this all together and make it work. That is why i would prefer some sample code please.

      Why don't you at least take a stab at it so you can bring something to the table? I don't think anyone here wants to do your work for you.
      $ perldoc -f sort

      Amel - f.k.a. - kel

        Let me see if i can put something together as a starter..