derby scratches his head and tries to remember all that mainframe crap he was forced to learn early in his career. We had a similar set-up (M204, VSAM, MVS, JCL, etc etc). I got damn tired of login into TSO, tweaking some JCL, submitting etc. so I wrote a perl script (4.x) that would (no DBI back then) merge a bunch of JCL, submit it to the mainframe and then wait for the data to be deposited in his *nix account.

Now I had some major differences from your set-up. First, our M204 database were backed up to VSAM files on a nightly basis and there were existing JCL snippets (what were those called exactly) that could query M204 directly if necessary (rarely was). So having to go against VSAM only (via JCL) simplified things immensely.

The second difference was we had snippets of JCL that could download data to a *nix box. That was fantastic. My perl script could just spin it's wheels waiting for the file to show up. I would then do some follow on processing in *nix.

Now for your application, you don't have to necessarily download all the data but you would need something to let you know the newly created dataset was available (which given the size of you're data, you may want to asynch that somehow). Once that temp dataset is there, you could retrieve it in chunks.

Sorry if this isn't much help. Just wanted to let you know I feel your pain. I think if you can get as much possible into a VSAM dataset and then use JCL (or perl on the mainframe) you should be able to easily develop a DBD driver for fetching.

-derby

update: I also remember being excited about EDA/SQL from ibi but it never made it out of our labs. I think that product has been rolled into a new company iwaysoftware. But I cannot comment one way or another about their products.