in reply to Re: Perl / Win32 / VC++ / USE_LARGE_FILES
in thread Perl / Win32 / VC++ / USE_LARGE_FILES

That is exactly why I brought this up here rather than on p5p.

From the p5p perspective, Borland doesn't support huge files - so don't use that. Trouble is, the seperation of the build-time options isn't good enough, and even trying to build with PERLIO and not LARGE_FILES, breaks, because one or more of the three calls you mention is used by PERLIO!

From Borland's perspective, the compiler is free and doesn't support huge files. If you want that then buy their professional compiler suite.

That's the correct view from their (p5p and Borland) point of view, but I still want to build perl with Borland. I don't have, and have no wish to purchase VC++! Given that the resultant code is going to run on the same target, the OS obviously has the capability and given that PERLIO already isolates all the CRT IO components into win32_* wrappers, it becomes a fairly trivial matter to bypass the CRT for those three calls and go direct to the OS. This is what I have done, and it's working great!

The point of this post is that in addition to the various varients of tell(), seek() and stat() that must be transmuted to their 64-bit equivalents. The perl sources also use fgetpos() and fsetpos(). They use the definition of fpos_t which by default is 32-bit. It was when I went looking to see how these calls handle the transition to using a __i64 when using VC++ (so as to get a clue as to the best way for me to handle the same under Borland) that I came up empty.

The VC++ documentation (and now confirmed by halley above. Thanks!) indicates that the selection of which definition of fpos_t is used is determine by the target platform. If your building for a 64-bit platform, then the natural integer size is 64-bits and so that gets used. The problem is, the majority of Win32 platforms are 32-bit, but they still support huge files. The question that arose was how do you pursuade the VC++ compiler to select 64-bit semantics for fgetpos and fsetpos whilst building for a target where the natural integer size is still 32-bit.

This question is still unanswered! If I am right, then it is at possible that when building perl using VC++ using USE_LARGE_FILES and/or USE_PERLIO, that these two calls are still using the 32-bit definition of fpos_t. If that is the case, given the number of uses of fgetpos/fsetpos that appear in the sources, including it's use in the implementation of one or more of the _*i64() routines you mentioned above, then it would appear that there is a hole in perl's support for huge files even when using VC++. I have patched these two calls also for use with Borland, and I now have a fully specified, working version of perl built with Borland that correctly handles tell(), seek(), stat() (and -s) on files up to at least 10 GB, so I am a happy bunny.

However, if this was a bug with VC++, then that would be a fairly major bug and would be very much a matter for the attention of p5p. It is to this end that I have posted this here in the hope of verifying my suspicions with the help of other monks who have VC++ (& experience) before sending the very busy p5p guys on a wild goose chase.

That said, I have used AS 802 (built with VC++) to test this and it also handles my 10 GB file correctly. The only question I am now trying to clarify is how does it do this.


Examine what is said, not who speaks.
"Efficiency is intelligent laziness." -David Dunham
"When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
If I understand your problem, I can solve it! Of course, the same can be said for you.

  • Comment on Re: Re: Perl / Win32 / VC++ / USE_LARGE_FILES

Replies are listed 'Best First'.
Re: Re: Re: Perl / Win32 / VC++ / USE_LARGE_FILES
by Courage (Parson) on Aug 27, 2003 at 01:19 UTC
    Dear BrowserUK,

    Your answer proves that you exactly should bring this up on p5p rather than here :)

    Of course it is not good that 5.8.1-pre do not compiles now with Borland C++ without touching source code tree (need some *.h patching), yet 5.8.0 builds with Borland fine -- no USE_LARGE_FILES, but source tree builds fine. Hence incompatibility was introduced somewhere in between. At least this needs immediate patches in p5p, may be you'll be in time before RC5 snapshot.
    And 5.10.0-to-be also needs to be updated.

    It is wrong to state that p5p is not aware of Borland compiler, it is really supported, but sometimes there is no Borland wisdom handy. You're very welcomed to add your efforts. Honestly.

    I also must disagree with your "From Borland's perspective, the compiler is free and doesn't support huge files. If you want that then buy their professional compiler suite". Our department payed for professinal compiler suite, as of Borland C++ Builder 4, and it uses *exactly same compiler* as free compiler suite. If you buy commercial suite, you pay for IDE, VCL (another Borland CRT, but also lacking _*i64()). It has absolutely same and interchangeable compiler inside!

    Now I'll do a humble attempt to answer your question, but don't expect much from stupid dog :)

    You see many wrapper functions in ./win32/ sources. You don't even realize which of those are used and wrapped to what. The only I can say to you right now with probability of 99% is that fgetpos function from win32.c is *not* used, instead there is a forest of wrappers and ifdefs, and it is far from obvious what namely is called, look at occurences of fgetpos in miscellaneous files, especially this from perl.h:

    #ifdef USE_64_BIT_STDIO .... # if defined(USE_FGETPOS64) # define fgetpos fgetpos64 # endif and #ifdef USE_64_BIT_STDIO # ifdef HAS_FPOS64_T # undef Fpos_t # define Fpos_t fpos64_t # endif yet in config_H.vc contains #define Fpos_t fpos_t /* File position type */
    So underlying API finally will be called with fpos64_t which is no longer 32 bit.

    Anyhow, this becomes more and more complicated and I gave up when I looked into this last time, and I just beleived that "all works if API has 64 bit support and do not work otherwise".

    However, still source code needs fixing to revive Borland build. The problem exists because many people propose patches into ./win32 files and they do not check Borland build, so they break it from time to time

    And please do not mystify p5p and go discuss your thoughts there. You have your patches, so go and propose them.
    Even I speak there. Hence anyone could speak there (provied there is something on-topic to say).
    www.perl.com clearly suggests subscribing and discussing on p5p, so why not just go and do this?

    Courage, the Cowardly Dog

      Courage++ I think you have explained the fgetpos() fsetpos() connundrum.

      As both calls are called with the address of the fpos_t parameter, if you pass the address of a 64-bit space to the function that only uses 32-bits of it, and then you pass that back to another function that only inspects the first 32-bits of it, then neither will care whether the space pointed at was 32-bits or 64-bits in size. Hence, that line from config_H.vc

      #define Fpos_t fpos_t /* File position type */
      which as you say means that the underlying API will be called with a 64-bit chunck of memory regardless explains why it works.

      Thanks :) You put that one to bed nicely and (hee hee!) now I don't have to bother p5p with a wild goose chase:)

      As for submitting the patches, the only fly in the ointment there is that whilst I know which 3 files and 56 lines (all inserts) are effected, I am having trouble producing the patches in the form required by p5p. The diff utility I have (from The Unix in Perl project unfortunately cannot handle diff'ing the 5100 lines of win32.c. So it will have to wait until I find an alternative. Also, I need to (find where to) download bleedperl? and ensure that my patches are applied to that base rather then rc4.

      RE: The Borland's professional CRT not supporting huge files. Frankly I am surpised and disappointed by this. Shame on them:)

      Thanks for your help. It's good to know I am not the only one who finds the perl source codebase somewhat confusing:)


      Examine what is said, not who speaks.
      "Efficiency is intelligent laziness." -David Dunham
      "When I'm working on a problem, I never think about beauty. I think only how to solve the problem. But when I have finished, if the solution is not beautiful, I know it is wrong." -Richard Buckminster Fuller
      If I understand your problem, I can solve it! Of course, the same can be said for you.