AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows?

sectokia has asked for the wisdom of the Perl Monks concerning the following question:

Hi Monks,

I was tracking down my issue with IP's in hosts file not resolving when using AnyEvent on Windows.

I was eventually lead to this code in AnyEvent::Socket.pm:

sub _parse_hosts($) {
   %HOSTS = ();
   
  for (split /\n/, $_[0]) {
      s/#.*$//;
      s/^[ \t]+//;
      y/A-Z/a-z/;

      my ($addr, @aliases) = split /[ \t]+/;

...
[download]

The problem is on windows the lines in the hosts file end with '\r\n'. But this code only splits on '\n'. So whats happening is '\r' ends up literally in the (last) alias, and thus it will never match when later the attempt is made to match the hostname to the alias.

Is this a mistake in this module? Or have I somehow messed up and perl on windows? Is perl meant to treat '\n' in regex as '\r\n' when running on windows?

Comment on AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? Download Code

Replies are listed 'Best First'.
Re: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? (updated) by haukex (Archbishop) on Feb 08, 2022 at 09:28 UTC
Is this a mistake in this module? Or have I somehow messed up and perl on windows? Is perl meant to treat '\n' in regex as '\r\n' when running on windows? As others have pointed out, Perl normally uses the `:crlf` PerlIO layer on Windows. However, if you look at how `_parse_hosts` gets called, it's using AnyEvent::IO's `aio_load`, which does `open $fh, "<:raw:perlio"`, or AFAICT its XS equivalent when the IO::AIO backend is used. So while you could hack it by changing the file to use LF instead of CRLF - I have no idea whether other Windows libraries will like that, though! - or by monkey-patching `_parse_hosts`, I'm not sure you'll be able to change this behavior without getting in touch with the module's author. Just for fun, this ugly hack should work: `{ my $oldsub = \&AnyEvent::Socket::_parse_hosts; no warnings 'redefine'; AnyEvent::Socket::_parse_hosts = sub ($) { (my $in = shift) =~ s/\r\n/\n/g; $oldsub->($in) }; }` [download] Update:* Looking at the code, another possible workaround is to point the environment variable `PERL_ANYEVENT_HOSTS` at a hosts file that has only LF line endings.	[reply] [d/l] [select]
Re^2: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? (updated) by sectokia (Pilgrim) on Feb 09, 2022 at 10:12 UTC
Thanks for the great answer, I have certainly learned more about IO layers now. Your hack idea is better than my redefine hack - I had not thought of calling the old sub, I just redefined it with a fixed version.	[reply]
Re: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by choroba (Cardinal) on Feb 08, 2022 at 08:06 UTC
It depends on how the file was opened. If you open it with `:crlf` (which is the default on MSWin), `\r\n` will be converted to `\n` when reading the file. I have no idea how the hosts file is opened in AnyEvent::Socket, though. `map{substr$_->[0],$_->[1]\|\|0,1}[\\|\|{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^ARGV,3]`	[reply] [d/l] [select]
Re^2: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by syphilis (Archbishop) on Feb 08, 2022 at 10:53 UTC
If you open it with :crlf (which is the default on MSWin) If you're running perl "on Cygwin on MSWin" it looks to me that `:crlf` is not the default. I therefore deduce that you regard running perl "on Cygwin on MSWin" as not running perl "on MSWin". (Just trying to clear up exactly what it means to be "on MSWin".) Cheers, Rob	[reply] [d/l]
Re^3: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by choroba (Cardinal) on Feb 08, 2022 at 11:01 UTC
Yeah, by "running perl on MSWin" I mean `$^O eq 'MSWin32'` [download] `map{substr$_->[0],$_->[1]\|\|0,1}[\\|\|{},3],[[]],[ref qr-1,-,-1],[{}],[sub{}^ARGV,3]`	[reply] [d/l] [select]
Re^4: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by LanX (Saint) on Feb 08, 2022 at 11:26 UTC
Re^5: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by ikegami (Patriarch) on Feb 08, 2022 at 14:47 UTC
Some notes below your chosen depth have not been shown here
Re: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by syphilis (Archbishop) on Feb 08, 2022 at 05:31 UTC
Is perl meant to treat '\n' in regex as '\r\n' when running on windows? I think the answer depends upon whether perl has been built for native windows (eg Strawberry Perl or ActivePerl) or whether perl has been built for (eg) Cygwin or MSYS. What's your `perl -V` ? Sounds like your problem would be fixed by running `dos2unix` on the hosts file. (Running `unix2dos` on the same file will then revert it to its original form.) Cheers, Rob	[reply] [d/l] [select]
Re^2: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by sectokia (Pilgrim) on Feb 09, 2022 at 10:18 UTC
I encountered the issue running standard strawberry perl 64bit. Modifying hosts file wasn't really an option for me because the idea was the program is being pp'd into an exe and run by various users on their own systems.	[reply]
Re: AnyEvent::Socket -> Hosts File -> CR/LF Issue on windows? by ikegami (Patriarch) on Feb 08, 2022 at 14:59 UTC
On Windows, Perl automatically converts CRLF to LF on read, and LF to CRLF on write (by default). But you might be using a `perl` built as part of unix emulation environment like Cygwin and I think MSYS/MSYS2. Such a Perl wouldn't perform such translations, even though it could be launched from a Windows program (e.g. the Windows shell). Possible solutions: You could install a native build of Perl (Strawberry Perl or ActivePerl). You could convert the file's line endings (e.g. using Perl or `dos2unix`). You could monkeypatch the module. (See haukex's answer.) You could patch the module.	[reply] [d/l] [select]


Think about Loose Coupling
	PerlMonks