|
Re: XS Modules - why and when?⭐
by LanX (Saint) on Dec 05, 2023 at 00:41 UTC
|
> is for speed as compiled languages of the C family are faster than interpreted languages such as Perl. Is that about right so far?
No, it's primarily about interfacing with an existing C library.
Speed is only one possible reason for using/creating that library.
Another is connecting to an API without native Perl support.
| [reply] |
|
|
Compatibility with defined behaviour of variables in another programming language (overflow etc) is another reason, like recently discussed in Trying to translate overflowing JS code to Perl. It might just be easier to load an existing library or interpreter than to recreate all that behaviour in Perl.
Another popular reason is security. I'd rather trust some XS bindings to the always updated and audited OpenSSL library, written by security experts, used by some of the biggest companies in the world) to secure my connections than i would to some pure-perl reimplementation made by a one-person team in their spare time.
Plus, why go to all the hassle of designing a complicated set of Perl modules to, say, implement an interpreter/interface/protocol/crypto library when some nice chaps on the internet have already gone through all the hard parts and gave you a perfectly fine C library with all the trimmings and bugfixes and workarounds. All you have to do is write a nice XS wrapper(*) around it and you're done. Unless the developers of that lib decide to change the interface, you even get all the updates for free ;-)
(*) If you're lucky and the interface is simple enough, h2xs might even do a lot of the work for you. Thumbs up for Perl scripts that writes Perl scripts for you!
| [reply] |
|
|
No, it's primarily about interfacing with an existing C library
Thanks...that makes a lot of sense. cavac's point about security resonates here -> Re^2: XS Modules - why and when?
But in RPi::DHT11, there doesn't seem to be any reason to use XS unless the C code already existed. I had assumed stevieb wrote the C code as well as the Perl code, but I have nothing to base that on.
If we look at RPi::PIGPIO::Device::DHT22 we can see a pure Perl method of reading the output from the device (DHT11 and DHT22 are read the same way). As Perl is rather good at bitwise and mathematical operations, it just seems weird to me that XS would be used in this situation.
| [reply] |
|
|
I quickly scanned the POD for RPi::DHT11 and found:
> This module requires the wiringPi library to be installed
And
http://wiringpi.com/ states:
>
WiringPi is a PIN based GPIO access library written in C
And
> is intended for use by experienced C/C++ programmers.
So, doesn't this answer your question?
| [reply] |
|
|
If we look at RPi::PIGPIO::Device::DHT22 we can see a pure Perl method of reading the output from the device (DHT11 and DHT22 are read the same way).
Ahhh, but wait! Understand that the library you mention is a TCP connector to a daemon that is running on the Pi hardware. That daemon is what does all of the GPIO register manipulation directly. That daemon is written in guess what? C! So the library you mention does not interact with the hardware directly. It communicates with software that does. My software directly interfaces the hardware, which is why I needed C/XS to do so. This library communicates with someone elses software via a TCP socket that does the down and dirty work.
See here for info on pigpio.
From the pigpio web page:
The pigpio library is written in the C programming language. The pigpio daemon offers a socket and pipe interface to the underlying C library.
So that's why the library you mention can be written in Pure Perl. Note that the author of the Perl library you mention and the author of pigpio are not the same person.
| [reply] |
|
|
|
|
|
Re: XS Modules - why and when? (XS References)⭐
by eyepopslikeamosquito (Archbishop) on Dec 05, 2023 at 01:14 UTC
|
Bod, surprisingly I don't seem to have a list of references on this topic yet.
This looks like a good place to start:
See also:
Other cool references welcome.
| [reply] |
|
|
Great, the WP article is already detailing 5 common reasons: (emphasis added)
>
It may be desirable for a Perl program to invoke a C subroutine in order to handle very CPU or memory intensive tasks , to interface with hardware or low-level system facilities , or to make use of existing C subroutine libraries.
| [reply] |
|
|
| [reply] |
Re: XS Modules - why and when?
by eyepopslikeamosquito (Archbishop) on Dec 05, 2023 at 05:44 UTC
|
> My understanding of an XS module is that it uses XSLoader to load C / C++ / C# code
Good luck with C# :)
Despite its name, C# is much closer to Java than C
(there's even a Comparison of C# and Java wikipedia page).
Historically, C# was invented by Microsoft after years of expensive legal battles with Sun over Java licensing ...
so they could hardly call it Java# :) - see this node for more details.
Note that the XS (Perl) wikipedia reference states:
XS is a Perl foreign function interface through which a program can call a C or C++ subroutine
Though XS is essentially C-based, it's possible to provide a C interface to C++ code
(BTW, the original 1983 Cfront C++ compiler converted C++ to C).
| [reply] |
|
|
Good luck with C# :)
I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory... So it could be used with Ada or Zig or anything in between.
| [reply] |
|
|
Not directly, no. From perlxs:
XS is an interface description file format used to create an extension interface between Perl and C code (or a C library) which one wishes to use with Perl.
...
The glue code pulls the arguments from the Perl stack, converts these Perl values to the formats expected by a C function, calls this C function, and then transfers the return values of the C function back to Perl.
So XS is inextricably tied to C. That is not to say that you could not use XS via C as a conduit to access other, non-C code but it would be a bit Heath Robinson. Perhaps FFI::Platypus would be a better bet for that.
FTAOD, I am in no way an XS expert and can count on the fingers of one hand the number of times I have written such code so treat all this with a little scepticism.
| [reply] |
|
|
I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory...
Unfortunately not. As someone already mentioned, it has the capability to play nice with C++ code (wiringPi is in fact C++ based), but there are a lot of gotchas that can prevent it from working, and often requires overriding/rewriting the functions that don't work in pure C before they can be presented through the Perl library.
I also write a fair amount of C# code (berrybrew would be my most complex example), but wrapping something like that with XS would be an impossible, futile, headdesking task.
| [reply] |
|
|
XS is always C, but can call into anything that exposes C-compatible functions in a shared library. So, C works, and C++ works as long as the author of the C++ module exposes plain-old-functions with C calling convention. There are other languages like Rust that can expose C functions from a compiled library even though the library was generated from Rust code. In fact, quite a lot of compiled languages can expose C library APIs just because C is sort of like a common meeting ground for interoperation and language designers want their modules to be widely usable. But XS can't just call into arbitrary other languages on their home turf.
Also I'd hesitate to call C# a compiled language. It's more like a bytecode interpreted language with optional just-in-time compilation. I don't think it can generate .so or .dll files with C calling convention. (but I'm too lazy to go find out)
| [reply] |
|
|
|
|
|
|
> I had assumed, perhaps wrongly, that the XS mechanism allows Perl to be connected to any compiled language, at least in theory
While I'm not an XS expert -- and further
to ++hippo's excellent reply --
I'd put both Java and C# in a separate category
due to their reliance on an underlying virtual machine.
I'm not aware of any plans to implement Perl 5 on the JVM (Java)
or the CLR (C#).
That this would require a huge technical effort is indicated at:
Update: from the FFI::Platypus docs:
Things not supported include languages that do not compile to machine code ... like .NET based languages and Java
| [reply] |
Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 18:07 UTC
|
Sometimes I use XS to wrap existing C libraries. WiringPi::API is probably one of my larger examples. Not only do I wrap all of WiringPi with it, but within the XS, I often override or wrap the wrapped C functions because I need something to work slightly differently. here is one example.
Other times, I interface with ICs and other devices where clocking bits in and out with precise timing is extremely critical to proper operation (example). That timing I found is often more reliable when written in a compiled language and then presented with a Perl interface.
Other times it's purely for speed for sure.
Sometimes, I've already written something in C for a project that I want to break out into a standalone project, so I put the C code into XS, then present it as a Perl interface. In these cases, I often will also write a Pure Perl port of it. Bit::Manip is one example. That distribution is purely written in C. The XS portion simply presents the C functions to the Perl library.
Sometimes I use XS for academic purposes. I don't write a whole lot of C code so I like to dabble in it from time to time before it fades from my memory completely. Personally I don't write the XS, I write C, then have Inline::C develop the XS for me.
In the case of RPi::DHT11, one has to turn high/low physical GPIO pins to read the data from the sensor. Using the wiringPi library allowed me to do that with an existing interface so I didn't have to write my own. That's why I spent two years wrapping the wiringPi C library in near its entirety. Writing something like that in pure Perl would have been impossible to deal with such low level hardware registers. The timing wasn't the issue here; the manipulation of the GPIO hardware registers was.
-stevieb
| [reply] |
Re: XS Modules - why and when?
by Marshall (Canon) on Dec 05, 2023 at 00:43 UTC
|
| [reply] |
Re: XS Modules - why and when?
by davido (Cardinal) on Dec 05, 2023 at 22:26 UTC
|
Another reason some things are done in XS is out of convenience in getting Perl to do things that Perl isn't made to do. There are modules written that mess around inside of Perl's internals in ways that would be really hard to do, if not nearly impossible to do, in Perl code.
| [reply] |
Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 18:41 UTC
|
I forgot another example of why someone may use C/XS that I've had experience with. One of my clients a few years ago had a Perl system that they deployed at client sites. The software was proprietary (ie. non-Open Source), but as we all know, there's no real way to obfuscate it so it's easily prone to theft and copying.
My client wanted a way to prevent this easy theft, so I suggested that we write some of the critical core components in C, compile it, then present it to Perl. That way, it's beyond trivial for the software to be stolen. Sure, the client could see all of the Perl code, but we didn't include the C library source, just the compiled shared library. Not perfect by any stretch, but pretty effective.
| [reply] |
Re: XS Modules - why and when?
by stevieb (Canon) on Dec 05, 2023 at 20:48 UTC
|
The DHT11 is a temperature and humidity sensor with a maximum frequency of 0.5Hz (one read per 2 seconds). So speed is not the issue here!
Yes, data can be read only once every two seconds, but that's not the timing one needs to be concerned with. The real timing issue is in issuing digital signals to the device to inform it that we're about to read data from it. This is measured in microseconds. Perl, at best can reliably deal in milliseconds.
You can extract two formats of data from this device. First, you have to hold the signal pin LOW for 18us (microseconds) to tell it you want to read data, then set it back HIGH. For the first format that is returned, the device sets the data pin low for exactly 50us (microseconds), then immediately hold it high for 26-28us. For format two, holds low for 50us, then high for 70us. If you're off even marginally sending or receiving these signals, you'll possibly get the incorrect format, corrupt or no data at all.
So on a Raspberry Pi, even while using C, you can still get mis-timed events because of possible contention of the CPU. This is why microcontrollers are so much better for this. They operate instructions in a linear fashion and never have the overhead of other processes competing for CPU cycles.
Also, Perl can't manipulate hardware registers like this directly. Even if it could, the timing would be too unreliable to be effective.
| [reply] |
Re: XS Modules - why and when?
by stevieb (Canon) on Dec 20, 2023 at 09:41 UTC
|
To bookend this whole thing, I want to say that I would stay away from the DHT* type sensors.
For almost 10 years, I had a significantly nice multi-stage indoor cannabis growing operation that I used them for to help me with my fully automated hydroponic system.
This is but my personal experience. Not only had I written software against them for the Pi, I also have software that I wrote for my real controllers, little Node MCU units, particularly the ESP-8266 and its derivatives (I still use frequently my WeMOS D1 Mini units). The DHT units required a lot of averaging and mediating of numbers to ensure a correct result. I learned by using them though. I learned a lot about math. Monks here taught me about "filtered moving average" and "probability normalization" and such. That's a fun game, literally, until I just want to code more.
I don't like to give advice unless asked, but I would say that instead of the DHT* units, based on my experience with the difficulties they presented me, it may be prudent to investigate other options on hardware for the rather delicate situation** that faces you.
-stevieb
** Inserting devices and custom software into your HVAC may influence how your insurance company looks at you, so I would definitely consider this as delicate.
| [reply] |