Beefy Boxes and Bandwidth Generously Provided by pair Networks
Your skill will accomplish
what the force of many cannot
 
PerlMonks  

Re: Web monitoring with Perl

by haukex (Archbishop)
on Jan 24, 2017 at 11:09 UTC ( [id://1180208]=note: print w/replies, xml ) Need Help??


in reply to Web moniitoring with Perl, comments and suggestions on general approach please

Hi predrag,

How it is about Perl installation and version? I expect Perl 5.10.1

While that Perl version may be "good enough", it's a little over 7 years old, so you should definitely look into installing a newer version.

Is 1 or 2GB RAM enough for this my application?

Of course that'll depend on the software you use, but just as a guess I'd say that it's probably enough for a start. Just in case, it would be good if your cloud provider offers you an easy way to increase that later if necessary.

2. The opposite: To create a small web server with PICs on the remote place, with MySQL base, so the website server should have to connect to it and read data two times a day.

While possible, I would say that it's probably better for the sensors to push their data to a central location (web server). That way, if you later have several sensor systems, the web server doesn't have to reach out to multiple remote locations to fetch the data.

I know about modules DBI and DBD:: mysql. Any recommendations for these or something other?

For your application I'd say MySQL is a good database, and those two modules are "the" modules for accessing databases. There are nowadays also more advanced modules, such as DBIx::Class, but I've found those are worth the effort when saving complex object-oriented data structures, whereas it sounds like your data will be mostly row-based with probably only one table or two. In that case DBI will be fine. Note: Make sure to use placeholders!

a cron job calling that Perl script that generates the charts twice a day and saves them as image files. ... But if I would like to do monitoring during the whole year, I have to find a solutions for that.

Generating static images and HTML pages is of course somewhat limited, but it's not all too difficult to write a CGI script* that allows the user to query and display certain ranges of data in a table dynamically. This could be done in the "traditional" way of having an HTML form, which is submitted to a CGI script which then fetches the data from the database and generates the HTML to send back to the browser. That's probably easiest for a first version, but later you could implement a more modern dynamic webpage where the JavaScript code (with something like jQuery) fetches the data in a format such as JSON and renders it in the browser dynamically.

I wonder if there is anything between Perl and PICs? ... Instead I could have a computer there, accept data to the serial port (?) and write Perl scripts that read data and present in a form of graphs and tables on HTML pages.

I would nowadays strongly suggest a small, single-board computer such as the Raspberry Pi, instead of a microcontroller like the PIC. I'm a little biased since I've been working with them for several months now to develop a data logging system (and I've released my software under the GPL), but there are also lots of other similar single-board systems. I think the advantage of such systems is that they feature full OSes, and most of the Linux software that runs on your PC will also run on an RPi, including Perl. Just a note, since RPis use SD cards to store data, backups of the data are important since SD cards are known to fail from time to time, especially when written to often.

You asked about the serial port, but in my experience it's easiest to use USB. Of course there are USB-to-Serial adapters which work fine with the RPi, so when selecting your sensors I'd recommend looking for ones that feature either USB interfaces out of the box, or at least RS-232/RS-485 so you can use an adapter. Many of these sensors will appear on the Linux system as a virtual serial port, so reading from them is just like reading from a regular serial port, but some require special drivers, this is also something to look out for when selecting sensors.

So here's an idea for the system architecture, based partially on my experience with the data logging system I'm currently working on. This is, of course, only One Way To Do It, there are lots of alternative ways to design the system. As mentioned, the sensors (USB, or RS-232/RS-485) would be connected to the RPi, where they can be read in one of two ways. First, for very low data rates (like twice a day, or hourly), probably a cron job that calls a Perl script which reads one sensor value and posts it to the web server is enough. Second, for higher data rates, it's probably better to stay connected to the sensor, continually reading from the serial port and then processing the data as needed: writing it to a text file, a database, etc. (this is what my code does) and then sending averages or summaries to be displayed on the web server. But since you said that for now, updates to the data will be infrequent, and because a cron job is easier to set up than a daemon, I'll assume the first solution.

So on the RPi, the script would have the tasks of connecting to the serial port (e.g. Device::SerialPort), if necessary requesting a value from the sensor, reading that value out, and if necessary parsing it into a usable format. Then, it would need to submit that data to the web server, for example with the standard module LWP::UserAgent, most likely with a POST request. A common data exchange format is JSON.

On the web server side, you'd have one CGI script* that accepts these data values - it should be protected with some kind of authentication mechanism to make sure only the RPis can submit data. This script could then enter the sensor values into the database. Second, there'd need to be another script that reads values from the database and generates the charts, tables, etc. How this script gets triggered depends on how real-time you want your data updates to be, but with data updates only twice a day, I think a cron job timed to run a little while after the RPis would have submitted their data is probably enough for a first version. For example, the RPis could be configured to submit their data at 11:45 UTC, and then web pages are generated at 12:00 UTC - note all systems should use NTP. A more advanced later version might have the updates to the database trigger the re-generation of the web pages. But the fairly simple set-up with cron would already allow your users to view "near-real-time" data using the static images and HTML files. A next step might then be a more dynamic web interface, as already described above.

* Note that when I say "CGI script", I don't mean specifically the CGI module, for new projects it's better to use one of the more modern web frameworks I mentioned in my previous post, such as Mojolicious.

Much of my suggestions describe a fairly simple mechanism because you said that this would be your first web based project, plus your data updates would be relatively infrequent. In several places I mentioned some future improvements, that might become necessary if you have data updates faster than roughly every half hour or so, or a larger number of sensor systems submitting data - maybe more than 10 sensor systems, but that's only an order of magnitude estimate. But hopefully this is one good place to start.

Hope this helps,
-- Hauke D

Replies are listed 'Best First'.
Re^2: Web monitoring with Perl
by predrag (Scribe) on Jan 24, 2017 at 16:42 UTC

    stevieb hippo marto and Hauke D, Thanks so much. I hope I will be able to help others one day too, if not by my knowledge, then by some experience in my future projects I expect will have

    I will answer to all replies soon, but first just to add few comments to my main post.

    Regarding data frequency it should be 3 or 4 times instead of 2, because not all measurement will be done at the same time, but that doesn't matter much. It is also possible for some data to be held for some time and then to send few data together. For my purposes, it is not so important to have all data immediately on the server

    In meantime, after sending my post yesterday, I've got one maybe a bit strange or even ridiculous idea (I still don't have a right sense of that) about situation with many locations for monitoring

    I could send SMS messages from PIC or some other monitoring devices to a small server that I should have in that case, in my home or some other place with internet connection. I don't have many data during the day, so there would not be so much messages. On the other side I think SMS messaging sometimes may be late or unsure, so that idea is maybe not recommended.

    Regarding representing graphs for longer time then for one month, I've got an simple idea. I would plot one graph in a month and when next month comes, its graph will be shown on the HTML page instead of previous, and other graphs in server directory would have links on that page, so visitor could choose what they want. I see now there are numerous options

    Kind regards

    predrag

Re^2: Web monitoring with Perl
by predrag (Scribe) on Jan 24, 2017 at 21:46 UTC

    Hi Hauke D

    Thanks for such great help again and again, I can't show how all that is useful for me

    Yes, I also see that 5.10.1 will probably be too old Perl version for the server now

    Just in case, it would be good if your cloud provider offers you an easy way to increase that later if necessary.

    Yes, it is possible, very quickly

    While possible, I would say that it's probably better for the sensors to push their data to a central location (web server).

    Yes, I agree. The idea of the opposite solution was from the past, when I wanted just to try connection to mysql database, whether internal or external, as an example on my cPanel shared hosting. The hosting company had very restricted rules for that kind of hosting, but with a short procedure, they would allow it, and maybe easier 2. The opposite then the first. Also, there was an already installed program for reading from external base to cPanel, so I wanted to try it. But unfortunatelly, at the end, I didn't make that test.

    Thanks for recommending DBI module. Yes, I need a simple tool for that project, no need for something large

    As you wrote, regarding images (graphs) for my first project, from more reasons then one is beter to work with static images. Also, I've already explained the additional idea I've got about several graphs. Although simple, I am sure it will give very good and nice results. I think everybody will agree that content is the most important, not the form. Of course, that doesn't mean we should not use very modern and powerful tools if we need that.

    You and other perl monks definitely convinced me to Raspberry Pi . The fact that I can work in Perl is maybe prevailing. Before my beginning with Perl, RPi, although attractive, wasn't enough. In that time I wanted to avoid learning Python but now, since I already learn something new (Perl) and love it, even a bit of Python would not be a problem

    I've visited your website about GPS Tools. Thank you for the link and I am happy to learn from your experience and your work and the experience of other perl monks. I will read that later more detailed.

    When I see where you work, I will write something OT, but very short, you will see how much I love all this. It was during 1977, I was in secondary school, and for my thesis (90 pages) I measured air temperature, temperature of the water in the river and water level every day during the whole year. Thesis were about our local river ecosystem.

    Thanks for suggesting USB as your better experience. I've mentioned serial port without any special reason. I didn't use it for such purposes yet, just know people used it before

    note all systems should use NTP

    Yes, I know that now, it become more clear to me on my Linux courses

    * Note that when I say "CGI script",

    Yes, I've understood that by intuition because you've already wrote about new modules for that

    I must say I am overwhelmed with information but I know it will settle down, especialy when I start with the project. Because of many reasons, I could not start that project before, and also, I can't start at this moment yet, but will use that time to learn

    Kind regards

    predrag

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://1180208]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others taking refuge in the Monastery: (4)
As of 2024-04-19 06:34 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found