You may be interested in App::RPi::EnvUI. It's a Dancer2 application that uses several external sensors (hygrometer, moisture etc) on the Raspberry Pi, with built-in async timers to fetch the data from the sensors, turn on/off fans, humidifiers when limits are reached, with a jQuery one-page front-end, with statistics graphing for temp/humidity. It also provides basic authentication for the "routes" that perform write operations. The data is stored locally on the Pi in the above cut, but it'd be trivial to write to a remote DB backend.

Even though the code isn't yet finished (as I'm still writing drivers for some sensors), it definitely works (as I'm using it currently in one of my grow rooms), but it might give you some ideas...

update: This software writes to the DB at least once every three seconds per sensor (by default), which doesn't include incidental writes (user toggles a UI switch, background maintenance etc, so there could be ~10 writes/sec or sometimes more), and reads happen every three for every browser pointed to the site. It's on an SQLite DB, running off of a micro SD card and it's been in operation for nearly six months.

Specs for the DB server and front-end web server will be determined by the speed of data transactions, and the number of visitors to the site (amongst other things). Because my app is essentially for me, it's quite speedy even on the RPi hardware, but I've at most had four different browsers to it open at once.

In essence, what you're wanting to do can most certainly be done in Perl (the backend and web front end for sure), and depending on the microcontroller, the sensors can be driven in Perl as well (in my case, Perl API that wraps over my (and others) C code that interacts with the actual sensors). If you're using a literal microcontroller (ie., eg: Arduino) that can't run Perl, the C code can gather the data, and send it back somewhere else where it can be processed./update

stevieb Thanks again for your comments.

After this first project, I will for sure need some data with high frequency

Specs for the DB server and front-end web server will be determined by the speed of data transactions, and the number of visitors to the site (amongst other things).

It is really interesting and of a big importance in my case. I didn't think in that way neither knew about that

I've didn't thought about Arduino, but about Raspberry Pi yes, and after your and other monks' comments here, I will for sure go for RPi too.

Useful to save your comments for my work. Good luck with your project.

regards, predrag

Thanks, and all the best to you as well in your endeavors.

If you're going the Pi route, I suggest you take a look at some of my CPAN modules that I've written for it:

• WiringPi::API: direct wrapper of wiringPi
• RPi::WiringPi: provides object oriented access to various pieces of hardware and the GPIO, with some safety built in, and includes direct interfaces to a BMP180 barometric pressure sensor, and 74HC595 series shift registers (up to 32 extra output pins using only 3 GPIO). Eventually, I'll break these two modules out of this distribution and make them standalone)
• RPi::ADC::ADS: interface to Adafruit's ADS1x15 analog to digital converters (allows you to read analog inputs on the Pi)
• RPi::DHT11: interface for the DHT11 hygrometer (temp & humidity)
• RPi::DigiPot::MCP4XXXX: interface to the MCP4xxxx series digital potentiometers, provides the Pi with a way to output analog signals, without having to use the single built-in PWM GPIO pin
• RPi::SPI: module that allows you to communicate on the serial peripheral interface bus, to aid in writing new drivers for devices that communicate using SPI

Note that some of the documentation may be a slight bit off, as I'm just now going through ensuring everything works properly and is documented correctly.

There are others I didn't mention above, as they don't seem like they'd be of any use to you for your current project. Others that may help though that are in various stages of completeness that aren't yet on the CPAN:

• MCP49xx series digital-to-analog converters (DAC), allows for analog output (similar to the digipot above). These units range from 8-12 bit resolution, and are packaged with either a single or dual onboard output channels
• MCP3008 (and MCP3004) series analog-to-digital converters, providing 4 or 8 analog input pins, with 10-bit resolution

Some points. Make life easier on yourself :) A Raspberry Pi 3 has Quad core ARM, @ 1.2GHz, 1GB RAM, has wired and wireless networking, Bluetooth and Blutooth LE, has GPIO and USB to talk to whatever else you wish. Storage, stick in a large Micro SD card, (optionally) add a couple of USB thumb drives, backup data (along with your config data & code) remotely should the bees attack (or you know, legitimate disaster recovery).

Weather/environment proof cases are relatively cheap, or if you have time fairly easy to build. You could run your data logging and code and host the website on the Pi, restricting access as you wish. I'd opt for SQLite over mysql (consider also a lighter web server such as lighttpd), you could have real time (ish) charting returning JSON data to some of todays modern JavaScript charting solutions. I've been playing with Mojolicious (see also Mojolicious::Lite) and find it to be a nice modern framework to build upon.

Update: added some more links.

marto Thanks again for the comments.

The reason why I was thinking about PICs is probably because I like the idea "Small is beautiful" (Schumacher) in general. Of course, I understand that while working on a project, we have to have in mind the whole picture and choose what is the most appropriate, whether it is small or big. And as I wrote in other replies, before, I was't sure whether RPi is for me to go, and now I see it is, especially for some next projects I dream about. I know about RPi for maybe 2 years or a bit longer, but am not informed about new versions. Wow, this Raspberry Pi 3 is the same as my Virtual machine for CentOS 6 where I work with Perl!

Weather/environment proof cases are relatively cheap, or if you have time fairly easy to build

I don't know just now, but maybe will build itmyself. It will need a heating during the winter too.

kind regards

predrag

Another option would be the ESP8266 clone boards, which should be fine for your purposes, alternatively the ESP32, is an upgraded board. They cost ~ 2 euro delivered from either ebay, aliexpress etc. They can be programmed using the Arduino IDE if you're already familiar with that, or via the other usual routes. Built in Wifi and so on, despite the relativity low specs you can do a lot with them. This tutorial give a short primer on the board and a working example of using it to read the values of 2 types of sensor, and display them on demand in a browser via a web server running on the board. Indeed there are lots of options open to you. Once you've had time to process them all please let me know which way you intend to go, or if you have further questions.

Hi predrag,

How it is about Perl installation and version? I expect Perl 5.10.1

While that Perl version may be "good enough", it's a little over 7 years old, so you should definitely look into installing a newer version.

Is 1 or 2GB RAM enough for this my application?

Of course that'll depend on the software you use, but just as a guess I'd say that it's probably enough for a start. Just in case, it would be good if your cloud provider offers you an easy way to increase that later if necessary.

2. The opposite: To create a small web server with PICs on the remote place, with MySQL base, so the website server should have to connect to it and read data two times a day.

While possible, I would say that it's probably better for the sensors to push their data to a central location (web server). That way, if you later have several sensor systems, the web server doesn't have to reach out to multiple remote locations to fetch the data.

I know about modules DBI and DBD:: mysql. Any recommendations for these or something other?

For your application I'd say MySQL is a good database, and those two modules are "the" modules for accessing databases. There are nowadays also more advanced modules, such as DBIx::Class, but I've found those are worth the effort when saving complex object-oriented data structures, whereas it sounds like your data will be mostly row-based with probably only one table or two. In that case DBI will be fine. Note: Make sure to use placeholders!

a cron job calling that Perl script that generates the charts twice a day and saves them as image files. ... But if I would like to do monitoring during the whole year, I have to find a solutions for that.

Generating static images and HTML pages is of course somewhat limited, but it's not all too difficult to write a CGI script* that allows the user to query and display certain ranges of data in a table dynamically. This could be done in the "traditional" way of having an HTML form, which is submitted to a CGI script which then fetches the data from the database and generates the HTML to send back to the browser. That's probably easiest for a first version, but later you could implement a more modern dynamic webpage where the JavaScript code (with something like jQuery) fetches the data in a format such as JSON and renders it in the browser dynamically.

I wonder if there is anything between Perl and PICs? ... Instead I could have a computer there, accept data to the serial port (?) and write Perl scripts that read data and present in a form of graphs and tables on HTML pages.

I would nowadays strongly suggest a small, single-board computer such as the Raspberry Pi, instead of a microcontroller like the PIC. I'm a little biased since I've been working with them for several months now to develop a data logging system (and I've released my software under the GPL), but there are also lots of other similar single-board systems. I think the advantage of such systems is that they feature full OSes, and most of the Linux software that runs on your PC will also run on an RPi, including Perl. Just a note, since RPis use SD cards to store data, backups of the data are important since SD cards are known to fail from time to time, especially when written to often.

You asked about the serial port, but in my experience it's easiest to use USB. Of course there are USB-to-Serial adapters which work fine with the RPi, so when selecting your sensors I'd recommend looking for ones that feature either USB interfaces out of the box, or at least RS-232/RS-485 so you can use an adapter. Many of these sensors will appear on the Linux system as a virtual serial port, so reading from them is just like reading from a regular serial port, but some require special drivers, this is also something to look out for when selecting sensors.

So here's an idea for the system architecture, based partially on my experience with the data logging system I'm currently working on. This is, of course, only One Way To Do It, there are lots of alternative ways to design the system. As mentioned, the sensors (USB, or RS-232/RS-485) would be connected to the RPi, where they can be read in one of two ways. First, for very low data rates (like twice a day, or hourly), probably a cron job that calls a Perl script which reads one sensor value and posts it to the web server is enough. Second, for higher data rates, it's probably better to stay connected to the sensor, continually reading from the serial port and then processing the data as needed: writing it to a text file, a database, etc. (this is what my code does) and then sending averages or summaries to be displayed on the web server. But since you said that for now, updates to the data will be infrequent, and because a cron job is easier to set up than a daemon, I'll assume the first solution.

So on the RPi, the script would have the tasks of connecting to the serial port (e.g. Device::SerialPort), if necessary requesting a value from the sensor, reading that value out, and if necessary parsing it into a usable format. Then, it would need to submit that data to the web server, for example with the standard module LWP::UserAgent, most likely with a POST request. A common data exchange format is JSON.

On the web server side, you'd have one CGI script* that accepts these data values - it should be protected with some kind of authentication mechanism to make sure only the RPis can submit data. This script could then enter the sensor values into the database. Second, there'd need to be another script that reads values from the database and generates the charts, tables, etc. How this script gets triggered depends on how real-time you want your data updates to be, but with data updates only twice a day, I think a cron job timed to run a little while after the RPis would have submitted their data is probably enough for a first version. For example, the RPis could be configured to submit their data at 11:45 UTC, and then web pages are generated at 12:00 UTC - note all systems should use NTP. A more advanced later version might have the updates to the database trigger the re-generation of the web pages. But the fairly simple set-up with cron would already allow your users to view "near-real-time" data using the static images and HTML files. A next step might then be a more dynamic web interface, as already described above.

* Note that when I say "CGI script", I don't mean specifically the CGI module, for new projects it's better to use one of the more modern web frameworks I mentioned in my previous post, such as Mojolicious.

Much of my suggestions describe a fairly simple mechanism because you said that this would be your first web based project, plus your data updates would be relatively infrequent. In several places I mentioned some future improvements, that might become necessary if you have data updates faster than roughly every half hour or so, or a larger number of sensor systems submitting data - maybe more than 10 sensor systems, but that's only an order of magnitude estimate. But hopefully this is one good place to start.

Hope this helps,
-- Hauke D

stevieb hippo marto and Hauke D, Thanks so much. I hope I will be able to help others one day too, if not by my knowledge, then by some experience in my future projects I expect will have

I will answer to all replies soon, but first just to add few comments to my main post.

Regarding data frequency it should be 3 or 4 times instead of 2, because not all measurement will be done at the same time, but that doesn't matter much. It is also possible for some data to be held for some time and then to send few data together. For my purposes, it is not so important to have all data immediately on the server

In meantime, after sending my post yesterday, I've got one maybe a bit strange or even ridiculous idea (I still don't have a right sense of that) about situation with many locations for monitoring

I could send SMS messages from PIC or some other monitoring devices to a small server that I should have in that case, in my home or some other place with internet connection. I don't have many data during the day, so there would not be so much messages. On the other side I think SMS messaging sometimes may be late or unsure, so that idea is maybe not recommended.

Regarding representing graphs for longer time then for one month, I've got an simple idea. I would plot one graph in a month and when next month comes, its graph will be shown on the HTML page instead of previous, and other graphs in server directory would have links on that page, so visitor could choose what they want. I see now there are numerous options

Kind regards

predrag

Hi Hauke D

Thanks for such great help again and again, I can't show how all that is useful for me

Yes, I also see that 5.10.1 will probably be too old Perl version for the server now

Just in case, it would be good if your cloud provider offers you an easy way to increase that later if necessary.

Yes, it is possible, very quickly

While possible, I would say that it's probably better for the sensors to push their data to a central location (web server).

Yes, I agree. The idea of the opposite solution was from the past, when I wanted just to try connection to mysql database, whether internal or external, as an example on my cPanel shared hosting. The hosting company had very restricted rules for that kind of hosting, but with a short procedure, they would allow it, and maybe easier 2. The opposite then the first. Also, there was an already installed program for reading from external base to cPanel, so I wanted to try it. But unfortunatelly, at the end, I didn't make that test.

Thanks for recommending DBI module. Yes, I need a simple tool for that project, no need for something large

As you wrote, regarding images (graphs) for my first project, from more reasons then one is beter to work with static images. Also, I've already explained the additional idea I've got about several graphs. Although simple, I am sure it will give very good and nice results. I think everybody will agree that content is the most important, not the form. Of course, that doesn't mean we should not use very modern and powerful tools if we need that.

You and other perl monks definitely convinced me to Raspberry Pi . The fact that I can work in Perl is maybe prevailing. Before my beginning with Perl, RPi, although attractive, wasn't enough. In that time I wanted to avoid learning Python but now, since I already learn something new (Perl) and love it, even a bit of Python would not be a problem

I've visited your website about GPS Tools. Thank you for the link and I am happy to learn from your experience and your work and the experience of other perl monks. I will read that later more detailed.

When I see where you work, I will write something OT, but very short, you will see how much I love all this. It was during 1977, I was in secondary school, and for my thesis (90 pages) I measured air temperature, temperature of the water in the river and water level every day during the whole year. Thesis were about our local river ecosystem.

Thanks for suggesting USB as your better experience. I've mentioned serial port without any special reason. I didn't use it for such purposes yet, just know people used it before

note all systems should use NTP

Yes, I know that now, it become more clear to me on my Linux courses

* Note that when I say "CGI script",

Yes, I've understood that by intuition because you've already wrote about new modules for that

I must say I am overwhelmed with information but I know it will settle down, especialy when I start with the project. Because of many reasons, I could not start that project before, and also, I can't start at this moment yet, but will use that time to learn

Kind regards

predrag

but the aplication among offered that I prefer is Cloud LAMP server: CentOS 6.x, Apache, MySQL, PHP… How it is about Perl installation and version? I expect Perl 5.10.1 installed On CentOS

Yes, that is correct and it should be installed by default. However, be aware that CentOS (like all the RedHat-based distros) does not include all of the Core Perl modules by default and if you require these they would need to be installed via yum. Many monks will advise you to avoid using the system perl anyway but I don't necessarily agree with that. So long as your personally-installed modules (ie. not via yum) are kept separate from the system and vendor modules then they can happily coexist.

Is 1 or 2GB RAM enough for this my application? Till now maximum number of site visits was around 1000 visits in a day in the peak ...

If, as discussed previously, you perform the image-generation on a schedule and not on-demand so that essentially the webserver is just serving static content then yes, you can probably get away with the low RAM indicated. Remember to strip out all the unnecessary modules from the Apache configuration to lower the memory footprint. Sensible caching and a hybrid MPM (worker or event) would help too. 1000 visits a day should be quite achievable (so long as they don't all arrive at once).

There's no harm in using something like RRDTool just to produce graphs if that's what you want.

Best of luck with your project.

hippo, thanks again for the help and for best wishes. I will have in mind your comments

As you suggested, I will pay a special attention to memory use

My website had maximum of around 14,000 hits per hour in peak season, but as I wrote before, I expect much more visits with that monitoring as the change of the weight hive data is extremelly attractive to beekeepers-visitors.

Best regards

predrag

Hi predrag!

Impressive suggestions... Nothing much to add, just my two cents:

Have fun!

Hi fishy

Thanks for the reply. Yes, I consider myself extremely lucky to have such support on perlmonks. For me, these comments and suggestions are worthier than any code

Hopefully, as Hauke D stated once before, the other monks and even non-monks can learn from this too

Thanks for the link. I've just visited and like it, very simple and I will try that for sure.

regards

Predrag

If you want to research PICs further, checkout the chipKIT PIC32 based Arduino clones at digilentinc.com. Also look at chipkit.net and chiplit.net/forums.

The RPi3 is more powerful and is a good choice if you are linux fluent already.

James

James, thank you for the comment

If you go with stevieb's libs, and switch to RPi:

* Working together is more fun. stevieb will get an incentive to mature his libraries (win for the community) and you will get good support.

* RPi's are more expensive. Think also about theft/raindamage/overheating of hardware and how to secure it (high pole, or inside a fake? hive or bury the hardware). Although, on second thought, I guess that bees guard the hive well. BACKUP your data!

* Graphics can also be made with javascript and because some have CDN links (which means the javascript code is downloaded from elsewhere) could potentially be lighter than generating images, as all you provide is the data with some initializing code.

I also like the fact you are going for RRD. Remember you can add data to more than 1 database, so if you collect each half an hour, and have 48 measurements, create another RRD that feeds on that first information, but summarizes the data into, say, 8 hour chunks (3 measurements per day). This way, you can have daily data, and weekly/monthly averages, easy to query and display on your webpage. And at fixed databasesizes.

FreeBeerReekingMonk thank you very much for your comments.

Yes, I think I will start my first project with RPi's. The reason why I was thinking about PICs solution for the hardware was because I would learn PICs and later may realize some other ideas that PICs are very suitable for. Also, I think power consumption will be less then with other solutions . And also, after years and decades of being not active in electronics, I am eager to do so many staf. A bit not so serious reason but…

I think I will not go with javacript, it will be too much for me.

RPi's are more expensive

For me as a beginner and an amatheur, the price is much more relative in comparison what it means to a professional. Even in the situation when I at this moment can't start project yet, because of the lack of the money and some other reasons. But I believe I will start this spring, till that I will preparing myself and learning. For me, it is the most important that I can finish that project and that I can do as much as possible of work myself, so I want to learn as much as possible.

Think also about theft/rain damage/overheating of hardware and how to secure it

Yes, I think about that. Till 5-10 years ago, that wasn't a serious problem here but now it became. I will maybe have monitoring hives close to someone's house…

I guess that bees guard the hive well

 Yes, of course, but in general our race of bees is not much aggressive. But if I would have installed the hive to crash down in the attempt of theft, thefts will be give up.

Since I didn't begin anything yet, it is pretty childish of me to tell that I have many other ideas about web monitoring in beekeeping. For one, it will be important to have daily average temperature. I will not use RRD for my first project but for sure will use in some next, I see how powerful it is.

Not childish at all. Remember that inventors have many ideas in their heads, but still go for one or two that seem the most interesting. Write them down though, and if one is good enough... you can even patent them!

Microcontrollers are booming with IoT around the corner. It pays to invert time in them to learn about it. And people that have cross-over skills (knows software and hardware (for example, how to protect a sensor with shrinkwrap so it lasts longer) AND has knowledge of, say, winddirection, so know how to place a box in the shade, away from rain) are valuable.

Also: the same set-up you make for monitoring your bees can be used in a tomato-greenhouse, where humidity (air AND soil) etc need to be measured too... so it pays to understand the full process (where to buy cheap hardware, how to set it up, how to read the data, software) and you can support other farmers. And you either get free tomatoes, or a bit of money for support/maintenance.

And it is not hard. I mean, you can go for the expensive humidity sensor or a very cheap sensor made of mostly recycled stuff.

See also:

www.gardenbot.org

On this website you learn about using bottles wrapped in white paper to keep the sun and rain out. Low tech does not mean bad tech...