Beefy Boxes and Bandwidth Generously Provided by pair Networks
Keep It Simple, Stupid
 
PerlMonks  

getting SMTP capability working

by Aldebaran (Curate)
on May 06, 2022 at 22:25 UTC ( [id://11143632]=perlquestion: print w/replies, xml ) Need Help??

Aldebaran has asked for the wisdom of the Perl Monks concerning the following question:

I'm seeking to be able to have a mail capability for both my droplet and my raspberry pi. I realize that there's a lot of ways to send email, but I need to have at least one of them. I have gotten closer on recent threads but not taken the ultimate step, likely because I don't understand how to configure it correctly. I've been able to do this in the past, but the dominance of giant corporations like google have altered the landscape (at least in my perception). I have to have some detente with evil corporations, but I don't like that they want everything to become proprietary and fee-promoting. I like to have alternatives, in particular, if I can roll my own.

I've had to use gmail in my sudden-onset case of becoming a secretary of something. It's a very odd calling for me, and I try to use perl to make tasks possible that would be tedious and error-prone without a script. Since then, I've been trying to get Mail::Webmail::Gmail installed on any of my 3 perl-endowed devices. Despite having made progress, cpanm keeps washing out before completion:

root@fourth:~# cpanm --verbose Mail::Webmail::Gmail cpanm (App::cpanminus) 1.7044 on perl 5.032001 built for x86_64-linux- +gnu-thread-multi ... Checking if you have Crypt::SSLeay 0.51 ... No ... ==> Found dependencies: Crypt::SSLeay Searching Crypt::SSLeay (0.51) on cpanmetadb ... --> Working on Crypt::SSLeay Fetching http://www.cpan.org/authors/id/N/NA/NANIS/Crypt-SSLeay-0.72.t +ar.gz ... OK Unpacking Crypt-SSLeay-0.72.tar.gz ... Configuring Crypt-SSLeay-0.72 ... *** THIS IS NOT AN ERROR, JUST A MESSAGE FOR YOUR INFORMATION *** Do you really need Crypt::SSLeay? Starting with version 6.02 of LWP, https support was unbundled int +o LWP::Protocol::https. This module specifies as one of its prerequi +sites IO::Socket::SSL which is automatically used by LWP::UserAgent unle +ss this preference is overridden separately. IO::Socket::SSL is a mor +e complete implementation, and, crucially, it allows hostname verification. Crypt::SSLeay does not support this. At this point, Crypt::SSLeay is maintained to support existing software that alre +ady depends on it. However, it is possible that your software does not really depend +on Crypt::SSLeay, only on the ability of LWP::UserAgent class to communicate with sites over SSL/TLS. If are using version LWP 6.02 or later, and therefore have install +ed LWP::Protocol::https and its dependencies, and do not explicitly u +se Net::SSL before loading LWP::UserAgent, or override the default so +cket class, you are probably using IO::Socket::SSL and do not really ne +ed Crypt::SSLeay. Before installing Crypt::SSLeay, you may want to try specifying a dependency on LWP::Protocol::https. root@fourth:~#

Q1) Do I really need Crypt::SSLeay? Is this a "bad logic game?"

As I read closer at DO-YOU-NEED-Crypt::SSLeay, I read this and heartbleed doesn't sound good, and superceded by LWP::Protocol::https for a reason. This sounds like a rock I should drop.

Another thing came up while I was grubbing around as root, having ssh'ed as another, which is the preferred-scheme.

I wonder what others might think of removing the back door, as it were, deleting authorized_keys in /root/.ssh/? PermitRootLogin is already set to No already wherever that counts. But if the setup was most-vulnerable where it rolled out, then erasing that access entirely seems like a proper finishing touch. (Yes, I've locked myself out of all kinds of things.)

root@fourth:~# cd .ssh/ root@fourth:~/.ssh# ll total 12 drwx------ 2 root root 4096 Jan 2 02:17 ./ drwx------ 9 root root 4096 Mar 5 04:47 ../ -rw------- 1 root root 419 Jan 2 02:17 authorized_keys root@fourth:~/.ssh#

Moving along, another way forward might be to go with something like this suggestion: Re^3: creating a secure environment for perl scripts to run. Here's my adaption of that source, with the minimal security of not having credentials in the same file:

#!/usr/bin/perl use v5.030; # strictness implied use warnings; use Email::Simple::Markdown; use Email::Sender::Simple qw(sendmail); use Email::Sender::Transport::SMTP qw(); use Try::Tiny; use Config::Tiny; my $ini_path = qw( /home/wilma/7.values.ini ); my $sub_hash = "my_smtp"; my $Config = Config::Tiny->new; $Config = Config::Tiny->read( $ini_path, 'utf8' ); # -> is optional between brackets my $host = $Config->{$sub_hash}{'host'}; my $port = $Config->{$sub_hash}{'port'}; my $username = $Config->{$sub_hash}{'sasl_username'}; my $pass = $Config->{$sub_hash}{'sasl_password'}; my $ssl = $Config->{$sub_hash}{'ssl'}; say "values are $host $port $username $ssl"; my $output = "Come here, Mr. Watson"; my $message = Email::Simple::Markdown->create( header => [ Subject => "Report", To => 'tallharry84@gmail.com', From => 'noreply@foo.com' ], body => $output ); try { sendmail( $message, { from => 'noreply@foo.com', transport => Email::Sender::Transport::SMTP->new( { host => $host, port => $port, sasl_username => $username, sasl_password => $pass, ssl => $ssl, } ) } ); } catch { warn "sending failed: $_"; }; __END__ wilma@fourth:~$

On execution, I'm unable to create the smtp connection:

wilma@fourth:~$ ./1.smtp.pl values are fourth 465 wilma starttls sending failed: unable to establish SMTP connection to (fourth) port 4 +65

So what values are not working here? I don't know, and it could be more than one of them. I think I get the hostname right:

wilma@fourth:~$ hostname fourth

I chose port 465 because that was what the module would default too. (Completely grasping at straws.)

Furthermore, I chose the username to be the actual name of an existing user on this machine. Does it have to be one? What I really don't like about this scheme is that the only password this user knows is also the one that would elevate her to root, and it's sitting there in a plain text file. This seems wildly wrong from a security vantage point.

While I'd like to get this straightened out, there is at least a third possibility for email on a server, which is to follow haukex's guide for rpi setup, section 5:

sudo apt-get install alpine postfix bsd-mailx sudo vi /etc/postfix/main.cf #=> correct "myhostname" if necessary #=> if it doesn't exist, add the line "smtp_tls_security_level = may" #=> if this option or the option "smtp_tls_CApath" doesn't exist, # add the line "smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt +" sudo dpkg-reconfigure postfix # and configure as appropriate echo "root: pi" | sudo tee -a /etc/aliases && echo "---" && cat /etc/a +liases sudo newaliases && sudo systemctl restart postfix echo "This is a mailx test" | mailx -s "mailx test" root alpine # Configure "User Domain" and anything else as needed

So, how do I best get email working on this server?

Replies are listed 'Best First'.
Re: getting SMTP capability working
by NERDVANA (Deacon) on May 08, 2022 at 01:25 UTC

    Mail::Webmail::Gmail has a date of 2006 on it. I'm going to guess that a 16 year old module probably isn't compatible with Google's current implementation of webmail. And yes, any modern "web" module should just depend on LWP::UserAgent and not have a direct dependency on SSLeay. (with the exception of event-driven modules which often have to re-invent things themselves)

    If you want to inspect a Gmail mailbox, you can currently go into Google settings and enable "insecure" protocols (for i.e. IMAP and SMTP) and then use Perl's IMAP modules to interact with the mailbox. This support is supposedly going away some time, but works for now. I don't know if there is a native API client for Google's mail back-end, in Perl.

    When you configure the outgoing SMTP host, you need to first make sure that that host is actually running an SMTP server. "fourth" looks like the host name of your raspberry Pi, so you should first make sure you installed Postfix correctly. But also, if you installed postfix locally, then you can deliver mail directly to postfix without using SMTP at all, and test this from the command line using the 'sendmail' or 'mail' commands. Also, using SSL from a local service to a local service is unnecessary complication. SSL needs a certificate, and it won't match for the local host name, forcing you to also set flags to disable host name checks.

    If you want to send mail through your gmail account, then the SMTP host would be whatever gmail defines for that, and you'd need to authenticate using your gmail password (but that also requires you to go into gmail settings and enable "insecure" clients)

    None of this yet touches on *receiving* email to your raspberry Pi, like giving the raspberry Pi its own DNS name with MX (mail exchange) records. That also involves Postfix (a very different configuration) and some way to locally store the email, like Dovecot IMAP.

      Mail::Webmail::Gmail has a date of 2006 on it. I'm going to guess that a 16 year old module probably isn't compatible with Google's current implementation of webmail. And yes, any modern "web" module should just depend on LWP::UserAgent and not have a direct dependency on SSLeay. (with the exception of event-driven modules which often have to re-invent things themselves)

      Ok, and thanks for your reply. I don't know how to say it in the language of perl modules, but I think it's clear that Google is shutting the door on the type of access that Mail::Webmail::Gmail provided before the free service became a magnet for bad actors. If I were putting up the sign it would say:

      STOP. Here swirls Charybdis. Do not play this logic game.

      If you want to inspect a Gmail mailbox, you can currently go into Google settings and enable "insecure" protocols (for i.e. IMAP and SMTP) and then use Perl's IMAP modules to interact with the mailbox. This support is supposedly going away some time, but works for now. I don't know if there is a native API client for Google's mail back-end, in Perl.

      Let me again repost what I discovered upthread from google:

      To help keep your account secure, starting May 30, 2022, Google will no longer support the use of third-party apps or devices w +hich ask you to sign in to your Google Account using only your user n +ame and password.

      Regarding what back ends there might be, my strong suspicion would be that you will be able to do this if you're a paying client. At least that's what I read into the following paragraph

      Please note this deadline does not apply to Google Workspace or Google Cloud Identity customers. The enforcement date for these customers will be announced on the Workspace blog at a later date.

      I'd be curious if someone among us had experience with either Google Workspace or Google Cloud Identity. I do not. My guess is that they might have backends for perl access, but the landscape is changing so fast. A lot of people's stuff is gonna break down at the end of this month. And it's not just gmail, it's anything without a subscription, as far as I can tell. I clicked around a bit to see what was still out there,and I fear that this type of access to gratis webmail is going the way of the dodo.

      When you configure the outgoing SMTP host, you need to first make sure that that host is actually running an SMTP server. "fourth" looks like the host name of your raspberry Pi, so you should first make sure you installed Postfix correctly. But also, if you installed postfix locally, then you can deliver mail directly to postfix without using SMTP at all, and test this from the command line using the 'sendmail' or 'mail' commands. Also, using SSL from a local service to a local service is unnecessary complication. SSL needs a certificate, and it won't match for the local host name, forcing you to also set flags to disable host name checks.

      Now that I'm halfway through this thread, I'd like to refine the task I'm doing to be one where I'm sending an email from my rpi to my droplet. I think of my rpi as being "out in the field" and my droplet as a never-blinking observer, and an appropriate place to build what I need to house emails. Since there is so much involved in setting up the server, the folks at digital ocean try to pour water on it with why-you-may-not-want-to-run-your-own-mail-server. This article also neatly states what a person needs to do, including the (sometimes confusing) acronyms. So far, I've successfully created the LAMP stack following install-linux-apache-mysql-php-lamp-stack-on-ubuntu. What's more, I created an MX record to point to mail.(domain name) and am trying to get my ducks in a row to install postfix, where my install had been hanging. I've learned over the years not to present a question to the monastery without doing an internet search on it, so I did: just-f***ing-google.it and found installing-postfix-hangs-at-postfix-configuration-screen. I was shocked to find that the culprit was the size of my terminal. I was unable to select anything when it was small, but maximized, the install moves forward without complaint.

      None of this yet touches on *receiving* email to your raspberry Pi, like giving the raspberry Pi its own DNS name with MX (mail exchange) records. That also involves Postfix (a very different configuration) and some way to locally store the email, like Dovecot IMAP.

      Ok, so I just broke through with postfix and have Dovecot on the install list.

      Thanks again for your comment,

        I clicked around a bit to see what was still out there,and I fear that this type of access to gratis webmail is going the way of the dodo.

        Not really, just the death of one-step authentication. I've been looking at workarounds and took the bold step of just f***ing googling it, which revealed this reddit thread. The poster describes the workaround for Auth 2.0. 2-step auth has to be authorized, so you're ok if you have a phone number to give to google:

        Log into gmail Click the user icon in the top-right corner and select "Manage your Go +ogle Account" Go to the security page (from the list at the left) [This is necessary for the next step, but will stop your scripts from +logging into smtp with your regular password. This is sort of the poi +nt of no return. Make sure you are ready to go all the way and update + all of your email-sending scripts before continuing.] Enable two-ste +p authentication. Once two-step authentication is enabled, go back to the security page. + There should now be a new section called "App Passwords". Select it. + You will probably be asked to log in again. Generate a new app password. I put in "Mail" under "select app" and "O +ther" under "select device," but I don't think any of the information + you put it matters. It will give you a 16-character app password which will be shown in ye +llow. Have your scripts use this password to log into smtp instead of your r +egular password and everything should work. It will not prompt you fo +r 2-step verification. After doing this, my scripts seem to work. I did not have to make any +other changes to the code other than substituting the app password fo +r the original one. Hopefully it helps you too.

        Gmailpocalypse this is not....

Re: getting SMTP capability working
by davies (Prior) on May 07, 2022 at 19:35 UTC

    My RPi (model 1) email server runs Postfix, Dovecot & opendkim, but does not use Crypt::SSLeay. The obvious use is on the Google side, but there I can't help you.

    Regards,

    John Davies

Re: getting SMTP capability working
by afoken (Chancellor) on May 08, 2022 at 10:04 UTC

    If you have some other mailserver available, a tool like nullmailer can be helpful. It is a minimal solution, accepting mails from the local system and delegating the delivery to some smarter mailserver. The Debian wiki explains how to set it up. Basically, install the package and write account data for the smarter mailserver into a configuration file.

    The Debian postfix package comes with a good configuration assistant, which makes setting up a fully featured mail server quite easy. Again, see the Debian wiki, which also has details on all the bells and whistles you probably won't need.

    Yes, I'm aware that you are using a Raspi. Guess which Linux distribution is the base for Raspbian a.k.a. Raspberry Pi OS.

    Both nullmailer and postfix allow you to submit mails via SMTP, that's the traditional and cleanest way. Any "solution" that tries to automate interacting with some kind of webmailer will ultimately fail, simply because there is no standard, only guesswork. Postfix comes with a sendmail executable that emulates the original "sendmail" as much as needed to send mail. Nullmailer has nullmailer-inject, which is not as compatible, but accepts emails from STDIN like sendmail does, and accepts a few parameters like sendmail does.

    Alexander

    --
    Today I will gladly share my knowledge and experience, for there are no sweeter words than "I told you so". ;-)
      If you have some other mailserver available, a tool like nullmailer can be helpful. It is a minimal solution, accepting mails from the local system and delegating the delivery to some smarter mailserver. The Debian wiki explains how to set it up. Basically, install the package and write account data for the smarter mailserver into a configuration file.

      Ok, thanks Alexander, so I'm still in process of creating the mail server. I've had steps forward, but a couple back, as I updated systems at in opportune time and lost apache2. Working on that. I looked at the nullmailer install. I like the date on it: 2018. (Not too old, not too new.) Read the HOWTO to glean what I could.

      The Debian postfix package comes with a good configuration assistant, which makes setting up a fully featured mail server quite easy. Again, see the Debian wiki, which also has details on all the bells and whistles you probably won't need.

      Those were very helpful links. I'll repost this one: Installing_and_Configuring_Postfix_on_Debian.

      Yes, I'm aware that you are using a Raspi. Guess which Linux distribution is the base for Raspbian a.k.a. Raspberry Pi OS.

      Your comments made me realize that the problem space here was wall-to-wall debian. I'll go ahead and write this like we used to on usenet when the subject became implementation-specific:

      [specific to debian implementations]

      I had been unable to completely update my droplet until I found a bit of extra update syntax from ubuntu. I crammed it all into a new alias:

      alias release='sudo apt-get update && sudo apt-get upgrade && apt-get dist-upgrade && sudo do-release-upgrade'

      I had been in a limbo zone where I almost had 22.04 but now do:

      root@fourth:~# release ... Checking for a new Ubuntu release There is no development version of an LTS available. To upgrade to the latest non-LTS development release set Prompt=normal in /etc/update-manager/release-upgrades. root@fourth:~# apt autoremove
      Both nullmailer and postfix allow you to submit mails via SMTP, that's the traditional and cleanest way. Any "solution" that tries to automate interacting with some kind of webmailer will ultimately fail, simply because there is no standard, only guesswork. Postfix comes with a sendmail executable that emulates the original "sendmail" as much as needed to send mail. Nullmailer has nullmailer-inject, which is not as compatible, but accepts emails from STDIN like sendmail does, and accepts a few parameters like sendmail does.

      Ok, and thanks for your comment. I'm happy to see your responses as I know they are likely to contain useful and exotic details of what is happening underneath the hood with unix. One issue I need to get clear is whether Digital Ocean is giving me port 25, so I will fire off this to them:

      Dear Digital Ocean Support, I've been renting space from you for over a year now, and, because google will disallow 3rd-party use of their smtp server on May 30, am working up an email server with my DO site. I'm not entirely convinced that I have access to port 25, as it is blocked during the first 2 months. Do I? Thank you for your attention,

      Beyond this, I got some partial results and hope to not need to start over. I think I got the MX record set up right with DO. I had to access the way back machine for some of my sources. This is from 5 years ago: way back machine on mx records.

      Here's a nice, new fresh one: how-to-install-and-configure-postfix-on-ubuntu-22-04. I'll end up going through this again. Since the beginning of this write-up, I tore out and replaced apache2, changing Postfix twice, so I'll be due to go over it again.

      One final thing. I told a friend that google was restricting 3rd-party access to their smtp server, and she asked me whether her iOS mail app that gets her gmail will still be able to do so. Does anyone have the expertise to speculate? (I hemmed and hawed.)

        One issue I need to get clear is whether Digital Ocean is giving me port 25

        Very simple to test this for yourself, but do note that most virtual host providers block SMTP traffic. Log into your server, and run the netcat command, listening on port 25 (need sudo because the port number is less-than 1024).

        steve@cesium:~$ sudo nc -l 25

        Now, on a remote machine, using netcat again, create a connection to the server, and type something, and hit ENTER:

        steve@maezi ~ >nc cesium 25 hello, world!

        If port 25 is open on your server, you'll see your typed message on its console:

        steve@cesium:~$ sudo nc -l 25 hello, world!

        You can also type something while on the server while the nc command is still running on both machines, and the message will show up on the machine you connected to the server with; in essence, a little chat program ;)

Re: getting SMTP capability working
by Bod (Parson) on May 07, 2022 at 17:37 UTC

    To be able to send mail from a Raspberry Pi, you need to install software to allow this. You could use sendmail but I understand this is difficult to configure.

    I have one Raspberry Pi that operates remotely which has email capability and sends me a weekly report. To do this I installed msmtp and used MIME::Lite to manipulate the emails.

    My installation came from following these instructions:
    https://www.howtoraspberry.com/2021/06/how-to-send-mail-from-a-raspberry-pi/

    I am not claiming this is the best way to do it but it certainly works.

      [re-ordered]

      My installation came from following these instructions: https://www.howtoraspberry.com/2021/06/how-to-send-mail-from-a-raspberry-pi/

      Thanks, Bod, I lurk on a lot of your threads, as we seem to be doing a lot of similar things. I thought that this was gonna work for me, but the folks at google are gonna rain on this parade:

      To help keep your account secure, starting May 30, 2022, ​&#8203 +;Google will no longer support the use of third-party apps or devices + which ask you to sign in to your Google Account using only your user +name and password. Please note this deadline does not apply to Google Workspace or Google + Cloud Identity customers. The enforcement date for these customers w +ill be announced on the Workspace blog at a later date. For more information, please continue reading. Special Note on Apple Device Sign-Ins. Users who have not recently sig +ned into their Google Account using only username and password will b +e able to only make new sign in attempts using the Google account typ +e starting from February 28, 2022. Existing users may continue to sig +n into their Google Account using their username and password until M +ay 30, 2022.

      I did follow through on the first part, so I'm not completely shut out:

      sudo apt install bsd-mailx msmtp msmtp-mta

      I ran:

      msmtp --configure=tallharry84@gmail.com

      Tried to generate a key properly:

      gpg --full-generate-key

      Followed by:

      gpg --encrypt --output=msmtp-password.gpg --recipient=tallharry84@gmail.com

      I had to wrestle with STDOUT here, and this is what I thought I did, and what I thought was right, but gpg was unable to decrypt it at runtime:

      password newline CTRL-D $

      I know the key is there, and I found the questions asked very exotic for what my idea of a "key" is.

      $ gpg --list-secret-keys --keyid-format LONG gpg: checking the trustdb gpg: marginals needed: 3 completes needed: 1 trust model: pgp gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u /home/pi/.gnupg/pubring.kbx --------------------------- sec rsa3072/7D027B03F4826BFA 2022-05-09 [SC] D4E5BC39CE4FBA0C694BBF837D027B03F4826BFA uid [ultimate] harry (tja) <tallharry84@gmail.com> ssb rsa3072/017426C56FA76146 2022-05-09 [E]

      I couldn't get the encryption to work on my first try. I think I'll delete this and give it a fresh try. I did, however get something going with this as .msmtprc :

      #Set default values for all accounts. defaults auth on tls on tls_starttls on tls_trust_file /etc/ssl/certs/ca-certificates.crt logfile /var/log/msmtp.log #Gmail settings account gmail host smtp.gmail.com port 587 from tallharry84gmail.com user tallharry84 #passwordeval gpg --decrypt /home/pi/msmtp-password.gpg password redacted #Set a default account account default : gmail

      and, lo and behold, I received output with this command:

      echo "BEEP BEEP" | mailx -s "Subject: This is a test!" tallharry84@gmail.com

      As I look at STDOUT, I see that I still have work to do in terms of setting permissions:

      $ echo "BEEP BEEP" | mailx -s "Subject: This is a test!" tallharry84@g +mail.com send-mail: cannot log to /var/log/msmtp.log: cannot open: Permission d +enied send-mail: log info was: host=smtp.gmail.com tls=on auth=on user=tallh +arry84 from=tallharry84@gmail.com recipients=tallharry84@gmail.com ma +ilsize=229 smtpstatus=250 smtpmsg='250 2.0.0 OK 1652063682 bh2-20020 +a170902a98200b0015e8d4eb2desm5726348plb.296 - gsmtp' exitcode=EX_OK $

      What type of persistence would text written to /var/log/msmtp.log have? Does a person need to "wipe his own logs" on occasion? It looks like I'm gonna have to get several different mail programs with varying capabilities, but they all seem to log, and I might generate a ginormous amount of data while forgetting about the software entirely.

      To be able to send mail from a Raspberry Pi, you need to install software to allow this. You could use sendmail but I understand this is difficult to configure.

      I looked at several of the ones mentioned on this thread. I read up on alpine only to find that the treatment had a dependence on gmail just like this one. alpine-gmail-imap-in-linux

      I have one Raspberry Pi that operates remotely which has email capability and sends me a weekly report. To do this I installed msmtp and used MIME::Lite to manipulate the emails.

      I'd like to see your script that sends your mail, edited appropriately. One big question here at the end:

      How does one send email without using the Google?

Re: getting SMTP capability working
by kaldor (Beadle) on May 09, 2022 at 19:09 UTC
    The only tip I can give on that topic : I've more than once solved SMTP issues with 'sendEmail'.

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: perlquestion [id://11143632]
Approved by kcott
Front-paged by davies
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others sharing their wisdom with the Monastery: (9)
As of 2024-03-28 12:24 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found