Hello, SpaceCowboy. Welcome to the Monastery.

Is there a way to encrypt a Perl file containing password?

There are many ways to encrypt a file in Perl - the real question is how do you decrypt it again? That will require a password and it will need to be stored somewhere, say in a file which then needs to be encrypted, and so ad infinitum. See How can a script use a password without making the password visible? and Password encryption for discussion of some approaches to this.

can I import a million rows and transpose, rename, transform and do a bunch of joins -- is this possible in Perl?

Yes, but why would you? As you say SQL is likely the better tool for this, that's why we have databases. Your reason for avoiding the database seems vague to me. Why not try the SQL approach and if you need help re-framing the results that could be another question.

any advice on books on perl deployment practices?

I'm not aware of any books specifically about that but deployment is touched on in various other books and articles. You might find the "Managing Real Programs" chapter of Modern Perl to be useful, for example.

Thank you @hippo for your response. I have data from more than one database and needed to bring them over to drop, rename, perform calculations, joins and export it as flat file... does not seem possible with SQL alone, need some kind of "pandas" environment; I am willing to learn and try other approaches - storing millions of rows and hundreds of columns in 2d array if possible... any advice on books related to data tables processing?

I would still be looking to do all that within an RDBMS.

storing millions of rows and hundreds of columns in 2d array if possible

Sure it's possible - if you have enough memory. I just don't see why you would do that when a database is the right tool for the job. There is nothing you have said here to suggest to me that it isn't.

---

🦛

Perl arrays and hashtables support nesting (e.g. hash-of-hash-of-array-of-hash etc.), mixed "data types" and slices (a convenient form of indexing). There is also PDL if your data is numeric. Additionally, there is Inline::Python if you want to inline python code you already have (untested by me).

storing millions of rows and hundreds of columns in 2d array if possible

well that's illegal in some states and some of us Perl-ers still work on Spectrums, :)

Update: There's also DBIx::Array which offers this:

foreach my $row ($dbx->sqlarrayhash($sql, @bind)) {
  do_something($row->{"id"}, $row->{"column"});
}
[download]

And there's DBIx::Class as a long-term investment.

bw, bliako

I have data from more than one database and needed to bring them over to drop, rename, perform calculations, joins and export it as flat file

That sounds like a use for a Temporary Table in the RDMS to me...

hippo has answered #2, but you could also have a look at Data::Frame for a general structure. It does not do things like joins and the API is marked as experimental, but it might still be interesting.

Data::Frame uses PDL for the numeric data, which is something else you could look at if you do not have mixed strings and numbers (but then you would probably be asking about numpy equivalents).

It also has a high number of fails on cpan testers, but those seem to have been cleared up with recent releases of PDL.

Thanks for your response. I am getting data from multiple databases. I needed to extract them, store them in a variable, drop a few columns, rename a few, calculate bunch of fields and prepare a final dataset. IS this still possible with Data::Frame and PDL as a whole?

Possibly, but have a look at the documentation. It's probably better in the long run to work directly with the database, as per hippo's suggestion in 11137712.

One place to start would be DBI, but stated with the caveat that I've never used it or interacted with databases via perl. However, many of the other monks have and could give you good advice.

If PDL processing is still desired, there is PDL::IO::DBI to bring in data from RDBMS.

1. Is there a way to encrypt a Perl file containing password? I need to import database credentials from a file and would like to know more about encrypting passwords

Since you mentioned Docker... If you need something that scales in a large environment, i.e. multiple nodes requiring credentials and that also works for unattended (re-)boots, have a look at HashiCorps Vault (client: i.e. WebService::HashiCorp::Vault, server: https://github.com/hashicorp/vault). There is still a secret to be kept (i.e. token), however, management of credentials becomes much more feasible at the cost of running a dedicated centralised (hopefully redundant) vault service.

Cheers.