in reply to Re: redacting from config hash
in thread redacting from config hash

I'm very grateful for 2 excellent responses. I've coded toward both haukex's and soonix's suggestions, and I haven't completely disentangled ideas that are still crowding out the good ones here. I think I can lay out what I've done, state a better specification, and outline the things that cause trouble.

I looked also at config1.pm and realized that my claim that I only had sensitive values in one file per workspace was false. My new specification for the copy to the temp directory is that any file that matches "config", then a number, a dot, and a "pm" shall not be copied. Meanwhile, I have a created a template file, config3.tmpl, that is of the form it needs to be less the definition of the hash, which is to be templated in. I don't see why Text::Template wouldn't be fine. The longer arc of my question is how to both to populate and depopulate such hashes. What makes it very difficult is that we seem to be acting on perl syntax itself.

If we look haukex's output, the hash begins

$CONFIG = { my_github => {

, yet if we're to imitate the syntax in my source it begins:

our %config = ( my_sftp => {

I'm more than a little confused about what seem to be competing definitions of a hash.

$ cat config3.tmpl package config3; use Exporter qw(import); our @EXPORT = qw(%config); {$config_hash} 1; ---------- $ cat util2.pm package utils2; require Exporter; use utils1; our @ISA = qw(Exporter); our @EXPORT = qw( redact make_ini); sub redact{ use strict; use warnings; use 5.010; use Path::Tiny; my $file = shift; #use $file; resulted in error use config2; my $sub_hash = "my_github"; my $email = $config{$sub_hash}->{'email'}; my $password = $config{$sub_hash}->{'password'}; my $tempdir = Path::Tiny->tempdir('backup_XXXXXX'); say "temp dir is $tempdir"; my $scratch_file = $tempdir->child('batch_01', '1.manifest.txt')->touc +hpath; say "scratch_file is $scratch_file"; my $parent = $scratch_file->parent; say "parent is $parent"; chdir $tempdir unless $tempdir->subsumes(Path::Tiny->cwd); system("pwd"); use Data::Dumper; print Dumper \%config; my $b = "dummy"; return $b; } sub make_ini{ use strict; use warnings; use 5.010; use Path::Tiny; my $file = shift; use config2; use Config::Tiny; use Data::Dumper; my $path = shift; say "path is $path"; say "is the stuff here?"; print Dumper \%config; $config->write( $path, 'utf8' ); #this line draws error chdir "/home/bob/Documents"; system("cat *.ini"); my $b = "dummy"; return $b; } 1;

What I tried to do was extend what I have now to create a Config::Tiny ini file. I think the install went well, and judging by how quickly it went, it truly seems tiny:

All tests successful. Files=5, Tests=49, 3 wallclock secs ( 0.06 usr 0.01 sys + 0.45 cusr + 0.10 csys = 0.62 CPU) Result: PASS RSAVAGE/Config-Tiny-2.23.tgz /usr/bin/make test -- OK Running make install Manifying 1 pod document Installing /usr/local/share/perl/5.26.1/Config/Tiny.pm Installing /usr/local/man/man3/Config::Tiny.3pm Appending installation info to /usr/local/lib/x86_64-linux-gnu/perl/5. +26.1/perllocal.pod RSAVAGE/Config-Tiny-2.23.tgz /usr/bin/make install -- OK cpan[2]> q Terminal does not support GetHistory. Lockfile removed. $ man Config::Tiny $ ./1.redact.pl Variable "$config" is not imported at template_stuff/utils2.pm line 61 +. Global symbol "$config" requires explicit package name (did you forget + to declare "my $config"?) at template_stuff/utils2.pm line 61. Compilation failed in require at ./1.redact.pl line 6. BEGIN failed--compilation aborted at ./1.redact.pl line 6.
$ cat config2.pm package config2; use Exporter qw(import); our @EXPORT = qw(%config); our %config = ( my_sftp => { domain => '', username => '', password => '', }, my_github => { email => '', password => '', }, ); 1; $

I don't seem to be sharing is the "our" on the definition of "our %config". Why is my syntax on the config hash clashing with that provided by man Config::Tiny and haukex?

Replies are listed 'Best First'.
Re^3: redacting from config hash
by haukex (Archbishop) on Aug 01, 2018 at 09:16 UTC
    if we're to imitate the syntax in my source it begins: our %config = (

    You can get this kind of output from my code by changing Data::Dumper->new([$CONFIG],['$CONFIG']) to Data::Dumper->new([$CONFIG],['*config']) (change the variable names as you like); you'd just have to prepend the our yourself. See also the Data::Dumper docs.

    What makes it very difficult is that we seem to be acting on perl syntax itself.

    Yes, that's a pretty complex topic, so soonix's suggestion to use a different configuration file format is a good way to avoid that issue.

    Variable "$config" is not imported at template_stuff/utils2.pm line 61 +. Global symbol "$config" requires explicit package name (did you forget + to declare "my $config"?) at template_stuff/utils2.pm line 61.
    Why is my syntax on the config hash clashing with that provided by man Config::Tiny and haukex?

    You've done use config2;, which should give you a hash %config, which you can confirm if you comment out the line with ->write: do you see the output of print Dumper \%config;? Then, you're doing $config->write, which means "call the method write on the object $config" - not the hash %config! That's why Perl complains about the missing variable $config.

    It's possible to create a new Config::Tiny object and then copy over the config from config2.pm's %config into that object. However, I am wondering what the goal of your code is here. Are you trying to automate the conversion from config2.pm into an INI format? If the config is long, I can understand that, but if it's as short as you showed, why not do that step once, by hand? You can then use a method similar to what I showed to create a redacted version, or, again, since that's something you'd probably only be doing once (?), you can do that by hand as well.

    Taking a step back: My understanding so far is that you want to provide your code for download for others to use, is that right? And you're figuring out a way to disentangle the stuff you've written for yourself (config files etc.) from the general stuff that you want to distribute to everyone? That's pretty common, and a situation I've been in plenty of times. I've found the best approach is to look at it from a different angle and put yourself in the user's shoes, as if you knew nothing about the internals of the package: what steps would a user who downloads the package have to take to set it up? How would you describe those steps in your README for the user to follow? And then, what do you as the developer have to provide to make that as easy as possible?

      what steps would a user who downloads the package have to take to set it up? How would you describe those steps in your README for the user to follow? And then, what do you as the developer have to provide to make that as easy as possible?

      It seems to me that a user would want a one session experience to populate his/her critical data. Thus, I've been removing hard-coded values one by one. What remains is a hard-coded path to an ini file, which I think is on the verge of working, but vexingly, does not. This is an analog to what I've used before to instantiate the sftp object. Perl thinks this is illegal:

      sub get_tiny{ use 5.011; use warnings; use Net::SFTP::Foreign; use Config::Tiny; use Data::Dumper; my $ini_path = qw( /home/bob/Documents/html_template_data/3.values.ini + ); say "ini path is $ini_path"; my $sub_hash = "my_sftp"; my $Config = Config::Tiny->new; $Config = Config::Tiny->read( $ini_path , 'utf8' ); say Dumper \%Config; my $domain = $Config{$sub_hash}->{'domain'}; my $username = $Config{$sub_hash}->{'username'}; my $password = $Config{$sub_hash}->{'password'}; my $port = $Config{$sub_hash}->{'port'}; #dial up the server say "values are $domain $username $password $port"; my $sftp = Net::SFTP::Foreign->new( $domain, user => $username, port => $port, password => $password) or die "Can't connect: $!\n"; return $sftp; }

      List of compiler complaints:

      $ ./1.qy1.pl Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 215. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 216. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 217. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 218. BEGIN not safe after errors--compilation aborted at template_stuff/htm +l4.pm line 239. Compilation failed in require at ./1.qy1.pl line 5. BEGIN failed--compilation aborted at ./1.qy1.pl line 5. $

      I just don't see where I'm not correctly imitating the syntax shown on Config::Tiny.

        I just don't see where I'm not correctly imitating the syntax shown on Config::Tiny

        Part way through you've stopped using a hashref and started using a hash. Just stick with the hashref. eg:

        my $Config = Config::Tiny->new; $Config = Config::Tiny->read( $ini_path , 'utf8' ); say Dumper $Config; my $domain = $Config->{$sub_hash}->{'domain'}; my $username = $Config->{$sub_hash}->{'username'}; my $password = $Config->{$sub_hash}->{'password'}; my $port = $Config->{$sub_hash}->{'port'};

        HTH.

        ... my $Config = Config::Tiny->new; $Config = Config::Tiny->read( $ini_path , 'utf8' ); say Dumper \%Config; my $domain = $Config{$sub_hash}->{'domain'}; my $username = $Config{$sub_hash}->{'username'}; my $password = $Config{$sub_hash}->{'password'}; my $port = $Config{$sub_hash}->{'port'}; ... Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 215. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 216. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 217. Global symbol "%Config" requires explicit package name (did you forget + to declare "my %Config"?) at template_stuff/html4.pm line 218. ...

        Config::Tiny returns a (blessed) reference to a hash, which is stored in the scalar $Config, so there is no %Config hash available. Both \%Config and $Config{$sub_hash} are however attempting to access a hash named %Config. Instead of Dumper \%Config, do Dumper $Config, and instead of $Config{$sub_hash}, say $Config->{$sub_hash}, and it should work. See perlreftut and perlref.

Re^4: redacting from config hash
by soonix (Chancellor) on Aug 01, 2018 at 12:53 UTC
    I cobbled together an example for using your %config as a hash as you used it rather than as a hashref.
    #!/usr/bin/env perl use 5.011; # implies strict + feature 'say' use warnings; use Data::Dumper; use Config::Tiny; use constant { INIFILE => 'example.ini', ENCODING => 'encoding(Windows-1252)' }; sub create { my %config = ( my_sftp => { domain => 'example.com', username => 'myname', password => 'topsecret', }, my_github => { email => "me\@example.com", password => "confidential", }, ); say Dumper \%config; my $ini = bless \%config, 'Config::Tiny'; $ini->write(INIFILE, ENCODING); say 'created ', INIFILE; } sub show { say 'reading ', INIFILE; my $ini = Config::Tiny->read(INIFILE, ENCODING); my %config = %$ini; say Dumper \%config; } create() unless -e INIFILE; show()

    There are two configuration values which don't belong into the config file :-) which I defined using constant.

    If you don't want to mess with Config::Tiny's internals, you could replace the my $ini = bless \%config, 'Config::Tiny'; line with
    my $ini = Config::Tiny->new; $ini->{$_} = $config{$_} for keys %config;
    Edit: Sorry, this was meant as a reply to Re^2: redacting from config hash...

    Update: changed 'example.ini' to INIFILE in line 40 🤦

      I am very pleased to have 2 more excellent responses from the same sources. I hope I'm not trying anyone's patience for not latching on quicker, but we really are in the thick of it now and gaining ground. I present with an error that I don't understand after redesigning what I'm doing to be more descriptive.

      The questions that haukex raised were partially dealt with by code that soonix wrote. What he wrote first worked, but the one where

      my $ini = Config::Tiny->new; $ini->{$_} = $config{$_} for keys %config;

      were substituted for the line used to instantiate the object, I got an error on line 38, which I marked. I'd be curious to know why, but having one way to do things is enough for now. It's probably the first time I've blessed anything in the world of bits and bytes, maybe appropriate to my future friar status

      I won't be "redacting" anything at all now. I'm gonna use Config::Tiny to create and then draw values from an .ini file that is not in the filespace to be copied. It's better data hygiene and really the path forward if the goal is to share with others in ways where they can recreate their own critical values without need to be aware of the internals. Redaction will occur by not copying anything of the form configx.pm . Then I'm gonna alter my script to read in the values from that other location. As it's written, that will be indicated by the return value in the current caller.

      What follow are 3 listings. The first is a working script that created an ini file successfully. The second is the current driver, 1.initialize.pl which calls the module that I'm writing to capture these functions, the third listing, called utils2.pm. What used to be called "redact" is being renamed "archive", and isn't even being called yet, because I have an error with "make_ini".

      Where I seem to have stumbled is combining the Path::Tiny and the Config::Tiny techniques:

      $ ./1.initialize.pl Global symbol "$ini_path" requires explicit package name (did you forg +et to declare "my $ini_path"?) at template_stuff/utils2.pm line 47. BEGIN not safe after errors--compilation aborted at template_stuff/uti +ls2.pm line 53. Compilation failed in require at ./1.initialize.pl line 5. BEGIN failed--compilation aborted at ./1.initialize.pl line 5. $

      To my eye, I have declared "my $ini_path" in the presence of the appropriate modules in the create() function. Maybe fresh eyes in the morning will help....