in reply to Turning a handful of scripts into a distro

Q1) How do I get this onto gitlab?

I used perl instead and got the same story.

#!/usr/bin/perl

use strict;
use warnings;
use Git::Repository;
use File::Spec;

# Hard-coded values for GitLab access
my $repo_path    = '/home/fritz/Desktop/dl2/repack'; # Path to your lo
+cal git repository
my $commit_msg   = 'Automated commit from Perl script';
my $branch       = 'main';
my $git_user     = '@redacted';
my $git_email    = 'redacted@gmail.com';
my $remote_name  = 'origin';
my $remote_url   = 'git@gitlab.com:aardvark/hja.git'; # Update with yo
+ur GitLab repo URL

# Set up Git repository object
my $repo = Git::Repository->new( work_tree => $repo_path );

# Set user configuration
$repo->run( config => 'user.name',  $git_user );
$repo->run( config => 'user.email', $git_email );

# Add all changes to the staging area
$repo->run( add => '.' );

# Commit changes
$repo->run( commit => '-m', $commit_msg );

# Check if the remote exists, add if not
my $remotes = $repo->run( 'remote' );
if ($remotes !~ /$remote_name/) {
    $repo->run( remote => 'add', $remote_name, $remote_url );
}

# Push changes to the remote repository
$repo->run( push => $remote_name, $branch );

print "Changes pushed to GitLab successfully.\n";

[download]

The message is always that the remote url doesn't exist, or I don't have access to it. I'm all but sure that I'm missing a setting at the gitlab site somewhere. (Frustrated.)

I want perl to pull the wagon from dl to watermark to postprocess.

I made a lot of progress with making this conform to what we might call a distro. I fired up Module Starter and created Acme::Frobnitz, where this is the manifest:

# Project Root
Dockerfile
ignore.txt
Makefile.PL
MANIFEST
README
requirements.txt

# Scripts and Configuration
bin/2.start_docker.sh
bin/4.caller.py
bin/4.wrap.pl
bin/7.wrap.sh
conf/app_config.json

# Libraries
lib/Acme/Frobnitz.pm
lib/python_utils/downloader5.py
lib/python_utils/utilities1.py

# Test Files
t/00-load.t
t/manifest.t
t/pod-coverage.t
t/pod.t
t/download.t

# Extended Tests
xt/boilerplate.t

[download]

Got a cool illustration:

Acme-Frobnitz/
├── bin/
│   ├── 7.wrap.sh
│   └── ... (other scripts)
├── lib/
│   ├── Acme/
│   │   └── Frobnitz.pm
├── t/
│   ├── download.t
├── Makefile.PL
├── ...

Simple driver: 1.driver.pl:

#!/usr/bin/perl
use strict;
use warnings;

use FindBin;
use lib "$FindBin::Bin/../lib";

use Acme::Frobnitz;

# Parse command-line arguments
my $hyperlink = shift @ARGV or die "Usage: $0 <hyperlink>\n";

# Call the download method
print "Starting download process...\n";
my $file = Acme::Frobnitz->download($hyperlink);

print "File downloaded and processed: $file\n";

[download]

Typical output, we dip into python quickly, avert your gaze if necessary:

(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz/bin$ perl 1.driver.pl h
+ttps://www.youtube.com/shorts/CFqehDVY_zQ
Starting download process...
2024-12-06 05:19:13,823 - __main__ - INFO - Current Python Path: ['/ap
+p/bin', '/app/lib/python_utils', '/app/lib', '/usr/local/lib/python31
+0.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-d
+ynload', '/usr/local/lib/python3.10/site-packages', '/app/lib/python_
+utils']
2024-12-06 05:19:17,284 - __main__ - INFO - Attempting to create direc
+tory: /media/fritz/BALLARDT1/2024-12-06/
2024-12-06 05:19:17,285 - __main__ - INFO - Directory created or alrea
+dy exists: /media/fritz/BALLARDT1/2024-12-06/
2024-12-06 05:19:17,285 - __main__ - INFO - Entering function: mask_me
+tadata
2024-12-06 05:19:17,286 - downloader5 - INFO - Masking metadata
2024-12-06 05:19:17,286 - downloader5 - INFO - Received parameters for
+ metadata 
[youtube] CFqehDVY_zQ: Downloading m3u8 information
[youtube] Extracting URL: https://www.youtube.com/shorts/CFqehDVY_zQ
[youtube] CFqehDVY_zQ: Downloading webpage
[youtube] CFqehDVY_zQ: Downloading ios player API JSON
[youtube] CFqehDVY_zQ: Downloading mweb player API JSON
[youtube] CFqehDVY_zQ: Downloading m3u8 information
[info] CFqehDVY_zQ: Downloading 1 format(s): 399+251
[download] Destination: /media/fritz/BALLARDT1/2024-12-06/Mormon_Stori
+es_Podcast_20241204.f399.mp4
[download] 100% of    9.39MiB in 00:00:03 at 2.72MiB/s   
[download] Destination: /media/fritz/BALLARDT1/2024-12-06/Mormon_Stori
+es_Podcast_20241204.f251.webm
[download] 100% of  788.73KiB in 00:00:00 at 1.64MiB/s   

(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz/bin$

[download]

But, the whole reason to bring it to the world of perl was to do testing, and this is where it starts getting fun, IMO.

It truly was fun to write the test. Sure I failed the whole thing, but it's pretty to me:

#!/usr/bin/perl
use strict;
use warnings;
use Test::More tests => 3;
use File::Path qw(remove_tree);
use File::Spec;
use FindBin;
use lib "$FindBin::Bin/../lib";

use Acme::Frobnitz;

# Setup
my $output_dir = "downloads";
remove_tree($output_dir) if -d $output_dir;  # Clean up any previous d
+ownloads

# Test 1: Ensure the method exists
can_ok('Acme::Frobnitz', 'download');

# Test 2: Attempt to download a YouTube video
my $youtube_link = 'https://www.youtube.com/shorts/CFqehDVY_zQ';
eval {
    Acme::Frobnitz->download($youtube_link);
};
ok(!$@, "YouTube video download did not throw an error");

# Test 3: Verify output directory and files
ok(-d $output_dir, "Output directory exists");
my @files = glob(File::Spec->catfile($output_dir, '*'));
ok(@files > 0, "Downloaded files exist in the output directory");

# Teardown
remove_tree($output_dir);  # Clean up after test

done_testing();

[download]

(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ prove -lv t/download.t
t/download.t .. 
1..3
ok 1 - Acme::Frobnitz->can('download')
bash: 7.wrap.sh: No such file or directory
not ok 2 - YouTube video download did not throw an error

#   Failed test 'YouTube video download did not throw an error'
#   at t/download.t line 24.
not ok 3 - Output directory exists

#   Failed test 'Output directory exists'
#   at t/download.t line 27.
not ok 4 - Downloaded files exist in the output directory

#   Failed test 'Downloaded files exist in the output directory'
#   at t/download.t line 29.
not ok 5 - planned to run 3 but done_testing() expects 4

#   Failed test 'planned to run 3 but done_testing() expects 4'
#   at t/download.t line 34.
# Looks like you planned 3 tests but ran 5.
# Looks like you failed 4 tests of 5 run.
Dubious, test returned 4 (wstat 1024, 0x400)
Failed 2/3 subtests

[download]

Let's see where I am now:

(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ prove -lv t/download.t
t/download.t .. 
not ok 1 - 7.wrap.sh exists and is executable

#   Failed test '7.wrap.sh exists and is executable'
#   at t/download.t line 13.
ok 2 - Acme::Frobnitz->can('download')
bash: 7.wrap.sh: No such file or directory
not ok 3 - YouTube video download did not throw an error

#   Failed test 'YouTube video download did not throw an error'
#   at t/download.t line 23.
ok 4 - Output directory exists
ok 5 - Downloaded files exist in the output directory
1..5
# Looks like you failed 2 tests of 5.
Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/5 subtests 

...
(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ ls -l bin/7.wrap.sh
-rw-r--r-- 1 fritz fritz 1933 Dec  5 21:55 bin/7.wrap.sh
(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ chmod +x bin/7.wrap.sh
(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ ls -l bin/7.wrap.sh
-rwxr-xr-x 1 fritz fritz 1933 Dec  5 21:55 bin/7.wrap.sh
(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$

[download]

I think that might help me...

(new-env) fritz@laptop:~/Desktop/Acme-Frobnitz$ prove -lv t/download.t
t/download.t .. 
ok 1 - 7.wrap.sh exists and is executable
ok 2 - Acme::Frobnitz->can('download')
...

[download]

Alright, one more test passed. At the durch Khyulo/Musk Goetterdaemmerung of a brazenly-hacked American election, this is my idea of fun. Using perl to call python makes sense to me, because I am that guy who wants a return value. And the progress I make is proportional to my astute logging. You've got to mask out the metadata from these things or get buried by them.

Question: where should I put this code if this is meant to be distro-ey? I find myself wanting to add it to Frobnitz.pm, but that can't be right. (Nobody else does that, right?) 

use Log::Log4perl;

# Initialize Log4perl
Log::Log4perl->init(\<<'END');
log4perl.logger                    = INFO, FileAppender

log4perl.appender.FileAppender     = Log::Log4perl::Appender::File
log4perl.appender.FileAppender.filename = ~/Desktop/Acme-Frobnitz/logs
+/acme-frobnitz.log
log4perl.appender.FileAppender.layout   = Log::Log4perl::Layout::Patte
+rnLayout
log4perl.appender.FileAppender.layout.ConversionPattern = %d [%p] %m%n
END

# Create logger
my $logger = Log::Log4perl->get_logger();

# Log messages
$logger->info("This is an info message.");
$logger->error("This is an error message.");

[download]

Also, what tests would you add here? I've asked before and gotten several responses, but now I can actually implement them. I also wonder whether this gets read in Berlin still. It well might not. Almost everything that Americans use to warn of cybercrimes whilst determining the fate of the free world has been flared up to dead silence. Look for sources on substack.

Keep yourself healthy; if helps me to write perl. Gruss aus yehacktem Amiland....

In Section Seekers of Perl Wisdom