in reply to Back to programming? A gossamer hope to automate SSL cert renewals

Your request sounds a bit like "What is the best way to write a wrapper around a HTTP REST API". (well, the part about a database to track clients and their details is sort of unrelated, but I think the first step to this project is a solid understanding of the web service and the workflows it enables, then the in-house database and cli tools just sort of follow from that)

I have strong opinions on writing web service wrappers, which you might find useful. This opinion is not necessarily "what is the ultimate way" but more "what is the most pragmatic way that gets the job done and is debuggable and reusable", because I run into this task over and over and over again. (everyone has a web API these days) I've applied this pattern probably 20-30 times, and it gets a little more refined each time.

Step 1 : make a request work with curl

The first step is to make sure you understand the documentation that the service has provided for you and are able to get a working API key and make requests work. Services often show 'curl' examples, so do that. Pay attention to the response format, especially the response codes and json structure for error messages. Lots of people don't really understand the HTTP spec, even the ones writing APIs for large companies.

Step 2 : Set up the boilerplate for making these requests in perl

Choose a namespace for your module, usually starting with the company name. Your suggestion of Digicert::Automation is fine. I might choose Digicert::CertCentral::Client or just Digicert::CertCentral. Think of the basic attributes you need to make a request, and then write 
package DigiCert::Automation;

=head1 DESCRIPTION

Blah blah about Digicert.

You need an API key to make calls.  API keys blah blah blah
Set L</api_key> to the secret value.

The API is documented at https://dev.digicert.com/en/certcentral-apis.
+html

=cut

use v5.36;
use Moo;
use Carp;
use JSON::MaybeXS;
use LWP::UserAgent;
use Log::Any '$log', default_adapter => 'Stderr';
use URI;

has base_uri     => is => 'rw', default => 'https://caas.digicert.com/
+automationws/v1';
has api_key      => is => 'rw', required => 1;

has http_timeout => is => 'rw', default => 60,
  trigger => sub ($self, @) {
    $self->_user_agent->timeout($self->http_timeout)
      if $self->_has_user_agent;
  };

has _user_agent => ( is => 'lazy', predicate => 1 );
sub _build__user_agent($self, @) {
  my $ua= LWP::UserAgent->new(cookie_jar => {});
  $ua->timeout($self->http_timeout)
    if defined $self->http_timeout;
  $ua->default_header(Accept => 'application/json');
  $ua->default_header('X-DC-DEVKEY' => $self->api_key);
  $ua->add_handler('request_send', sub ($req, $ua, $h, @) {
    if ($log->is_trace) {
      my $req_text= $req->as_string;
      $req_text =~ s/^/    /mg;
      $log->trace("Digicert HTTP Request:\n$req_text");
    }
    return;
  });
  $ua->add_handler('response_done', sub ($res, $ua, $h, @) {
    if ($log->is_trace) {
      my $res_text= $res->as_string;
      $res_text =~ s/^/    /mg;
      $log->trace("Digicert HTTP Response:\n$res_text");
    }
    return;
  });
  push @{ $ua->requests_redirectable }, 'POST';
  return $ua;
}

[download]

Adapt that boilerplate to set whatever default headers you need, which you discovered during the 'curl' experiments above.

Note the logging integration - use whatever logger you like, but this example uses the log levels of Log::Any to control whether it dumps verbose request/response packets on every API call. I combine this with Log::Any::Adapter::Daemontools (which I wrote) so that I can use environment variables or SIGUSR1 to bump up the log level and see the API requests.

Step 3 : Add utility methods for the most common API pattern

This depends on the service, but often there will be a very common 'GET' and 'POST' pattern where you compose parameters and JSON content, then decode the JSON response. There are usually "expected errors" (like the API replying that your certificate doesn't exist or can't be renewed) and "unexpected errors" like the server returning 500 due to back-end problems or calling an API method that no longer exists. I usually have these utility methods die on the unexpected errors, and return the expected errors up one more level. But, this depends on the service. I've seen some services that return a 404 for queries of a keyword that can't be found, even when the URL is perfectly valid. So this is where you account for all those quirks and provide an API that makes sense for what you're doing. 

# Shorthand for attaching the auth token and running an API request.
sub _user_agent_request($self, $req) {
  $log->debug("Digicert API: ".$req->method.' '.$req->uri)
    if $log->is_debug;
  my $startT= time;
  my $res= $self->_user_agent->request($req);
  my $endT= time;
  $log->infof("%s  response: %s (took %.3f sec)",
      (!$log->is_debug? "Digicert API: ".$req->method.' '.$req->uri : 
+''),
      $res->status_line, $endT - $startT)
    if $log->is_debug or $endT-$startT > 3 or !$res->is_success;
  return $res;
}

# Shorthand that takes an HTTP response and either unpacks the data or
+ dies.
sub _return_data_or_die_informatively($self, $res, $req) {
  my $data;
  $data= JSON->new->allow_nonref->decode($res->decoded_content)
    if $res->content_type =~ m,^application/.*?json\z,;
  if ($res->is_success && defined $data && defined $data->{data}) {
    return $data->{data};
  } else {
    croak "DigiCert ".$res->status_line." -- ".$res->decoded_content;
  }
}

[download]
Then I add wrappers around the common parameter patterns: 
# Shorthand private method for GET request with URL parameters
sub _api_get($self, $uri_tail, $query_params) {
  my $uri= URI->new($self->base_uri . '/' . $uri_tail);
  $uri->query_form(%$query_params) if $query_params;
  my $req= HTTP::Request->new(GET => $uri, []);
  my $res= $self->_user_agent_request($req);
  return $self->_return_data_or_die_informatively($res, $req);
}

# Shorthand private method for POST request with JSON packet of parame
+ters
sub _api_post($self, $uri_tail, $query_params, $body_data) {
  my $uri= URI->new($self->base_uri . '/' . $uri_tail);
  $uri->query_form(%$query_params) if $query_params;
  my $req= HTTP::Request->new(POST => $uri,
    [Content_Type => 'application/json'],
    encode_json($body_data)
  );
  my $res= $self->_user_agent_request($req);
  return $self->_return_data_or_die_informatively($res, $req);
}

[download]
In some APIs, POST requests won't have query parameters, only the json body, or maybe no json body and only the query parameters, or maybe its a mix of HTTP verbs like PATCH/DELETE/PUT. Whatever the API is doing, adapt these helpers for that.

Step 4 : Experiment!

Now that you have the boilerplate set up, start experimenting with API calls so you can get a good feel for how things are working: 

perl -Ilib -MDDP -MDigicert::CertCentral -E '\
  my $d= Digicert::CertCentral->new(api_key => $ENV{DIGICERT_API_KEY})
+; \
  my $data= $d->_api_post("automation/viewAutomationDetails", { accoun
+tId => ... }); \
  p $data;'

[download]

Step 5 : Wrap the behavior to be more friendly to perl

Once you have a good feel for how things work, start making methods for the API calls. Future-you (and coworkers) will thank you if you copy the documentation from the API: 

=head2 api_viewAutomationDetails

Input structure:

  {
    "accountId": 5153184,
    "automationId": 298577,
    "divisionId": 677793
  }

Output structure:

  "autoStatusResponse": {
      "automationStatus": "INSTALL_VALIDATION_FAILED",
      "requestedOn": "1599716536705",
      "installSettings": {
          "installationType": "AUTO_INSTALL_AFTER_APPROVAL",
          "isAlwaysOn": false
      },
      "userDetailsResponse": {
          "firstName": "CertCentral",
          "lastName": "Admin",
          "container": {
              "id": 677793,
              "name": "Cert Testing Inc."
          }
      }
  },
  "autoOrderProgressResponse": {
      "csrGenResponse": {
          "csrProgress": "CSR_GENERATION_SUCCEEDED"
      },
      "certInstallResponse": {
          "certInstallProgress": "INSTALL_VALIDATION_FAILED",
          "causeOfFailure": "Post install validation is unsuccessful, 
+could be due to installation failure / scan failure",
          "solution": "Verify certificate manually on website and canc
+el this automation request if installed, otherwise retry. Contact cus
+tomer support for additional help."
      },
      "isRetryApplicable": true
  },
  "certDetailsResponse": {
      "commonName": "cert-testing.com",
      "signatureHash": "",
      "orgDetails": {
          "name": "Cert Testing Inc.",
          "address": "2801 N Thanksgiving Way",
          "address2": "Suite 500",
          "city": "Lehi",
          "state": "Utah",
          "telephone": "801-701-9600"
      },
      "sans": "",
      "automationProfileName": "22jul2020 01",
      "automationProfileId": 741.0,
      "automationProfileStatus": "ACTIVE",
      "productType": "SSL_SECURESITE_FLEX",
      "validityPeriod": "1Y"
  },
  "isRequestApproval": false

https://dev.digicert.com/en/certcentral-apis/automation-api/view-autom
+ation-details.html

=cut

sub api_viewAutomationDetails($self, %params) {
  $params{accountId} //= $self->account_id;
  $params{divisionId} //= $self->division_id;
  length $params{automationId} or croak "automationId required";
  $self->_api_post("automation/viewAutomationDetails", \%params);
}

[download]

Step 6 : Repeat 4 & 5 until you solved the entire business problem

Just keep making api methods until you achieve what you set out to accomplish. Don't get distracted wrapping things with objects. If you run out of time here, you're at a good stopping point anyway.

Step 7 : Decide whether to wrap things with Objects

If you think this API is going to be used so frequently that it warrants additional object structure, then you can begin to decide what is appropriate for that. One method is to exhaustively wrap every logical entity in the API with a Moo object and declare every attribute. For a stable API, this might not be a bad idea. In fact, now that we have AI, you can basically show it an example object and a JSON packet from the API docs and ask it to write the attributes for you, and need almost no keyboard work on your part.

Another option is to write objects that wrap the JSON output and provide convenient access to certain parameters, and ignore the rest: 

package Digicert::CertCentral;

=head2 automation

  $atn= $api->automation($id);

Return a new or cached Automation object for the given ID.

=cut

sub automation($self, $id) {
  return $self->{_automation_cache}{$id}->refresh(10)
    if $self->{_automation_cache}{$id};
  my $atn= Digicert::CertCentral::Automation->new(api => $self, id => 
+$id);
  Scalar::Util::weaken($self->{_automation_cache}{$id}= $atn);
  $atn;
}

package Digicert::CertCentral::Automation;
use v5.36;
use Moo;

has api     => ( is => 'ro', required => 1 );
has id      => ( is => 'ro', required => 1 );
has data    => ( is => 'rw' );
has data_ts => ( is => 'rw' );

sub BUILD ($self, @) {
  # die if automation doesn't exist, before finishing constructor
  $self->refresh;
}

sub refresh($self, $max_age=0) {
  if (!$self->data_ts || !$max_age || time - $self->data_ts > $max_age
+) {
    $self->data($self->api->api_viewAutomationDetails(automationId => 
+$self->id));
    $self->data_ts(time);
  }
  $self;
}

sub status($self) {
  $self->data->{autoStatusResponse}{automationStatus};
}

[download]

of course there are a long list of decision to make about how an object can most usefully help the code that is using it, especially caching problems, which is why you want to make sure you solved the actual problem before you get down into all this mental debate about the perl API :-)

Hope that helps.

In Section Meditations