Beefy Boxes and Bandwidth Generously Provided by pair Networks
laziness, impatience, and hubris
 
PerlMonks  

Re: Private Utilities

by graff (Chancellor)
on Dec 01, 2005 at 02:08 UTC ( [id://513172]=note: print w/replies, xml ) Need Help??


in reply to Private Utilities

Seems like just about every day I have to pull something out of an oracle or mysql database, or do some sort of alteration to table data. But leaving the shell command line behind to use the oracle or mysql native CLI just causes too much constraint and discomfort.

So I have two generic Perl/DBI utilities -- one to run "select/describe/show" type queries, one to run "update/insert/delete" type queries -- that just take the query text as a command-line arg, or take a file name that contains the query text. Here's the usage message for the mysql version of the "select" tool:

Usage: mdbget -db database -s 'select sql' [options] [query_param.list +] or: [-db database] -f sql.file [options] [query_param.list +] options: -l : label columns (print list of column names as first line of ou +tput) -d delim : set field delimiter to 'delim' (default delim is tab) "-d sp" and "-d ' '" both work to produce single-space delimite +d data "-d csv" will produce "true CSV" output, adding double-quotes a +s needed NOTE: only "-d csv" will quote fields containing the delimiter +character (csv also turns on "-l", so column names are listed on fi +rst line) -w {norm|mark|keep} : control the treatment of whitespace within f +ields: "-w norm" (default) convert all whitespace strings to single sp +ace "-w mark" keep spaces as-is, convert newline to '\n', tab to '\ +t' "-w keep" make no alterations to white space at all query_param.list is needed if sql contains "?" placeholder(s) for par +am(s)
The tools both use a special "DBI-wrapper" module I cooked up, that maps the "database" name to an appropriate user-id and password for connecting to the server, so both utils can be used with all the different databases I have to deal with.

The placeholder support for the sql statements means I can pipe a list of field data from any other shell command or data file, which saves me tons of time. (update: and of course the big win for me is the ability to pipe query output to other shell commands)

But it's definitely the sort of thing I tend to keep to myself and just a few trusted colleagues (especially the update/insert/delete tool).

Update: here's the code for the "select" tool (minus the special module that defines the user/password settings per database)

... oops! just noticed that this was written before I habituated to "use strict" -- well, I'll post it anyway and take my lumps (it does work for me, after all)...

#!/usr/bin/perl # Program: mdbget # Written by: David Graff # Purpose: general utility for running sql select statements # Command-line args give the mysql database name and an sql-statement # for doing a selection. Selection output is sent to stdout. # The sql-statement can be stored in a file and used by means of the # "-f sql.file" option; if the environment variable SQL_PATH is # defined, this will be searched for the sql.file if the file is not # found in the current directory, or in the directory that holds mdbge +t. # An sql.file may contain comment lines beginning with "#"; it is # possible to include a database name in the file as well (so # it won't be needed on the command line) on a line starting with # "##", e.g.: # ## -a account_name $Usage = <<ENDUSE; Usage: mdbget -db database -s 'select sql' [options] [query_param.list +] or: [-db database] -f sql.file [options] [query_param.list +] (-a database is a synonym for -db database) options: -l : label columns (print list of column names as first line of ou +tput) -d delim : set field delimiter to 'delim' (default delim is tab) "-d sp" and "-d ' '" both work to produce single-space delimite +d data "-d csv" will produce "true CSV" output, adding double-quotes a +s needed NOTE: only "-d csv" will quote fields containing the delimiter +character (csv also turns on "-l", so column names are listed on fi +rst line) -w {norm|mark|keep} : control the treatment of whitespace within f +ields: "-w norm" (default) convert all whitespace strings to single sp +ace "-w mark" keep spaces as-is, convert newline to '\\n', tab to ' +\\t' "-w keep" make no alterations to white space at all query_param.list is needed if sql contains "?" placeholder(s) for par +am(s) ENDUSE use Getopt::Long; use MysqlFunc; ### this is where user-ids/passwds are kept die $Usage unless ( &GetOptions( \%opt, 'db=s', 'a=s', 'd=s', 'f=s', 's=s', ' +w=s', 'l' ) && ( $opt{f} ne "" ^ ( $opt{s} =~ /^\s*(select|show|describe) +\s+/i && ( $opt{a} =~ /^\w+$/ or $opt{db} =~ /^\w ++$/ )))); die $Usage if ( $opt{w} ne "" && $opt{w} !~ /norm|mark|keep/ ); $opt{w} ||= "norm"; $opt{db} ||= $opt{a}; if ( $opt{s} ne "" ) { $sql = $opt{s}; } else { unless ( -r $opt{f} ) { $base = $0; $base =~ s%[^/]*$%%; $sfile = ""; if ( $ENV{SQL_PATH} ne "" ) { foreach $p ( split( /:/, $ENV{SQL_PATH} )) { if ( -r "$p/$opt{f}" ) { $sfile = "$p/$opt{f}"; last; } } } if ( $sfile eq "" ) { if ( -r "$base$opt{f}" ) { $sfile = "$base$opt{f}"; } else { $sqldir = ( $ENV{SQL_PATH} ne "" ) ? $ENV{SQL_PATH} : +"not set"; die "No script file $opt{f} in $ENV{PWD}, $base or SQL +_PATH ($sqldir)\n"; } } $opt{f} = $sfile; } open( SCRPT, "<$opt{f}" ); while (<SCRPT>) { if ( /^(?:\#|--)/ && / -(db|a) (\w+)/ ) { $opt{db} = $1; } next if ( /^(?:\#|--)|^\s*$/ ); s/^\s*//; # remove leading whitespace s/\s*$/ /; # normalize trailing whitespace $sql .= $_; } close SCRPT; die "No suitable sql statement found in $opt{f}\n" unless ( $sql =~ /^\s*(select|show|describe)\s+/i ); die "No database name has been provided\n" unless ( $opt{db} =~ /^\w+$/ ); } $delim = "\t"; if ( $opt{d} ) { $delim = ( $opt{d} eq "sp" ) ? " " : $opt{d}; } @qm = ( $sql =~ /\?/g ); $nq = scalar(@qm); if ( $nq ) { $infile = ( @ARGV ) ? join( " ", @ARGV) : "stdin"; print STDERR "looking for $nq params per line from $infile...\n"; while (<>) { s/[\r\n]+$//; if ( $nq > 1 ) { @row = split( /$delim/ ); if ( scalar( @row ) != $nq ) { warn scalar(@row)," fields instead of $nq at line $. - + skipped: $_\n"; next; } push( @data, [ @row ] ); } else { push( @data, [ $_ ] ); } } die "No usable input data for query parameters\n" unless (@data); } my $db = MysqlFunc->new( $opt{db} ); my $hdrline = ( $delim eq "csv" or $opt{l} ); if ( $hdrline and $sql =~ /\s*describe/i ) { @fields = qw/Field Type Null Key Default Extra/; printline( \@fields ); } if ( @data ) { $sqlop = $db->sqlPrep( $sql ); foreach $param ( @data ) { $grid = $db->queryGrid( $sqlop, $hdrline, $param ); ### (just a wrapper method for DBI::prepare/execute/fetcha +ll_arrayref) foreach $row ( @$grid ) { printline( $row ); } } $db->sqlDone( $sqlop ); } else { $grid = $db->queryGrid( $sql, $hdrline ); foreach $row ( @$grid ) { printline( $row ); } } # In case output delimiter is set to CSV, implement these rules # (from www.perlmonks.org node 151784): # # Here is a description of the basic CSV spec as implemented in most M +icrosoft products: # # 1. Rows are delimited by returns. (\r\n or \n depending on the platf +orm, binmode, etc.) # 2. Fields within a row are delimited by ",". # 3. Fields may be quoted or unquoted. # 4. Quoted fields are literal text that start and end with an unpaire +d ". # Separators, returns, etc can appear within a quoted field, and " +can appear doubled. # 5. Unquoted fields cannot contain the separator, returns, or quotati +on marks. # They are also subject to some interpretation. For instance number +s may appear # in floating point, and an empty field is a null (represented with +in Perl by undef # - very few parsers get this right). # 6. It is customary for the first row to be the field names, and for +all rows to have # the same number of fields. sub printline { my @cols = @{$_[0]}; my $sep = $delim; if ( $opt{w} eq "norm" ) { foreach ( @cols ) { s/\s+/ /g; } } elsif ( $opt{w} eq "mark" ) { foreach ( @cols ) { s/\n/\\n/g; s/\r/\\r/g; s/\t/\\t/g; } } if ( $delim eq "csv" ) { $sep = ","; foreach ( @cols ) { s/\"/\"\"/g; # double all quotation marks present in fiel +d content $_ = '"' . $_ . '"' if ( /[\"\'\,]/ ); # bracket with "... +" if necessary } } print join( $sep, @cols ) . "\n"; }

Log In?
Username:
Password:

What's my password?
Create A New User
Domain Nodelet?
Node Status?
node history
Node Type: note [id://513172]
help
Chatterbox?
and the web crawler heard nothing...

How do I use this?Last hourOther CB clients
Other Users?
Others chilling in the Monastery: (4)
As of 2024-04-23 06:53 GMT
Sections?
Information?
Find Nodes?
Leftovers?
    Voting Booth?

    No recent polls found