NAME
    netdisco.pm - Utility functions used for back and front ends

DESCRIPTION
    This module provides utility functions for use with netdisco in both the
    front and backend. Front-end specific features are limited to the mason
    (.html) files, and the back-end specific features are limited to
    netdisco.

AUTHOR
    Max Baker

SYNOPSIS
GLOBALS
    %netdisco::DBH
        Holds Database Handles, key is db name as set in config file.

    %netdisco::DB
        Index of current Database Handle. Default 'Pg';

    %netdisco::CONFIG
        Holds config info from "netdisco.conf"

    %netdisco::GRAPH
        Holds vertex information for "make_graph()"

    $netdisco::SENDMAIL
        Full path to sendmail executable

    $netdisco::SQLCARP - Carps SQL!
        This will "carp()" the SQL sent off to the server for Debugging.

        If running under mason, the output of "carp()" goes to the Apache
        Error Log. From the shell it goes to STDERR.

        Note that if you set this on a MASON page, the value will remain
        cached across most of the current httpd proccesses. Make sure you
        set it back to 0 via mason when you're done, unless you like
        watching Apache's error_log grow.

    %PORT_CONTROL_REASONS
        Reason why a port would be shutdown. These get fed into
        "port_control_log"

    $VERSION - Sync'ed with Netdisco releases

Exportable Functions
  General Functions
    add_arp(mac,ip)
        Manipulates entries in 'node_ip' table.

        Expires old entries for given IP address.

        Adds new entry or time stamps matching one.

    add_node(mac,ip,port)
        Manipulates entries in "node" table.

        Expires old entries matching given arguments.

        Adds a new entry or time stamps matching old entry.

    add_nbt(ip,mac,nbname,domain,server,nbuser)
        Manipulates entries in 'node_nbt' table.

        Expires old entries for given MAC address.

        Adds new entry or time stamps matching one.

    bits_to_mask(bits)
        Takes a CIDR style network mask in number of bits (/24) and returns
        the older style bitmask.

    get_community(type,host,ip)
        Get Community depending on type (ro,rw). If "get_community" is
        defined, then get the try to get the community from shell-command.
        If "get_community" is undefined or nothing is returned from the
        command use "community" or "community_rw".

        The command specified in "get_community" must return in stdout a
        string like

            community=<list of readonly-communities>
            setCommunity=<list of write-communities>

        Returns Community-List as Array reference

        Options: type => 'ro'|'rw' for the type of community host => name of
        the device ip => device ip-address

    config()
        Reads the config file and fills the %CONFIG hash.

    updateconfig()
        Checks the modification time of the configuration file and re-reads
        it if needed. (Note: for now, defaults are not reset - i.e., if
        there was an item in the config file before, and it is missing when
        we reread it, it keeps its old value and doesn't get set to the
        default.)

        Uses eval to run config, so that we can keep running with the old
        config if there's a problem with the config file.

    has_layer(bit string,layer)
        Takes ascii encoded string of eight bits, and checks for the
        specific layer being true. Most significant bit first.

            has_layer(00000100,3) = true

    hostname(ip)
        Returns the DNS server entry for the given ip or hostname.

    getip(host)
        Returns the IP Address of a given IP or hostname. If the given
        argument appears to be in dotted octet notation, it does no DNS hits
        and just returns it.

        It also just returns an IP address with a subnet mask. Subnet masks
        are not permitted on host names.

    in_device(device,to_match)
        First argument can either be:

            1. plain text IP or hostname
            2. A row from the device table as returned from sql_hash

        Second argument is an array ref as returned from config, eg.
        "bulkwalk_no".

    in_subnet(subnet,ip)
        Returns Boolean. Checks to see if IP address is in subnet. Subnet is
        defined as single IP address, or CIDR block. Partial CIDR format
        (192.168/16) is NOT supported.

         in_subnet('192.168.0.0/24','192.168.0.3') = 1;
         in_subnet('192.168.0.3','192.168.0.3') = 1;

    in_subnets(ip,config_directive)
        Returns Boolean. Checks a given IP address against all the IPs and
        subnet blocks listed for a config file directive.

         print in_subnets('192.168.0.1','macsuck_no');

    active_subnets()
        Returns array ref containing all rows from the subnets table that
        have a node or device in them.

    dump_subnet(cidr style subnet)
        Serves you all the possible IP addresses in a subnet.

        Returns reference to hash. Keys are IP addresses in dotted decimal
        that are in the subnet.

        Gateway and Broadcast (.0 .255) addresses are not included.

          $hash_ref = dump_subnet('192.168.0.0/24');
          scalar keys %$hash_ref == 254;

        Also accepted :

          dump_subnet('14.0/16');
          dump_subnet('4/24');

    is_mac(mac)
        Returns Boolean. Checks if argument appears to be a mac address.

        Checks for types :

            08002b:010203
            08002b-010203
            0800.2b01.0203
            08-00-2b-01-02-03
            08:00:2b:01:02:03

    log(class,text)
        Inserts an entry in the "log" table.

            log('error',"this is an error");

    mail(to,subject,body)
        Sends an E-Mail as Netdisco

    is_private(ip)
        Returns true if a given IP address is in the RFC1918 private address
        range.

    cidr(ip, mask)
        Takes an IP address and netmask and returns the CIDR format subnet.

    mask_to_bits(mask)
        Takes a netmask and returns the CIDR integer number of bits.

            mask_to_bits('255.255.0.0') = 16

    is_secure
        To be run under mason only.

        Returns true if the server want's to be secure and is, or true if
        the server doesn't want to be secure.

        Returns false if the server is not secure but wants to be.

    url_secure(url)
    sort_ip()
        Used by "sort {}" calls to sort by IP octet.

        If passed two hashes, will sort on the key "ip" or "remote_ip".

    sort_port()
        Used by "sort()" - Sort port names with the following formatting
        types :

            A5
            5
            FastEthernet0/1
            FastEthernet0/1-atm
            5.5
            Port:3

        Works on hashes if a key named port exists.

        Cheers to Bradley Baetz (bbaetz) for improvements in this sub.

    make_graph()
        Returns "Graph::Undirected" object that represents the discovered
        network.

        Graph is made by loading all the "device_port" entries that have a
        neighbor, using them as edges. Then each device seen in those
        entries is added as a vertex.

        Nodes without topology information are not included.

    root_device(ip)
        If the given IP Address matches a device IP, returns it.

        If the given IP Address matches an alias of a device, returns the IP
        of the device the alias belongs to.

  User Functions
    user_add(user,%args)
        Adds or changes a user account.

        %args can have key values of { pw, admin, port, ldap, full_name,
        note }

        Returns error message if problem.

    user_del(user)
        Deletes a user from netdisco.

        Returns result from "DBI->do()"

        Integer for number of rows deleted, or undef if error.

    user_ldap_verify(user,password)
        Test a user from netdisco.

        Returns 1 if user and password are OK, or undef if error.

    ldap_search(filter,attrs,user,pass)
        Perform an LDAP search from the configured ldap_base with the
        specified filter.

        Uses an anonymous bind if the user is 'anonymous' or undefined.

        Returns reference to an array of Net::LDAP::Entry objects which
        match the search filter. Each entry will contain the accessible
        attributes as defined in attrs array reference. If attrs is
        undefined, then the server will return the attributes that are
        specified as accessible by default given the bind credentials.

  SQL Functions
    dbh()
        Creates and returns a database handle. Creates once, then returns
        the cached copy.

        Select database handle in use by localizing $netdisco::DB

    dbh_quote($text)
        Runs DBI::dbh->quote() on the text and returns it.

    hash_diff($hashref_orig,$hashref_new)
        Sees if items to change in second hash are different or new compared
        to first.

    insert_or_update(table, {matching}, {values} )
        Checks for Existing entry in table using "\%matching" and either
        updates or inserts into table with "\%values" accodringly.

            insert_or_update('device', { 'ip'=>'127.0.0.1' },
                             { 'ip' => '127.0.0.1', dns => 'dog' }
                            );

        First time called it will insert the new entry

        Second time called it will modify the entry with the values.

        Supports

            * Auto Quoting of Values

        Returns undef if problem.

        On inserts in PostgreSQL, returns the OID of the row inserted.

        Or returns value from "DBD::St::execute()"

    sql_column(table,[key_col,val_col],{where})
        Returns reference to hash. Hash has form $hash{key_val}={val_val}

        If multiple matches are found for key_col, only the last one is
        kept.

        Usage is the same as sql_rows() -- See for Usage.

            $OldDevices = sql_column('device',['ip','layers']);

        Creates the hash %$OldDevices where the key is the IP address and
        the Value is the Layers

    sql_columns(table,[key_col,val_col,...],{where})
        Returns reference to hash. Hash has form $hash{key_val}=hash of all
        columns}

        If multiple matches are found for key_col, only the last one is
        kept.

        Usage is the same as sql_rows() -- See for Usage.

            $OldDevices = sql_columns('device',['ip','layers','dns']);

        Creates the hash %$OldDevices where the key is the IP address and
        the Value is a hash with the keys 'ip', 'layers' and 'dns'.

    sql_do(sql)
        Simple wrapper to "$dbh->do()"

        No quoting.

    sql_begin()
        Start an SQL transaction.

        Pass an array reference to the list of tables that should be locked
        in EXCLUSIVE mode for this transaction. Normally, row locking is
        sufficient, so no tables need be locked. However, netdisco's macsuck
        and arpnip processes perform statements like

            UPDATE table SET active='f' WHERE (ip|mac)='foo'

        If two such statements are executed concurrently for the same value
        of 'foo', they can visit the same rows in different order, resulting
        in a deadlock. It's tempting to say that you can solve this by
        making the UPDATE only touch one row at a time by adding "AND
        active" to the WHERE clause; however, it's possible to get multiple
        rows with active=true and open up the window for deadlock again.
        Without a significant rewrite, the best option is to lock the
        appropriate table in exclusive mode (which still allows readers,
        such as the web front end, but blocks any inserts or updates). Since
        netdisco performs updates in bulk, the table will not spend a
        significant amount of time locked.

    sql_commit()
        Finish an SQL transaction and return to AutoCommit mode

    sql_rollback()
        If an SQL transaction is in progress, roll it back and return to
        AutoCommit mode. Suitable to be called in a generic error situation
        when you don't know what has been done, since it is a noop if there
        is no transaction occurring.

    sql_disconnect()
        Disconnect from the SQL database, e.g., before forking.

    sql_hash(table, [columns], {where})
        Returns reference to hash representing single row.

        Usage is the same as sql_rows() -- See for Usage.

            my $hashref = sql_hash('device',['ip','ports'], {'ip'=>'127.0.0.1'});

    sql_match(text,exact_flag)
        Parses text to substitue wildcards * and ? for % and _

        Optional exact_flag specifies whether or not to search for that
        exact text search or to do a *text*.

        Default is non_exact.

    sql_rows(table, [columns] , {where} ,OR, orderbystring)
        Returns a reference to an array of hash references. Each hash
        reference is the return of "$dbh->fetchrow_hashref"

        Supports

            * Joins
            * Pattern Matching
            * NULL/NOT NULL 
            * Boolean OR or AND criteria
            * Auto-Quotes all values and Override
            * IN (list) and NOT IN (list) clauses

        Pass a true value for the OR argument to join constraints in the
        WHERE clause by OR instead of AND.

        SIMPLE QUERY:
            Select info for every device:

                $matches = sql_rows('device',['ip','dns','uptime]);

        DISABLE AUTOQUOTING:
            Pass the where value as a reference:

                sql_rows('device d, device e',['d.ip,d.dns,e.ip,e.dns'],
                        {'d.dns' => \'e.dns'});

            Creates the SQL:

                SELECT d.ip,d.dns,e.ip,e.dns FROM device d, device e WHERE d.dns = e.dns;

            This also leaves a security hole if the where value is coming
            from the outside world because someone could stuff in
            'dog';delete from node where true;...> as a value. If you turn
            off quoting make sure the program is feeding the where value.

        DISABLE AUTOQUOTING AND CONNECTOR
            Pass the where value as a double scalar reference:

                sql_rows('device',['*'], {'age(creation)' => \\"< interval '1 day'"})

            Creates the sql:

               SELECT * FROM device WHERE age(creation) < interval '1 day';

        NULL VALUES
            Select all the devices without reverse dns entries:

              $matches = sql_rows('device',['ip','name'],{'dns'=>'IS NULL'});

        JOINS
                $matches = sql_rows('device d left join device_ip i on d.ip = i.ip',
                                    ['distinct(d.ip)','d.dns','i.alias','i.ip'],
                                    {'d.ip/i.alias'=>'127.0.0.1', 'd.dns/i.dns' => 'dog%'},
                                    1);

            Selects all rows within "device" and "device_ip" where
            "device_ip.alias" or "device.ip" are "localhost" or
            "device_ip.dns" or "device.dns" has "dog" in it.

            Where columns with slashes are split into separate search terms
            combined with "OR":

               { 'd.ip/i.alias' => '127.0.0.1' }

            Creates the SQL

               "WHERE (d.ip = '127.0.0.1' OR i.alias = '127.0.0.1')"

        MULTIPLE CONTSTRAINTS ON SAME COLUMN
            Pass the where values as an array reference

                $matches = sql_rows('device',['ip'],{'dns' => ['cat%','-g%'] },1 );

            Creates the SQL:

                SELECT ip FROM device WHERE (dns like 'cat%') OR (dns like '-g%');

        IN (list) CLAUSE
            Pass the value as a double array reference. Values are
            auto-quoted.

            Single array reference is for creating multiple "WHERE" entries
            (see above)

                $matches = sql_rows('node',['mac'], {'substr(mac,1,8)' => [['00:00:00','00:00:0c']]})

            Creates the SQL:

                SELECT mac FROM node WHERE (substr(mac,1,8) IN ('00:00:00','00:00:0c'));

        NOT IN (list)
            Pass the value as a double array reference. Prepend one of the
            array values with a "!"

                $matches = sql_rows('device',['name'], {'vendor' => [['!cisco','hp']]  });

            Will find all devices that are neither cisco or hp.

        ORDER BY CLAUSE
                $matches = sql_rows('device',['ip','dns'], undef, undef, 'order by dns limit 10');

    sql_query(table, [columns] , {where} ,OR, orderbystring)
        Returns a DBI state handle on which the SQL query has been
        prepare()d and execute()d. This function is good for large queries
        instead of sql_rows(), as the whole result set does not need to be
        read into memory.

        Code such as

            my $nodes = sql_rows(...);
            foreach my $row (@$nodes) {
                ...
            }

        can be replaced by

            my $sth = sql_query(...);
            while (my $row = $sth->fetchrow_hashref()) {
                ...
            }

        The arguments are exactly the same as sql_rows().

    sql_scalar(table,[column],{where})
        Returns a scalar of value of first column given.

        Internally calls "sql_hash()" which calls "sql_rows()"

        All arguments are passed directly to "sql_hash()"

            my $count_ip = sql_scalar('device',['COUNT(ip)'],{'name' => 'dog'});

    sql_vacuum(table,%opts)
        Runs a VACUUM ANALYZE if we are Postgres

        Pass the table name as '' if you want to vacuum all tables.

        Options:

            verbose => 1  (set if DEBUG)
            full    => 1

    homepath(config,[default])
        Return the full path of the given file as specified in the config
        file by prepending $CONFIG{home} to it, if it doesn't already start
        with a slash. If no value is specified in the config file, the
        default is used.

    tryuse(module,%opts)
        Try to use the given module.

        Returns two values: success / failure, and the error message if
        failure. Caches values if it's non fatal.

        Options:

            ver => version of module required
            die => 1 if you want to die instead of recover yourself