<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<!--Converted with LaTeX2HTML 2008 (1.71)
original version by: Nikos Drakos, CBLU, University of Leeds
* revised and updated by: Marcus Hennecke, Ross Moore, Herb Swan
* with significant contributions from:
Jens Lippmann, Marek Rouchal, Martin Wilck and others -->
<HTML>
<HEAD>
<TITLE>It comes in the night and sucks the essence from your computers. </TITLE>
<META NAME="description" CONTENT="It comes in the night and sucks the essence from your computers. ">
<META NAME="keywords" CONTENT="developers">
<META NAME="resource-type" CONTENT="document">
<META NAME="distribution" CONTENT="global">
<META NAME="Generator" CONTENT="LaTeX2HTML v2008">
<META HTTP-EQUIV="Content-Style-Type" CONTENT="text/css">
<LINK REL="STYLESHEET" HREF="developers.css">
</HEAD>
<BODY >
<!--Navigation Panel-->
<IMG WIDTH="81" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next_inactive"
SRC="file:/usr/lib/latex2html/icons/nx_grp_g.png">
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up"
SRC="file:/usr/lib/latex2html/icons/up_g.png">
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous"
SRC="file:/usr/lib/latex2html/icons/prev_g.png">
<BR>
<BR>
<BR>
<!--End of Navigation Panel-->
<P>
<P>
<P>
<P>
<H1 ALIGN="CENTER"><IMG
WIDTH="457" HEIGHT="99" ALIGN="BOTTOM" BORDER="0"
SRC="bacula-logo.png"
ALT="\includegraphics{./bacula-logo.eps}">
<BR><P><P>
<BR>
<DIV ALIGN="CENTER">
<FONT SIZE="+1">It comes in the night and sucks
the essence from your computers.
</FONT></DIV></H1>
<DIV>
<P ALIGN="CENTER"><STRONG>Kern Sibbald</STRONG></P>
<P ALIGN="CENTER"><STRONG>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
<BR>
September 10, 2009
<BR>
This manual documents Bacula version 2.4.3 (10 October 2008)
<BR>
<BR>
<BR>
Copyright ©1999-2007, Free Software Foundation Europe
e.V.
<BR> <BR>
<BR>
Permission is granted to copy, distribute and/or modify this document under the terms of the
GNU Free Documentation License, Version 1.2 published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU Free Documentation License".
</STRONG></P>
</DIV>
<P>
<BR>
<H2><A NAME="SECTION00010000000000000000">
Contents</A>
</H2>
<!--Table of Contents-->
<UL>
<LI><A NAME="tex2html198"
HREF="developers.html#SECTION00020000000000000000">List of Figures</A>
<LI><A NAME="tex2html199"
HREF="developers.html#SECTION00030000000000000000">List of Tables</A>
<LI><A NAME="tex2html200"
HREF="developers.html#SECTION00040000000000000000">Bacula Developer Notes</A>
<UL>
<LI><A NAME="tex2html201"
HREF="developers.html#SECTION00041000000000000000">General</A>
<LI><A NAME="tex2html202"
HREF="developers.html#SECTION00042000000000000000">The Development Cycle</A>
<LI><A NAME="tex2html203"
HREF="developers.html#SECTION00043000000000000000">Bacula Code Submissions and Projects</A>
<LI><A NAME="tex2html204"
HREF="developers.html#SECTION00044000000000000000">Patches for Released Versions</A>
<LI><A NAME="tex2html205"
HREF="developers.html#SECTION00045000000000000000">SVN Usage</A>
<LI><A NAME="tex2html206"
HREF="developers.html#SECTION00046000000000000000">Subversion Resources</A>
<LI><A NAME="tex2html207"
HREF="developers.html#SECTION00047000000000000000">Developing Bacula</A>
</UL>
<BR>
<LI><A NAME="tex2html208"
HREF="developers.html#SECTION00050000000000000000">Platform Support</A>
<UL>
<LI><A NAME="tex2html209"
HREF="developers.html#SECTION00051000000000000000">General</A>
<LI><A NAME="tex2html210"
HREF="developers.html#SECTION00052000000000000000">Requirements to become a Supported Platform</A>
</UL>
<BR>
<LI><A NAME="tex2html211"
HREF="developers.html#SECTION00060000000000000000">Daemon Protocol</A>
<UL>
<LI><A NAME="tex2html212"
HREF="developers.html#SECTION00061000000000000000">General</A>
<LI><A NAME="tex2html213"
HREF="developers.html#SECTION00062000000000000000">Low Level Network Protocol</A>
<LI><A NAME="tex2html214"
HREF="developers.html#SECTION00063000000000000000">General Daemon Protocol</A>
<LI><A NAME="tex2html215"
HREF="developers.html#SECTION00064000000000000000">The Protocol Used Between the Director and the Storage Daemon</A>
<LI><A NAME="tex2html216"
HREF="developers.html#SECTION00065000000000000000">The Protocol Used Between the Director and the File Daemon</A>
<LI><A NAME="tex2html217"
HREF="developers.html#SECTION00066000000000000000">The Save Protocol Between the File Daemon and the Storage Daemon</A>
</UL>
<BR>
<LI><A NAME="tex2html218"
HREF="developers.html#SECTION00070000000000000000">Director Services Daemon</A>
<LI><A NAME="tex2html219"
HREF="developers.html#SECTION00080000000000000000">File Services Daemon</A>
<UL>
<LI><A NAME="tex2html220"
HREF="developers.html#SECTION00081000000000000000">Commands Received from the Director for a Backup</A>
<LI><A NAME="tex2html221"
HREF="developers.html#SECTION00082000000000000000">Commands Received from the Director for a Restore</A>
</UL>
<BR>
<LI><A NAME="tex2html222"
HREF="developers.html#SECTION00090000000000000000">Storage Daemon Design</A>
<UL>
<LI><A NAME="tex2html223"
HREF="developers.html#SECTION00091000000000000000">SD Design Introduction</A>
<LI><A NAME="tex2html224"
HREF="developers.html#SECTION00092000000000000000">SD Development Outline</A>
<LI><A NAME="tex2html225"
HREF="developers.html#SECTION00093000000000000000">SD Connections and Sessions</A>
<LI><A NAME="tex2html226"
HREF="developers.html#SECTION00094000000000000000">SD Data Structures</A>
</UL>
<BR>
<LI><A NAME="tex2html227"
HREF="developers.html#SECTION000100000000000000000">Catalog Services</A>
<UL>
<LI><A NAME="tex2html228"
HREF="developers.html#SECTION000101000000000000000">General</A>
<LI><A NAME="tex2html229"
HREF="developers.html#SECTION000102000000000000000">Sequence of Creation of Records for a Save Job</A>
<LI><A NAME="tex2html230"
HREF="developers.html#SECTION000103000000000000000">Database Tables</A>
</UL>
<BR>
<LI><A NAME="tex2html231"
HREF="developers.html#SECTION000110000000000000000">Storage Media Output Format</A>
<UL>
<LI><A NAME="tex2html232"
HREF="developers.html#SECTION000111000000000000000">General</A>
<LI><A NAME="tex2html233"
HREF="developers.html#SECTION000112000000000000000">Definitions</A>
<LI><A NAME="tex2html234"
HREF="developers.html#SECTION000113000000000000000">Storage Daemon File Output Format</A>
<LI><A NAME="tex2html235"
HREF="developers.html#SECTION000114000000000000000">Overall Format</A>
<LI><A NAME="tex2html236"
HREF="developers.html#SECTION000115000000000000000">Serialization</A>
<LI><A NAME="tex2html237"
HREF="developers.html#SECTION000116000000000000000">Block Header</A>
<LI><A NAME="tex2html238"
HREF="developers.html#SECTION000117000000000000000">Record Header</A>
<LI><A NAME="tex2html239"
HREF="developers.html#SECTION000118000000000000000">Version BB02 Block Header</A>
<LI><A NAME="tex2html240"
HREF="developers.html#SECTION000119000000000000000">Version 2 Record Header</A>
<LI><A NAME="tex2html241"
HREF="developers.html#SECTION0001110000000000000000">Volume Label Format</A>
<LI><A NAME="tex2html242"
HREF="developers.html#SECTION0001111000000000000000">Session Label</A>
<LI><A NAME="tex2html243"
HREF="developers.html#SECTION0001112000000000000000">Overall Storage Format</A>
<LI><A NAME="tex2html244"
HREF="developers.html#SECTION0001113000000000000000">Unix File Attributes</A>
<LI><A NAME="tex2html245"
HREF="developers.html#SECTION0001114000000000000000">Old Depreciated Tape Format</A>
</UL>
<BR>
<LI><A NAME="tex2html246"
HREF="developers.html#SECTION000120000000000000000">Bacula Porting Notes</A>
<UL>
<LI><A NAME="tex2html247"
HREF="developers.html#SECTION000121000000000000000">Porting Requirements</A>
<LI><A NAME="tex2html248"
HREF="developers.html#SECTION000122000000000000000">Steps to Take for Porting</A>
</UL>
<BR>
<LI><A NAME="tex2html249"
HREF="developers.html#SECTION000130000000000000000">Implementing a Bacula GUI Interface</A>
<UL>
<LI><A NAME="tex2html250"
HREF="developers.html#SECTION000131000000000000000">General</A>
</UL>
<BR>
<LI><A NAME="tex2html251"
HREF="developers.html#SECTION000140000000000000000">TLS</A>
<UL>
<LI><A NAME="tex2html252"
HREF="developers.html#SECTION000141000000000000000">Introduction to TLS</A>
<LI><A NAME="tex2html253"
HREF="developers.html#SECTION000142000000000000000">New Configuration Directives</A>
<LI><A NAME="tex2html254"
HREF="developers.html#SECTION000143000000000000000">TLS API Implementation</A>
<LI><A NAME="tex2html255"
HREF="developers.html#SECTION000144000000000000000">Bnet API Changes</A>
<LI><A NAME="tex2html256"
HREF="developers.html#SECTION000145000000000000000">Authentication Negotiation</A>
</UL>
<BR>
<LI><A NAME="tex2html257"
HREF="developers.html#SECTION000150000000000000000">Bacula Regression Testing</A>
<UL>
<LI><A NAME="tex2html258"
HREF="developers.html#SECTION000151000000000000000">General</A>
<LI><A NAME="tex2html259"
HREF="developers.html#SECTION000152000000000000000">Running the Regression Script</A>
<LI><A NAME="tex2html260"
HREF="developers.html#SECTION000153000000000000000">Running a Single Test</A>
<LI><A NAME="tex2html261"
HREF="developers.html#SECTION000154000000000000000">Writing a Regression Test</A>
</UL>
<BR>
<LI><A NAME="tex2html262"
HREF="developers.html#SECTION000160000000000000000">Bacula MD5 Algorithm</A>
<UL>
<LI><A NAME="tex2html263"
HREF="developers.html#SECTION000161000000000000000">Command Line Message Digest Utility </A>
<LI><A NAME="tex2html264"
HREF="developers.html#SECTION000162000000000000000">Download md5.zip
(Zipped
archive)</A>
</UL>
<BR>
<LI><A NAME="tex2html265"
HREF="developers.html#SECTION000170000000000000000">Bacula Memory Management</A>
<UL>
<LI><A NAME="tex2html266"
HREF="developers.html#SECTION000171000000000000000">General</A>
</UL>
<BR>
<LI><A NAME="tex2html267"
HREF="developers.html#SECTION000180000000000000000">TCP/IP Network Protocol</A>
<UL>
<LI><A NAME="tex2html268"
HREF="developers.html#SECTION000181000000000000000">General</A>
<LI><A NAME="tex2html269"
HREF="developers.html#SECTION000182000000000000000">bnet and Threads</A>
<LI><A NAME="tex2html270"
HREF="developers.html#SECTION000183000000000000000">bnet_open</A>
<LI><A NAME="tex2html271"
HREF="developers.html#SECTION000184000000000000000">bnet_send</A>
<LI><A NAME="tex2html272"
HREF="developers.html#SECTION000185000000000000000">bnet_fsend</A>
<LI><A NAME="tex2html273"
HREF="developers.html#SECTION000186000000000000000">Additional Error information</A>
<LI><A NAME="tex2html274"
HREF="developers.html#SECTION000187000000000000000">bnet_recv</A>
<LI><A NAME="tex2html275"
HREF="developers.html#SECTION000188000000000000000">bnet_sig</A>
<LI><A NAME="tex2html276"
HREF="developers.html#SECTION000189000000000000000">bnet_strerror</A>
<LI><A NAME="tex2html277"
HREF="developers.html#SECTION0001810000000000000000">bnet_close</A>
<LI><A NAME="tex2html278"
HREF="developers.html#SECTION0001811000000000000000">Becoming a Server</A>
<LI><A NAME="tex2html279"
HREF="developers.html#SECTION0001812000000000000000">Higher Level Conventions</A>
</UL>
<BR>
<LI><A NAME="tex2html280"
HREF="developers.html#SECTION000190000000000000000">Smart Memory Allocation With Orphaned Buffer Detection </A>
<UL>
<LI><A NAME="tex2html281"
HREF="developers.html#SECTION000191000000000000000">Download smartall.zip
(Zipped archive)</A>
</UL>
<BR>
<LI><A NAME="tex2html282"
HREF="developers.html#SECTION000200000000000000000">GNU Free Documentation License</A>
<LI><A NAME="tex2html283"
HREF="developers.html#SECTION000210000000000000000">General Index</A>
<LI><A NAME="tex2html284"
HREF="developers.html#SECTION000220000000000000000">About this document ...</A>
</UL>
<!--End of Table of Contents-->
<BR>
<H2><A NAME="SECTION00020000000000000000">
List of Figures</A>
</H2><OL>
<LI><A HREF="developers.html#5309">Smart Memory Allocation with Orphaned BufferDetection</A>
</OL>
<BR>
<H2><A NAME="SECTION00030000000000000000">
List of Tables</A>
</H2><OL>
<LI><A HREF="developers.html#412">Message Error Code Classes</A>
<LI><A HREF="developers.html#1873">Filename Table Layout</A>
<LI><A HREF="developers.html#1898">Path Table Layout</A>
<LI><A HREF="developers.html#1923">File Table Layout</A>
<LI><A HREF="developers.html#1967">Job Table Layout</A>
<LI><A HREF="developers.html#2052">Job Types</A>
<LI><A HREF="developers.html#2076">Job Statuses</A>
<LI><A HREF="developers.html#2126">File Sets Table Layout</A>
<LI><A HREF="developers.html#2157">JobMedia Table Layout</A>
<LI><A HREF="developers.html#2206">Media Table Layout</A>
<LI><A HREF="developers.html#2294">Pool Table Layout</A>
<LI><A HREF="developers.html#2361">Client Table Layout</A>
<LI><A HREF="developers.html#2398">Unsaved Files Table Layout</A>
<LI><A HREF="developers.html#2429">Counter Table Layout</A>
<LI><A HREF="developers.html#2463">Version Table Layout</A>
<LI><A HREF="developers.html#2485">Base Files Table Layout</A>
<LI><A HREF="developers.html#3040">File Attributes</A>
</OL>
<P>
<P>
<H1><A NAME="SECTION00040000000000000000"></A>
<A NAME="_ChapterStart10"></A><A NAME="63"></A>
<A NAME="64"></A>
<BR>
Bacula Developer Notes
</H1>
<A NAME="67"></A>
<P>
<H2><A NAME="SECTION00041000000000000000"></A>
<A NAME="69"></A>
<BR>
General
</H2>
<A NAME="72"></A>
<P>
This document is intended mostly for developers and describes the the general
framework of making Bacula source changes.
<P>
<H3><A NAME="SECTION00041100000000000000"></A>
<A NAME="74"></A>
<BR>
Contributions
</H3>
<A NAME="77"></A>
<P>
Contributions from programmers are broken into two groups. The first are
contributions that are aids and not essential to Bacula. In general, these
will be scripts or will go into and examples or contributions directory.
For these kinds of non-essential contributions there is no obligation to do
a copyright assignment as described below. However, a copyright assignment
would still be appreciated.
<P>
The second class of contributions are those which will be integrated with
Bacula and become an essential part. Within this class of contributions, there
are two hurdles to surmount. One is getting your patch accepted, and two is
dealing with copyright issues. The following text describes some of the
requirements for such code.
<P>
<H3><A NAME="SECTION00041200000000000000"></A>
<A NAME="79"></A>
<BR>
Patches
</H3>
<A NAME="82"></A>
<P>
Subject to the copyright assignment described below, your patches should be
sent in <B>diff -u</B> format relative to the current contents of the Source
Forge SVN, which is the easiest to understand and integrate.
Please be sure to use the Bacula indenting standard (see below).
If you have checked out the source with SVN, you can get a diff using:
<P>
<PRE>
svn update
svn diff > change.patch
</PRE>
<P>
If you plan on doing significant development work over a period of time,
after having your first patch reviewed and approved, you will be eligible
for having developer SVN access so that you can commit your changes
directly to the SVN repository. To do so, you will need a userid on Source
Forge.
<P>
<H3><A NAME="SECTION00041300000000000000"></A>
<A NAME="87"></A>
<BR>
Copyrights
</H3>
<A NAME="90"></A>
<P>
To avoid future problems concerning changing licensing or
copyrights, all code contributions more than a hand full of lines
must be in the Public Domain or have the copyright transferred to
the Free Software Foundation Europe e.V. with a Fiduciary License
Agreement (FLA) as in the current code. Note, prior to
November 2004, the code was copyrighted by Kern Sibbald and John
Walker. After November 2004, the code was copyrighted by Kern
Sibbald, then on the 15th of November 2006, the copyright was
transferred to the Free Software Foundation Europe e.V.
<P>
Your name should be clearly indicated as the author of the code, and you
must be extremely careful not to violate any copyrights or use other
people's code without acknowledging it. The purpose of this requirement is
to avoid future copyright, patent, or intellectual property problems.
Please read the LICENSE agreement in the main source code
directory. When you sign the Fiduciary License Agreement (FLA)
and send it in, you are argeeing to the terms of that LICENSE
file.
<P>
To understand the possible source of future problems, please
examine the difficulties Mozilla is (was?) having finding
previous contributors at <A NAME="tex2html1"
HREF="http://www.mozilla.org/MPL/missing.html">http://www.mozilla.org/MPL/missing.html</A>. The other important issue is to
avoid copyright, patent, or intellectual property violations as are currently
(May 2003) being claimed by SCO against IBM.
<P>
Although the copyright will be held by the Free Software
Foundation Europe e.V., each developer is expected to indicate
that he wrote and/or modified a particular module (or file) and
any other sources. The copyright assignment may seem a bit
unusual, but in reality, it is not. Most large projects require
this.
<P>
If you have any doubts about this, please don't hesitate to ask. The
objective is to assure the long term servival of the Bacula project.
<P>
Items not needing a copyright assignment are: most small changes,
enhancements, or bug fixes of 5-10 lines of code, which amount to
less than 20
<H3><A NAME="SECTION00041400000000000000"></A>
<A NAME="94"></A>
<A NAME="95"></A>
<BR>
Copyright Assignment -- Fiduciary License Agreement
</H3>
<A NAME="98"></A>
<P>
Since this is not a commercial enterprise, and we prefer to believe in
everyone's good faith, previously developers could assign the copyright by
explicitly acknowledging that they do so in their first submission. This
was sufficient if the developer is independent, or an employee of a
not-for-profit organization or a university. However, in an effort to
ensure that the Bacula code is really clean, beginning in August 2006, all
previous and future developers with SVN access will be asked to submit a
copyright assignment (or Fiduciary License Agreement -- FLA),
which means you agree to the LICENSE in the main source
directory. It also means that you receive back the right to use
the code that you have submitted.
<P>
Any developer who wants to contribute and is employed by a company should
either list the employer as the owner of the code, or get
explicit permission from him to sign the copyright assignment.
This is because in many
countries, all work that an employee does whether on company time or in the
employee's free time is considered to be Intellectual Property of the
company. Obtaining official approval or an FLA from the company will avoid
misunderstandings between the employee, the company, and the Bacula
project. A good number of companies have already followed this procedure.
<P>
The Fiduciary License Agreement is posted on the Bacula web site at:
<A NAME="tex2html2"
HREF="http://www.bacula.org/FLA-bacula.en.pdf">http://www.bacula.org/FLA-bacula.en.pdf</A>
<P>
The instructions for filling out this agreement are also at:
<A NAME="tex2html3"
HREF="http://www.bacula.org/?page=fsfe">http://www.bacula.org/?page=fsfe</A>
<P>
It should be filled out, then sent to:
<P>
<PRE>
Free Software Foundation Europe
Freedom Task Force
Sumatrastrasse 25
8006 Zürich
Switzerland
</PRE>
<P>
Please note that the above address is different from the officially
registered office mentioned in the document. When you send in such a
complete document, please notify me: kern at sibbald dot com.
<P>
<H2><A NAME="SECTION00042000000000000000"></A>
<A NAME="106"></A>
<A NAME="107"></A>
<BR>
The Development Cycle
</H2>
<A NAME="110"></A>
<P>
As I noted in the 1.38 ReleaseNotes, version 1.38 was different from prior
versions because it had a lot more contributions. I expect that this trend
will continue. As a consequence, I am going to modify how I normally do
development, and instead of making a list of all the features that I will
implement in the next version, I will personally sign up for one (maybe
two) projects at a time, and when they are complete, I will release a new
version.
<P>
The difference is that I will have more time to review the new code that is
being contributed, and will be able to devote more time to a smaller number
of projects (1.38 had too many new features for me to handle correctly).
<P>
I expect that future release schedules will be much the same, and the
number of new features will also be much the same providing that the
contributions continue to come -- and they show no signs of let up :-)
<P>
<A NAME="111"></A>
<B>Feature Requests:</B>
<BR>
In addition, I would like to "formalize" the feature requests a bit.
<P>
Instead of me maintaining an informal list of everything I run into
(kernstodo), I would like to maintain a "formal" list of projects. This
means that all new feature requests, including those recently discussed on
the email lists, must be formally submitted and approved.
<P>
Formal submission of feature requests will take two forms:
<BR>
1. non-mandatory, but highly recommended is to discuss proposed new features
on the mailing list.
<BR>
2. Formal submission of an Feature Request in a special format.
I'll give an example of this below, but you can also find it on the web
site under "Support -> Feature Requests". Since it takes a bit of time to
properly fill out a Feature Request form, you probably should check on the email list
first.
<P>
Once the Feature Request is received by the keeper of the projects list, it
will be sent to me, and I will either accept it, send it back
asking for clarification, send it to the email list asking for opinions, or
reject it.
<P>
If it is accepted, it will go in the "projects" file (a simple ASCII file)
maintained in the main Bacula source directory.
<P>
<B>Implementation of Feature Requests:</B>
<BR>
Any qualified developer can sign up for a project. The project must have
an entry in the projects file, and the developer's name will appear in the
Status field.
<P>
<B>How Feature Requests are accepted:</B>
<BR>
Acceptance of Feature Requests depends on several things:
<BR>
1. feedback from users. If it is negative, the Feature Request will probably not be
accepted.
<BR>
2. the difficulty of the project. A project that is so
difficult that I cannot imagine finding someone to implement probably won't
be accepted.
<BR>
3. whether or not the Feature Request fits within the
current stategy of Bacula (for example an Feature Request that requests changing the
tape to tar format would not be accepted, ...)
<P>
<B>How Feature Requests are prioritized:</B>
<BR>
Once an Feature Request is accepted, it needs to be implemented. If you
can find a developer for it, or one signs up for implementing it, then the
Feature Request becomes top priority (at least for that developer).
<P>
Between releases of Bacula, we will generally solicit Feature Request input
for the next version, and by way of this email, we suggest that you send
discuss and send in your Feature Requests for the next release. Please
verify that the Feature Request is not in the current list (attached to this email).
<P>
Once users have had several weeks to submit Feature Requests, the keeper of the
projects list will
organize them, and request users to vote on them. This will allow fixing
prioritizing the Feature Requests. Having a priority is one thing, but
getting it implement is another thing -- we are hoping that the Bacula
community will take more responsibility for assuring the implementation of
accepted Feature Requests.
<P>
Feature Request format:
<PRE>
============= Empty Feature Request form ===========
Item n: One line summary ...
Date: Date submitted
Origin: Name and email of originator.
Status:
What: More detailed explanation ...
Why: Why it is important ...
Notes: Additional notes or features (omit if not used)
============== End Feature Request form ==============
</PRE>
<P>
<PRE>
============= Example Completed Feature Request form ===========
Item 1: Implement a Migration job type that will move the job
data from one device to another.
Origin: Sponsored by Riege Sofware International GmbH. Contact:
Daniel Holtkamp <holtkamp at riege dot com>
Date: 28 October 2005
Status: Partially coded in 1.37 -- much more to do. Assigned to
Kern.
What: The ability to copy, move, or archive data that is on a
device to another device is very important.
Why: An ISP might want to backup to disk, but after 30 days
migrate the data to tape backup and delete it from
disk. Bacula should be able to handle this
automatically. It needs to know what was put where,
and when, and what to migrate -- it is a bit like
retention periods. Doing so would allow space to be
freed up for current backups while maintaining older
data on tape drives.
Notes: Migration could be triggered by:
Number of Jobs
Number of Volumes
Age of Jobs
Highwater size (keep total size)
Lowwater mark
=================================================
</PRE>
<P>
<H2><A NAME="SECTION00043000000000000000"></A>
<A NAME="122"></A>
<BR>
Bacula Code Submissions and Projects
</H2>
<A NAME="125"></A>
<P>
Getting code implemented in Bacula works roughly as follows:
<P>
<UL>
<LI>Kern is the project manager, but prefers not to be a "gate keeper".
This means that the developers are expected to be self-motivated,
and once they have experience submit directly to the SVN. However,
it is a good idea to have your patches reviewed prior to submitting,
and it is a bad idea to submit monster patches because no one will
be able to properly review them. See below for more details on this.
<P>
</LI>
<LI>There are growing numbers of contributions (very good).
<P>
</LI>
<LI>Some contributions come in the form of relatively small patches,
which Kern reviews, integrates, documents, tests, and maintains.
<P>
</LI>
<LI>All Bacula developers take full
responsibility for writing the code, posting as patches so that I can
review it as time permits, integrating it at an appropriate time,
responding to my requests for tweaking it (name changes, ...),
document it in the code, document it in the manual (even though
their mother tongue is not English), test it, develop and commit
regression scripts, and answer in a timely fashion all bug reports --
even occassionally accepting additional bugs :-)
<P>
This is a sustainable way of going forward with Bacula, and the
direction that the project will be taking more and more. For
example, in the past, we have had some very dedicated programmers
who did major projects. However, these
programmers due to outside obligations (job responsibilities change of
job, school duties, ...) could not continue to maintain the code. In
those cases, the code suffers from lack of maintenance, sometimes I
patch it, sometimes not. In the end, the code gets dropped from the
project (there are two such contributions that are heading in that
direction). When ever possible, we would like to avoid this, and
ensure a continuation of the code and a sharing of the development,
debugging, documentation, and maintenance responsibilities.
</LI>
</UL>
<P>
<H2><A NAME="SECTION00044000000000000000"></A>
<A NAME="129"></A>
<BR>
Patches for Released Versions
</H2>
<A NAME="132"></A>
If you fix a bug in a released version, you should, unless it is
an absolutely trivial bug, create and release a patch file for the
bug. The procedure is as follows:
<P>
Fix the bug in the branch and in the trunk.
<P>
Make a patch file for the branch and add the branch patch to
the patches directory in both the branch and the trunk.
The name should be 2.2.4-xxx.patch where xxx is unique, in this case it can
be "restore", e.g. 2.2.4-restore.patch. Add to the top of the
file a brief description and instructions for applying it -- see for example
2.2.4-poll-mount.patch. The best way to create the patch file is as
follows:
<P>
<PRE>
(edit) 2.2.4-restore.patch
(input description)
(end edit)
svn diff >>2.2.4-restore.patch
</PRE>
<P>
check to make sure no extra junk got put into the patch file (i.e.
it should have the patch for that bug only).
<P>
If there is not a bug report on the problem, create one, then add the
patch to the bug report.
<P>
Uthen upload it to the 2.2.x release of bacula-patches.
<P>
So, end the end, the patch file is:
<UL>
<LI>Attached to the bug report
<P>
</LI>
<LI>In Branch-2.2/bacula/patches/...
<P>
</LI>
<LI>In the trunk
<P>
</LI>
<LI>Loaded on Source Forge bacula-patches 2.2.x release. When
you add it, click on the check box to send an Email so that all the
users that are monitoring SF patches get notified.
</LI>
</UL>
<P>
<H2><A NAME="SECTION00045000000000000000"></A>
<A NAME="138"></A>
<BR>
SVN Usage
</H2>
<A NAME="141"></A>
<P>
Please note that if you are familar with CVS, SVN is very
similar (and better), but there can be a few surprising
differences.
<P>
The *entire* Bacula SourceForge.net Subversion repository can be
checked out through SVN with the following command:
<P>
<PRE>
svn checkout https://bacula.svn.sourceforge.net/svnroot/bacula bacula
</PRE>
<P>
With the above command, you will get everything, which is a very large
amount of data:
<P>
<PRE>
branches/
Branch-1.32a/
...
Branch-2.0/
import/
vendor/
tags/
Release-1.1/
...
Release-2.0.2/
trunk/
bacula/
docs/
gui/
regress/
rescue/
</PRE>
<P>
Note, you should NEVER commit code to any checkout that you have
done of a tag. All tags (e.g. Release-1.1, ... Release-2.0.2)
should be considered read-only.
<P>
You may commit code to the most recent item in
branches (in the above the most recent one is Branch-2.0). If
you want to commit code to an older branch, then please contact
Kern first.
<P>
You may create your own tags and/or branches, but they should
have a name clearly distinctive from Branch-, Release-, or Beta-,
which are official names used by the project. If you create a
tag, then you should NEVER commit code to it, for the same
reason noted above -- it should serve as a marker for something
you released. If you create a branch, then you are free to
commit to it as you wish.
<P>
You may, of course, commit to the trunk.
<P>
In summary:
<P>
<PRE>
branches
Branch-nnn
tags
Release-nnn
Beta-nnn
</PRE>
<P>
are reserved names to be created only by the project manager (or
with his OK), where the nnn is any sequence of numbers and
periods (e.g. 2.0, 2.0.1, ...).
<P>
In addition all tags even those that you create are read-only
forever. Typically tags represent release points either in the
trunc or in a branch.
<P>
Coming back to getting source code.
If you only want the current Bacula source code, you could use:
<P>
<PRE>
svn checkout https://bacula.svn.sourceforge.net/svnroot/bacula/trunk/bacula bacula
</PRE>
<P>
To view what is in the SVN, point your browser at the following URL:
http://bacula.svn.sourceforge.net/viewvc/bacula/
<P>
Many of the Subversion (svn) commands are almost identical to those that
you have used for cvs, but some (such as a checkout) can have surprising
results, so you should take a careful look at the documentation.
<P>
Robert has kindly provided the following documentation on the new
svn repository and how to use it:
<P>
Here is the list of branches:
<PRE>
Branch-1.32a
Branch-1.32e
Branch-1.34.2
Branch-1.34.5
Branch-1.36
Branch-1.36.1
Branch-1.36.2
Branch-1.38
Branch-2.0
import
vendor
</PRE>
<P>
The list of tags is:
<PRE>
Release-1.1 Release-1.19 Release-1.19a Release-1.19b
Release-1.20 Release-1.21 Release-1.22 Release-1.23
Release-1.23a Release-1.24 Release-1.25 Release-1.25a
Release-1.26 Release-1.27 Release-1.27a Release-1.27b
Release-1.27c Release-1.28 Release-1.29 Release-1.30
Release-1.31 Release-1.31a Release-1.32 Release-1.32a
Release-1.32b Release-1.32c Release-1.32d Release-1.32e
Release-1.32f Release-1.32f-2 Release-1.32f-3 Release-1.32f-4
Release-1.32f-5 Release-1.34.0 Release-1.34.1 Release-1.34.3
Release-1.34.4 Release-1.34.5 Release-1.34.6 Release-1.35.1
Release-1.35.2 Release-1.35.3 Release-1.35.6 Release-1.35.7
Release-1.35.8 Release-1.36.0 Release-1.36.1 Release-1.36.2
Release-1.36.3 Release-1.38.0 Release-1.38.1 Release-1.38.10
Release-1.38.11 Release-1.38.2 Release-1.38.3 Release-1.38.4
Release-1.38.5 Release-1.38.6 Release-1.38.7 Release-1.38.8
Release-1.38.9 Release-1.8.1 Release-1.8.2 Release-1.8.3
Release-1.8.4 Release-1.8.5 Release-1.8.6 Release-2.0.0
Release-2.0.1 Release-2.0.2
</PRE>
<P>
Here is a list of commands to get you started. The recommended book is
"Version Control with Subversion", by Ben Collins-Sussmann,
Brian W. Fitzpatrick, and Michael Pilato, O'Reilly. The book is
Open Source, so it is also available on line at:
<P>
<PRE>
http://svnbook.red-bean.com
</PRE>
<P>
Get a list of commands
<P>
<PRE>
svn help
</PRE>
<P>
Get a help with a command
<P>
<PRE>
svn help command
</PRE>
<P>
Checkout the HEAD revision of all modules from the project into the
directory bacula-new
<P>
<PRE>
svn co https://bacula.svn.sourceforge.net/svnroot/bacula/trunk bacula.new
</PRE>
<P>
Checkout the HEAD revision of the bacula module into the bacula subdirectory
<P>
<PRE>
svn checkout https://bacula.svn.sourceforge.net/svnroot/bacula/trunk/bacula
</PRE>
<P>
See which files have changed in the working copy
<P>
<PRE>
svn status
</PRE>
<P>
See which files are out of date
<P>
<PRE>
svn status -u
</PRE>
<P>
Add a new file file.c
<P>
<PRE>
svn add file.c
</PRE>
<P>
Create a new directory
<P>
<PRE>
svn mkdir newdir
</PRE>
<P>
Delete an obsolete file
<P>
<PRE>
svn delete file.c
</PRE>
<P>
Rename a file
<P>
<PRE>
svn move file.c newfile.c
</PRE>
<P>
Move a file to a new location
<P>
<PRE>
svn move file.c ../newdir/file.c
</PRE>
<P>
Copy a file retaining the original history in the new file
<P>
<PRE>
svn copy file.c newfile.c
</PRE>
<P>
Update the working copy with the outstanding changes
<P>
<PRE>
svn update
</PRE>
<P>
Compare working copy with the repository
<P>
<PRE>
svn diff file.c
</PRE>
<P>
Commit the changes in the local working copy
<P>
<PRE>
svn commit
</PRE>
<P>
Specify which files are ignored in the current directory
<P>
<PRE>
svn propedit svn:ignore .
</PRE>
<P>
Mark a file to be executable
<P>
<PRE>
svn propset svn:executable '*' prog.sh
</PRE>
<P>
Unmark a file as executable
<P>
<PRE>
svn propdel svn:executable prog.sh
</PRE>
<P>
List a file's properties
<P>
<PRE>
svn proplist file.c
</PRE>
<P>
Create a branch for a new version
<P>
<PRE>
svn copy https://bacula.svn.sourceforge.net/svnroot/bacula/trunk \
https://bacula.svn.sourceforge.net/svnroot/bacula/branches/Branch-2.1
</PRE>
<P>
Tag a version for a new release
<P>
<PRE>
svn copy https://bacula.svn.sourceforge.net/svnroot/bacula/branches/Branch-2.1 \
https://bacula.svn.sourceforge.net/svnroot/bacula/branches/Release-2.1
</PRE>
<P>
Let's say you are working in the directory scripts. You would then do:
<P>
<PRE>
cd scripts
(edit some files)
</PRE>
<P>
when you are happy with your changes, you can do the following:
<P>
<PRE>
cd bacula (to your top level directory)
svn diff my-changes.patch
</PRE>
<P>
When the command is done, you can look in the file my-changes.patch
and you will see all the changes you have made to your copy of the
repository. Make sure that you understand all the changes that
it reports before proceeding. If you modified files that you do
do not want to commit to the main repository, you can simply delete
them from your local directory, and they will be restored from the
repository with the "svn update" that is shown below. Normally, you
should not find changes to files that you do not want to commit, and
if you find yourself in that position a lot, you are probably doing
something wrong.
<P>
Let's assume that now you want to commit your changes to the main
SVN repository.
<P>
First do:
<P>
<PRE>
cd bacula
svn update
</PRE>
<P>
When you do this, it will pull any changes made by other developers into
your local copy of the repository, and it will check for conflicts. If there
are any, it will tell you, and you will need to resolve them. The problems
of resolving conflicts are a bit more than this document can cover, but
you can examine the files it claims have conflicts and look for <<<<
or look in the .rej files that it creates. If you have problems, just ask
on the developer's list.
<P>
Note, doing the above "svn update" is not absolutely necessary. There are
times when you may be working on code and you want to commit it, but you
explicitly do not want to move up to the latest version of the code in
the SVN. If that is the case, you can simply skip the "svn update" and
do the commit shown below. If the commit fails because of a conflict, it
will tell you, and you must resolve the conflict before it will permit
you to do the commit.
<P>
Once your local copy of the repository has been updated, you can now
commit your changes:
<P>
<PRE>
svn commit -m "Some comment about what you changed"
</PRE>
<P>
or if you really only want to commit a single file, you can
do:
<P>
<PRE>
svn commit -m "comment" scripts/file-I-edited
</PRE>
<P>
Note, if you have done a build in your directory, or you have added
other new files, the commit will update only the files that are
actually in the repository. For example, none of the object files
are stored in the repository, so when you do a commit, those object
files will simply be ignored.
<P>
If you want to add new files or remove files from the main SVN
repository, and you are not experienced with SVN, please ask Kern
to do it. If you follow the simple steps above, it is unlikely that
you will do any damage to the repository, and if you do, it is always
possible for us to recover, but it can be painful.
<P>
If you are only working in one subdirectory of say the bacula project,
for example, the scripts directory, you can do your commit from
that subdirectory, and only the changes in that directory and all its
subdirectories will be committed. This can be helpful for translators.
If you are doing a French translation, you will be working in
docs/manual-fr, and if you are always cd'ed into that directory when
doing your commits, your commit will effect only that directory. As
long as you are careful only to change files that you want changed,
you have little to worry about.
<P>
<H2><A NAME="SECTION00046000000000000000"></A>
<A NAME="213"></A>
<BR>
Subversion Resources
</H2>
<A NAME="216"></A>
<P>
<PRE>
cvs2svn Statistics:
------------------
Total CVS Files: 3286
Total CVS Revisions: 28924
Total Unique Tags: 63
Total Unique Branches: 11
CVS Repos Size in KB: 232421
Total SVN Commits: 4116
First Revision Date: Tue Apr 23 12:42:57 2002
Last Revision Date: Tue Feb 6 06:37:57 2007
</PRE>
<P>
The new Subversion repository size on Robert's machine:
<P>
<PRE>
4.0K bacula-tst/dav
12K bacula-tst/locks
40K bacula-tst/hooks
16K bacula-tst/conf
190M bacula-tst/db/revs
17M bacula-tst/db/revprops
4.0K bacula-tst/db/transactions
206M bacula-tst/db
206M bacula-tst
</PRE>
<P>
Main Subversion Web Page
<A NAME="tex2html4"
HREF="http://subversion.tigris.org">http://subversion.tigris.org</A>
<P>
Subversion Book
<A NAME="tex2html5"
HREF="http://svnbook.red-bean.com">http://svnbook.red-bean.com</A>
<P>
Subversion Clients
<A NAME="tex2html6"
HREF="http://subversion.tigris.org/project_packages.html">http://subversion.tigris.org/project_packages.html</A>
<P>
(For Windows users the TortoiseSVN package is awesome)
<P>
GUI UNIX client link
<A NAME="tex2html7"
HREF="http://rapidsvn.tigris.org/">http://rapidsvn.tigris.org/</A>
<P>
A nice KDE GUI client:
kdesvn
<P>
<H2><A NAME="SECTION00047000000000000000"></A>
<A NAME="230"></A>
<A NAME="231"></A>
<BR>
Developing Bacula
</H2>
<A NAME="234"></A>
<P>
Typically the simplest way to develop Bacula is to open one xterm window
pointing to the source directory you wish to update; a second xterm window at
the top source directory level, and a third xterm window at the bacula
directory <top>/src/bacula. After making source changes in one of the
directories, in the top source directory xterm, build the source, and start
the daemons by entering:
<P>
make and
<P>
./startit then in the enter:
<P>
./console or
<P>
./gnome-console to start the Console program. Enter any commands for testing.
For example: run kernsverify full.
<P>
Note, the instructions here to use <B>./startit</B> are different from using a
production system where the administrator starts Bacula by entering <B>./bacula start</B>. This difference allows a development version of <B>Bacula</B>
to be run on a computer at the same time that a production system is running.
The <B>./startit</B> strip starts <B>Bacula</B> using a different set of
configuration files, and thus permits avoiding conflicts with any production
system.
<P>
To make additional source changes, exit from the Console program, and in the
top source directory, stop the daemons by entering:
<P>
./stopit then repeat the process.
<P>
<H3><A NAME="SECTION00047100000000000000"></A>
<A NAME="243"></A>
<BR>
Debugging
</H3>
<A NAME="246"></A>
<P>
Probably the first thing to do is to turn on debug output.
<P>
A good place to start is with a debug level of 20 as in <B>./startit -d20</B>.
The startit command starts all the daemons with the same debug level.
Alternatively, you can start the appropriate daemon with the debug level you
want. If you really need more info, a debug level of 60 is not bad, and for
just about everything a level of 200.
<P>
<H3><A NAME="SECTION00047200000000000000"></A>
<A NAME="249"></A>
<A NAME="250"></A>
<BR>
Using a Debugger
</H3>
<A NAME="253"></A>
<P>
If you have a serious problem such as a segmentation fault, it can usually be
found quickly using a good multiple thread debugger such as <B>gdb</B>. For
example, suppose you get a segmentation violation in <B>bacula-dir</B>. You
might use the following to find the problem:
<P>
<start the Storage and File daemons>
cd dird
gdb ./bacula-dir
run -f -s -c ./dird.conf
<it dies with a segmentation fault>
where
The <B>-f</B> option is specified on the <B>run</B> command to inhibit <B>dird</B> from going into the background. You may also want to add the <B>-s</B>
option to the run command to disable signals which can potentially interfere
with the debugging.
<P>
As an alternative to using the debugger, each <B>Bacula</B> daemon has a built
in back trace feature when a serious error is encountered. It calls the
debugger on itself, produces a back trace, and emails the report to the
developer. For more details on this, please see the chapter in the main Bacula
manual entitled ``What To Do When Bacula Crashes (Kaboom)''.
<P>
<H3><A NAME="SECTION00047300000000000000"></A>
<A NAME="266"></A>
<A NAME="267"></A>
<BR>
Memory Leaks
</H3>
<A NAME="270"></A>
<P>
Because Bacula runs routinely and unattended on client and server machines, it
may run for a long time. As a consequence, from the very beginning, Bacula
uses SmartAlloc to ensure that there are no memory leaks. To make detection of
memory leaks effective, all Bacula code that dynamically allocates memory MUST
have a way to release it. In general when the memory is no longer needed, it
should be immediately released, but in some cases, the memory will be held
during the entire time that Bacula is executing. In that case, there MUST be a
routine that can be called at termination time that releases the memory. In
this way, we will be able to detect memory leaks. Be sure to immediately
correct any and all memory leaks that are printed at the termination of the
daemons.
<P>
<H3><A NAME="SECTION00047400000000000000"></A>
<A NAME="272"></A>
<A NAME="273"></A>
<BR>
Special Files
</H3>
<A NAME="276"></A>
<P>
Kern uses files named 1, 2, ... 9 with any extension as scratch files. Thus
any files with these names are subject to being rudely deleted at any time.
<P>
<H3><A NAME="SECTION00047500000000000000"></A>
<A NAME="278"></A>
<A NAME="279"></A>
<BR>
When Implementing Incomplete Code
</H3>
<A NAME="282"></A>
<P>
Please identify all incomplete code with a comment that contains
<P>
<PRE>
***FIXME***
</PRE>
<P>
where there are three asterisks (*) before and after the word
FIXME (in capitals) and no intervening spaces. This is important as it allows
new programmers to easily recognize where things are partially implemented.
<P>
<H3><A NAME="SECTION00047600000000000000"></A>
<A NAME="286"></A>
<A NAME="287"></A>
<BR>
Bacula Source File Structure
</H3>
<A NAME="290"></A>
<P>
The distribution generally comes as a tar file of the form <B>bacula.x.y.z.tar.gz</B> where x, y, and z are the version, release, and update
numbers respectively.
<P>
Once you detar this file, you will have a directory structure as follows:
<P>
<PRE>
|
Tar file:
|- depkgs
|- mtx (autochanger control program + tape drive info)
|- sqlite (SQLite database program)
Tar file:
|- depkgs-win32
|- pthreads (Native win32 pthreads library -- dll)
|- zlib (Native win32 zlib library)
|- wx (wxWidgets source code)
Project bacula:
|- bacula (main source directory containing configuration
| and installation files)
|- autoconf (automatic configuration files, not normally used
| by users)
|- intl (programs used to translate)
|- platforms (OS specific installation files)
|- redhat (Red Hat installation)
|- solaris (Sun installation)
|- freebsd (FreeBSD installation)
|- irix (Irix installation -- not tested)
|- unknown (Default if system not identified)
|- po (translations of source strings)
|- src (source directory; contains global header files)
|- cats (SQL catalog database interface directory)
|- console (bacula user agent directory)
|- dird (Director daemon)
|- filed (Unix File daemon)
|- win32 (Win32 files to make bacula-fd be a service)
|- findlib (Unix file find library for File daemon)
|- gnome-console (GNOME version of console program)
|- lib (General Bacula library)
|- stored (Storage daemon)
|- tconsole (Tcl/tk console program -- not yet working)
|- testprogs (test programs -- normally only in Kern's tree)
|- tools (Various tool programs)
|- win32 (Native Win32 File daemon)
|- baculafd (Visual Studio project file)
|- compat (compatibility interface library)
|- filed (links to src/filed)
|- findlib (links to src/findlib)
|- lib (links to src/lib)
|- console (beginning of native console program)
|- wx-console (wxWidget console Win32 specific parts)
|- wx-console (wxWidgets console main source program)
Project regress:
|- regress (Regression scripts)
|- bin (temporary directory to hold Bacula installed binaries)
|- build (temporary directory to hold Bacula source)
|- scripts (scripts and .conf files)
|- tests (test scripts)
|- tmp (temporary directory for temp files)
|- working (temporary working directory for Bacula daemons)
Project docs:
|- docs (documentation directory)
|- developers (Developer's guide)
|- home-page (Bacula's home page source)
|- manual (html document directory)
|- manual-fr (French translation)
|- manual-de (German translation)
|- techlogs (Technical development notes);
Project rescue:
|- rescue (Bacula rescue CDROM)
|- linux (Linux rescue CDROM)
|- cdrom (Linux rescue CDROM code)
...
|- solaris (Solaris rescue -- incomplete)
|- freebsd (FreeBSD rescue -- incomplete)
Project gui:
|- gui (Bacula GUI projects)
|- bacula-web (Bacula web php management code)
|- bimagemgr (Web application for burning CDROMs)
</PRE>
<P>
<H3><A NAME="SECTION00047700000000000000"></A>
<A NAME="295"></A>
<A NAME="296"></A>
<BR>
Header Files
</H3>
<A NAME="299"></A>
<P>
Please carefully follow the scheme defined below as it permits in general only
two header file includes per C file, and thus vastly simplifies programming.
With a large complex project like Bacula, it isn't always easy to ensure that
the right headers are invoked in the right order (there are a few kludges to
make this happen -- i.e. in a few include files because of the chicken and egg
problem, certain references to typedefs had to be replaced with <B>void</B> ).
<P>
Every file should include <B>bacula.h</B>. It pulls in just about everything,
with very few exceptions. If you have system dependent ifdefing, please do it
in <B>baconfig.h</B>. The version number and date are kept in <B>version.h</B>.
<P>
Each of the subdirectories (console, cats, dird, filed, findlib, lib, stored,
...) contains a single directory dependent include file generally the name of
the directory, which should be included just after the include of <B>bacula.h</B>. This file (for example, for the dird directory, it is <B>dird.h</B>)
contains either definitions of things generally needed in this directory, or
it includes the appropriate header files. It always includes <B>protos.h</B>.
See below.
<P>
Each subdirectory contains a header file named <B>protos.h</B>, which contains
the prototypes for subroutines exported by files in that directory. <B>protos.h</B> is always included by the main directory dependent include file.
<P>
<H3><A NAME="SECTION00047800000000000000"></A>
<A NAME="310"></A>
<A NAME="311"></A>
<BR>
Programming Standards
</H3>
<A NAME="314"></A>
<P>
For the most part, all code should be written in C unless there is a burning
reason to use C++, and then only the simplest C++ constructs will be used.
Note, Bacula is slowly evolving to use more and more C++.
<P>
Code should have some documentation -- not a lot, but enough so that I can
understand it. Look at the current code, and you will see that I document more
than most, but am definitely not a fanatic.
<P>
I prefer simple linear code where possible. Gotos are strongly discouraged
except for handling an error to either bail out or to retry some code, and
such use of gotos can vastly simplify the program.
<P>
Remember this is a C program that is migrating to a <B>tiny</B> subset of C++,
so be conservative in your use of C++ features.
<P>
<H3><A NAME="SECTION00047900000000000000"></A>
<A NAME="317"></A>
<A NAME="318"></A>
<BR>
Do Not Use
</H3>
<A NAME="321"></A>
<P>
<UL>
<LI>STL -- it is totally incomprehensible.
</LI>
</UL>
<P>
<H3><A NAME="SECTION000471000000000000000"></A>
<A NAME="325"></A>
<A NAME="326"></A>
<BR>
Avoid if Possible
</H3>
<A NAME="329"></A>
<P>
<UL>
<LI>Using <B>void *</B> because this generally means that one must
using casting, and in C++ casting is rather ugly. It is OK to use
void * to pass structure address where the structure is not known
to the routines accepting the packet (typically callback routines).
However, declaring "void *buf" is a bad idea. Please use the
correct types whenever possible.
<P>
</LI>
<LI>Using undefined storage specifications such as (short, int, long,
long long, size_t ...). The problem with all these is that the number of bytes
they allocate depends on the compiler and the system. Instead use
Bacula's types (int8_t, uint8_t, int32_t, uint32_t, int64_t, and
uint64_t). This guarantees that the variables are given exactly the
size you want. Please try at all possible to avoid using size_t ssize_t
and the such. They are very system dependent. However, some system
routines may need them, so their use is often unavoidable.
<P>
</LI>
<LI>Returning a malloc'ed buffer from a subroutine -- someone will forget
to release it.
<P>
</LI>
<LI>Heap allocation (malloc) unless needed -- it is expensive. Use
POOL_MEM instead.
<P>
</LI>
<LI>Templates -- they can create portability problems.
<P>
</LI>
<LI>Fancy or tricky C or C++ code, unless you give a good explanation of
why you used it.
<P>
</LI>
<LI>Too much inheritance -- it can complicate the code, and make reading it
difficult (unless you are in love with colons)
<P>
</LI>
</UL>
<P>
<H3><A NAME="SECTION000471100000000000000"></A>
<A NAME="334"></A>
<A NAME="335"></A>
<BR>
Do Use Whenever Possible
</H3>
<A NAME="338"></A>
<P>
<UL>
<LI>Locking and unlocking within a single subroutine.
<P>
</LI>
<LI>A single point of exit from all subroutines. A goto is
perfectly OK to use to get out early, but only to a label
named bail_out, and possibly an ok_out. See current code
examples.
<P>
</LI>
<LI>Malloc and free within a single subroutine.
<P>
</LI>
<LI>Comments and global explanations on what your code or algorithm does.
<P>
</LI>
</UL>
<P>
<H3><A NAME="SECTION000471200000000000000"></A>
<A NAME="342"></A>
<A NAME="343"></A>
<BR>
Indenting Standards
</H3>
<A NAME="346"></A>
<P>
I cannot stand code indented 8 columns at a time. This makes the code
unreadable. Even 4 at a time uses a lot of space, so I have adopted indenting
3 spaces at every level. Note, indention is the visual appearance of the
source on the page, while tabbing is replacing a series of up to 8 spaces from
a tab character.
<P>
The closest set of parameters for the Linux <B>indent</B> program that will
produce reasonably indented code are:
<P>
<PRE>
-nbad -bap -bbo -nbc -br -brs -c36 -cd36 -ncdb -ce -ci3 -cli0
-cp36 -d0 -di1 -ndj -nfc1 -nfca -hnl -i3 -ip0 -l85 -lp -npcs
-nprs -npsl -saf -sai -saw -nsob -nss -nbc -ncs -nbfda
</PRE>
<P>
You can put the above in your .indent.pro file, and then just invoke indent on
your file. However, be warned. This does not produce perfect indenting, and it
will mess up C++ class statements pretty badly.
<P>
Braces are required in all if statements (missing in some very old code). To
avoid generating too many lines, the first brace appears on the first line
(e.g. of an if), and the closing brace is on a line by itself. E.g.
<P>
<PRE>
if (abc) {
some_code;
}
</PRE>
<P>
Just follow the convention in the code. Originally I indented case clauses
under a switch(), but now I prefer non-indented cases.
<P>
<PRE>
switch (code) {
case 'A':
do something
break;
case 'B':
again();
break;
default:
break;
}
</PRE>
<P>
Avoid using // style comments except for temporary code or turning off debug
code. Standard C comments are preferred (this also keeps the code closer to
C).
<P>
Attempt to keep all lines less than 85 characters long so that the whole line
of code is readable at one time. This is not a rigid requirement.
<P>
Always put a brief description at the top of any new file created describing
what it does and including your name and the date it was first written. Please
don't forget any Copyrights and acknowledgments if it isn't 100% your code.
Also, include the Bacula copyright notice that is in <B>src/c</B>.
<P>
In general you should have two includes at the top of the an include for the
particular directory the code is in, for includes are needed, but this should
be rare.
<P>
In general (except for self-contained packages), prototypes should all be put
in <B>protos.h</B> in each directory.
<P>
Always put space around assignment and comparison operators.
<P>
<PRE>
a = 1;
if (b >= 2) {
cleanup();
}
</PRE>
<P>
but your can compress things in a <B>for</B> statement:
<P>
<PRE>
for (i=0; i < del.num_ids; i++) {
...
</PRE>
<P>
Don't overuse the inline if (?:). A full <B>if</B> is preferred, except in a
print statement, e.g.:
<P>
<PRE>
if (ua->verbose \&& del.num_del != 0) {
bsendmsg(ua, _("Pruned %d %s on Volume %s from catalog.\n"), del.num_del,
del.num_del == 1 ? "Job" : "Jobs", mr->VolumeName);
}
</PRE>
<P>
Leave a certain amount of debug code (Dmsg) in code you submit, so that future
problems can be identified. This is particularly true for complicated code
likely to break. However, try to keep the debug code to a minimum to avoid
bloating the program and above all to keep the code readable.
<P>
Please keep the same style in all new code you develop. If you include code
previously written, you have the option of leaving it with the old indenting
or re-indenting it. If the old code is indented with 8 spaces, then please
re-indent it to Bacula standards.
<P>
If you are using <B>vim</B>, simply set your tabstop to 8 and your shiftwidth
to 3.
<P>
<H3><A NAME="SECTION000471300000000000000"></A>
<A NAME="366"></A>
<BR>
Tabbing
</H3>
<A NAME="369"></A>
<P>
Tabbing (inserting the tab character in place of spaces) is as normal on all
Unix systems -- a tab is converted space up to the next column multiple of 8.
My editor converts strings of spaces to tabs automatically -- this results in
significant compression of the files. Thus, you can remove tabs by replacing
them with spaces if you wish. Please don't confuse tabbing (use of tab
characters) with indenting (visual alignment of the code).
<P>
<H3><A NAME="SECTION000471400000000000000"></A>
<A NAME="371"></A>
<BR>
Don'ts
</H3>
<A NAME="374"></A>
<P>
Please don't use:
<P>
<PRE>
strcpy()
strcat()
strncpy()
strncat();
sprintf()
snprintf()
</PRE>
<P>
They are system dependent and un-safe. These should be replaced by the Bacula
safe equivalents:
<P>
<PRE>
char *bstrncpy(char *dest, char *source, int dest_size);
char *bstrncat(char *dest, char *source, int dest_size);
int bsnprintf(char *buf, int32_t buf_len, const char *fmt, ...);
int bvsnprintf(char *str, int32_t size, const char *format, va_list ap);
</PRE>
<P>
See src/lib/bsys.c for more details on these routines.
<P>
Don't use the <B>%lld</B> or the <B>%q</B> printf format editing types to edit
64 bit integers -- they are not portable. Instead, use <B>%s</B> with <B>edit_uint64()</B>. For example:
<P>
<PRE>
char buf[100];
uint64_t num = something;
char ed1[50];
bsnprintf(buf, sizeof(buf), "Num=%s\n", edit_uint64(num, ed1));
</PRE>
<P>
The edit buffer <B>ed1</B> must be at least 27 bytes long to avoid overflow.
See src/lib/edit.c for more details. If you look at the code, don't start
screaming that I use <B>lld</B>. I actually use subtle trick taught to me by
John Walker. The <B>lld</B> that appears in the editing routine is actually
<B>#define</B> to a what is needed on your OS (usually ``lld'' or ``q'') and
is defined in autoconf/configure.in for each OS. C string concatenation causes
the appropriate string to be concatenated to the ``%''.
<P>
Also please don't use the STL or Templates or any complicated C++ code.
<P>
<H3><A NAME="SECTION000471500000000000000"></A>
<A NAME="390"></A>
<A NAME="391"></A>
<BR>
Message Classes
</H3>
<A NAME="394"></A>
<P>
Currently, there are five classes of messages: Debug, Error, Job, Memory,
and Queued.
<P>
<H3><A NAME="SECTION000471600000000000000"></A>
<A NAME="396"></A>
<A NAME="397"></A>
<BR>
Debug Messages
</H3>
<A NAME="400"></A>
<P>
Debug messages are designed to be turned on at a specified debug level and are
always sent to STDOUT. There are designed to only be used in the development
debug process. They are coded as:
<P>
DmsgN(level, message, arg1, ...) where the N is a number indicating how many
arguments are to be substituted into the message (i.e. it is a count of the
number arguments you have in your message -- generally the number of percent
signs (%)). <B>level</B> is the debug level at which you wish the message to
be printed. message is the debug message to be printed, and arg1, ... are the
arguments to be substituted. Since not all compilers support #defines with
varargs, you must explicitly specify how many arguments you have.
<P>
When the debug message is printed, it will automatically be prefixed by the
name of the daemon which is running, the filename where the Dmsg is, and the
line number within the file.
<P>
Some actual examples are:
<P>
Dmsg2(20, ``MD5len=%d MD5=%s\n'', strlen(buf), buf);
<P>
Dmsg1(9, ``Created client %s record\n'', client->hdr.name);
<P>
<H3><A NAME="SECTION000471700000000000000"></A>
<A NAME="405"></A>
<A NAME="406"></A>
<BR>
Error Messages
</H3>
<A NAME="409"></A>
<P>
Error messages are messages that are related to the daemon as a whole rather
than a particular job. For example, an out of memory condition my generate an
error message. They should be very rarely needed. In general, you should be
using Job and Job Queued messages (Jmsg and Qmsg). They are coded as:
<P>
EmsgN(error-code, level, message, arg1, ...) As with debug messages, you must
explicitly code the of arguments to be substituted in the message. error-code
indicates the severity or class of error, and it may be one of the following:
<P>
<A NAME="412"></A>
<TABLE CELLPADDING=3>
<TR><TD ALIGN="LEFT"><B>M_ABORT</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Causes the daemon to immediately abort. This should be
used only in extreme cases. It attempts to produce a traceback.</TD>
</TR>
<TR><TD ALIGN="LEFT"><B>M_ERROR_TERM</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Causes the daemon to immediately terminate. This
should be used only in extreme cases. It does not produce a traceback.</TD>
</TR>
<TR><TD ALIGN="LEFT"><B>M_FATAL</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Causes the daemon to terminate the current job, but the
daemon keeps running</TD>
</TR>
<TR><TD ALIGN="LEFT"><B>M_ERROR</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Reports the error. The daemon and the job continue
running</TD>
</TR>
<TR><TD ALIGN="LEFT"><B>M_WARNING</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Reports an warning message. The daemon and the job
continue running</TD>
</TR>
<TR><TD ALIGN="LEFT"><B>M_INFO</B></TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=216>Reports an informational message.
<P></TD>
</TR>
</TABLE>
<P>
There are other error message classes, but they are in a state of being
redesigned or deprecated, so please do not use them. Some actual examples are:
<P>
Emsg1(M_ABORT, 0, ``Cannot create message thread: %s\n'',
strerror(status));
<P>
Emsg3(M_WARNING, 0, ``Connect to File daemon %s at %s:%d failed. Retrying
...\n'', client->hdr.name, client->address,
client->port);
<P>
Emsg3(M_FATAL, 0, ``bdird<filed: bad response from Filed to %s command:
%d %s\n'', cmd, n, strerror(errno));
<P>
<H3><A NAME="SECTION000471800000000000000"></A>
<A NAME="436"></A>
<A NAME="437"></A>
<BR>
Job Messages
</H3>
<A NAME="440"></A>
<P>
Job messages are messages that pertain to a particular job such as a file that
could not be saved, or the number of files and bytes that were saved. They
Are coded as:
<PRE>
Jmsg(jcr, M\_FATAL, 0, "Text of message");
</PRE>
A Jmsg with M_FATAL will fail the job. The Jmsg() takes varargs so can
have any number of arguments for substituted in a printf like format.
Output from the Jmsg() will go to the Job report.
<br>
If the Jmsg is followed with a number such as Jmsg1(...), the number
indicates the number of arguments to be substituted (varargs is not
standard for #defines), and what is more important is that the file and
line number will be prefixed to the message. This permits a sort of debug
from user's output.
<P>
<H3><A NAME="SECTION000471900000000000000"></A>
<A NAME="444"></A>
<A NAME="445"></A>
<BR>
Queued Job Messages
</H3>
<A NAME="448"></A>
Queued Job messages are similar to Jmsg()s except that the message is
Queued rather than immediately dispatched. This is necessary within the
network subroutines and in the message editing routines. This is to prevent
recursive loops, and to ensure that messages can be delivered even in the
event of a network error.
<P>
<H3><A NAME="SECTION000472000000000000000"></A>
<A NAME="450"></A>
<A NAME="451"></A>
<BR>
Memory Messages
</H3>
<A NAME="454"></A>
<P>
Memory messages are messages that are edited into a memory buffer. Generally
they are used in low level routines such as the low level device file dev.c in
the Storage daemon or in the low level Catalog routines. These routines do not
generally have access to the Job Control Record and so they return error
essages reformatted in a memory buffer. Mmsg() is the way to do this.
<P>
<H1><A NAME="SECTION00050000000000000000"></A>
<A NAME="_PlatformChapter"></A><A NAME="1134"></A>
<A NAME="1135"></A>
<BR>
Platform Support
</H1>
<A NAME="1138"></A>
<P>
<H2><A NAME="SECTION00051000000000000000"></A>
<A NAME="1140"></A>
<BR>
General
</H2>
<A NAME="1143"></A>
<P>
This chapter describes the requirements for having a
supported platform (Operating System). In general, Bacula is
quite portable. It supports 32 and 64 bit architectures as well
as bigendian and littleendian machines. For full
support, the platform (Operating System) must implement POSIX Unix
system calls. However, for File daemon support only, a small
compatibility library can be written to support almost any
architecture.
<P>
Currently Linux, FreeBSD, and Solaris are fully supported
platforms, which means that the code has been tested on those
machines and passes a full set of regression tests.
<P>
In addition, the Windows File daemon is supported on most versions
of Windows, and finally, there are a number of other platforms
where the File daemon (client) is known to run: NetBSD, OpenBSD,
Mac OSX, SGI, ...
<P>
<H2><A NAME="SECTION00052000000000000000"></A>
<A NAME="1145"></A>
<A NAME="1146"></A>
<BR>
Requirements to become a Supported Platform
</H2>
<A NAME="1149"></A>
<P>
As mentioned above, in order to become a fully supported platform, it
must support POSIX Unix system calls. In addition, the following
requirements must be met:
<P>
<UL>
<LI>The principal developer (currently Kern) must have
non-root ssh access to a test machine running the platform.
</LI>
<LI>The ideal requirements and minimum requirements
for this machine are given below.
</LI>
<LI>There must be a defined platform champion who is normally
a system administrator for the machine that is available. This
person need not be a developer/programmer but must be familiar
with system administration of the platform.
</LI>
<LI>There must be at least one person designated who will
run regression tests prior to each release. Releases occur
approximately once every 6 months, but can be more frequent.
It takes at most a day's effort to setup the regression scripts
in the beginning, and after that, they can either be run daily
or on demand before a release. Running the regression scripts
involves only one or two command line commands and is fully
automated.
</LI>
<LI>Ideally there are one or more persons who will package
each Bacula release.
</LI>
<LI>Ideally there are one or more developers who can respond to
and fix platform specific bugs.
</LI>
</UL>
<P>
Ideal requirements for a test machine:
<UL>
<LI>The principal developer will have non-root ssh access to
the test machine at all times.
</LI>
<LI>The pricipal developer will have a root password.
</LI>
<LI>The test machine will provide approximately 200 MB of
disk space for continual use.
</LI>
<LI>The test machine will have approximately 500 MB of free
disk space for temporary use.
</LI>
<LI>The test machine will run the most common version of the OS.
</LI>
<LI>The test machine will have an autochanger of DDS-4 technology
or later having two or more tapes.
</LI>
<LI>The test machine will have MySQL and/or PostgreSQL database
access for account "bacula" available.
</LI>
<LI>The test machine will have sftp access.
</LI>
<LI>The test machine will provide an smtp server.
</LI>
</UL>
<P>
Minimum requirements for a test machine:
<UL>
<LI>The principal developer will have non-root ssh access to
the test machine when requested approximately once a month.
</LI>
<LI>The pricipal developer not have root access.
</LI>
<LI>The test machine will provide approximately 80 MB of
disk space for continual use.
</LI>
<LI>The test machine will have approximately 300 MB of free
disk space for temporary use.
</LI>
<LI>The test machine will run the the OS.
</LI>
<LI>The test machine will have a tape drive of DDS-4 technology
or later that can be scheduled for access.
</LI>
<LI>The test machine will not have MySQL and/or PostgreSQL database
access.
</LI>
<LI>The test machine will have no sftp access.
</LI>
<LI>The test machine will provide no email access.
</LI>
</UL>
<P>
Bare bones test machine requirements:
<UL>
<LI>The test machine is available only to a designated
test person (your own machine).
</LI>
<LI>The designated test person runs the regession
tests on demand.
</LI>
<LI>The test machine has a tape drive available.
</LI>
</UL>
<P>
<H1><A NAME="SECTION00060000000000000000"></A>
<A NAME="_ChapterStart2"></A><A NAME="1182"></A>
<A NAME="1183"></A>
<BR>
Daemon Protocol
</H1>
<A NAME="1186"></A>
<P>
<H2><A NAME="SECTION00061000000000000000"></A>
<A NAME="1188"></A>
<BR>
General
</H2>
<A NAME="1191"></A>
<P>
This document describes the protocols used between the various daemons. As
Bacula has developed, it has become quite out of date. The general idea still
holds true, but the details of the fields for each command, and indeed the
commands themselves have changed considerably.
<P>
It is intended to be a technical discussion of the general daemon protocols
and as such is not targeted at end users but rather at developers and system
administrators that want or need to know more of the working details of <B>Bacula</B>.
<P>
<H2><A NAME="SECTION00062000000000000000"></A>
<A NAME="1194"></A>
<A NAME="1195"></A>
<BR>
Low Level Network Protocol
</H2>
<A NAME="1198"></A>
<P>
At the lowest level, the network protocol is handled by <B>BSOCK</B> packets
which contain a lot of information about the status of the network connection:
who is at the other end, etc. Each basic <B>Bacula</B> network read or write
actually consists of two low level network read/writes. The first write always
sends four bytes of data in machine independent byte order. If data is to
follow, the first four bytes are a positive non-zero integer indicating the
length of the data that follow in the subsequent write. If the four byte
integer is zero or negative, it indicates a special request, a sort of network
signaling capability. In this case, no data packet will follow. The low level
BSOCK routines expect that only a single thread is accessing the socket at a
time. It is advised that multiple threads do not read/write the same socket.
If you must do this, you must provide some sort of locking mechanism. It would
not be appropriate for efficiency reasons to make every call to the BSOCK
routines lock and unlock the packet.
<P>
<H2><A NAME="SECTION00063000000000000000"></A>
<A NAME="1202"></A>
<A NAME="1203"></A>
<BR>
General Daemon Protocol
</H2>
<A NAME="1206"></A>
<P>
In general, all the daemons follow the following global rules. There may be
exceptions depending on the specific case. Normally, one daemon will be
sending commands to another daemon (specifically, the Director to the Storage
daemon and the Director to the File daemon).
<P>
<UL>
<LI>Commands are always ASCII commands that are upper/lower case dependent
as well as space sensitive.
</LI>
<LI>All binary data is converted into ASCII (either with printf statements
or using base64 encoding).
</LI>
<LI>All responses to commands sent are always prefixed with a return
numeric code where codes in the 1000's are reserved for the Director, the
2000's are reserved for the File daemon, and the 3000's are reserved for the
Storage daemon.
</LI>
<LI>Any response that is not prefixed with a numeric code is a command (or
subcommand if you like) coming from the other end. For example, while the
Director is corresponding with the Storage daemon, the Storage daemon can
request Catalog services from the Director. This convention permits each side
to send commands to the other daemon while simultaneously responding to
commands.
</LI>
<LI>Any response that is of zero length, depending on the context, either
terminates the data stream being sent or terminates command mode prior to
closing the connection.
</LI>
<LI>Any response that is of negative length is a special sign that normally
requires a response. For example, during data transfer from the File daemon
to the Storage daemon, normally the File daemon sends continuously without
intervening reads. However, periodically, the File daemon will send a packet
of length -1 indicating that the current data stream is complete and that the
Storage daemon should respond to the packet with an OK, ABORT JOB, PAUSE,
etc. This permits the File daemon to efficiently send data while at the same
time occasionally ``polling'' the Storage daemon for his status or any
special requests.
<P>
Currently, these negative lengths are specific to the daemon, but shortly,
the range 0 to -999 will be standard daemon wide signals, while -1000 to
-1999 will be for Director user, -2000 to -2999 for the File daemon, and
-3000 to -3999 for the Storage daemon.
</LI>
</UL>
<P>
<H2><A NAME="SECTION00064000000000000000"></A>
<A NAME="1210"></A>
<A NAME="1211"></A>
<BR>
The Protocol Used Between the Director and the Storage Daemon
</H2>
<A NAME="1214"></A>
<P>
Before sending commands to the File daemon, the Director opens a Message
channel with the Storage daemon, identifies itself and presents its password.
If the password check is OK, the Storage daemon accepts the Director. The
Director then passes the Storage daemon, the JobId to be run as well as the
File daemon authorization (append, read all, or read for a specific session).
The Storage daemon will then pass back to the Director a enabling key for this
JobId that must be presented by the File daemon when opening the job. Until
this process is complete, the Storage daemon is not available for use by File
daemons.
<P>
<PRE>
SD: listens
DR: makes connection
DR: Hello <Director-name> calling <password>
SD: 3000 OK Hello
DR: JobId=nnn Allow=(append, read) Session=(*, SessionId)
(Session not implemented yet)
SD: 3000 OK Job Authorization=<password>
DR: use device=<device-name> media_type=<media-type>
pool_name=<pool-name> pool_type=<pool_type>
SD: 3000 OK use device
</PRE>
<P>
For the Director to be authorized, the <Director-name> and the
<password> must match the values in one of the Storage daemon's
Director resources (there may be several Directors that can access a single
Storage daemon).
<P>
<H2><A NAME="SECTION00065000000000000000"></A>
<A NAME="1222"></A>
<A NAME="1223"></A>
<BR>
The Protocol Used Between the Director and the File Daemon
</H2>
<A NAME="1226"></A>
<P>
A typical conversation might look like the following:
<P>
<PRE>
FD: listens
DR: makes connection
DR: Hello <Director-name> calling <password>
FD: 2000 OK Hello
DR: JobId=nnn Authorization=<password>
FD: 2000 OK Job
DR: storage address = <Storage daemon address> port = <port-number>
name = <DeviceName> mediatype = <MediaType>
FD: 2000 OK storage
DR: include
DR: <directory1>
DR: <directory2>
...
DR: Null packet
FD: 2000 OK include
DR: exclude
DR: <directory1>
DR: <directory2>
...
DR: Null packet
FD: 2000 OK exclude
DR: full
FD: 2000 OK full
DR: save
FD: 2000 OK save
FD: Attribute record for each file as sent to the
Storage daemon (described above).
FD: Null packet
FD: <append close responses from Storage daemon>
e.g.
3000 OK Volumes = <number of volumes>
3001 Volume = <volume-id> <start file> <start block>
<end file> <end block> <volume session-id>
3002 Volume data = <date/time of last write> <Number bytes written>
<number errors>
... additional Volume / Volume data pairs for volumes 2 .. n
FD: Null packet
FD: close socket
</PRE>
<P>
<H2><A NAME="SECTION00066000000000000000"></A>
<A NAME="1230"></A>
<A NAME="1231"></A>
<BR>
The Save Protocol Between the File Daemon and the Storage Daemon
</H2>
<A NAME="1234"></A>
<P>
Once the Director has send a <B>save</B> command to the File daemon, the File
daemon will contact the Storage daemon to begin the save.
<P>
In what follows: FD: refers to information set via the network from the File
daemon to the Storage daemon, and SD: refers to information set from the
Storage daemon to the File daemon.
<P>
<H3><A NAME="SECTION00066100000000000000"></A>
<A NAME="1237"></A>
<A NAME="1238"></A>
<BR>
Command and Control Information
</H3>
<A NAME="1241"></A>
<P>
Command and control information is exchanged in human readable ASCII commands.
<P>
<PRE>
FD: listens
SD: makes connection
FD: append open session = <JobId> [<password>]
SD: 3000 OK ticket = <number>
FD: append data <ticket-number>
SD: 3000 OK data address = <IPaddress> port = <port>
</PRE>
<P>
<H3><A NAME="SECTION00066200000000000000"></A>
<A NAME="1245"></A>
<A NAME="1246"></A>
<BR>
Data Information
</H3>
<A NAME="1249"></A>
<P>
The Data information consists of the file attributes and data to the Storage
daemon. For the most part, the data information is sent one way: from the File
daemon to the Storage daemon. This allows the File daemon to transfer
information as fast as possible without a lot of handshaking and network
overhead.
<P>
However, from time to time, the File daemon needs to do a sort of checkpoint
of the situation to ensure that everything is going well with the Storage
daemon. To do so, the File daemon sends a packet with a negative length
indicating that he wishes the Storage daemon to respond by sending a packet of
information to the File daemon. The File daemon then waits to receive a packet
from the Storage daemon before continuing.
<P>
All data sent are in binary format except for the header packet, which is in
ASCII. There are two packet types used data transfer mode: a header packet,
the contents of which are known to the Storage daemon, and a data packet, the
contents of which are never examined by the Storage daemon.
<P>
The first data packet to the Storage daemon will be an ASCII header packet
consisting of the following data.
<P>
<File-Index> <Stream-Id> <Info> where <B><File-Index></B> is a sequential number beginning from one that
increments with each file (or directory) sent.
<P>
where <B><Stream-Id></B> will be 1 for the Attributes record and 2 for
uncompressed File data. 3 is reserved for the MD5 signature for the file.
<P>
where <B><Info></B> transmit information about the Stream to the
Storage Daemon. It is a character string field where each character has a
meaning. The only character currently defined is 0 (zero), which is simply a
place holder (a no op). In the future, there may be codes indicating
compressed data, encrypted data, etc.
<P>
Immediately following the header packet, the Storage daemon will expect any
number of data packets. The series of data packets is terminated by a zero
length packet, which indicates to the Storage daemon that the next packet will
be another header packet. As previously mentioned, a negative length packet is
a request for the Storage daemon to temporarily enter command mode and send a
reply to the File daemon. Thus an actual conversation might contain the
following exchanges:
<P>
<PRE>
FD: <1 1 0> (header packet)
FD: <data packet containing file-attributes>
FD: Null packet
FD: <1 2 0>
FD: <multiple data packets containing the file data>
FD: Packet length = -1
SD: 3000 OK
FD: <2 1 0>
FD: <data packet containing file-attributes>
FD: Null packet
FD: <2 2 0>
FD: <multiple data packets containing the file data>
FD: Null packet
FD: Null packet
FD: append end session <ticket-number>
SD: 3000 OK end
FD: append close session <ticket-number>
SD: 3000 OK Volumes = <number of volumes>
SD: 3001 Volume = <volumeid> <start file> <start block>
<end file> <end block> <volume session-id>
SD: 3002 Volume data = <date/time of last write> <Number bytes written>
<number errors>
SD: ... additional Volume / Volume data pairs for
volumes 2 .. n
FD: close socket
</PRE>
<P>
The information returned to the File daemon by the Storage daemon in response
to the <B>append close session</B> is transmit in turn to the Director.
<P>
<H1><A NAME="SECTION00070000000000000000"></A>
<A NAME="_ChapterStart6"></A><A NAME="1370"></A>
<A NAME="1371"></A>
<BR>
Director Services Daemon
</H1>
<A NAME="1374"></A>
<P>
This chapter is intended to be a technical discussion of the Director services
and as such is not targeted at end users but rather at developers and system
administrators that want or need to know more of the working details of <B>Bacula</B>.
<P>
The <B>Bacula Director</B> services consist of the program that supervises all
the backup and restore operations.
<P>
To be written ...
<P>
<H1><A NAME="SECTION00080000000000000000"></A>
<A NAME="_ChapterStart11"></A><A NAME="1391"></A>
<A NAME="1392"></A>
<BR>
File Services Daemon
</H1>
<A NAME="1395"></A>
<P>
Please note, this section is somewhat out of date as the code has evolved
significantly. The basic idea has not changed though.
<P>
This chapter is intended to be a technical discussion of the File daemon
services and as such is not targeted at end users but rather at developers and
system administrators that want or need to know more of the working details of
<B>Bacula</B>.
<P>
The <B>Bacula File Services</B> consist of the programs that run on the system
to be backed up and provide the interface between the Host File system and
Bacula -- in particular, the Director and the Storage services.
<P>
When time comes for a backup, the Director gets in touch with the File daemon
on the client machine and hands it a set of ``marching orders'' which, if
written in English, might be something like the following:
<P>
OK, <B>File daemon</B>, it's time for your daily incremental backup. I want you
to get in touch with the Storage daemon on host archive.mysite.com and perform
the following save operations with the designated options. You'll note that
I've attached include and exclude lists and patterns you should apply when
backing up the file system. As this is an incremental backup, you should save
only files modified since the time you started your last backup which, as you
may recall, was 2000-11-19-06:43:38. Please let me know when you're done and
how it went. Thank you.
<P>
So, having been handed everything it needs to decide what to dump and where to
store it, the File daemon doesn't need to have any further contact with the
Director until the backup is complete providing there are no errors. If there
are errors, the error messages will be delivered immediately to the Director.
While the backup is proceeding, the File daemon will send the file coordinates
and data for each file being backed up to the Storage daemon, which will in
turn pass the file coordinates to the Director to put in the catalog.
<P>
During a <B>Verify</B> of the catalog, the situation is different, since the
File daemon will have an exchange with the Director for each file, and will
not contact the Storage daemon.
<P>
A <B>Restore</B> operation will be very similar to the <B>Backup</B> except that
during the <B>Restore</B> the Storage daemon will not send storage coordinates
to the Director since the Director presumably already has them. On the other
hand, any error messages from either the Storage daemon or File daemon will
normally be sent directly to the Directory (this, of course, depends on how
the Message resource is defined).
<P>
<H2><A NAME="SECTION00081000000000000000"></A>
<A NAME="1404"></A>
<A NAME="1405"></A>
<BR>
Commands Received from the Director for a Backup
</H2>
<A NAME="1408"></A>
<P>
To be written ...
<P>
<H2><A NAME="SECTION00082000000000000000"></A>
<A NAME="1410"></A>
<A NAME="1411"></A>
<BR>
Commands Received from the Director for a Restore
</H2>
<A NAME="1414"></A>
<P>
To be written ...
<P>
<H1><A NAME="SECTION00090000000000000000"></A>
<A NAME="_ChapterStart3"></A><A NAME="1457"></A>
<A NAME="1458"></A>
<BR>
Storage Daemon Design
</H1>
<A NAME="1461"></A>
<P>
This chapter is intended to be a technical discussion of the Storage daemon
services and as such is not targeted at end users but rather at developers and
system administrators that want or need to know more of the working details of
<B>Bacula</B>.
<P>
This document is somewhat out of date.
<P>
<H2><A NAME="SECTION00091000000000000000"></A>
<A NAME="1464"></A>
<A NAME="1465"></A>
<BR>
SD Design Introduction
</H2>
<A NAME="1468"></A>
<P>
The Bacula Storage daemon provides storage resources to a Bacula installation.
An individual Storage daemon is associated with a physical permanent storage
device (for example, a tape drive, CD writer, tape changer or jukebox, etc.),
and may employ auxiliary storage resources (such as space on a hard disk file
system) to increase performance and/or optimize use of the permanent storage
medium.
<P>
Any number of storage daemons may be run on a given machine; each associated
with an individual storage device connected to it, and BACULA operations may
employ storage daemons on any number of hosts connected by a network, local or
remote. The ability to employ remote storage daemons (with appropriate
security measures) permits automatic off-site backup, possibly to publicly
available backup repositories.
<P>
<H2><A NAME="SECTION00092000000000000000"></A>
<A NAME="1470"></A>
<A NAME="1471"></A>
<BR>
SD Development Outline
</H2>
<A NAME="1474"></A>
<P>
In order to provide a high performance backup and restore solution that scales
to very large capacity devices and networks, the storage daemon must be able
to extract as much performance from the storage device and network with which
it interacts. In order to accomplish this, storage daemons will eventually
have to sacrifice simplicity and painless portability in favor of techniques
which improve performance. My goal in designing the storage daemon protocol
and developing the initial prototype storage daemon is to provide for these
additions in the future, while implementing an initial storage daemon which is
very simple and portable to almost any POSIX-like environment. This original
storage daemon (and its evolved descendants) can serve as a portable solution
for non-demanding backup requirements (such as single servers of modest size,
individual machines, or small local networks), while serving as the starting
point for development of higher performance configurable derivatives which use
techniques such as POSIX threads, shared memory, asynchronous I/O, buffering
to high-speed intermediate media, and support for tape changers and jukeboxes.
<P>
<H2><A NAME="SECTION00093000000000000000"></A>
<A NAME="1476"></A>
<A NAME="1477"></A>
<BR>
SD Connections and Sessions
</H2>
<A NAME="1480"></A>
<P>
A client connects to a storage server by initiating a conventional TCP
connection. The storage server accepts the connection unless its maximum
number of connections has been reached or the specified host is not granted
access to the storage server. Once a connection has been opened, the client
may make any number of Query requests, and/or initiate (if permitted), one or
more Append sessions (which transmit data to be stored by the storage daemon)
and/or Read sessions (which retrieve data from the storage daemon).
<P>
Most requests and replies sent across the connection are simple ASCII strings,
with status replies prefixed by a four digit status code for easier parsing.
Binary data appear in blocks stored and retrieved from the storage. Any
request may result in a single-line status reply of ``<TT>3201 Notification pending</TT>'', which indicates the client must send a ``Query notification''
request to retrieve one or more notifications posted to it. Once the
notifications have been returned, the client may then resubmit the request
which resulted in the 3201 status.
<P>
The following descriptions omit common error codes, yet to be defined, which
can occur from most or many requests due to events like media errors,
restarting of the storage daemon, etc. These details will be filled in, along
with a comprehensive list of status codes along with which requests can
produce them in an update to this document.
<P>
<H3><A NAME="SECTION00093100000000000000"></A>
<A NAME="1483"></A>
<A NAME="1484"></A>
<BR>
SD Append Requests
</H3>
<A NAME="1487"></A>
<P>
<DL>
<DT><STRONG>append open session = <JobId> [ <Password> ] </STRONG></DT>
<DD><A NAME="1493"></A>
A data append session is opened with the Job ID given by <I>JobId</I> with
client password (if required) given by <I>Password</I>. If the session is
successfully opened, a status of <TT>3000 OK</TT> is returned with a ``<TT>ticket = </TT><I>number</I>'' reply used to identify subsequent messages in the
session. If too many sessions are open, or a conflicting session (for
example, a read in progress when simultaneous read and append sessions are
not permitted), a status of ``<TT>3502 Volume busy</TT>'' is returned. If no
volume is mounted, or the volume mounted cannot be appended to, a status of
``<TT>3503 Volume not mounted</TT>'' is returned.
<P>
</DD>
<DT><STRONG>append data = <ticket-number> </STRONG></DT>
<DD><A NAME="1503"></A>
If the append data is accepted, a status of <TT>3000 OK data address =
<IPaddress> port = <port></TT> is returned, where the <TT>IPaddress</TT> and <TT>port</TT> specify the IP address and port number of the data
channel. Error status codes are <TT>3504 Invalid ticket number</TT> and <TT>3505 Session aborted</TT>, the latter of which indicates the entire append
session has failed due to a daemon or media error.
<P>
Once the File daemon has established the connection to the data channel
opened by the Storage daemon, it will transfer a header packet followed by
any number of data packets. The header packet is of the form:
<P>
<TT><file-index> <stream-id> <info></TT>
<P>
The details are specified in the
<A HREF="#_ChapterStart2">Daemon Protocol</A> section of this
document.
<P>
</DD>
<DT><STRONG>*append abort session = <ticket-number> </STRONG></DT>
<DD><A NAME="1522"></A>
The open append session with ticket <I>ticket-number</I> is aborted; any blocks
not yet written to permanent media are discarded. Subsequent attempts to
append data to the session will receive an error status of <TT>3505 Session aborted</TT>.
<P>
</DD>
<DT><STRONG>append end session = <ticket-number> </STRONG></DT>
<DD><A NAME="1527"></A>
The open append session with ticket <I>ticket-number</I> is marked complete; no
further blocks may be appended. The storage daemon will give priority to
saving any buffered blocks from this session to permanent media as soon as
possible.
<P>
</DD>
<DT><STRONG>append close session = <ticket-number> </STRONG></DT>
<DD><A NAME="1531"></A>
The append session with ticket <I>ticket</I> is closed. This message does not
receive an <TT>3000 OK</TT> reply until all of the content of the session are
stored on permanent media, at which time said reply is given, followed by a
list of volumes, from first to last, which contain blocks from the session,
along with the first and last file and block on each containing session data
and the volume session key identifying data from that session in lines with
the following format:
<P>
<TT><TT>Volume = </TT><Volume-id> <start-file>
<start-block> <end-file> <end-block>
<volume-session-id></TT>where <I>Volume-id</I> is the volume label, <I>start-file</I> and <I>start-block</I> are the file and block containing the first
data from that session on the volume, <I>end-file</I> and <I>end-block</I> are
the file and block with the last data from the session on the volume and <I>volume-session-id</I> is the volume session ID for blocks from the session
stored on that volume.
</DD>
</DL>
<P>
<H3><A NAME="SECTION00093200000000000000"></A>
<A NAME="1555"></A>
<A NAME="1556"></A>
<BR>
SD Read Requests
</H3>
<A NAME="1559"></A>
<P>
<DL>
<DT><STRONG>Read open session = <JobId> <Volume-id>
<start-file> <start-block> <end-file>
<end-block> <volume-session-id> <password> </STRONG></DT>
<DD><A NAME="1577"></A>
where <I>Volume-id</I> is the volume label, <I>start-file</I> and <I>start-block</I> are the file and block containing the first data from that
session on the volume, <I>end-file</I> and <I>end-block</I> are the file and
block with the last data from the session on the volume and <I>volume-session-id</I> is the volume session ID for blocks from the session
stored on that volume.
<P>
If the session is successfully opened, a status of
<P>
<TT><TT>3100 OK Ticket = </TT><I>number</I>``</TT>
<P>
is returned with a reply used to identify subsequent messages in the session.
If too many sessions are open, or a conflicting session (for example, an
append in progress when simultaneous read and append sessions are not
permitted), a status of ''<TT>3502 Volume busy</TT>`` is returned. If no
volume is mounted, or the volume mounted cannot be appended to, a status of
''<TT>3503 Volume not mounted</TT>`` is returned. If no block with the given
volume session ID and the correct client ID number appears in the given first
file and block for the volume, a status of ''<TT>3505 Session not found</TT>`` is returned.
<P>
</DD>
<DT><STRONG>Read data = <Ticket> > <Block> </STRONG></DT>
<DD><A NAME="1594"></A>
The specified Block of data from open read session with the specified Ticket
number is returned, with a status of <TT>3000 OK</TT> followed by a ''<TT>Length = </TT><I>size</I>`` line giving the length in bytes of the block data
which immediately follows. Blocks must be retrieved in ascending order, but
blocks may be skipped. If a block number greater than the largest stored on
the volume is requested, a status of ''<TT>3201 End of volume</TT>`` is
returned. If a block number greater than the largest in the file is
requested, a status of ''<TT>3401 End of file</TT>`` is returned.
<P>
</DD>
<DT><STRONG>Read close session = <Ticket> </STRONG></DT>
<DD><A NAME="1602"></A>
The read session with Ticket number is closed. A read session may be closed
at any time; you needn't read all its blocks before closing it.
</DD>
</DL>
<P>
<I>by
<A NAME="tex2html8"
HREF="http://www.fourmilab.ch/">John Walker</A>
January 30th, MM </I>
<P>
<H2><A NAME="SECTION00094000000000000000"></A>
<A NAME="1607"></A>
<BR>
SD Data Structures
</H2>
<A NAME="1610"></A>
<P>
In the Storage daemon, there is a Device resource (i.e. from conf file)
that describes each physical device. When the physical device is used it
is controled by the DEVICE structure (defined in dev.h), and typically
refered to as dev in the C++ code. Anyone writing or reading a physical
device must ultimately get a lock on the DEVICE structure -- this controls
the device. However, multiple Jobs (defined by a JCR structure src/jcr.h)
can be writing a physical DEVICE at the same time (of course they are
sequenced by locking the DEVICE structure). There are a lot of job
dependent "device" variables that may be different for each Job such as
spooling (one job may spool and another may not, and when a job is
spooling, it must have an i/o packet open, each job has its own record and
block structures, ...), so there is a device control record or DCR that is
the primary way of interfacing to the physical device. The DCR contains
all the job specific data as well as a pointer to the Device resource
(DEVRES structure) and the physical DEVICE structure.
<P>
Now if a job is writing to two devices (it could be writing two separate
streams to the same device), it must have two DCRs. Today, the code only
permits one. This won't be hard to change, but it is new code.
<P>
Today three jobs (threads), two physical devices each job
writes to only one device:
<P>
<PRE>
Job1 -> DCR1 -> DEVICE1
Job2 -> DCR2 -> DEVICE1
Job3 -> DCR3 -> DEVICE2
</PRE>
<P>
To be implemented three jobs, three physical devices, but
job1 is writing simultaneously to three devices:
<P>
<PRE>
Job1 -> DCR1 -> DEVICE1
-> DCR4 -> DEVICE2
-> DCR5 -> DEVICE3
Job2 -> DCR2 -> DEVICE1
Job3 -> DCR3 -> DEVICE2
Job = job control record
DCR = Job contorl data for a specific device
DEVICE = Device only control data
</PRE>
<P>
<P>
<H1><A NAME="SECTION000100000000000000000"></A>
<A NAME="_ChapterStart30"></A>
<BR>
Catalog Services
</H1>
<A NAME="1795"></A>
<A NAME="1796"></A>
<A NAME="1799"></A>
<P>
<H2><A NAME="SECTION000101000000000000000">
General</A>
</H2>
<A NAME="1801"></A>
<A NAME="1804"></A>
<P>
This chapter is intended to be a technical discussion of the Catalog services
and as such is not targeted at end users but rather at developers and system
administrators that want or need to know more of the working details of <B>Bacula</B>.
<P>
The <B>Bacula Catalog</B> services consist of the programs that provide the SQL
database engine for storage and retrieval of all information concerning files
that were backed up and their locations on the storage media.
<P>
We have investigated the possibility of using the following SQL engines for
Bacula: Beagle, mSQL, GNU SQL, PostgreSQL, SQLite, Oracle, and MySQL. Each
presents certain problems with either licensing or maturity. At present, we
have chosen for development purposes to use MySQL, PostgreSQL and SQLite.
MySQL was chosen because it is fast, proven to be reliable, widely used, and
actively being developed. MySQL is released under the GNU GPL license.
PostgreSQL was chosen because it is a full-featured, very mature database, and
because Dan Langille did the Bacula driver for it. PostgreSQL is distributed
under the BSD license. SQLite was chosen because it is small, efficient, and
can be directly embedded in <B>Bacula</B> thus requiring much less effort from
the system administrator or person building <B>Bacula</B>. In our testing
SQLite has performed very well, and for the functions that we use, it has
never encountered any errors except that it does not appear to handle
databases larger than 2GBytes. That said, we would not recommend it for
serious production use.
<P>
The Bacula SQL code has been written in a manner that will allow it to be
easily modified to support any of the current SQL database systems on the
market (for example: mSQL, iODBC, unixODBC, Solid, OpenLink ODBC, EasySoft
ODBC, InterBase, Oracle8, Oracle7, and DB2).
<P>
If you do not specify either <B><code>--</code>with-mysql</B> or <B><code>--</code>with-postgresql</B> or
<B><code>--</code>with-sqlite</B> on the ./configure line, Bacula will use its minimalist
internal database. This database is kept for build reasons but is no longer
supported. Bacula <B>requires</B> one of the three databases (MySQL,
PostgreSQL, or SQLite) to run.
<P>
<H3><A NAME="SECTION000101100000000000000">
Filenames and Maximum Filename Length</A>
</H3>
<A NAME="1814"></A>
<A NAME="1815"></A>
<A NAME="1818"></A>
<P>
In general, either MySQL, PostgreSQL or SQLite permit storing arbitrary long
path names and file names in the catalog database. In practice, there still
may be one or two places in the Catalog interface code that restrict the
maximum path length to 512 characters and the maximum file name length to 512
characters. These restrictions are believed to have been removed. Please note,
these restrictions apply only to the Catalog database and thus to your ability
to list online the files saved during any job. All information received and
stored by the Storage daemon (normally on tape) allows and handles arbitrarily
long path and filenames.
<P>
<H3><A NAME="SECTION000101200000000000000">
Installing and Configuring MySQL</A>
</H3>
<A NAME="1820"></A>
<A NAME="1821"></A>
<A NAME="1824"></A>
<P>
For the details of installing and configuring MySQL, please see the
<A HREF="#_ChapterStart">Installing and Configuring MySQL</A> chapter of
this manual.
<P>
<H3><A NAME="SECTION000101300000000000000">
Installing and Configuring PostgreSQL</A>
</H3>
<A NAME="1828"></A>
<A NAME="1829"></A>
<A NAME="1832"></A>
<P>
For the details of installing and configuring PostgreSQL, please see the
<A HREF="#_ChapterStart10">Installing and Configuring PostgreSQL</A>
chapter of this manual.
<P>
<H3><A NAME="SECTION000101400000000000000">
Installing and Configuring SQLite</A>
</H3>
<A NAME="1836"></A>
<A NAME="1837"></A>
<A NAME="1840"></A>
<P>
For the details of installing and configuring SQLite, please see the
<A HREF="#_ChapterStart33">Installing and Configuring SQLite</A> chapter of
this manual.
<P>
<H3><A NAME="SECTION000101500000000000000">
Internal Bacula Catalog</A>
</H3>
<A NAME="1844"></A>
<A NAME="1845"></A>
<A NAME="1848"></A>
<P>
Please see the
<A HREF="#_ChapterStart42">Internal Bacula Database</A> chapter of this
manual for more details.
<P>
<H3><A NAME="SECTION000101600000000000000">
Database Table Design</A>
</H3>
<A NAME="1852"></A>
<A NAME="1853"></A>
<A NAME="1856"></A>
<P>
All discussions that follow pertain to the MySQL database. The details for the
PostgreSQL and SQLite databases are essentially identical except for that all
fields in the SQLite database are stored as ASCII text and some of the
database creation statements are a bit different. The details of the internal
Bacula catalog are not discussed here.
<P>
Because the Catalog database may contain very large amounts of data for large
sites, we have made a modest attempt to normalize the data tables to reduce
redundant information. While reducing the size of the database significantly,
it does, unfortunately, add some complications to the structures.
<P>
In simple terms, the Catalog database must contain a record of all Jobs run by
Bacula, and for each Job, it must maintain a list of all files saved, with
their File Attributes (permissions, create date, ...), and the location and
Media on which the file is stored. This is seemingly a simple task, but it
represents a huge amount interlinked data. Note: the list of files and their
attributes is not maintained when using the internal Bacula database. The data
stored in the File records, which allows the user or administrator to obtain a
list of all files backed up during a job, is by far the largest volume of
information put into the Catalog database.
<P>
Although the Catalog database has been designed to handle backup data for
multiple clients, some users may want to maintain multiple databases, one for
each machine to be backed up. This reduces the risk of confusion of accidental
restoring a file to the wrong machine as well as reducing the amount of data
in a single database, thus increasing efficiency and reducing the impact of a
lost or damaged database.
<P>
<H2><A NAME="SECTION000102000000000000000">
Sequence of Creation of Records for a Save Job</A>
</H2>
<A NAME="1858"></A>
<A NAME="1859"></A>
<A NAME="1862"></A>
<P>
Start with StartDate, ClientName, Filename, Path, Attributes, MediaName,
MediaCoordinates. (PartNumber, NumParts). In the steps below, ``Create new''
means to create a new record whether or not it is unique. ``Create unique''
means each record in the database should be unique. Thus, one must first
search to see if the record exists, and only if not should a new one be
created, otherwise the existing RecordId should be used.
<P>
<OL>
<LI>Create new Job record with StartDate; save JobId
</LI>
<LI>Create unique Media record; save MediaId
</LI>
<LI>Create unique Client record; save ClientId
</LI>
<LI>Create unique Filename record; save FilenameId
</LI>
<LI>Create unique Path record; save PathId
</LI>
<LI>Create unique Attribute record; save AttributeId
store ClientId, FilenameId, PathId, and Attributes
</LI>
<LI>Create new File record
store JobId, AttributeId, MediaCoordinates, etc
</LI>
<LI>Repeat steps 4 through 8 for each file
</LI>
<LI>Create a JobMedia record; save MediaId
</LI>
<LI>Update Job record filling in EndDate and other Job statistics
</LI>
</OL>
<P>
<H2><A NAME="SECTION000103000000000000000">
Database Tables</A>
</H2>
<A NAME="1866"></A>
<A NAME="1867"></A>
<A NAME="1870"></A>
<P>
<A NAME="1873"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Filename </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="LEFT" COLSPAN=1><B>Data Type </B></TD>
<TD ALIGN="LEFT" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">FilenameId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">Name</TD>
<TD ALIGN="LEFT">Blob</TD>
<TD ALIGN="LEFT">Filename</TD>
</TR>
</TABLE>
<P>
The <B>Filename</B> table shown above contains the name of each file backed up
with the path removed. If different directories or machines contain the same
filename, only one copy will be saved in this table.
<P>
<P>
<A NAME="1898"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Path </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">PathId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">Path</TD>
<TD ALIGN="LEFT">Blob</TD>
<TD ALIGN="LEFT">Full Path</TD>
</TR>
</TABLE>
<P>
The <B>Path</B> table contains shown above the path or directory names of all
directories on the system or systems. The filename and any MSDOS disk name are
stripped off. As with the filename, only one copy of each directory name is
kept regardless of how many machines or drives have the same directory. These
path names should be stored in Unix path name format.
<P>
Some simple testing on a Linux file system indicates that separating the
filename and the path may be more complication than is warranted by the space
savings. For example, this system has a total of 89,097 files, 60,467 of which
have unique filenames, and there are 4,374 unique paths.
<P>
Finding all those files and doing two stats() per file takes an average wall
clock time of 1 min 35 seconds on a 400MHz machine running RedHat 6.1 Linux.
<P>
Finding all those files and putting them directly into a MySQL database with
the path and filename defined as TEXT, which is variable length up to 65,535
characters takes 19 mins 31 seconds and creates a 27.6 MByte database.
<P>
Doing the same thing, but inserting them into Blob fields with the filename
indexed on the first 30 characters and the path name indexed on the 255 (max)
characters takes 5 mins 18 seconds and creates a 5.24 MB database. Rerunning
the job (with the database already created) takes about 2 mins 50 seconds.
<P>
Running the same as the last one (Path and Filename Blob), but Filename
indexed on the first 30 characters and the Path on the first 50 characters
(linear search done there after) takes 5 mins on the average and creates a 3.4
MB database. Rerunning with the data already in the DB takes 3 mins 35
seconds.
<P>
Finally, saving only the full path name rather than splitting the path and the
file, and indexing it on the first 50 characters takes 6 mins 43 seconds and
creates a 7.35 MB database.
<P>
<P>
<A NAME="1923"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>File </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">FileId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">FileIndex</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">The sequential file number in the Job</TD>
</TR>
<TR><TD ALIGN="LEFT">JobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Link to Job Record</TD>
</TR>
<TR><TD ALIGN="LEFT">PathId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Link to Path Record</TD>
</TR>
<TR><TD ALIGN="LEFT">FilenameId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Link to Filename Record</TD>
</TR>
<TR><TD ALIGN="LEFT">MarkId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Used to mark files during Verify Jobs</TD>
</TR>
<TR><TD ALIGN="LEFT">LStat</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">File attributes in base64 encoding</TD>
</TR>
<TR><TD ALIGN="LEFT">MD5</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">MD5 signature in base64 encoding</TD>
</TR>
</TABLE>
<P>
The <B>File</B> table shown above contains one entry for each file backed up by
Bacula. Thus a file that is backed up multiple times (as is normal) will have
multiple entries in the File table. This will probably be the table with the
most number of records. Consequently, it is essential to keep the size of this
record to an absolute minimum. At the same time, this table must contain all
the information (or pointers to the information) about the file and where it
is backed up. Since a file may be backed up many times without having changed,
the path and filename are stored in separate tables.
<P>
This table contains by far the largest amount of information in the Catalog
database, both from the stand point of number of records, and the stand point
of total database size. As a consequence, the user must take care to
periodically reduce the number of File records using the <B>retention</B>
command in the Console program.
<P>
<P>
<A NAME="1967"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Job </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">JobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">Job</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Unique Job Name</TD>
</TR>
<TR><TD ALIGN="LEFT">Name</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Job Name</TD>
</TR>
<TR><TD ALIGN="LEFT">PurgedFiles</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Used by Bacula for purging/retention periods</TD>
</TR>
<TR><TD ALIGN="LEFT">Type</TD>
<TD ALIGN="LEFT">binary(1)</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Job Type: Backup, Copy, Clone, Archive, Migration</TD>
</TR>
<TR><TD ALIGN="LEFT">Level</TD>
<TD ALIGN="LEFT">binary(1)</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Job Level</TD>
</TR>
<TR><TD ALIGN="LEFT">ClientId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Client index</TD>
</TR>
<TR><TD ALIGN="LEFT">JobStatus</TD>
<TD ALIGN="LEFT">binary(1)</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Job Termination Status</TD>
</TR>
<TR><TD ALIGN="LEFT">SchedTime</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Time/date when Job scheduled</TD>
</TR>
<TR><TD ALIGN="LEFT">StartTime</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Time/date when Job started</TD>
</TR>
<TR><TD ALIGN="LEFT">EndTime</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Time/date when Job ended</TD>
</TR>
<TR><TD ALIGN="LEFT">JobTDate</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Start day in Unix format but 64 bits; used for
Retention period.</TD>
</TR>
<TR><TD ALIGN="LEFT">VolSessionId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Unique Volume Session ID</TD>
</TR>
<TR><TD ALIGN="LEFT">VolSessionTime</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Unique Volume Session Time</TD>
</TR>
<TR><TD ALIGN="LEFT">JobFiles</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Number of files saved in Job</TD>
</TR>
<TR><TD ALIGN="LEFT">JobBytes</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Number of bytes saved in Job</TD>
</TR>
<TR><TD ALIGN="LEFT">JobErrors</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Number of errors during Job</TD>
</TR>
<TR><TD ALIGN="LEFT">JobMissingFiles</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Number of files not saved (not yet used)</TD>
</TR>
<TR><TD ALIGN="LEFT">PoolId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Link to Pool Record</TD>
</TR>
<TR><TD ALIGN="LEFT">FileSetId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Link to FileSet Record</TD>
</TR>
<TR><TD ALIGN="LEFT">PurgedFiles</TD>
<TD ALIGN="LEFT">tiny integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Set when all File records purged</TD>
</TR>
<TR><TD ALIGN="LEFT">HasBase</TD>
<TD ALIGN="LEFT">tiny integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Set when Base Job run</TD>
</TR>
</TABLE>
<P>
The <B>Job</B> table contains one record for each Job run by Bacula. Thus
normally, there will be one per day per machine added to the database. Note,
the JobId is used to index Job records in the database, and it often is shown
to the user in the Console program. However, care must be taken with its use
as it is not unique from database to database. For example, the user may have
a database for Client data saved on machine Rufus and another database for
Client data saved on machine Roxie. In this case, the two database will each
have JobIds that match those in another database. For a unique reference to a
Job, see Job below.
<P>
The Name field of the Job record corresponds to the Name resource record given
in the Director's configuration file. Thus it is a generic name, and it will
be normal to find many Jobs (or even all Jobs) with the same Name.
<P>
The Job field contains a combination of the Name and the schedule time of the
Job by the Director. Thus for a given Director, even with multiple Catalog
databases, the Job will contain a unique name that represents the Job.
<P>
For a given Storage daemon, the VolSessionId and VolSessionTime form a unique
identification of the Job. This will be the case even if multiple Directors
are using the same Storage daemon.
<P>
The Job Type (or simply Type) can have one of the following values:
<P>
<A NAME="2052"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Value </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Meaning </B></TD>
</TR>
<TR><TD ALIGN="LEFT">B</TD>
<TD ALIGN="LEFT">Backup Job</TD>
</TR>
<TR><TD ALIGN="LEFT">V</TD>
<TD ALIGN="LEFT">Verify Job</TD>
</TR>
<TR><TD ALIGN="LEFT">R</TD>
<TD ALIGN="LEFT">Restore Job</TD>
</TR>
<TR><TD ALIGN="LEFT">C</TD>
<TD ALIGN="LEFT">Console program (not in database)</TD>
</TR>
<TR><TD ALIGN="LEFT">D</TD>
<TD ALIGN="LEFT">Admin Job</TD>
</TR>
<TR><TD ALIGN="LEFT">A</TD>
<TD ALIGN="LEFT">Archive Job (not implemented)</TD>
</TR>
</TABLE>
<P>
The JobStatus field specifies how the job terminated, and can be one of the
following:
<P>
<A NAME="2076"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Value </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Meaning </B></TD>
</TR>
<TR><TD ALIGN="LEFT">C</TD>
<TD ALIGN="LEFT">Created but not yet running</TD>
</TR>
<TR><TD ALIGN="LEFT">R</TD>
<TD ALIGN="LEFT">Running</TD>
</TR>
<TR><TD ALIGN="LEFT">B</TD>
<TD ALIGN="LEFT">Blocked</TD>
</TR>
<TR><TD ALIGN="LEFT">T</TD>
<TD ALIGN="LEFT">Terminated normally</TD>
</TR>
<TR><TD ALIGN="LEFT">E</TD>
<TD ALIGN="LEFT">Terminated in Error</TD>
</TR>
<TR><TD ALIGN="LEFT">e</TD>
<TD ALIGN="LEFT">Non-fatal error</TD>
</TR>
<TR><TD ALIGN="LEFT">f</TD>
<TD ALIGN="LEFT">Fatal error</TD>
</TR>
<TR><TD ALIGN="LEFT">D</TD>
<TD ALIGN="LEFT">Verify Differences</TD>
</TR>
<TR><TD ALIGN="LEFT">A</TD>
<TD ALIGN="LEFT">Canceled by the user</TD>
</TR>
<TR><TD ALIGN="LEFT">F</TD>
<TD ALIGN="LEFT">Waiting on the File daemon</TD>
</TR>
<TR><TD ALIGN="LEFT">S</TD>
<TD ALIGN="LEFT">Waiting on the Storage daemon</TD>
</TR>
<TR><TD ALIGN="LEFT">m</TD>
<TD ALIGN="LEFT">Waiting for a new Volume to be mounted</TD>
</TR>
<TR><TD ALIGN="LEFT">M</TD>
<TD ALIGN="LEFT">Waiting for a Mount</TD>
</TR>
<TR><TD ALIGN="LEFT">s</TD>
<TD ALIGN="LEFT">Waiting for Storage resource</TD>
</TR>
<TR><TD ALIGN="LEFT">j</TD>
<TD ALIGN="LEFT">Waiting for Job resource</TD>
</TR>
<TR><TD ALIGN="LEFT">c</TD>
<TD ALIGN="LEFT">Waiting for Client resource</TD>
</TR>
<TR><TD ALIGN="LEFT">d</TD>
<TD ALIGN="LEFT">Wating for Maximum jobs</TD>
</TR>
<TR><TD ALIGN="LEFT">t</TD>
<TD ALIGN="LEFT">Waiting for Start Time</TD>
</TR>
<TR><TD ALIGN="LEFT">p</TD>
<TD ALIGN="LEFT">Waiting for higher priority job to finish</TD>
</TR>
</TABLE>
<P>
<P>
<A NAME="2126"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>FileSet </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">FileSetId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">FileSet</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">FileSet name</TD>
</TR>
<TR><TD ALIGN="LEFT">MD5</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">MD5 checksum of FileSet</TD>
</TR>
<TR><TD ALIGN="LEFT">CreateTime</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT">Time and date Fileset created</TD>
</TR>
</TABLE>
<P>
The <B>FileSet</B> table contains one entry for each FileSet that is used. The
MD5 signature is kept to ensure that if the user changes anything inside the
FileSet, it will be detected and the new FileSet will be used. This is
particularly important when doing an incremental update. If the user deletes a
file or adds a file, we need to ensure that a Full backup is done prior to the
next incremental.
<P>
<P>
<A NAME="2157"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>JobMedia </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">JobMediaId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">JobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Link to Job Record</TD>
</TR>
<TR><TD ALIGN="LEFT">MediaId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>Link to Media Record</TD>
</TR>
<TR><TD ALIGN="LEFT">FirstIndex</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The index (sequence number) of the first file
written for this Job to the Media</TD>
</TR>
<TR><TD ALIGN="LEFT">LastIndex</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The index of the last file written for this
Job to the Media</TD>
</TR>
<TR><TD ALIGN="LEFT">StartFile</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The physical media (tape) file number of the
first block written for this Job</TD>
</TR>
<TR><TD ALIGN="LEFT">EndFile</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The physical media (tape) file number of the
last block written for this Job</TD>
</TR>
<TR><TD ALIGN="LEFT">StartBlock</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The number of the first block written for
this Job</TD>
</TR>
<TR><TD ALIGN="LEFT">EndBlock</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The number of the last block written for this
Job</TD>
</TR>
<TR><TD ALIGN="LEFT">VolIndex</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=180>The Volume use sequence number within the Job</TD>
</TR>
</TABLE>
<P>
The <B>JobMedia</B> table contains one entry at the following: start of
the job, start of each new tape file, start of each new tape, end of the
job. Since by default, a new tape file is written every 2GB, in general,
you will have more than 2 JobMedia records per Job. The number can be
varied by changing the "Maximum File Size" specified in the Device
resource. This record allows Bacula to efficiently position close to
(within 2GB) any given file in a backup. For restoring a full Job,
these records are not very important, but if you want to retrieve
a single file that was written near the end of a 100GB backup, the
JobMedia records can speed it up by orders of magnitude by permitting
forward spacing files and blocks rather than reading the whole 100GB
backup.
<P>
<P>
<A NAME="2206"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Media </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">MediaId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">VolumeName</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Volume name</TD>
</TR>
<TR><TD ALIGN="LEFT">Slot</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Autochanger Slot number or zero</TD>
</TR>
<TR><TD ALIGN="LEFT">PoolId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Link to Pool Record</TD>
</TR>
<TR><TD ALIGN="LEFT">MediaType</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>The MediaType supplied by the user</TD>
</TR>
<TR><TD ALIGN="LEFT">FirstWritten</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Time/date when first written</TD>
</TR>
<TR><TD ALIGN="LEFT">LastWritten</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Time/date when last written</TD>
</TR>
<TR><TD ALIGN="LEFT">LabelDate</TD>
<TD ALIGN="LEFT">datetime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Time/date when tape labeled</TD>
</TR>
<TR><TD ALIGN="LEFT">VolJobs</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of jobs written to this media</TD>
</TR>
<TR><TD ALIGN="LEFT">VolFiles</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of files written to this media</TD>
</TR>
<TR><TD ALIGN="LEFT">VolBlocks</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of blocks written to this media</TD>
</TR>
<TR><TD ALIGN="LEFT">VolMounts</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of time media mounted</TD>
</TR>
<TR><TD ALIGN="LEFT">VolBytes</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of bytes saved in Job</TD>
</TR>
<TR><TD ALIGN="LEFT">VolErrors</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of errors during Job</TD>
</TR>
<TR><TD ALIGN="LEFT">VolWrites</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of writes to media</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolBytes</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Maximum bytes to put on this media</TD>
</TR>
<TR><TD ALIGN="LEFT">VolCapacityBytes</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Capacity estimate for this volume</TD>
</TR>
<TR><TD ALIGN="LEFT">VolStatus</TD>
<TD ALIGN="LEFT">enum</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Status of media: Full, Archive, Append, Recycle,
Read-Only, Disabled, Error, Busy</TD>
</TR>
<TR><TD ALIGN="LEFT">Recycle</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Whether or not Bacula can recycle the Volumes:
Yes, No</TD>
</TR>
<TR><TD ALIGN="LEFT">VolRetention</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>64 bit seconds until expiration</TD>
</TR>
<TR><TD ALIGN="LEFT">VolUseDuration</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>64 bit seconds volume can be used</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolJobs</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>maximum jobs to put on Volume</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolFiles</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>maximume EOF marks to put on Volume</TD>
</TR>
</TABLE>
<P>
The <B>Volume</B> table (internally referred to as the Media table) contains
one entry for each volume, that is each tape, cassette (8mm, DLT, DAT, ...),
or file on which information is or was backed up. There is one Volume record
created for each of the NumVols specified in the Pool resource record.
<P>
<P>
<A NAME="2294"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Pool </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">PoolId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">Name</TD>
<TD ALIGN="LEFT">Tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Pool Name</TD>
</TR>
<TR><TD ALIGN="LEFT">NumVols</TD>
<TD ALIGN="LEFT">Integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Number of Volumes in the Pool</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVols</TD>
<TD ALIGN="LEFT">Integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Maximum Volumes in the Pool</TD>
</TR>
<TR><TD ALIGN="LEFT">UseOnce</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Use volume once</TD>
</TR>
<TR><TD ALIGN="LEFT">UseCatalog</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Set to use catalog</TD>
</TR>
<TR><TD ALIGN="LEFT">AcceptAnyVolume</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Accept any volume from Pool</TD>
</TR>
<TR><TD ALIGN="LEFT">VolRetention</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>64 bit seconds to retain volume</TD>
</TR>
<TR><TD ALIGN="LEFT">VolUseDuration</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>64 bit seconds volume can be used</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolJobs</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>max jobs on volume</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolFiles</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>max EOF marks to put on Volume</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxVolBytes</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>max bytes to write on Volume</TD>
</TR>
<TR><TD ALIGN="LEFT">AutoPrune</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>yes|no for autopruning</TD>
</TR>
<TR><TD ALIGN="LEFT">Recycle</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>yes|no for allowing auto recycling of Volume</TD>
</TR>
<TR><TD ALIGN="LEFT">PoolType</TD>
<TD ALIGN="LEFT">enum</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Backup, Copy, Cloned, Archive, Migration</TD>
</TR>
<TR><TD ALIGN="LEFT">LabelFormat</TD>
<TD ALIGN="LEFT">Tinyblob</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=173>Label format</TD>
</TR>
</TABLE>
<P>
The <B>Pool</B> table contains one entry for each media pool controlled by
Bacula in this database. One media record exists for each of the NumVols
contained in the Pool. The PoolType is a Bacula defined keyword. The MediaType
is defined by the administrator, and corresponds to the MediaType specified in
the Director's Storage definition record. The CurrentVol is the sequence
number of the Media record for the current volume.
<P>
<P>
<A NAME="2361"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Client </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">ClientId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">Name</TD>
<TD ALIGN="LEFT">TinyBlob</TD>
<TD ALIGN="LEFT">File Services Name</TD>
</TR>
<TR><TD ALIGN="LEFT">UName</TD>
<TD ALIGN="LEFT">TinyBlob</TD>
<TD ALIGN="LEFT">uname -a from Client (not yet used)</TD>
</TR>
<TR><TD ALIGN="LEFT">AutoPrune</TD>
<TD ALIGN="LEFT">tinyint</TD>
<TD ALIGN="LEFT">yes|no for autopruning</TD>
</TR>
<TR><TD ALIGN="LEFT">FileRetention</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT">64 bit seconds to retain Files</TD>
</TR>
<TR><TD ALIGN="LEFT">JobRetention</TD>
<TD ALIGN="LEFT">bigint</TD>
<TD ALIGN="LEFT">64 bit seconds to retain Job</TD>
</TR>
</TABLE>
<P>
The <B>Client</B> table contains one entry for each machine backed up by Bacula
in this database. Normally the Name is a fully qualified domain name.
<P>
<P>
<A NAME="2398"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>UnsavedFiles </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">UnsavedId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">JobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">JobId corresponding to this record</TD>
</TR>
<TR><TD ALIGN="LEFT">PathId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Id of path</TD>
</TR>
<TR><TD ALIGN="LEFT">FilenameId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Id of filename</TD>
</TR>
</TABLE>
<P>
The <B>UnsavedFiles</B> table contains one entry for each file that was not
saved. Note! This record is not yet implemented.
<P>
<P>
<A NAME="2429"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Counter </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">Counter</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">Counter name</TD>
</TR>
<TR><TD ALIGN="LEFT">MinValue</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Start/Min value for counter</TD>
</TR>
<TR><TD ALIGN="LEFT">MaxValue</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Max value for counter</TD>
</TR>
<TR><TD ALIGN="LEFT">CurrentValue</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Current counter value</TD>
</TR>
<TR><TD ALIGN="LEFT">WrapCounter</TD>
<TD ALIGN="LEFT">tinyblob</TD>
<TD ALIGN="LEFT">Name of another counter</TD>
</TR>
</TABLE>
<P>
The <B>Counter</B> table contains one entry for each permanent counter defined
by the user.
<P>
<P>
<A NAME="2463"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>Version </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">VersionId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
</TABLE>
<P>
The <B>Version</B> table defines the Bacula database version number. Bacula
checks this number before reading the database to ensure that it is compatible
with the Bacula binary file.
<P>
<P>
<A NAME="2485"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="LEFT" COLSPAN=3><B>BaseFiles </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Column Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Data Type
</B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Remark </B></TD>
</TR>
<TR><TD ALIGN="LEFT">BaseId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Primary Key</TD>
</TR>
<TR><TD ALIGN="LEFT">BaseJobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">JobId of Base Job</TD>
</TR>
<TR><TD ALIGN="LEFT">JobId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Reference to Job</TD>
</TR>
<TR><TD ALIGN="LEFT">FileId</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">Reference to File</TD>
</TR>
<TR><TD ALIGN="LEFT">FileIndex</TD>
<TD ALIGN="LEFT">integer</TD>
<TD ALIGN="LEFT">File Index number</TD>
</TR>
</TABLE>
<P>
The <B>BaseFiles</B> table contains all the File references for a particular
JobId that point to a Base file -- i.e. they were previously saved and hence
were not saved in the current JobId but in BaseJobId under FileId. FileIndex
is the index of the file, and is used for optimization of Restore jobs to
prevent the need to read the FileId record when creating the in memory tree.
This record is not yet implemented.
<P>
<P>
<H3><A NAME="SECTION000103100000000000000">
MySQL Table Definition</A>
</H3>
<A NAME="2518"></A>
<A NAME="2519"></A>
<A NAME="2522"></A>
<P>
The commands used to create the MySQL tables are as follows:
<P>
<PRE>
USE bacula;
CREATE TABLE Filename (
FilenameId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
Name BLOB NOT NULL,
PRIMARY KEY(FilenameId),
INDEX (Name(30))
);
CREATE TABLE Path (
PathId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
Path BLOB NOT NULL,
PRIMARY KEY(PathId),
INDEX (Path(50))
);
CREATE TABLE File (
FileId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
FileIndex INTEGER UNSIGNED NOT NULL DEFAULT 0,
JobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
PathId INTEGER UNSIGNED NOT NULL REFERENCES Path,
FilenameId INTEGER UNSIGNED NOT NULL REFERENCES Filename,
MarkId INTEGER UNSIGNED NOT NULL DEFAULT 0,
LStat TINYBLOB NOT NULL,
MD5 TINYBLOB NOT NULL,
PRIMARY KEY(FileId),
INDEX (JobId),
INDEX (PathId),
INDEX (FilenameId)
);
CREATE TABLE Job (
JobId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
Job TINYBLOB NOT NULL,
Name TINYBLOB NOT NULL,
Type BINARY(1) NOT NULL,
Level BINARY(1) NOT NULL,
ClientId INTEGER NOT NULL REFERENCES Client,
JobStatus BINARY(1) NOT NULL,
SchedTime DATETIME NOT NULL,
StartTime DATETIME NOT NULL,
EndTime DATETIME NOT NULL,
JobTDate BIGINT UNSIGNED NOT NULL,
VolSessionId INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolSessionTime INTEGER UNSIGNED NOT NULL DEFAULT 0,
JobFiles INTEGER UNSIGNED NOT NULL DEFAULT 0,
JobBytes BIGINT UNSIGNED NOT NULL,
JobErrors INTEGER UNSIGNED NOT NULL DEFAULT 0,
JobMissingFiles INTEGER UNSIGNED NOT NULL DEFAULT 0,
PoolId INTEGER UNSIGNED NOT NULL REFERENCES Pool,
FileSetId INTEGER UNSIGNED NOT NULL REFERENCES FileSet,
PurgedFiles TINYINT NOT NULL DEFAULT 0,
HasBase TINYINT NOT NULL DEFAULT 0,
PRIMARY KEY(JobId),
INDEX (Name(128))
);
CREATE TABLE FileSet (
FileSetId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
FileSet TINYBLOB NOT NULL,
MD5 TINYBLOB NOT NULL,
CreateTime DATETIME NOT NULL,
PRIMARY KEY(FileSetId)
);
CREATE TABLE JobMedia (
JobMediaId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
JobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
MediaId INTEGER UNSIGNED NOT NULL REFERENCES Media,
FirstIndex INTEGER UNSIGNED NOT NULL DEFAULT 0,
LastIndex INTEGER UNSIGNED NOT NULL DEFAULT 0,
StartFile INTEGER UNSIGNED NOT NULL DEFAULT 0,
EndFile INTEGER UNSIGNED NOT NULL DEFAULT 0,
StartBlock INTEGER UNSIGNED NOT NULL DEFAULT 0,
EndBlock INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolIndex INTEGER UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY(JobMediaId),
INDEX (JobId, MediaId)
);
CREATE TABLE Media (
MediaId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
VolumeName TINYBLOB NOT NULL,
Slot INTEGER NOT NULL DEFAULT 0,
PoolId INTEGER UNSIGNED NOT NULL REFERENCES Pool,
MediaType TINYBLOB NOT NULL,
FirstWritten DATETIME NOT NULL,
LastWritten DATETIME NOT NULL,
LabelDate DATETIME NOT NULL,
VolJobs INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolFiles INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolBlocks INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolMounts INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolBytes BIGINT UNSIGNED NOT NULL DEFAULT 0,
VolErrors INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolWrites INTEGER UNSIGNED NOT NULL DEFAULT 0,
VolCapacityBytes BIGINT UNSIGNED NOT NULL,
VolStatus ENUM('Full', 'Archive', 'Append', 'Recycle', 'Purged',
'Read-Only', 'Disabled', 'Error', 'Busy', 'Used', 'Cleaning') NOT NULL,
Recycle TINYINT NOT NULL DEFAULT 0,
VolRetention BIGINT UNSIGNED NOT NULL DEFAULT 0,
VolUseDuration BIGINT UNSIGNED NOT NULL DEFAULT 0,
MaxVolJobs INTEGER UNSIGNED NOT NULL DEFAULT 0,
MaxVolFiles INTEGER UNSIGNED NOT NULL DEFAULT 0,
MaxVolBytes BIGINT UNSIGNED NOT NULL DEFAULT 0,
InChanger TINYINT NOT NULL DEFAULT 0,
MediaAddressing TINYINT NOT NULL DEFAULT 0,
VolReadTime BIGINT UNSIGNED NOT NULL DEFAULT 0,
VolWriteTime BIGINT UNSIGNED NOT NULL DEFAULT 0,
PRIMARY KEY(MediaId),
INDEX (PoolId)
);
CREATE TABLE Pool (
PoolId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
Name TINYBLOB NOT NULL,
NumVols INTEGER UNSIGNED NOT NULL DEFAULT 0,
MaxVols INTEGER UNSIGNED NOT NULL DEFAULT 0,
UseOnce TINYINT NOT NULL,
UseCatalog TINYINT NOT NULL,
AcceptAnyVolume TINYINT DEFAULT 0,
VolRetention BIGINT UNSIGNED NOT NULL,
VolUseDuration BIGINT UNSIGNED NOT NULL,
MaxVolJobs INTEGER UNSIGNED NOT NULL DEFAULT 0,
MaxVolFiles INTEGER UNSIGNED NOT NULL DEFAULT 0,
MaxVolBytes BIGINT UNSIGNED NOT NULL,
AutoPrune TINYINT DEFAULT 0,
Recycle TINYINT DEFAULT 0,
PoolType ENUM('Backup', 'Copy', 'Cloned', 'Archive', 'Migration', 'Scratch') NOT NULL,
LabelFormat TINYBLOB,
Enabled TINYINT DEFAULT 1,
ScratchPoolId INTEGER UNSIGNED DEFAULT 0 REFERENCES Pool,
RecyclePoolId INTEGER UNSIGNED DEFAULT 0 REFERENCES Pool,
UNIQUE (Name(128)),
PRIMARY KEY (PoolId)
);
CREATE TABLE Client (
ClientId INTEGER UNSIGNED NOT NULL AUTO_INCREMENT,
Name TINYBLOB NOT NULL,
Uname TINYBLOB NOT NULL, /* full uname -a of client */
AutoPrune TINYINT DEFAULT 0,
FileRetention BIGINT UNSIGNED NOT NULL,
JobRetention BIGINT UNSIGNED NOT NULL,
UNIQUE (Name(128)),
PRIMARY KEY(ClientId)
);
CREATE TABLE BaseFiles (
BaseId INTEGER UNSIGNED AUTO_INCREMENT,
BaseJobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
JobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
FileId INTEGER UNSIGNED NOT NULL REFERENCES File,
FileIndex INTEGER UNSIGNED,
PRIMARY KEY(BaseId)
);
CREATE TABLE UnsavedFiles (
UnsavedId INTEGER UNSIGNED AUTO_INCREMENT,
JobId INTEGER UNSIGNED NOT NULL REFERENCES Job,
PathId INTEGER UNSIGNED NOT NULL REFERENCES Path,
FilenameId INTEGER UNSIGNED NOT NULL REFERENCES Filename,
PRIMARY KEY (UnsavedId)
);
CREATE TABLE Version (
VersionId INTEGER UNSIGNED NOT NULL
);
-- Initialize Version
INSERT INTO Version (VersionId) VALUES (7);
CREATE TABLE Counters (
Counter TINYBLOB NOT NULL,
MinValue INTEGER,
MaxValue INTEGER,
CurrentValue INTEGER,
WrapCounter TINYBLOB NOT NULL,
PRIMARY KEY (Counter(128))
);
</PRE>
<P>
<H1><A NAME="SECTION000110000000000000000"></A>
<A NAME="_ChapterStart9"></A><A NAME="2885"></A>
<A NAME="2886"></A>
<BR>
Storage Media Output Format
</H1>
<A NAME="2889"></A>
<P>
<H2><A NAME="SECTION000111000000000000000"></A>
<A NAME="2891"></A>
<BR>
General
</H2>
<A NAME="2894"></A>
<P>
This document describes the media format written by the Storage daemon. The
Storage daemon reads and writes in units of blocks. Blocks contain records.
Each block has a block header followed by records, and each record has a
record header followed by record data.
<P>
This chapter is intended to be a technical discussion of the Media Format and
as such is not targeted at end users but rather at developers and system
administrators that want or need to know more of the working details of <B>Bacula</B>.
<P>
<H2><A NAME="SECTION000112000000000000000"></A>
<A NAME="2897"></A>
<BR>
Definitions
</H2>
<A NAME="2900"></A>
<P>
<DL>
<DT><STRONG>Block</STRONG></DT>
<DD><A NAME="2902"></A>
A block represents the primitive unit of information that the Storage daemon
reads and writes to a physical device. Normally, for a tape device, it will
be the same as a tape block. The Storage daemon always reads and writes
blocks. A block consists of block header information followed by records.
Clients of the Storage daemon (the File daemon) normally never see blocks.
However, some of the Storage tools (bls, bscan, bextract, ...) may be use
block header information. In older Bacula tape versions, a block could
contain records (see record definition below) from multiple jobs. However,
all blocks currently written by Bacula are block level BB02, and a given
block contains records for only a single job. Different jobs simply have
their own private blocks that are intermingled with the other blocks from
other jobs on the Volume (previously the records were intermingled within
the blocks). Having only records from a single job in any give block
permitted moving the VolumeSessionId and VolumeSessionTime (see below) from
each record heading to the Block header. This has two advantages: 1. a block
can be quickly rejected based on the contents of the header without reading
all the records. 2. because there is on the average more than one record per
block, less data is written to the Volume for each job.
<P>
</DD>
<DT><STRONG>Record</STRONG></DT>
<DD><A NAME="2903"></A>
A record consists of a Record Header, which is managed by the Storage daemon
and Record Data, which is the data received from the Client. A record is the
primitive unit of information sent to and from the Storage daemon by the
Client (File daemon) programs. The details are described below.
<P>
</DD>
<DT><STRONG>JobId</STRONG></DT>
<DD><A NAME="2904"></A>
A number assigned by the Director daemon for a particular job. This number
will be unique for that particular Director (Catalog). The daemons use this
number to keep track of individual jobs. Within the Storage daemon, the JobId
may not be unique if several Directors are accessing the Storage daemon
simultaneously.
<P>
</DD>
<DT><STRONG>Session</STRONG></DT>
<DD><A NAME="2905"></A>
A Session is a concept used in the Storage daemon corresponds one to one to a
Job with the exception that each session is uniquely identified within the
Storage daemon by a unique SessionId/SessionTime pair (see below).
<P>
</DD>
<DT><STRONG>VolSessionId</STRONG></DT>
<DD><A NAME="2906"></A>
A unique number assigned by the Storage daemon to a particular session (Job)
it is having with a File daemon. This number by itself is not unique to the
given Volume, but with the VolSessionTime, it is unique.
<P>
</DD>
<DT><STRONG>VolSessionTime</STRONG></DT>
<DD><A NAME="2907"></A>
A unique number assigned by the Storage daemon to a particular Storage daemon
execution. It is actually the Unix time_t value of when the Storage daemon
began execution cast to a 32 bit unsigned integer. The combination of the
<B>VolSessionId</B> and the <B>VolSessionTime</B> for a given Storage daemon is
guaranteed to be unique for each Job (or session).
<P>
</DD>
<DT><STRONG>FileIndex</STRONG></DT>
<DD><A NAME="2910"></A>
A sequential number beginning at one assigned by the File daemon to the files
within a job that are sent to the Storage daemon for backup. The Storage
daemon ensures that this number is greater than zero and sequential. Note,
the Storage daemon uses negative FileIndexes to flag Session Start and End
Labels as well as End of Volume Labels. Thus, the combination of
VolSessionId, VolSessionTime, and FileIndex uniquely identifies the records
for a single file written to a Volume.
<P>
</DD>
<DT><STRONG>Stream</STRONG></DT>
<DD><A NAME="2911"></A>
While writing the information for any particular file to the Volume, there
can be any number of distinct pieces of information about that file, e.g. the
attributes, the file data, ... The Stream indicates what piece of data it
is, and it is an arbitrary number assigned by the File daemon to the parts
(Unix attributes, Win32 attributes, data, compressed data, ...) of a file
that are sent to the Storage daemon. The Storage daemon has no knowledge of
the details of a Stream; it simply represents a numbered stream of bytes. The
data for a given stream may be passed to the Storage daemon in single record,
or in multiple records.
<P>
</DD>
<DT><STRONG>Block Header</STRONG></DT>
<DD><A NAME="2912"></A>
A block header consists of a block identification (``BB02''), a block length
in bytes (typically 64,512) a checksum, and sequential block number. Each
block starts with a Block Header and is followed by Records. Current block
headers also contain the VolSessionId and VolSessionTime for the records
written to that block.
<P>
</DD>
<DT><STRONG>Record Header</STRONG></DT>
<DD><A NAME="2913"></A>
A record header contains the Volume Session Id, the Volume Session Time, the
FileIndex, the Stream, and the size of the data record which follows. The
Record Header is always immediately followed by a Data Record if the size
given in the Header is greater than zero. Note, for Block headers of level
BB02 (version 1.27 and later), the Record header as written to tape does not
contain the Volume Session Id and the Volume Session Time as these two
fields are stored in the BB02 Block header. The in-memory record header does
have those fields for convenience.
<P>
</DD>
<DT><STRONG>Data Record</STRONG></DT>
<DD><A NAME="2914"></A>
A data record consists of a binary stream of bytes and is always preceded by
a Record Header. The details of the meaning of the binary stream of bytes are
unknown to the Storage daemon, but the Client programs (File daemon) defines
and thus knows the details of each record type.
<P>
</DD>
<DT><STRONG>Volume Label</STRONG></DT>
<DD><A NAME="2915"></A>
A label placed by the Storage daemon at the beginning of each storage volume.
It contains general information about the volume. It is written in Record
format. The Storage daemon manages Volume Labels, and if the client wants, he
may also read them.
<P>
</DD>
<DT><STRONG>Begin Session Label</STRONG></DT>
<DD><A NAME="2916"></A>
The Begin Session Label is a special record placed by the Storage daemon on
the storage medium as the first record of an append session job with a File
daemon. This record is useful for finding the beginning of a particular
session (Job), since no records with the same VolSessionId and VolSessionTime
will precede this record. This record is not normally visible outside of the
Storage daemon. The Begin Session Label is similar to the Volume Label except
that it contains additional information pertaining to the Session.
<P>
</DD>
<DT><STRONG>End Session Label</STRONG></DT>
<DD><A NAME="2917"></A>
The End Session Label is a special record placed by the Storage daemon on the
storage medium as the last record of an append session job with a File
daemon. The End Session Record is distinguished by a FileIndex with a value
of minus two (-2). This record is useful for detecting the end of a
particular session since no records with the same VolSessionId and
VolSessionTime will follow this record. This record is not normally visible
outside of the Storage daemon. The End Session Label is similar to the Volume
Label except that it contains additional information pertaining to the
Session.
</DD>
</DL>
<P>
<H2><A NAME="SECTION000113000000000000000"></A>
<A NAME="2920"></A>
<A NAME="2921"></A>
<BR>
Storage Daemon File Output Format
</H2>
<A NAME="2924"></A>
<P>
The file storage and tape storage formats are identical except that tape
records are by default blocked into blocks of 64,512 bytes, except for the
last block, which is the actual number of bytes written rounded up to a
multiple of 1024 whereas the last record of file storage is not rounded up.
The default block size of 64,512 bytes may be overridden by the user (some
older tape drives only support block sizes of 32K). Each Session written to
tape is terminated with an End of File mark (this will be removed later).
Sessions written to file are simply appended to the end of the file.
<P>
<H2><A NAME="SECTION000114000000000000000"></A>
<A NAME="2926"></A>
<A NAME="2927"></A>
<BR>
Overall Format
</H2>
<A NAME="2930"></A>
<P>
A Bacula output file consists of Blocks of data. Each block contains a block
header followed by records. Each record consists of a record header followed
by the record data. The first record on a tape will always be the Volume Label
Record.
<P>
No Record Header will be split across Bacula blocks. However, Record Data may
be split across any number of Bacula blocks. Obviously this will not be the
case for the Volume Label which will always be smaller than the Bacula Block
size.
<P>
To simplify reading tapes, the Start of Session (SOS) and End of Session (EOS)
records are never split across blocks. If this is about to happen, Bacula will
write a short block before writing the session record (actually, the SOS
record should always be the first record in a block, excepting perhaps the
Volume label).
<P>
Due to hardware limitations, the last block written to the tape may not be
fully written. If your drive permits backspace record, Bacula will backup over
the last record written on the tape, re-read it and verify that it was
correctly written.
<P>
When a new tape is mounted Bacula will write the full contents of the
partially written block to the new tape ensuring that there is no loss of
data. When reading a tape, Bacula will discard any block that is not totally
written, thus ensuring that there is no duplication of data. In addition,
since Bacula blocks are sequentially numbered within a Job, it is easy to
ensure that no block is missing or duplicated.
<P>
<H2><A NAME="SECTION000115000000000000000"></A>
<A NAME="2932"></A>
<BR>
Serialization
</H2>
<A NAME="2935"></A>
<P>
All Block Headers, Record Headers, and Label Records are written using
Bacula's serialization routines. These routines guarantee that the data is
written to the output volume in a machine independent format.
<P>
<H2><A NAME="SECTION000116000000000000000"></A>
<A NAME="2937"></A>
<A NAME="2938"></A>
<BR>
Block Header
</H2>
<A NAME="2941"></A>
<P>
The format of the Block Header (version 1.27 and later) is:
<P>
<PRE>
uint32_t CheckSum; /* Block check sum */
uint32_t BlockSize; /* Block byte size including the header */
uint32_t BlockNumber; /* Block number */
char ID[4] = "BB02"; /* Identification and block level */
uint32_t VolSessionId; /* Session Id for Job */
uint32_t VolSessionTime; /* Session Time for Job */
</PRE>
<P>
The Block header is a fixed length and fixed format and is followed by Record
Headers and Record Data. The CheckSum field is a 32 bit checksum of the block
data and the block header but not including the CheckSum field. The Block
Header is always immediately followed by a Record Header. If the tape is
damaged, a Bacula utility will be able to recover as much information as
possible from the tape by recovering blocks which are valid. The Block header
is written using the Bacula serialization routines and thus is guaranteed to
be in machine independent format. See below for version 2 of the block header.
<P>
<H2><A NAME="SECTION000117000000000000000"></A>
<A NAME="2945"></A>
<A NAME="2946"></A>
<BR>
Record Header
</H2>
<A NAME="2949"></A>
<P>
Each binary data record is preceded by a Record Header. The Record Header is
fixed length and fixed format, whereas the binary data record is of variable
length. The Record Header is written using the Bacula serialization routines
and thus is guaranteed to be in machine independent format.
<P>
The format of the Record Header (version 1.27 or later) is:
<P>
<PRE>
int32_t FileIndex; /* File index supplied by File daemon */
int32_t Stream; /* Stream number supplied by File daemon */
uint32_t DataSize; /* size of following data record in bytes */
</PRE>
<P>
This record is followed by the binary Stream data of DataSize bytes, followed
by another Record Header record and the binary stream data. For the definitive
definition of this record, see record.h in the src/stored directory.
<P>
Additional notes on the above:
<P>
<DL>
<DT><STRONG>The <B>VolSessionId</B> </STRONG></DT>
<DD><A NAME="2954"></A>
is a unique sequential number that is assigned by the Storage Daemon to a
particular Job. This number is sequential since the start of execution of the
daemon.
<P>
</DD>
<DT><STRONG>The <B>VolSessionTime</B> </STRONG></DT>
<DD><A NAME="2956"></A>
is the time/date that the current execution of the Storage Daemon started. It
assures that the combination of VolSessionId and VolSessionTime is unique for
every jobs written to the tape, even if there was a machine crash between two
writes.
<P>
</DD>
<DT><STRONG>The <B>FileIndex</B> </STRONG></DT>
<DD><A NAME="2958"></A>
is a sequential file number within a job. The Storage daemon requires this
index to be greater than zero and sequential. Note, however, that the File
daemon may send multiple Streams for the same FileIndex. In addition, the
Storage daemon uses negative FileIndices to hold the Begin Session Label, the
End Session Label, and the End of Volume Label.
<P>
</DD>
<DT><STRONG>The <B>Stream</B> </STRONG></DT>
<DD><A NAME="2960"></A>
is defined by the File daemon and is used to identify separate parts of the
data saved for each file (Unix attributes, Win32 attributes, file data,
compressed file data, sparse file data, ...). The Storage Daemon has no idea
of what a Stream is or what it contains except that the Stream is required to
be a positive integer. Negative Stream numbers are used internally by the
Storage daemon to indicate that the record is a continuation of the previous
record (the previous record would not entirely fit in the block).
<P>
For Start Session and End Session Labels (where the FileIndex is negative),
the Storage daemon uses the Stream field to contain the JobId. The current
stream definitions are:
<P>
<PRE>
#define STREAM_UNIX_ATTRIBUTES 1 /* Generic Unix attributes */
#define STREAM_FILE_DATA 2 /* Standard uncompressed data */
#define STREAM_MD5_SIGNATURE 3 /* MD5 signature for the file */
#define STREAM_GZIP_DATA 4 /* GZip compressed file data */
/* Extended Unix attributes with Win32 Extended data. Deprecated. */
#define STREAM_UNIX_ATTRIBUTES_EX 5 /* Extended Unix attr for Win32 EX */
#define STREAM_SPARSE_DATA 6 /* Sparse data stream */
#define STREAM_SPARSE_GZIP_DATA 7
#define STREAM_PROGRAM_NAMES 8 /* program names for program data */
#define STREAM_PROGRAM_DATA 9 /* Data needing program */
#define STREAM_SHA1_SIGNATURE 10 /* SHA1 signature for the file */
#define STREAM_WIN32_DATA 11 /* Win32 BackupRead data */
#define STREAM_WIN32_GZIP_DATA 12 /* Gzipped Win32 BackupRead data */
#define STREAM_MACOS_FORK_DATA 13 /* Mac resource fork */
#define STREAM_HFSPLUS_ATTRIBUTES 14 /* Mac OS extra attributes */
#define STREAM_UNIX_ATTRIBUTES_ACCESS_ACL 15 /* Standard ACL attributes on UNIX */
#define STREAM_UNIX_ATTRIBUTES_DEFAULT_ACL 16 /* Default ACL attributes on UNIX */
</PRE>
<P>
</DD>
<DT><STRONG>The <B>DataSize</B> </STRONG></DT>
<DD><A NAME="2964"></A>
is the size in bytes of the binary data record that follows the Session
Record header. The Storage Daemon has no idea of the actual contents of the
binary data record. For standard Unix files, the data record typically
contains the file attributes or the file data. For a sparse file the first
64 bits of the file data contains the storage address for the data block.
</DD>
</DL>
<P>
The Record Header is never split across two blocks. If there is not enough
room in a block for the full Record Header, the block is padded to the end
with zeros and the Record Header begins in the next block. The data record, on
the other hand, may be split across multiple blocks and even multiple physical
volumes. When a data record is split, the second (and possibly subsequent)
piece of the data is preceded by a new Record Header. Thus each piece of data
is always immediately preceded by a Record Header. When reading a record, if
Bacula finds only part of the data in the first record, it will automatically
read the next record and concatenate the data record to form a full data
record.
<P>
<H2><A NAME="SECTION000118000000000000000"></A>
<A NAME="2967"></A>
<A NAME="2968"></A>
<BR>
Version BB02 Block Header
</H2>
<A NAME="2971"></A>
<P>
Each session or Job has its own private block. As a consequence, the SessionId
and SessionTime are written once in each Block Header and not in the Record
Header. So, the second and current version of the Block Header BB02 is:
<P>
<PRE>
uint32_t CheckSum; /* Block check sum */
uint32_t BlockSize; /* Block byte size including the header */
uint32_t BlockNumber; /* Block number */
char ID[4] = "BB02"; /* Identification and block level */
uint32_t VolSessionId; /* Applies to all records */
uint32_t VolSessionTime; /* contained in this block */
</PRE>
<P>
As with the previous version, the BB02 Block header is a fixed length and
fixed format and is followed by Record Headers and Record Data. The CheckSum
field is a 32 bit CRC checksum of the block data and the block header but not
including the CheckSum field. The Block Header is always immediately followed
by a Record Header. If the tape is damaged, a Bacula utility will be able to
recover as much information as possible from the tape by recovering blocks
which are valid. The Block header is written using the Bacula serialization
routines and thus is guaranteed to be in machine independent format.
<P>
<H2><A NAME="SECTION000119000000000000000"></A>
<A NAME="2975"></A>
<A NAME="2976"></A>
<BR>
Version 2 Record Header
</H2>
<A NAME="2979"></A>
<P>
Version 2 Record Header is written to the medium when using Version BB02 Block
Headers. The memory representation of the record is identical to the old BB01
Record Header, but on the storage medium, the first two fields, namely
VolSessionId and VolSessionTime are not written. The Block Header is filled
with these values when the First user record is written (i.e. non label
record) so that when the block is written, it will have the current and unique
VolSessionId and VolSessionTime. On reading each record from the Block, the
VolSessionId and VolSessionTime is filled in the Record Header from the Block
Header.
<P>
<H2><A NAME="SECTION0001110000000000000000"></A>
<A NAME="2981"></A>
<A NAME="2982"></A>
<BR>
Volume Label Format
</H2>
<A NAME="2985"></A>
<P>
Tape volume labels are created by the Storage daemon in response to a <B>label</B> command given to the Console program, or alternatively by the <B>btape</B> program. created. Each volume is labeled with the following information
using the Bacula serialization routines, which guarantee machine byte order
independence.
<P>
For Bacula versions 1.27 and later, the Volume Label Format is:
<P>
<PRE>
char Id[32]; /* Bacula 1.0 Immortal\n */
uint32_t VerNum; /* Label version number */
/* VerNum 11 and greater Bacula 1.27 and later */
btime_t label_btime; /* Time/date tape labeled */
btime_t write_btime; /* Time/date tape first written */
/* The following are 0 in VerNum 11 and greater */
float64_t write_date; /* Date this label written */
float64_t write_time; /* Time this label written */
char VolName[128]; /* Volume name */
char PrevVolName[128]; /* Previous Volume Name */
char PoolName[128]; /* Pool name */
char PoolType[128]; /* Pool type */
char MediaType[128]; /* Type of this media */
char HostName[128]; /* Host name of writing computer */
char LabelProg[32]; /* Label program name */
char ProgVersion[32]; /* Program version */
char ProgDate[32]; /* Program build date/time */
</PRE>
<P>
Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label, ...)
is stored in the record FileIndex field of the Record Header and does not
appear in the data part of the record.
<P>
<H2><A NAME="SECTION0001111000000000000000"></A>
<A NAME="2991"></A>
<A NAME="2992"></A>
<BR>
Session Label
</H2>
<A NAME="2995"></A>
<P>
The Session Label is written at the beginning and end of each session as well
as the last record on the physical medium. It has the following binary format:
<P>
<PRE>
char Id[32]; /* Bacula Immortal ... */
uint32_t VerNum; /* Label version number */
uint32_t JobId; /* Job id */
uint32_t VolumeIndex; /* sequence no of vol */
/* Prior to VerNum 11 */
float64_t write_date; /* Date this label written */
/* VerNum 11 and greater */
btime_t write_btime; /* time/date record written */
/* The following is zero VerNum 11 and greater */
float64_t write_time; /* Time this label written */
char PoolName[128]; /* Pool name */
char PoolType[128]; /* Pool type */
char JobName[128]; /* base Job name */
char ClientName[128];
/* Added in VerNum 10 */
char Job[128]; /* Unique Job name */
char FileSetName[128]; /* FileSet name */
uint32_t JobType;
uint32_t JobLevel;
</PRE>
<P>
In addition, the EOS label contains:
<P>
<PRE>
/* The remainder are part of EOS label only */
uint32_t JobFiles;
uint64_t JobBytes;
uint32_t start_block;
uint32_t end_block;
uint32_t start_file;
uint32_t end_file;
uint32_t JobErrors;
</PRE>
<P>
In addition, for VerNum greater than 10, the EOS label contains (in addition
to the above):
<P>
<PRE>
uint32_t JobStatus /* Job termination code */
</PRE>
<P>
: Note, the LabelType (Volume Label, Volume PreLabel, Session Start Label,
...) is stored in the record FileIndex field and does not appear in the data
part of the record. Also, the Stream field of the Record Header contains the
JobId. This permits quick filtering without actually reading all the session
data in many cases.
<P>
<H2><A NAME="SECTION0001112000000000000000"></A>
<A NAME="3003"></A>
<A NAME="3004"></A>
<BR>
Overall Storage Format
</H2>
<A NAME="3007"></A>
<P>
<PRE>
Current Bacula Tape Format
6 June 2001
Version BB02 added 28 September 2002
Version BB01 is the old deprecated format.
A Bacula tape is composed of tape Blocks. Each block
has a Block header followed by the block data. Block
Data consists of Records. Records consist of Record
Headers followed by Record Data.
:=======================================================:
| |
| Block Header (24 bytes) |
| |
|-------------------------------------------------------|
| |
| Record Header (12 bytes) |
| |
|-------------------------------------------------------|
| |
| Record Data |
| |
|-------------------------------------------------------|
| |
| Record Header (12 bytes) |
| |
|-------------------------------------------------------|
| |
| ... |
Block Header: the first item in each block. The format is
shown below.
Partial Data block: occurs if the data from a previous
block spills over to this block (the normal case except
for the first block on a tape). However, this partial
data block is always preceded by a record header.
Record Header: identifies the Volume Session, the Stream
and the following Record Data size. See below for format.
Record data: arbitrary binary data.
Block Header Format BB02
:=======================================================:
| CheckSum (uint32_t) |
|-------------------------------------------------------|
| BlockSize (uint32_t) |
|-------------------------------------------------------|
| BlockNumber (uint32_t) |
|-------------------------------------------------------|
| "BB02" (char [4]) |
|-------------------------------------------------------|
| VolSessionId (uint32_t) |
|-------------------------------------------------------|
| VolSessionTime (uint32_t) |
:=======================================================:
BBO2: Serves to identify the block as a
Bacula block and also servers as a block format identifier
should we ever need to change the format.
BlockSize: is the size in bytes of the block. When reading
back a block, if the BlockSize does not agree with the
actual size read, Bacula discards the block.
CheckSum: a checksum for the Block.
BlockNumber: is the sequential block number on the tape.
VolSessionId: a unique sequential number that is assigned
by the Storage Daemon to a particular Job.
This number is sequential since the start
of execution of the daemon.
VolSessionTime: the time/date that the current execution
of the Storage Daemon started. It assures
that the combination of VolSessionId and
VolSessionTime is unique for all jobs
written to the tape, even if there was a
machine crash between two writes.
Record Header Format BB02
:=======================================================:
| FileIndex (int32_t) |
|-------------------------------------------------------|
| Stream (int32_t) |
|-------------------------------------------------------|
| DataSize (uint32_t) |
:=======================================================:
FileIndex: a sequential file number within a job. The
Storage daemon enforces this index to be
greater than zero and sequential. Note,
however, that the File daemon may send
multiple Streams for the same FileIndex.
The Storage Daemon uses negative FileIndices
to identify Session Start and End labels
as well as the End of Volume labels.
Stream: defined by the File daemon and is intended to be
used to identify separate parts of the data
saved for each file (attributes, file data,
...). The Storage Daemon has no idea of
what a Stream is or what it contains.
DataSize: the size in bytes of the binary data record
that follows the Session Record header.
The Storage Daemon has no idea of the
actual contents of the binary data record.
For standard Unix files, the data record
typically contains the file attributes or
the file data. For a sparse file
the first 64 bits of the data contains
the storage address for the data block.
Volume Label
:=======================================================:
| Id (32 bytes) |
|-------------------------------------------------------|
| VerNum (uint32_t) |
|-------------------------------------------------------|
| label_date (float64_t) |
| label_btime (btime_t VerNum 11 |
|-------------------------------------------------------|
| label_time (float64_t) |
| write_btime (btime_t VerNum 11 |
|-------------------------------------------------------|
| write_date (float64_t) |
| 0 (float64_t) VerNum 11 |
|-------------------------------------------------------|
| write_time (float64_t) |
| 0 (float64_t) VerNum 11 |
|-------------------------------------------------------|
| VolName (128 bytes) |
|-------------------------------------------------------|
| PrevVolName (128 bytes) |
|-------------------------------------------------------|
| PoolName (128 bytes) |
|-------------------------------------------------------|
| PoolType (128 bytes) |
|-------------------------------------------------------|
| MediaType (128 bytes) |
|-------------------------------------------------------|
| HostName (128 bytes) |
|-------------------------------------------------------|
| LabelProg (32 bytes) |
|-------------------------------------------------------|
| ProgVersion (32 bytes) |
|-------------------------------------------------------|
| ProgDate (32 bytes) |
|-------------------------------------------------------|
:=======================================================:
Id: 32 byte Bacula identifier "Bacula 1.0 immortal\n"
(old version also recognized:)
Id: 32 byte Bacula identifier "Bacula 0.9 mortal\n"
LabelType (Saved in the FileIndex of the Header record).
PRE_LABEL -1 Volume label on unwritten tape
VOL_LABEL -2 Volume label after tape written
EOM_LABEL -3 Label at EOM (not currently implemented)
SOS_LABEL -4 Start of Session label (format given below)
EOS_LABEL -5 End of Session label (format given below)
VerNum: 11
label_date: Julian day tape labeled
label_time: Julian time tape labeled
write_date: Julian date tape first used (data written)
write_time: Julian time tape first used (data written)
VolName: "Physical" Volume name
PrevVolName: The VolName of the previous tape (if this tape is
a continuation of the previous one).
PoolName: Pool Name
PoolType: Pool Type
MediaType: Media Type
HostName: Name of host that is first writing the tape
LabelProg: Name of the program that labeled the tape
ProgVersion: Version of the label program
ProgDate: Date Label program built
Session Label
:=======================================================:
| Id (32 bytes) |
|-------------------------------------------------------|
| VerNum (uint32_t) |
|-------------------------------------------------------|
| JobId (uint32_t) |
|-------------------------------------------------------|
| write_btime (btime_t) VerNum 11 |
|-------------------------------------------------------|
| 0 (float64_t) VerNum 11 |
|-------------------------------------------------------|
| PoolName (128 bytes) |
|-------------------------------------------------------|
| PoolType (128 bytes) |
|-------------------------------------------------------|
| JobName (128 bytes) |
|-------------------------------------------------------|
| ClientName (128 bytes) |
|-------------------------------------------------------|
| Job (128 bytes) |
|-------------------------------------------------------|
| FileSetName (128 bytes) |
|-------------------------------------------------------|
| JobType (uint32_t) |
|-------------------------------------------------------|
| JobLevel (uint32_t) |
|-------------------------------------------------------|
| FileSetMD5 (50 bytes) VerNum 11 |
|-------------------------------------------------------|
Additional fields in End Of Session Label
|-------------------------------------------------------|
| JobFiles (uint32_t) |
|-------------------------------------------------------|
| JobBytes (uint32_t) |
|-------------------------------------------------------|
| start_block (uint32_t) |
|-------------------------------------------------------|
| end_block (uint32_t) |
|-------------------------------------------------------|
| start_file (uint32_t) |
|-------------------------------------------------------|
| end_file (uint32_t) |
|-------------------------------------------------------|
| JobErrors (uint32_t) |
|-------------------------------------------------------|
| JobStatus (uint32_t) VerNum 11 |
:=======================================================:
* => fields deprecated
Id: 32 byte Bacula Identifier "Bacula 1.0 immortal\n"
LabelType (in FileIndex field of Header):
EOM_LABEL -3 Label at EOM
SOS_LABEL -4 Start of Session label
EOS_LABEL -5 End of Session label
VerNum: 11
JobId: JobId
write_btime: Bacula time/date this tape record written
write_date: Julian date tape this record written - deprecated
write_time: Julian time tape this record written - deprecated.
PoolName: Pool Name
PoolType: Pool Type
MediaType: Media Type
ClientName: Name of File daemon or Client writing this session
Not used for EOM_LABEL.
</PRE>
<P>
<H2><A NAME="SECTION0001113000000000000000"></A>
<A NAME="3011"></A>
<A NAME="3012"></A>
<BR>
Unix File Attributes
</H2>
<A NAME="3015"></A>
<P>
The Unix File Attributes packet consists of the following:
<P>
<File-Index> <Type>
<Filename>@<File-Attributes>@<Link>
@<Extended-Attributes@> where
<P>
<DL>
<DT><STRONG>@</STRONG></DT>
<DD>represents a byte containing a binary zero.
<P>
</DD>
<DT><STRONG>FileIndex</STRONG></DT>
<DD><A NAME="3029"></A>
is the sequential file index starting from one assigned by the File daemon.
<P>
</DD>
<DT><STRONG>Type</STRONG></DT>
<DD><A NAME="3030"></A>
is one of the following:
<P>
<PRE>
#define FT_LNKSAVED 1 /* hard link to file already saved */
#define FT_REGE 2 /* Regular file but empty */
#define FT_REG 3 /* Regular file */
#define FT_LNK 4 /* Soft Link */
#define FT_DIR 5 /* Directory */
#define FT_SPEC 6 /* Special file -- chr, blk, fifo, sock */
#define FT_NOACCESS 7 /* Not able to access */
#define FT_NOFOLLOW 8 /* Could not follow link */
#define FT_NOSTAT 9 /* Could not stat file */
#define FT_NOCHG 10 /* Incremental option, file not changed */
#define FT_DIRNOCHG 11 /* Incremental option, directory not changed */
#define FT_ISARCH 12 /* Trying to save archive file */
#define FT_NORECURSE 13 /* No recursion into directory */
#define FT_NOFSCHG 14 /* Different file system, prohibited */
#define FT_NOOPEN 15 /* Could not open directory */
#define FT_RAW 16 /* Raw block device */
#define FT_FIFO 17 /* Raw fifo device */
</PRE>
<P>
</DD>
<DT><STRONG>Filename</STRONG></DT>
<DD><A NAME="3033"></A>
is the fully qualified filename.
<P>
</DD>
<DT><STRONG>File-Attributes</STRONG></DT>
<DD><A NAME="3034"></A>
consists of the 13 fields of the stat() buffer in ASCII base64 format
separated by spaces. These fields and their meanings are shown below. This
stat() packet is in Unix format, and MUST be provided (constructed) for ALL
systems.
<P>
</DD>
<DT><STRONG>Link</STRONG></DT>
<DD><A NAME="3035"></A>
when the FT code is FT_LNK or FT_LNKSAVED, the item in question is a Unix
link, and this field contains the fully qualified link name. When the FT code
is not FT_LNK or FT_LNKSAVED, this field is null.
<P>
</DD>
<DT><STRONG>Extended-Attributes</STRONG></DT>
<DD><A NAME="3036"></A>
The exact format of this field is operating system dependent. It contains
additional or extended attributes of a system dependent nature. Currently,
this field is used only on WIN32 systems where it contains a ASCII base64
representation of the WIN32_FILE_ATTRIBUTE_DATA structure as defined by
Windows. The fields in the base64 representation of this structure are like
the File-Attributes separated by spaces.
</DD>
</DL>
<P>
The File-attributes consist of the following:
<P>
<A NAME="3040"></A>
<TABLE CELLPADDING=3 BORDER="1">
<TR><TD ALIGN="CENTER" COLSPAN=1><B>Field No. </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Stat Name </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Unix </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>Win98/NT </B></TD>
<TD ALIGN="CENTER" COLSPAN=1><B>MacOS </B></TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>1</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_dev</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Device number of filesystem</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Drive number</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>vRefNum</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>2</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_ino</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Inode number</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Always 0</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>fileID/dirID</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>3</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_mode</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>File mode</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>File mode</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>777 dirs/apps; 666 docs; 444 locked docs</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>4</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_nlink</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Number of links to the file</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Number of link (only on NTFS)</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Always 1</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>5</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_uid</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Owner ID</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Always 0</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Always 0</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>6</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_gid</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Group ID</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Always 0</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Always 0</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>7</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_rdev</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Device ID for special files</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Drive No.</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Always 0</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>8</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_size</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>File size in bytes</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>File
size in bytes</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Data fork file size in bytes</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>9</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_blksize</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Preferred block size</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Always 0</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Preferred block size</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>10</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_blocks</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Number of blocks allocated</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Always 0</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Number of blocks allocated</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>11</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_atime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Last access time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Last access time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Last access time -66 years</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>12</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_mtime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Last modify time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Last modify time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>Last access time -66 years</TD>
</TR>
<TR><TD ALIGN="CENTER" COLSPAN=1>13</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=50>st_ctime</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>Inode change time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=72>File create time since epoch</TD>
<TD ALIGN="LEFT" VALIGN="TOP" WIDTH=101>File create time -66 years</TD>
</TR>
</TABLE>
<P>
<H2><A NAME="SECTION0001114000000000000000"></A>
<A NAME="3155"></A>
<A NAME="3156"></A>
<BR>
Old Depreciated Tape Format
</H2>
<A NAME="3159"></A>
<P>
The format of the Block Header (version 1.26 and earlier) is:
<P>
<PRE>
uint32_t CheckSum; /* Block check sum */
uint32_t BlockSize; /* Block byte size including the header */
uint32_t BlockNumber; /* Block number */
char ID[4] = "BB01"; /* Identification and block level */
</PRE>
<P>
The format of the Record Header (version 1.26 or earlier) is:
<P>
<PRE>
uint32_t VolSessionId; /* Unique ID for this session */
uint32_t VolSessionTime; /* Start time/date of session */
int32_t FileIndex; /* File index supplied by File daemon */
int32_t Stream; /* Stream number supplied by File daemon */
uint32_t DataSize; /* size of following data record in bytes */
</PRE>
<P>
<PRE>
Current Bacula Tape Format
6 June 2001
Version BB01 is the old deprecated format.
A Bacula tape is composed of tape Blocks. Each block
has a Block header followed by the block data. Block
Data consists of Records. Records consist of Record
Headers followed by Record Data.
:=======================================================:
| |
| Block Header |
| (16 bytes version BB01) |
|-------------------------------------------------------|
| |
| Record Header |
| (20 bytes version BB01) |
|-------------------------------------------------------|
| |
| Record Data |
| |
|-------------------------------------------------------|
| |
| Record Header |
| (20 bytes version BB01) |
|-------------------------------------------------------|
| |
| ... |
Block Header: the first item in each block. The format is
shown below.
Partial Data block: occurs if the data from a previous
block spills over to this block (the normal case except
for the first block on a tape). However, this partial
data block is always preceded by a record header.
Record Header: identifies the Volume Session, the Stream
and the following Record Data size. See below for format.
Record data: arbitrary binary data.
Block Header Format BB01 (deprecated)
:=======================================================:
| CheckSum (uint32_t) |
|-------------------------------------------------------|
| BlockSize (uint32_t) |
|-------------------------------------------------------|
| BlockNumber (uint32_t) |
|-------------------------------------------------------|
| "BB01" (char [4]) |
:=======================================================:
BBO1: Serves to identify the block as a
Bacula block and also servers as a block format identifier
should we ever need to change the format.
BlockSize: is the size in bytes of the block. When reading
back a block, if the BlockSize does not agree with the
actual size read, Bacula discards the block.
CheckSum: a checksum for the Block.
BlockNumber: is the sequential block number on the tape.
VolSessionId: a unique sequential number that is assigned
by the Storage Daemon to a particular Job.
This number is sequential since the start
of execution of the daemon.
VolSessionTime: the time/date that the current execution
of the Storage Daemon started. It assures
that the combination of VolSessionId and
VolSessionTime is unique for all jobs
written to the tape, even if there was a
machine crash between two writes.
Record Header Format BB01 (deprecated)
:=======================================================:
| VolSessionId (uint32_t) |
|-------------------------------------------------------|
| VolSessionTime (uint32_t) |
|-------------------------------------------------------|
| FileIndex (int32_t) |
|-------------------------------------------------------|
| Stream (int32_t) |
|-------------------------------------------------------|
| DataSize (uint32_t) |
:=======================================================:
VolSessionId: a unique sequential number that is assigned
by the Storage Daemon to a particular Job.
This number is sequential since the start
of execution of the daemon.
VolSessionTime: the time/date that the current execution
of the Storage Daemon started. It assures
that the combination of VolSessionId and
VolSessionTime is unique for all jobs
written to the tape, even if there was a
machine crash between two writes.
FileIndex: a sequential file number within a job. The
Storage daemon enforces this index to be
greater than zero and sequential. Note,
however, that the File daemon may send
multiple Streams for the same FileIndex.
The Storage Daemon uses negative FileIndices
to identify Session Start and End labels
as well as the End of Volume labels.
Stream: defined by the File daemon and is intended to be
used to identify separate parts of the data
saved for each file (attributes, file data,
...). The Storage Daemon has no idea of
what a Stream is or what it contains.
DataSize: the size in bytes of the binary data record
that follows the Session Record header.
The Storage Daemon has no idea of the
actual contents of the binary data record.
For standard Unix files, the data record
typically contains the file attributes or
the file data. For a sparse file
the first 64 bits of the data contains
the storage address for the data block.
Volume Label
:=======================================================:
| Id (32 bytes) |
|-------------------------------------------------------|
| VerNum (uint32_t) |
|-------------------------------------------------------|
| label_date (float64_t) |
|-------------------------------------------------------|
| label_time (float64_t) |
|-------------------------------------------------------|
| write_date (float64_t) |
|-------------------------------------------------------|
| write_time (float64_t) |
|-------------------------------------------------------|
| VolName (128 bytes) |
|-------------------------------------------------------|
| PrevVolName (128 bytes) |
|-------------------------------------------------------|
| PoolName (128 bytes) |
|-------------------------------------------------------|
| PoolType (128 bytes) |
|-------------------------------------------------------|
| MediaType (128 bytes) |
|-------------------------------------------------------|
| HostName (128 bytes) |
|-------------------------------------------------------|
| LabelProg (32 bytes) |
|-------------------------------------------------------|
| ProgVersion (32 bytes) |
|-------------------------------------------------------|
| ProgDate (32 bytes) |
|-------------------------------------------------------|
:=======================================================:
Id: 32 byte Bacula identifier "Bacula 1.0 immortal\n"
(old version also recognized:)
Id: 32 byte Bacula identifier "Bacula 0.9 mortal\n"
LabelType (Saved in the FileIndex of the Header record).
PRE_LABEL -1 Volume label on unwritten tape
VOL_LABEL -2 Volume label after tape written
EOM_LABEL -3 Label at EOM (not currently implemented)
SOS_LABEL -4 Start of Session label (format given below)
EOS_LABEL -5 End of Session label (format given below)
label_date: Julian day tape labeled
label_time: Julian time tape labeled
write_date: Julian date tape first used (data written)
write_time: Julian time tape first used (data written)
VolName: "Physical" Volume name
PrevVolName: The VolName of the previous tape (if this tape is
a continuation of the previous one).
PoolName: Pool Name
PoolType: Pool Type
MediaType: Media Type
HostName: Name of host that is first writing the tape
LabelProg: Name of the program that labeled the tape
ProgVersion: Version of the label program
ProgDate: Date Label program built
Session Label
:=======================================================:
| Id (32 bytes) |
|-------------------------------------------------------|
| VerNum (uint32_t) |
|-------------------------------------------------------|
| JobId (uint32_t) |
|-------------------------------------------------------|
| *write_date (float64_t) VerNum 10 |
|-------------------------------------------------------|
| *write_time (float64_t) VerNum 10 |
|-------------------------------------------------------|
| PoolName (128 bytes) |
|-------------------------------------------------------|
| PoolType (128 bytes) |
|-------------------------------------------------------|
| JobName (128 bytes) |
|-------------------------------------------------------|
| ClientName (128 bytes) |
|-------------------------------------------------------|
| Job (128 bytes) |
|-------------------------------------------------------|
| FileSetName (128 bytes) |
|-------------------------------------------------------|
| JobType (uint32_t) |
|-------------------------------------------------------|
| JobLevel (uint32_t) |
|-------------------------------------------------------|
| FileSetMD5 (50 bytes) VerNum 11 |
|-------------------------------------------------------|
Additional fields in End Of Session Label
|-------------------------------------------------------|
| JobFiles (uint32_t) |
|-------------------------------------------------------|
| JobBytes (uint32_t) |
|-------------------------------------------------------|
| start_block (uint32_t) |
|-------------------------------------------------------|
| end_block (uint32_t) |
|-------------------------------------------------------|
| start_file (uint32_t) |
|-------------------------------------------------------|
| end_file (uint32_t) |
|-------------------------------------------------------|
| JobErrors (uint32_t) |
|-------------------------------------------------------|
| JobStatus (uint32_t) VerNum 11 |
:=======================================================:
* => fields deprecated
Id: 32 byte Bacula Identifier "Bacula 1.0 immortal\n"
LabelType (in FileIndex field of Header):
EOM_LABEL -3 Label at EOM
SOS_LABEL -4 Start of Session label
EOS_LABEL -5 End of Session label
VerNum: 11
JobId: JobId
write_btime: Bacula time/date this tape record written
write_date: Julian date tape this record written - deprecated
write_time: Julian time tape this record written - deprecated.
PoolName: Pool Name
PoolType: Pool Type
MediaType: Media Type
ClientName: Name of File daemon or Client writing this session
Not used for EOM_LABEL.
</PRE>
<P>
<H1><A NAME="SECTION000120000000000000000"></A>
<A NAME="_ChapterStart1"></A><A NAME="3433"></A>
<A NAME="3434"></A>
<BR>
Bacula Porting Notes
</H1>
<A NAME="3437"></A>
<P>
This document is intended mostly for developers who wish to port Bacula to a
system that is not <B>officially</B> supported.
<P>
It is hoped that Bacula clients will eventually run on every imaginable system
that needs backing up (perhaps even a Palm). It is also hoped that the Bacula
Directory and Storage daemons will run on every system capable of supporting
them.
<P>
<H2><A NAME="SECTION000121000000000000000"></A>
<A NAME="3440"></A>
<A NAME="3441"></A>
<BR>
Porting Requirements
</H2>
<A NAME="3444"></A>
<P>
In General, the following holds true:
<P>
<UL>
<LI><B>Bacula</B> has been compiled and run on Linux RedHat, FreeBSD, and
Solaris systems.
</LI>
<LI>In addition, clients exist on Win32, and Irix
</LI>
<LI>It requires GNU C++ to compile. You can try with other compilers, but
you are on your own. The Irix client is built with the Irix complier, but, in
general, you will need GNU.
</LI>
<LI>Your compiler must provide support for 64 bit signed and unsigned
integers.
</LI>
<LI>You will need a recent copy of the <B>autoconf</B> tools loaded on your
system (version 2.13 or later). The <B>autoconf</B> tools are used to build
the configuration program, but are not part of the Bacula source
distribution.
</LI>
<LI>There are certain third party packages that Bacula needs. Except for
MySQL, they can all be found in the <B>depkgs</B> and <B>depkgs1</B> releases.
</LI>
<LI>To build the Win32 binaries, we use Microsoft VC++ standard
2003. Please see the instructions in
bacula-source/src/win32/README.win32 for more details. If you
want to use VC++ Express, please see README.vc8. Our build is
done under the most recent version of Cygwin, but Cygwin is
not used in the Bacula binaries that are produced.
Unfortunately, we do not have the resources to help you build
your own version of the Win32 FD, so you are pretty much on
your own. You can ask the bacula-devel list for help, but
please don't expect much.
</LI>
<LI><B>Bacula</B> requires a good implementation of pthreads to work.
</LI>
<LI>The source code has been written with portability in mind and is mostly
POSIX compatible. Thus porting to any POSIX compatible operating system
should be relatively easy.
</LI>
</UL>
<P>
<H2><A NAME="SECTION000122000000000000000"></A>
<A NAME="3454"></A>
<A NAME="3455"></A>
<BR>
Steps to Take for Porting
</H2>
<A NAME="3458"></A>
<P>
<UL>
<LI>The first step is to ensure that you have version 2.13 or later of the
<B>autoconf</B> tools loaded. You can skip this step, but making changes to
the configuration program will be difficult or impossible.
</LI>
<LI>The run a <B>./configure</B> command in the main source directory and
examine the output. It should look something like the following:
<P>
<PRE>
Configuration on Mon Oct 28 11:42:27 CET 2002:
Host: i686-pc-linux-gnu -- redhat 7.3
Bacula version: 1.27 (26 October 2002)
Source code location: .
Install binaries: /sbin
Install config files: /etc/bacula
C Compiler: gcc
C++ Compiler: c++
Compiler flags: -g -O2
Linker flags:
Libraries: -lpthread
Statically Linked Tools: no
Database found: no
Database type: Internal
Database lib:
Job Output Email: root@localhost
Traceback Email: root@localhost
SMTP Host Address: localhost
Director Port 9101
File daemon Port 9102
Storage daemon Port 9103
Working directory /etc/bacula/working
SQL binaries Directory
Large file support: yes
readline support: yes
cweb support: yes /home/kern/bacula/depkgs/cweb
TCP Wrappers support: no
ZLIB support: yes
enable-smartalloc: yes
enable-gnome: no
gmp support: yes
</PRE>
<P>
The details depend on your system. The first thing to check is that it
properly identified your host on the <B>Host:</B> line. The first part (added
in version 1.27) is the GNU four part identification of your system. The part
after the -- is your system and the system version. Generally, if your system
is not yet supported, you must correct these.
</LI>
<LI>If the <B>./configure</B> does not function properly, you must determine
the cause and fix it. Generally, it will be because some required system
routine is not available on your machine.
</LI>
<LI>To correct problems with detection of your system type or with routines
and libraries, you must edit the file <B> <bacula-src>/autoconf/configure.in</B>. This is the ``source'' from
which <B>configure</B> is built. In general, most of the changes for your
system will be made in <B>autoconf/aclocal.m4</B> in the routine <B>BA_CHECK_OPSYS</B> or in the routine <B>BA_CHECK_OPSYS_DISTNAME</B>. I have
already added the necessary code for most systems, but if yours shows up as
<B>unknown</B> you will need to make changes. Then as mentioned above, you
will need to set a number of system dependent items in <B>configure.in</B> in
the <B>case</B> statement at approximately line 1050 (depending on the Bacula
release).
</LI>
<LI>The items to in the case statement that corresponds to your system are
the following:
<P>
<UL>
<LI>DISTVER -- set to the version of your operating system. Typically some
form of <B>uname</B> obtains it.
</LI>
<LI>TAPEDRIVE -- the default tape drive. Not too important as the user can
set it as an option.
</LI>
<LI>PSCMD -- set to the <B>ps</B> command that will provide the PID in the
first field and the program name in the second field. If this is not set
properly, the <B>bacula stop</B> script will most likely not be able to stop
Bacula in all cases.
</LI>
<LI>hostname -- command to return the base host name (non-qualified) of
your system. This is generally the machine name. Not too important as the
user can correct this in his configuration file.
</LI>
<LI>CFLAGS -- set any special compiler flags needed. Many systems need a
special flag to make pthreads work. See cygwin for an example.
</LI>
<LI>LDFLAGS -- set any special loader flags. See cygwin for an example.
</LI>
<LI>PTHREAD_LIB -- set for any special pthreads flags needed during
linking. See freebsd as an example.
</LI>
<LI>lld -- set so that a ``long long int'' will be properly edited in a
printf() call.
</LI>
<LI>llu -- set so that a ``long long unsigned'' will be properly edited in
a printf() call.
</LI>
<LI>PFILES -- set to add any files that you may define is your platform
subdirectory. These files are used for installation of automatic system
startup of Bacula daemons.
</LI>
</UL>
<P>
</LI>
<LI>To rebuild a new version of <B>configure</B> from a changed <B> autoconf/configure.in</B> you enter <B>make configure</B> in the top level Bacula
source directory. You must have done a ./configure prior to trying to rebuild
the configure script or it will get into an infinite loop.
</LI>
<LI>If the <B>make configure</B> gets into an infinite loop, ctl-c it, then
do <B>./configure</B> (no options are necessary) and retry the <B>make
configure</B>, which should now work.
</LI>
<LI>To rebuild <B>configure</B> you will need to have <B>autoconf</B> version
2.57-3 or higher loaded. Older versions of autoconf will complain about
unknown or bad options, and won't work.
</LI>
<LI>After you have a working <B>configure</B> script, you may need to make a
few system dependent changes to the way Bacula works. Generally, these are
done in <B>src/baconfig.h</B>. You can find a few examples of system dependent
changes toward the end of this file. For example, on Irix systems, there is
no definition for <B>socklen_t</B>, so it is made in this file. If your
system has structure alignment requirements, check the definition of BALIGN
in this file. Currently, all Bacula allocated memory is aligned on a <B>double</B> boundary.
</LI>
<LI>If you are having problems with Bacula's type definitions, you might
look at <B>src/bc_types.h</B> where all the types such as <B>uint32_t</B>,
<B>uint64_t</B>, etc. that Bacula uses are defined.
</LI>
</UL>
<P>
<H1><A NAME="SECTION000130000000000000000"></A>
<A NAME="_ChapterStart"></A>
<BR>
Implementing a Bacula GUI Interface
</H1>
<A NAME="3601"></A>
<A NAME="3602"></A>
<A NAME="3605"></A>
<P>
<H2><A NAME="SECTION000131000000000000000">
General</A>
</H2>
<A NAME="3607"></A>
<A NAME="3610"></A>
<P>
This document is intended mostly for developers who wish to develop a new GUI
interface to <B>Bacula</B>.
<P>
<H3><A NAME="SECTION000131100000000000000">
Minimal Code in Console Program</A>
</H3>
<A NAME="3613"></A>
<A NAME="3614"></A>
<A NAME="3617"></A>
<P>
Until now, I have kept all the Catalog code in the Directory (with the
exception of dbcheck and bscan). This is because at some point I would like to
add user level security and access. If we have code spread everywhere such as
in a GUI this will be more difficult. The other advantage is that any code you
add to the Director is automatically available to both the tty console program
and the GNOME program. The major disadvantage is it increases the size of the
code -- however, compared to Networker the Bacula Director is really tiny.
<P>
<H3><A NAME="SECTION000131200000000000000">
GUI Interface is Difficult</A>
</H3>
<A NAME="3619"></A>
<A NAME="3620"></A>
<A NAME="3623"></A>
<P>
Interfacing to an interactive program such as Bacula can be very difficult
because the interfacing program must interpret all the prompts that may come.
This can be next to impossible. There are are a number of ways that Bacula is
designed to facilitate this:
<P>
<UL>
<LI>The Bacula network protocol is packet based, and thus pieces of
information sent can be ASCII or binary.
</LI>
<LI>The packet interface permits knowing where the end of a list is.
</LI>
<LI>The packet interface permits special ``signals'' to be passed rather
than data.
</LI>
<LI>The Director has a number of commands that are non-interactive. They
all begin with a period, and provide things such as the list of all Jobs,
list of all Clients, list of all Pools, list of all Storage, ... Thus the GUI
interface can get to virtually all information that the Director has in a
deterministic way. See <bacula-source>/src/dird/ua_dotcmds.c for
more details on this.
</LI>
<LI>Most console commands allow all the arguments to be specified on the
command line: e.g. <B>run job=NightlyBackup level=Full</B>
</LI>
</UL>
<P>
One of the first things to overcome is to be able to establish a conversation
with the Director. Although you can write all your own code, it is probably
easier to use the Bacula subroutines. The following code is used by the
Console program to begin a conversation.
<P>
<PRE>
static BSOCK *UA_sock = NULL;
static JCR *jcr;
...
read-your-config-getting-address-and-pasword;
UA_sock = bnet_connect(NULL, 5, 15, "Director daemon", dir->address,
NULL, dir->DIRport, 0);
if (UA_sock == NULL) {
terminate_console(0);
return 1;
}
jcr.dir_bsock = UA_sock;
if (!authenticate_director(\&jcr, dir)) {
fprintf(stderr, "ERR=%s", UA_sock->msg);
terminate_console(0);
return 1;
}
read_and_process_input(stdin, UA_sock);
if (UA_sock) {
bnet_sig(UA_sock, BNET_TERMINATE); /* send EOF */
bnet_close(UA_sock);
}
exit 0;
</PRE>
<P>
Then the read_and_process_input routine looks like the following:
<P>
<PRE>
get-input-to-send-to-the-Director;
bnet_fsend(UA_sock, "%s", input);
stat = bnet_recv(UA_sock);
process-output-from-the-Director;
</PRE>
<P>
For a GUI program things will be a bit more complicated. Basically in the very
inner loop, you will need to check and see if any output is available on the
UA_sock. For an example, please take a look at the GNOME GUI interface code
in: <bacula-source&gt/src/gnome-console/console.c
<P>
<H1><A NAME="SECTION000140000000000000000"></A>
<A NAME="_Chapter_TLS"></A><A NAME="3674"></A>
<BR>
TLS
</H1>
<A NAME="3677"></A>
<P>
Written by Landon Fuller
<P>
<H2><A NAME="SECTION000141000000000000000"></A>
<A NAME="3679"></A>
<A NAME="3680"></A>
<BR>
Introduction to TLS
</H2>
<A NAME="3683"></A>
<P>
This patch includes all the back-end code necessary to add complete TLS
data encryption support to Bacula. In addition, support for TLS in
Console/Director communications has been added as a proof of concept.
Adding support for the remaining daemons will be straight-forward.
Supported features of this patchset include:
<P>
<UL>
<LI>Client/Server TLS Requirement Negotiation
</LI>
<LI>TLSv1 Connections with Server and Client Certificate
Validation
</LI>
<LI>Forward Secrecy Support via Diffie-Hellman Ephemeral Keying
</LI>
</UL>
<P>
This document will refer to both ``server'' and ``client'' contexts. These
terms refer to the accepting and initiating peer, respectively.
<P>
Diffie-Hellman anonymous ciphers are not supported by this patchset. The
use of DH anonymous ciphers increases the code complexity and places
explicit trust upon the two-way Cram-MD5 implementation. Cram-MD5 is
subject to known plaintext attacks, and is should be considered
considerably less secure than PKI certificate-based authentication.
<P>
Appropriate autoconf macros have been added to detect and use OpenSSL. Two
additional preprocessor defines have been added: <I>HAVE_TLS</I> and
<I>HAVE_OPENSSL</I>. All changes not specific to OpenSSL rely on
<I>HAVE_TLS</I>. OpenSSL-specific code is constrained to
<I>src/lib/tls.c</I> to facilitate the support of alternative TLS
implementations.
<P>
<H2><A NAME="SECTION000142000000000000000"></A>
<A NAME="3691"></A>
<A NAME="3692"></A>
<BR>
New Configuration Directives
</H2>
<A NAME="3695"></A>
<P>
Additional configuration directives have been added to both the Console and
Director resources. These new directives are defined as follows:
<P>
<UL>
<LI><U>TLS Enable</U> <I>(yes/no)</I>
Enable TLS support.
<P>
</LI>
<LI><U>TLS Require</U> <I>(yes/no)</I>
Require TLS connections.
<P>
</LI>
<LI><U>TLS Certificate</U> <I>(path)</I>
Path to PEM encoded TLS certificate. Used as either a client or server
certificate.
<P>
</LI>
<LI><U>TLS Key</U> <I>(path)</I>
Path to PEM encoded TLS private key. Must correspond with the TLS
certificate.
<P>
</LI>
<LI><U>TLS Verify Peer</U> <I>(yes/no)</I>
Verify peer certificate. Instructs server to request and verify the
client's x509 certificate. Any client certificate signed by a known-CA
will be accepted unless the TLS Allowed CN configuration directive is used.
Not valid in a client context.
<P>
</LI>
<LI><U>TLS Allowed CN</U> <I>(string list)</I>
Common name attribute of allowed peer certificates. If directive is
specified, all client certificates will be verified against this list.
This directive may be specified more than once. Not valid in a client
context.
<P>
</LI>
<LI><U>TLS CA Certificate File</U> <I>(path)</I>
Path to PEM encoded TLS CA certificate(s). Multiple certificates are
permitted in the file. One of <I>TLS CA Certificate File</I> or <I>TLS
CA Certificate Dir</I> are required in a server context if <U>TLS
Verify Peer</U> is also specified, and are always required in a client
context.
<P>
</LI>
<LI><U>TLS CA Certificate Dir</U> <I>(path)</I>
Path to TLS CA certificate directory. In the current implementation,
certificates must be stored PEM encoded with OpenSSL-compatible hashes.
One of <I>TLS CA Certificate File</I> or <I>TLS CA Certificate Dir</I> are
required in a server context if <I>TLS Verify Peer</I> is also specified,
and are always required in a client context.
<P>
</LI>
<LI><U>TLS DH File</U> <I>(path)</I>
Path to PEM encoded Diffie-Hellman parameter file. If this directive is
specified, DH ephemeral keying will be enabled, allowing for forward
secrecy of communications. This directive is only valid within a server
context. To generate the parameter file, you may use openssl:
<PRE>
openssl dhparam -out dh1024.pem -5 1024
</PRE>
</LI>
</UL>
<P>
<H2><A NAME="SECTION000143000000000000000"></A>
<A NAME="3725"></A>
<A NAME="3726"></A>
<BR>
TLS API Implementation
</H2>
<A NAME="3729"></A>
<P>
To facilitate the use of additional TLS libraries, all OpenSSL-specific
code has been implemented within <I>src/lib/tls.c</I>. In turn, a generic
TLS API is exported.
<P>
<H3><A NAME="SECTION000143100000000000000"></A>
<A NAME="3732"></A>
<A NAME="3733"></A>
<BR>
Library Initialization and Cleanup
</H3>
<A NAME="3736"></A>
<P>
<PRE>
int init_tls (void);
</PRE>
<P>
Performs TLS library initialization, including seeding of the PRNG. PRNG
seeding has not yet been implemented for win32.
<P>
<PRE>
int cleanup_tls (void);
</PRE>
<P>
Performs TLS library cleanup.
<P>
<H3><A NAME="SECTION000143200000000000000"></A>
<A NAME="3742"></A>
<A NAME="3743"></A>
<BR>
Manipulating TLS Contexts
</H3>
<A NAME="3746"></A>
<P>
<PRE>
TLS_CONTEXT *new_tls_context (const char *ca_certfile,
const char *ca_certdir, const char *certfile,
const char *keyfile, const char *dhfile, bool verify_peer);
</PRE>
<P>
Allocates and initalizes a new opaque <I>TLS_CONTEXT</I> structure. The
<I>TLS_CONTEXT</I> structure maintains default TLS settings from which
<I>TLS_CONNECTION</I> structures are instantiated. In the future the
<I>TLS_CONTEXT</I> structure may be used to maintain the TLS session
cache. <I>ca_certfile</I> and <I>ca_certdir</I> arguments are used to
initialize the CA verification stores. The <I>certfile</I> and
<I>keyfile</I> arguments are used to initialize the local certificate and
private key. If <I>dhfile</I> is non-NULL, it is used to initialize
Diffie-Hellman ephemeral keying. If <I>verify_peer</I> is <I>true</I> ,
client certificate validation is enabled.
<P>
<PRE>
void free_tls_context (TLS_CONTEXT *ctx);
</PRE>
<P>
Deallocated a previously allocated <I>TLS_CONTEXT</I> structure.
<P>
<H3><A NAME="SECTION000143300000000000000"></A>
<A NAME="3764"></A>
<A NAME="3765"></A>
<BR>
Performing Post-Connection Verification
</H3>
<A NAME="3768"></A>
<P>
<PRE>
bool tls_postconnect_verify_host (TLS_CONNECTION *tls, const char *host);
</PRE>
<P>
Performs post-connection verification of the peer-supplied x509
certificate. Checks whether the <I>subjectAltName</I> and
<I>commonName</I> attributes match the supplied <I>host</I> string.
Returns <I>true</I> if there is a match, <I>false</I> otherwise.
<P>
<PRE>
bool tls_postconnect_verify_cn (TLS_CONNECTION *tls, alist *verify_list);
</PRE>
<P>
Performs post-connection verification of the peer-supplied x509
certificate. Checks whether the <I>commonName</I> attribute matches any
strings supplied via the <I>verify_list</I> parameter. Returns
<I>true</I> if there is a match, <I>false</I> otherwise.
<P>
<H3><A NAME="SECTION000143400000000000000"></A>
<A NAME="3783"></A>
<A NAME="3784"></A>
<BR>
Manipulating TLS Connections
</H3>
<A NAME="3787"></A>
<P>
<PRE>
TLS_CONNECTION *new_tls_connection (TLS_CONTEXT *ctx, int fd);
</PRE>
<P>
Allocates and initializes a new <I>TLS_CONNECTION</I> structure with
context <I>ctx</I> and file descriptor <I>fd</I>.
<P>
<PRE>
void free_tls_connection (TLS_CONNECTION *tls);
</PRE>
<P>
Deallocates memory associated with the <I>tls</I> structure.
<P>
<PRE>
bool tls_bsock_connect (BSOCK *bsock);
</PRE>
<P>
Negotiates a a TLS client connection via <I>bsock</I>. Returns <I>true</I>
if successful, <I>false</I> otherwise. Will fail if there is a TLS
protocol error or an invalid certificate is presented
<P>
<PRE>
bool tls_bsock_accept (BSOCK *bsock);
</PRE>
<P>
Accepts a TLS client connection via <I>bsock</I>. Returns <I>true</I> if
successful, <I>false</I> otherwise. Will fail if there is a TLS protocol
error or an invalid certificate is presented.
<P>
<PRE>
bool tls_bsock_shutdown (BSOCK *bsock);
</PRE>
<P>
Issues a blocking TLS shutdown request to the peer via <I>bsock</I>. This function may not wait for the peer's reply.
<P>
<PRE>
int tls_bsock_writen (BSOCK *bsock, char *ptr, int32_t nbytes);
</PRE>
<P>
Writes <I>nbytes</I> from <I>ptr</I> via the <I>TLS_CONNECTION</I>
associated with <I>bsock</I>. Due to OpenSSL's handling of <I>EINTR</I>,
<I>bsock</I> is set non-blocking at the start of the function, and restored
to its original blocking state before the function returns. Less than
<I>nbytes</I> may be written if an error occurs. The actual number of
bytes written will be returned.
<P>
<PRE>
int tls_bsock_readn (BSOCK *bsock, char *ptr, int32_t nbytes);
</PRE>
<P>
Reads <I>nbytes</I> from the <I>TLS_CONNECTION</I> associated with
<I>bsock</I> and stores the result in <I>ptr</I>. Due to OpenSSL's
handling of <I>EINTR</I>, <I>bsock</I> is set non-blocking at the start of
the function, and restored to its original blocking state before the
function returns. Less than <I>nbytes</I> may be read if an error occurs.
The actual number of bytes read will be returned.
<P>
<H2><A NAME="SECTION000144000000000000000"></A>
<A NAME="3828"></A>
<A NAME="3829"></A>
<BR>
Bnet API Changes
</H2>
<A NAME="3832"></A>
<P>
A minimal number of changes were required in the Bnet socket API. The BSOCK
structure was expanded to include an associated TLS_CONNECTION structure,
as well as a flag to designate the current blocking state of the socket.
The blocking state flag is required for win32, where it does not appear
possible to discern the current blocking state of a socket.
<P>
<H3><A NAME="SECTION000144100000000000000"></A>
<A NAME="3834"></A>
<A NAME="3835"></A>
<BR>
Negotiating a TLS Connection
</H3>
<A NAME="3838"></A>
<P>
<I>bnet_tls_server()</I> and <I>bnet_tls_client()</I> were both
implemented using the new TLS API as follows:
<P>
<PRE>
int bnet_tls_client(TLS_CONTEXT *ctx, BSOCK * bsock);
</PRE>
<P>
Negotiates a TLS session via <I>bsock</I> using the settings from
<I>ctx</I>. Returns 1 if successful, 0 otherwise.
<P>
<PRE>
int bnet_tls_server(TLS_CONTEXT *ctx, BSOCK * bsock, alist *verify_list);
</PRE>
<P>
Accepts a TLS client session via <I>bsock</I> using the settings from
<I>ctx</I>. If <I>verify_list</I> is non-NULL, it is passed to
<I>tls_postconnect_verify_cn()</I> for client certificate verification.
<P>
<H3><A NAME="SECTION000144200000000000000"></A>
<A NAME="3852"></A>
<A NAME="3853"></A>
<A NAME="3854"></A>
<BR>
Manipulating Socket Blocking State
</H3>
<A NAME="3857"></A>
<P>
Three functions were added for manipulating the blocking state of a socket
on both Win32 and Unix-like systems. The Win32 code was written according
to the MSDN documentation, but has not been tested.
<P>
These functions are prototyped as follows:
<P>
<PRE>
int bnet_set_nonblocking (BSOCK *bsock);
</PRE>
<P>
Enables non-blocking I/O on the socket associated with <I>bsock</I>.
Returns a copy of the socket flags prior to modification.
<P>
<PRE>
int bnet_set_blocking (BSOCK *bsock);
</PRE>
<P>
Enables blocking I/O on the socket associated with <I>bsock</I>. Returns a
copy of the socket flags prior to modification.
<P>
<PRE>
void bnet_restore_blocking (BSOCK *bsock, int flags);
</PRE>
<P>
Restores blocking or non-blocking IO setting on the socket associated with
<I>bsock</I>. The <I>flags</I> argument must be the return value of either
<I>bnet_set_blocking()</I> or <I>bnet_restore_blocking()</I>.
<P>
<P>
<H2><A NAME="SECTION000145000000000000000"></A>
<A NAME="3871"></A>
<A NAME="3872"></A>
<BR>
Authentication Negotiation
</H2>
<A NAME="3875"></A>
<P>
Backwards compatibility with the existing SSL negotiation hooks implemented
in src/lib/cram-md5.c have been maintained. The
<I>cram_md5_get_auth()</I> function has been modified to accept an
integer pointer argument, tls_remote_need. The TLS requirement
advertised by the remote host is returned via this pointer.
<P>
After exchanging cram-md5 authentication and TLS requirements, both the
client and server independently decide whether to continue:
<P>
<PRE>
if (!cram_md5_get_auth(dir, password, &tls_remote_need) ||
!cram_md5_auth(dir, password, tls_local_need)) {
[snip]
/* Verify that the remote host is willing to meet our TLS requirements */
if (tls_remote_need < tls_local_need && tls_local_need != BNET_TLS_OK &&
tls_remote_need != BNET_TLS_OK) {
sendit(_("Authorization problem:"
" Remote server did not advertise required TLS support.\n"));
auth_success = false;
goto auth_done;
}
/* Verify that we are willing to meet the remote host's requirements */
if (tls_remote_need > tls_local_need && tls_local_need != BNET_TLS_OK &&
tls_remote_need != BNET_TLS_OK) {
sendit(_("Authorization problem:"
" Remote server requires TLS.\n"));
auth_success = false;
goto auth_done;
}
</PRE>
<P>
<H1><A NAME="SECTION000150000000000000000"></A>
<A NAME="_ChapterStart8"></A><A NAME="4109"></A>
<A NAME="4110"></A>
<BR>
Bacula Regression Testing
</H1>
<A NAME="4113"></A>
<P>
<H2><A NAME="SECTION000151000000000000000"></A>
<A NAME="4115"></A>
<BR>
General
</H2>
<A NAME="4118"></A>
<P>
This document is intended mostly for developers who wish to ensure that their
changes to Bacula don't introduce bugs in the base code. However, you
don't need to be a developer to run the regression scripts. They are
simply shell scripts that drive Bacula through bconsole and then typically
compare the input and output with diff.
<P>
You can find the existing regression script in the Bacula SVN on the
SourceForge SVN in the project tree named <B>regress</B>. I strongly
recommend that you check it out of the SVN because afterwards, you can
easily get updates made to them.
<P>
To get started, we recommend that you create a directory named <B>bacula</B>, under which you will put the current source code and the current
set of regression scripts. Below, we will describe how to set this up for
the SVN trunk, but you can also set it up for any branch, such as the
current Branch-2.2, where the currently released code resides.
<P>
The top level directory that we call <B>bacula</B> can be named anything you
want. Note, all the standard regression scripts run as non-root and can be
run on the same machine as a production Bacula system (Kern runs it this
way).
<P>
To create the directory structure for the current trunk code and to
checkout the necessary files, do the following (note, we assume you
are working in your home directory in a non-root account):
<P>
<PRE>
cd
mkdir bacula
cd bacula
svn checkout https://bacula.svn.sourceforge.net/svnroot/bacula/trunk/bacula
svn checkout https://bacula.svn.sourceforge.net/svnroot/bacula/trunk/regress
</PRE>
<P>
This will get the base files that you need. The above should be needed only
once. Thereafter to update to the latest code, you do:
<P>
<PRE>
cd bacula/bacula
svn update
cd ../regress
svn update
</PRE>
<P>
If you want to test with SQLite and it is not installed on your system,
you will need to download the latest depkgs release from Source Forge and
unpack it into <B>bacula/depkgs</B>, then simply:
<P>
<PRE>
cd bacula/depkgs
make
</PRE>
<P>
There are two different aspects of regression testing that this document will
discuss: 1. Running the Regression Script, 2. Writing a Regression test.
<P>
<H2><A NAME="SECTION000152000000000000000"></A>
<A NAME="4130"></A>
<A NAME="4131"></A>
<BR>
Running the Regression Script
</H2>
<A NAME="4134"></A>
<P>
There are a number of different tests that may be run, such as: the standard
set that uses disk Volumes and runs under any userid; a small set of tests
that write to tape; another set of tests where you must be root to run them.
Normally, I run all my tests as non-root and very rarely run the root
tests. The tests vary in length, and running the full tests including disk
based testing, tape based testing, autochanger based testing, and multiple
drive autochanger based testing can take 3 or 4 hours.
<P>
<H3><A NAME="SECTION000152100000000000000"></A>
<A NAME="4136"></A>
<A NAME="4137"></A>
<BR>
Setting the Configuration Parameters
</H3>
<A NAME="4140"></A>
<P>
There is nothing you need to change in the source directory.
<P>
To begin:
<P>
<PRE>
cd bacula/regress
</PRE>
<P>
The
very first time you are going to run the regression scripts, you will
need to create a custom config file for your system.
We suggest that you start by:
<P>
<PRE>
cp prototype.conf config
</PRE>
<P>
Then you can edit the <B>config</B> file directly.
<P>
<PRE>
# Where to get the source to be tested
BACULA_SOURCE="${HOME}/bacula/bacula"
# Where to send email !!!!! Change me !!!!!!!
EMAIL=your-name@your-domain.com
SMTP_HOST="localhost"
# Full "default" path where to find sqlite (no quotes!)
SQLITE3_DIR=${HOME}/bacula/depkgs/sqlite3
SQLITE_DIR=${HOME}/bacula/depkgs/sqlite
TAPE_DRIVE="/dev/nst0"
# if you don't have an autochanger set AUTOCHANGER to /dev/null
AUTOCHANGER="/dev/sg0"
# For two drive tests -- set to /dev/null if you do not have it
TAPE_DRIVE1="/dev/null"
# This must be the path to the autochanger including its name
AUTOCHANGER_PATH="/usr/sbin/mtx"
# Set your database here
#WHICHDB="--with-sqlite=${SQLITE_DIR}"
#WHICHDB="--with-sqlite3=${SQLITE3_DIR}"
#WHICHDB="--with-mysql"
WHICHDB="--with-postgresql"
# Set this to "--with-tcp-wrappers" or "--without-tcp-wrappers"
TCPWRAPPERS="--with-tcp-wrappers"
# Set this to "" to disable OpenSSL support, "--with-openssl=yes"
# to enable it, or provide the path to the OpenSSL installation,
# eg "--with-openssl=/usr/local"
OPENSSL="--with-openssl"
# You may put your real host name here, but localhost is valid also
# and it has the advantage that it works on a non-newtworked machine
HOST="localhost"
</PRE>
<P>
<UL>
<LI><B>BACULA_SOURCE</B> should be the full path to the Bacula source code
that you wish to test. It will be loaded configured, compiled, and
installed with the "make setup" command, which needs to be done only
once each time you change the source code.
<P>
</LI>
<LI><B>EMAIL</B> should be your email addres. Please remember to change this
or I will get a flood of unwanted messages. You may or may not want to see
these emails. In my case, I don't need them so I direct it to the bit bucket.
<P>
</LI>
<LI><B>SMTP_HOST</B> defines where your SMTP server is.
<P>
</LI>
<LI><B>SQLITE_DIR</B> should be the full path to the sqlite package, must
be build before running a Bacula regression, if you are using SQLite. This
variable is ignored if you are using MySQL or PostgreSQL. To use PostgreSQL,
edit the Makefile and change (or add) WHICHDB?=``<code>--</code>with-postgresql''. For
MySQL use ``WHICHDB=''<code>--</code>with-mysql``.
<P>
The advantage of using SQLite is that it is totally independent of any
installation you may have running on your system, and there is no
special configuration or authorization that must be done to run it.
With both MySQL and PostgreSQL, you must pre-install the packages,
initialize them and ensure that you have authorization to access the
database and create and delete tables.
<P>
</LI>
<LI><B>TAPE_DRIVE</B> is the full path to your tape drive. The base set of
regression tests do not use a tape, so this is only important if you want to
run the full tests. Set this to /dev/null if you do not have a tape drive.
<P>
</LI>
<LI><B>TAPE_DRIVE1</B> is the full path to your second tape drive, if
have one. The base set of
regression tests do not use a tape, so this is only important if you want to
run the full two drive tests. Set this to /dev/null if you do not have a
second tape drive.
<P>
</LI>
<LI><B>AUTOCHANGER</B> is the name of your autochanger control device. Set this to
/dev/null if you do not have an autochanger.
<P>
</LI>
<LI><B>AUTOCHANGER_PATH</B> is the full path including the program name for
your autochanger program (normally <B>mtx</B>. Leave the default value if you
do not have one.
<P>
</LI>
<LI><B>TCPWRAPPERS</B> defines whether or not you want the ./configure
to be performed with tcpwrappers enabled.
<P>
</LI>
<LI><B>OPENSSL</B> used to enable/disable SSL support for Bacula
communications and data encryption.
<P>
</LI>
<LI><B>HOST</B> is the hostname that it will use when building the
scripts. The Bacula daemons will be named <HOST>-dir, <HOST>-fd,
... It is also the name of the HOST machine that to connect to the
daemons by the network. Hence the name should either be your real
hostname (with an appropriate DNS or /etc/hosts entry) or <B> localhost</B> as it is in the default file.
<P>
</LI>
</UL>
<P>
<H3><A NAME="SECTION000152200000000000000"></A>
<A NAME="4164"></A>
<A NAME="4165"></A>
<BR>
Building the Test Bacula
</H3>
<A NAME="4168"></A>
<P>
Once the above variables are set, you can build the Makefile by entering:
<P>
<PRE>
./config xxx.conf
</PRE>
<P>
Where xxx.conf is the name of the conf file containing your system parameters.
This will build a Makefile from Makefile.in, and you should not need to
do this again unless you want to change the database or other regression
configuration parameter.
<P>
<H3><A NAME="SECTION000152300000000000000"></A>
<A NAME="4172"></A>
<BR>
Setting up your SQL engine
</H3>
<A NAME="4175"></A>
If you are using SQLite or SQLite3, there is nothing more to do; you can
simply run the tests as described in the next section.
<P>
If you are using MySQL or PostgreSQL, you will need to establish an
account with your database engine for the user name <B>regress</B> and
you will need to manually create a database named <B>regress</B> that can be
used by user name regress, which means you will have to give the user
regress sufficient permissions to use the database named regress.
There is no password on the regress account.
<P>
You have probably already done this procedure for the user name and
database named bacula. If not, the manual describes roughly how to
do it, and the scripts in bacula/regress/build/src/cats named
create_mysql_database, create_postgresql_database, grant_mysql_privileges,
and grant_postgresql_privileges may be of a help to you.
<P>
Generally, to do the above, you will need to run under root to
be able to create databases and modify permissions within MySQL and
PostgreSQL.
<P>
<H3><A NAME="SECTION000152400000000000000"></A>
<A NAME="4179"></A>
<A NAME="4180"></A>
<BR>
Running the Disk Only Regression
</H3>
<A NAME="4183"></A>
<P>
Every time the Bacula source code has changed, you will need to recompile
it under the regression directory. You do so with:
<P>
<PRE>
make setup
</PRE>
<P>
The above will then copy the source code within
the regression tree (in directory regress/build), configure it, and build it.
There should be no errors. If there are, please correct them before
continuing. From this point on, as long as you don't change the Bacula
source code, you should not need to repeat any of the above steps. If
you pull down a new version of the source code, simply run <B>make setup</B>
again.
<P>
Once Bacula is built, you can run the basic disk only non-root regression test
by entering:
<P>
<PRE>
make test
</PRE>
<P>
This will run the base set of tests using disk Volumes.
If you are testing on a
non-Linux machine several of the of the tests may not be run. In any case,
as we add new tests, the number will vary. It will take about 1 hour
and you don't need to be root
to run these tests (I run under my regular userid). The result should be
something similar to:
<P>
<PRE>
Test results
===== auto-label-test OK 12:31:33 =====
===== backup-bacula-test OK 12:32:32 =====
===== bextract-test OK 12:33:27 =====
===== bscan-test OK 12:34:47 =====
===== bsr-opt-test OK 12:35:46 =====
===== compressed-test OK 12:36:52 =====
===== compressed-encrypt-test OK 12:38:18 =====
===== concurrent-jobs-test OK 12:39:49 =====
===== data-encrypt-test OK 12:41:11 =====
===== encrypt-bug-test OK 12:42:00 =====
===== fifo-test OK 12:43:46 =====
===== backup-bacula-fifo OK 12:44:54 =====
===== differential-test OK 12:45:36 =====
===== four-concurrent-jobs-test OK 12:47:39 =====
===== four-jobs-test OK 12:49:22 =====
===== incremental-test OK 12:50:38 =====
===== query-test OK 12:51:37 =====
===== recycle-test OK 12:53:52 =====
===== restore2-by-file-test OK 12:54:53 =====
===== restore-by-file-test OK 12:55:40 =====
===== restore-disk-seek-test OK 12:56:29 =====
===== six-vol-test OK 12:57:44 =====
===== span-vol-test OK 12:58:52 =====
===== sparse-compressed-test OK 13:00:00 =====
===== sparse-test OK 13:01:04 =====
===== two-jobs-test OK 13:02:39 =====
===== two-vol-test OK 13:03:49 =====
===== verify-vol-test OK 13:04:56 =====
===== weird-files2-test OK 13:05:47 =====
===== weird-files-test OK 13:06:33 =====
===== migration-job-test OK 13:08:15 =====
===== migration-jobspan-test OK 13:09:33 =====
===== migration-volume-test OK 13:10:48 =====
===== migration-time-test OK 13:12:59 =====
===== hardlink-test OK 13:13:50 =====
===== two-pool-test OK 13:18:17 =====
===== fast-two-pool-test OK 13:24:02 =====
===== two-volume-test OK 13:25:06 =====
===== incremental-2disk OK 13:25:57 =====
===== 2drive-incremental-2disk OK 13:26:53 =====
===== scratch-pool-test OK 13:28:01 =====
Total time = 0:57:55 or 3475 secs
</PRE>
<P>
and the working tape tests are run with
<P>
<PRE>
make full_test
</PRE>
<P>
<PRE>
Test results
===== Bacula tape test OK =====
===== Small File Size test OK =====
===== restore-by-file-tape test OK =====
===== incremental-tape test OK =====
===== four-concurrent-jobs-tape OK =====
===== four-jobs-tape OK =====
</PRE>
<P>
Each separate test is self contained in that it initializes to run Bacula from
scratch (i.e. newly created database). It will also kill any Bacula session
that is currently running. In addition, it uses ports 8101, 8102, and 8103 so
that it does not intefere with a production system.
<P>
<H3><A NAME="SECTION000152500000000000000"></A>
<A NAME="4196"></A>
<A NAME="4197"></A>
<BR>
Other Tests
</H3>
<A NAME="4200"></A>
<P>
There are a number of other tests that can be run as well. All the tests are a
simply shell script keep in the regress directory. For example the ''make
test`` simply executes <B>./all-non-root-tests</B>. The other tests, which
are invoked by directly running the script are:
<P>
<DL>
<DT><STRONG>all_non-root-tests</STRONG></DT>
<DD><A NAME="4203"></A>
All non-tape tests not requiring root. This is the standard set of tests,
that in general, backup some data, then restore it, and finally compares the
restored data with the original data.
<P>
</DD>
<DT><STRONG>all-root-tests</STRONG></DT>
<DD><A NAME="4204"></A>
All non-tape tests requiring root permission. These are a relatively small
number of tests that require running as root. The amount of data backed up
can be quite large. For example, one test backs up /usr, another backs up
/etc. One or more of these tests reports an error -- I'll fix it one day.
<P>
</DD>
<DT><STRONG>all-non-root-tape-tests</STRONG></DT>
<DD><A NAME="4205"></A>
All tape test not requiring root. There are currently three tests, all run
without being root, and backup to a tape. The first two tests use one volume,
and the third test requires an autochanger, and uses two volumes. If you
don't have an autochanger, then this script will probably produce an error.
<P>
</DD>
<DT><STRONG>all-tape-and-file-tests</STRONG></DT>
<DD><A NAME="4206"></A>
All tape and file tests not requiring root. This includes just about
everything, and I don't run it very often.
</DD>
</DL>
<P>
<H3><A NAME="SECTION000152600000000000000"></A>
<A NAME="4209"></A>
<A NAME="4210"></A>
<BR>
If a Test Fails
</H3>
<A NAME="4213"></A>
<P>
If you one or more tests fail, the line output will be similar to:
<P>
<PRE>
!!!!! concurrent-jobs-test failed!!! !!!!!
</PRE>
<P>
If you want to determine why the test failed, you will need to rerun the
script with the debug output turned on. You do so by defining the
environment variable <B>REGRESS_DEBUG</B> with commands such as:
<P>
<PRE>
REGRESS_DEBUG=1
export REGRESS_DEBUG
</PRE>
<P>
Then from the "regress" directory (all regression scripts assume that
you have "regress" as the current directory), enter:
<P>
<PRE>
tests/test-name
</PRE>
<P>
where test-name should be the name of a test script -- for example:
<B>tests/backup-bacula-test</B>.
<P>
<H2><A NAME="SECTION000153000000000000000"></A>
<A NAME="4223"></A>
<BR>
Running a Single Test
</H2>
<A NAME="4226"></A>
<P>
If you wish to run a single test, you can simply:
<P>
<PRE>
cd regress
tests/<name-of-test>
</PRE>
<P>
or, if the source code has been updated, you would do:
<P>
<PRE>
cd bacula/bacula
svn update
cd ../regress
svn update
make setup
tests/backup-to-null
</PRE>
<P>
<H2><A NAME="SECTION000154000000000000000"></A>
<A NAME="4232"></A>
<A NAME="4233"></A>
<BR>
Writing a Regression Test
</H2>
<A NAME="4236"></A>
<P>
Any developer, who implements a major new feature, should write a regression
test that exercises and validates the new feature. Each regression test is a
complete test by itself. It terminates any running Bacula, initializes the
database, starts Bacula, then runs the test by using the console program.
<P>
<H3><A NAME="SECTION000154100000000000000"></A>
<A NAME="4238"></A>
<A NAME="4239"></A>
<BR>
Running the Tests by Hand
</H3>
<A NAME="4242"></A>
<P>
You can run any individual test by hand by cd'ing to the <B>regress</B>
directory and entering:
<P>
<PRE>
tests/<test-name>
</PRE>
<P>
<H3><A NAME="SECTION000154200000000000000"></A>
<A NAME="4247"></A>
<A NAME="4248"></A>
<BR>
Directory Structure
</H3>
<A NAME="4251"></A>
<P>
The directory structure of the regression tests is:
<P>
<PRE>
regress - Makefile, scripts to start tests
|------ scripts - Scripts and conf files
|-------tests - All test scripts are here
|
|------------------ -- All directories below this point are used
| for testing, but are created from the
| above directories and are removed with
| "make distclean"
|
|------ bin - This is the install directory for
| Bacula to be used testing
|------ build - Where the Bacula source build tree is
|------ tmp - Most temp files go here
|------ working - Bacula working directory
|------ weird-files - Weird files used in two of the tests.
</PRE>
<P>
<H3><A NAME="SECTION000154300000000000000"></A>
<A NAME="4255"></A>
<A NAME="4256"></A>
<BR>
Adding a New Test
</H3>
<A NAME="4259"></A>
<P>
If you want to write a new regression test, it is best to start with one of
the existing test scripts, and modify it to do the new test.
<P>
When adding a new test, be extremely careful about adding anything to any of
the daemons' configuration files. The reason is that it may change the prompts
that are sent to the console. For example, adding a Pool means that the
current scripts, which assume that Bacula automatically selects a Pool, will
now be presented with a new prompt, so the test will fail. If you need to
enhance the configuration files, consider making your own versions.
<P>
<H3><A NAME="SECTION000154400000000000000"></A>
<A NAME="4261"></A>
<BR>
Running a Test Under The Debugger
</H3>
<A NAME="4264"></A>
You can run a test under the debugger (actually run a Bacula daemon
under the debugger) by first setting the environment variable
<B>REGRESS_WAIT</B> with commands such as:
<P>
<PRE>
REGRESS_WAIT=1
export REGRESS_WAIT
</PRE>
<P>
Then executing the script. When the script prints the following line:
<P>
<PRE>
Start Bacula under debugger and enter anything when ready ...
</PRE>
<P>
You start the Bacula component you want to run under the debugger in a
different shell window. For example:
<P>
<PRE>
cd .../regress/bin
gdb bacula-sd
(possibly set breakpoints, ...)
run -s -f
</PRE>
<P>
Then enter any character in the window with the above message.
An error message will appear saying that the daemon you are debugging
is already running, which is the case. You can simply ignore the
error message.
<P>
<H1><A NAME="SECTION000160000000000000000"></A>
<A NAME="MD5Chapter"></A>
<BR>
Bacula MD5 Algorithm
</H1>
<A NAME="4576"></A>
<P>
<H2><A NAME="SECTION000161000000000000000"></A>
<A NAME="4578"></A>
<A NAME="4579"></A>
<BR>
Command Line Message Digest Utility
</H2>
<A NAME="4582"></A>
<P>
This page describes <B>md5</B>, a command line utility usable on either Unix or
MS-DOS/Windows, which generates and verifies message digests (digital
signatures) using the MD5 algorithm. This program can be useful when
developing shell scripts or Perl programs for software installation, file
comparison, and detection of file corruption and tampering.
<P>
<H3><A NAME="SECTION000161100000000000000"></A>
<A NAME="4585"></A>
<BR>
Name
</H3>
<A NAME="4588"></A>
<P>
<B>md5</B> - generate / check MD5 message digest
<P>
<H3><A NAME="SECTION000161200000000000000"></A>
<A NAME="4591"></A>
<BR>
Synopsis
</H3>
<A NAME="4594"></A>
<P>
<B>md5</B> [ <B>-c</B><I>signature</I> ] [ <B>-u</B> ] [ <B>-d</B><I>input_text</I>
| <I>infile</I> ] [ <I>outfile</I> ]
<P>
<H3><A NAME="SECTION000161300000000000000"></A>
<A NAME="4604"></A>
<BR>
Description
</H3>
<A NAME="4607"></A>
<P>
A <I>message digest</I> is a compact digital signature for an arbitrarily long
stream of binary data. An ideal message digest algorithm would never generate
the same signature for two different sets of input, but achieving such
theoretical perfection would require a message digest as long as the input
file. Practical message digest algorithms compromise in favour of a digital
signature of modest size created with an algorithm designed to make
preparation of input text with a given signature computationally infeasible.
Message digest algorithms have much in common with techniques used in
encryption, but to a different end; verification that data have not been
altered since the signature was published.
<P>
Many older programs requiring digital signatures employed 16 or 32 bit <I>cyclical redundancy codes</I> (CRC) originally developed to verify correct
transmission in data communication protocols, but these short codes, while
adequate to detect the kind of transmission errors for which they were
intended, are insufficiently secure for applications such as electronic
commerce and verification of security related software distributions.
<P>
The most commonly used present-day message digest algorithm is the 128 bit MD5
algorithm, developed by Ron Rivest of the
<A NAME="tex2html9"
HREF="http://web.mit.edu/">MIT</A>
<A NAME="tex2html10"
HREF="http://www.lcs.mit.edu/">Laboratory for Computer Science</A>
and
<A NAME="tex2html11"
HREF="http://www.rsa.com/">RSA Data Security, Inc.</A>
The algorithm, with a
reference implementation, was published as Internet
<A NAME="tex2html12"
HREF="http://www.fourmilab.ch/md5/rfc1321.html">RFC 1321</A>
in April 1992, and
was placed into the public domain at that time. Message digest algorithms such
as MD5 are not deemed ``encryption technology'' and are not subject to the
export controls some governments impose on other data security products.
(Obviously, the responsibility for obeying the laws in the jurisdiction in
which you reside is entirely your own, but many common Web and Mail utilities
use MD5, and I am unaware of any restrictions on their distribution and use.)
<P>
The MD5 algorithm has been implemented in numerous computer languages
including C,
<A NAME="tex2html13"
HREF="http://www.perl.org/">Perl</A>, and
<A NAME="tex2html14"
HREF="http://www.javasoft.com/">Java</A>; if you're writing a program in such a
language, track down a suitable subroutine and incorporate it into your
program. The program described on this page is a <I>command line</I>
implementation of MD5, intended for use in shell scripts and Perl programs (it
is much faster than computing an MD5 signature directly in Perl). This <B>md5</B> program was originally developed as part of a suite of tools intended to
monitor large collections of files (for example, the contents of a Web site)
to detect corruption of files and inadvertent (or perhaps malicious) changes.
That task is now best accomplished with more comprehensive packages such as
<A NAME="tex2html15"
HREF="ftp://coast.cs.purdue.edu/pub/COAST/Tripwire/">Tripwire</A>, but the
command line <B>md5</B> component continues to prove useful for verifying
correct delivery and installation of software packages, comparing the contents
of two different systems, and checking for changes in specific files.
<P>
<H3><A NAME="SECTION000161400000000000000"></A>
<A NAME="4628"></A>
<BR>
Options
</H3>
<A NAME="4631"></A>
<P>
<DL>
<DT><STRONG><B>-c</B><I>signature</I> </STRONG></DT>
<DD><A NAME="4635"></A>
Computes the signature of the specified <I>infile</I> or the string supplied
by the <B>-d</B> option and compares it against the specified <I>signature</I>.
If the two signatures match, the exit status will be zero, otherwise the exit
status will be 1. No signature is written to <I>outfile</I> or standard
output; only the exit status is set. The signature to be checked must be
specified as 32 hexadecimal digits.
<P>
</DD>
<DT><STRONG><B>-d</B><I>input_text</I> </STRONG></DT>
<DD><A NAME="4642"></A>
A signature is computed for the given <I>input_text</I> (which must be quoted
if it contains white space characters) instead of input from <I>infile</I> or
standard input. If input is specified with the <B>-d</B> option, no <I>infile</I> should be specified.
<P>
</DD>
<DT><STRONG><B>-u</B> </STRONG></DT>
<DD>Print how-to-call information.
</DD>
</DL>
<P>
<H3><A NAME="SECTION000161500000000000000"></A>
<A NAME="4650"></A>
<BR>
Files
</H3>
<A NAME="4653"></A>
<P>
If no <I>infile</I> or <B>-d</B> option is specified or <I>infile</I> is a single
``-'', <B>md5</B> reads from standard input; if no <I>outfile</I> is given, or
<I>outfile</I> is a single ``-'', output is sent to standard output. Input and
output are processed strictly serially; consequently <B>md5</B> may be used in
pipelines.
<P>
<H3><A NAME="SECTION000161600000000000000"></A>
<A NAME="4662"></A>
<BR>
Bugs
</H3>
<A NAME="4665"></A>
<P>
The mechanism used to set standard input to binary mode may be specific to
Microsoft C; if you rebuild the DOS/Windows version of the program from source
using another compiler, be sure to verify binary files work properly when read
via redirection or a pipe.
<P>
This program has not been tested on a machine on which <TT>int</TT> and/or <TT>long</TT> are longer than 32 bits.
<P>
<H2><A NAME="SECTION000162000000000000000"></A>
<A NAME="4670"></A>
<A NAME="4671"></A>
<BR>
<A NAME="tex2html16"
HREF="http://www.fourmilab.ch/md5/md5.zip">Download md5.zip</A>
(Zipped
archive)
</H2>
<A NAME="4674"></A>
<P>
The program is provided as
<A NAME="tex2html17"
HREF="http://www.fourmilab.ch/md5/md5.zip">md5.zip</A>, a
<A NAME="tex2html18"
HREF="http://www.pkware.com/">Zipped</A>
archive containing an ready-to-run
Win32 command-line executable program, <TT>md5.exe</TT> (compiled using Microsoft
Visual C++ 5.0), and in source code form along with a <TT>Makefile</TT> to build
the program under Unix.
<P>
<H3><A NAME="SECTION000162100000000000000"></A>
<A NAME="4682"></A>
<A NAME="4683"></A>
<BR>
See Also
</H3>
<A NAME="4686"></A>
<P>
<B>sum</B>(1)
<P>
<H3><A NAME="SECTION000162200000000000000"></A>
<A NAME="4689"></A>
<A NAME="4690"></A>
<BR>
Exit Status
</H3>
<A NAME="4693"></A>
<P>
<B>md5</B> returns status 0 if processing was completed without errors, 1 if
the <B>-c</B> option was specified and the given signature does not match that
of the input, and 2 if processing could not be performed at all due, for
example, to a nonexistent input file.
<P>
<H3><A NAME="SECTION000162300000000000000"></A>
<A NAME="4697"></A>
<BR>
Copying
</H3>
<A NAME="4700"></A>
<P>
<BLOCKQUOTE>
This software is in the public domain. Permission to use, copy, modify, and
distribute this software and its documentation for any purpose and without
fee is hereby granted, without any conditions or restrictions. This software
is provided ``as is'' without express or implied warranty.
</BLOCKQUOTE>
<P>
<H3><A NAME="SECTION000162400000000000000"></A>
<A NAME="4704"></A>
<BR>
Acknowledgements
</H3>
<A NAME="4707"></A>
<P>
The MD5 algorithm was developed by Ron Rivest. The public domain C language
implementation used in this program was written by Colin Plumb in 1993.
<I><A NAME="tex2html19"
HREF="http://www.fourmilab.ch/">by John Walker</A>
January 6th, MIM </I>
<P>
<H1><A NAME="SECTION000170000000000000000"></A>
<A NAME="_ChapterStart7"></A><A NAME="4864"></A>
<A NAME="4865"></A>
<BR>
Bacula Memory Management
</H1>
<A NAME="4868"></A>
<P>
<H2><A NAME="SECTION000171000000000000000"></A>
<A NAME="4870"></A>
<BR>
General
</H2>
<A NAME="4873"></A>
<P>
This document describes the memory management routines that are used in Bacula
and is meant to be a technical discussion for developers rather than part of
the user manual.
<P>
Since Bacula may be called upon to handle filenames of varying and more or
less arbitrary length, special attention needs to be used in the code to
ensure that memory buffers are sufficiently large. There are four
possibilities for memory usage within <B>Bacula</B>. Each will be described in
turn. They are:
<P>
<UL>
<LI>Statically allocated memory.
</LI>
<LI>Dynamically allocated memory using malloc() and free().
</LI>
<LI>Non-pooled memory.
</LI>
<LI>Pooled memory.
</LI>
</UL>
<P>
<H3><A NAME="SECTION000171100000000000000"></A>
<A NAME="4878"></A>
<A NAME="4879"></A>
<BR>
Statically Allocated Memory
</H3>
<A NAME="4882"></A>
<P>
Statically allocated memory is of the form:
<P>
<PRE>
char buffer[MAXSTRING];
</PRE>
<P>
The use of this kind of memory is discouraged except when you are 100% sure
that the strings to be used will be of a fixed length. One example of where
this is appropriate is for <B>Bacula</B> resource names, which are currently
limited to 127 characters (MAX_NAME_LENGTH). Although this maximum size may
change, particularly to accommodate Unicode, it will remain a relatively small
value.
<P>
<H3><A NAME="SECTION000171200000000000000"></A>
<A NAME="4887"></A>
<A NAME="4888"></A>
<BR>
Dynamically Allocated Memory
</H3>
<A NAME="4891"></A>
<P>
Dynamically allocated memory is obtained using the standard malloc() routines.
As in:
<P>
<PRE>
char *buf;
buf = malloc(256);
</PRE>
<P>
This kind of memory can be released with:
<P>
<PRE>
free(buf);
</PRE>
<P>
It is recommended to use this kind of memory only when you are sure that you
know the memory size needed and the memory will be used for short periods of
time -- that is it would not be appropriate to use statically allocated
memory. An example might be to obtain a large memory buffer for reading and
writing files. When <B>SmartAlloc</B> is enabled, the memory obtained by
malloc() will automatically be checked for buffer overwrite (overflow) during
the free() call, and all malloc'ed memory that is not released prior to
termination of the program will be reported as Orphaned memory.
<P>
<H3><A NAME="SECTION000171300000000000000"></A>
<A NAME="4898"></A>
<A NAME="4899"></A>
<BR>
Pooled and Non-pooled Memory
</H3>
<A NAME="4902"></A>
<P>
In order to facility the handling of arbitrary length filenames and to
efficiently handle a high volume of dynamic memory usage, we have implemented
routines between the C code and the malloc routines. The first is called
``Pooled'' memory, and is memory, which once allocated and then released, is
not returned to the system memory pool, but rather retained in a Bacula memory
pool. The next request to acquire pooled memory will return any free memory
block. In addition, each memory block has its current size associated with the
block allowing for easy checking if the buffer is of sufficient size. This
kind of memory would normally be used in high volume situations (lots of
malloc()s and free()s) where the buffer length may have to frequently change
to adapt to varying filename lengths.
<P>
The non-pooled memory is handled by routines similar to those used for pooled
memory, allowing for easy size checking. However, non-pooled memory is
returned to the system rather than being saved in the Bacula pool. This kind
of memory would normally be used in low volume situations (few malloc()s and
free()s), but where the size of the buffer might have to be adjusted
frequently.
<P>
<H4><A NAME="SECTION000171310000000000000">
Types of Memory Pool:</A>
</H4>
<P>
Currently there are three memory pool types:
<P>
<UL>
<LI>PM_NOPOOL -- non-pooled memory.
</LI>
<LI>PM_FNAME -- a filename pool.
</LI>
<LI>PM_MESSAGE -- a message buffer pool.
</LI>
<LI>PM_EMSG -- error message buffer pool.
</LI>
</UL>
<P>
<H4><A NAME="SECTION000171320000000000000">
Getting Memory:</A>
</H4>
<P>
To get memory, one uses:
<P>
<PRE>
void *get_pool_memory(pool);
</PRE>
<P>
where <B>pool</B> is one of the above mentioned pool names. The size of the
memory returned will be determined by the system to be most appropriate for
the application.
<P>
If you wish non-pooled memory, you may alternatively call:
<P>
<PRE>
void *get_memory(size_t size);
</PRE>
<P>
The buffer length will be set to the size specified, and it will be assigned
to the PM_NOPOOL pool (no pooling).
<P>
<H4><A NAME="SECTION000171330000000000000">
Releasing Memory:</A>
</H4>
<P>
To free memory acquired by either of the above two calls, use:
<P>
<PRE>
void free_pool_memory(void *buffer);
</PRE>
<P>
where buffer is the memory buffer returned when the memory was acquired. If
the memory was originally allocated as type PM_NOPOOL, it will be released to
the system, otherwise, it will be placed on the appropriate Bacula memory pool
free chain to be used in a subsequent call for memory from that pool.
<P>
<H4><A NAME="SECTION000171340000000000000">
Determining the Memory Size:</A>
</H4>
<P>
To determine the memory buffer size, use:
<P>
<PRE>
size_t sizeof_pool_memory(void *buffer);
</PRE>
<P>
<H4><A NAME="SECTION000171350000000000000">
Resizing Pool Memory:</A>
</H4>
<P>
To resize pool memory, use:
<P>
<PRE>
void *realloc_pool_memory(void *buffer);
</PRE>
<P>
The buffer will be reallocated, and the contents of the original buffer will
be preserved, but the address of the buffer may change.
<P>
<H4><A NAME="SECTION000171360000000000000">
Automatic Size Adjustment:</A>
</H4>
<P>
To have the system check and if necessary adjust the size of your pooled
memory buffer, use:
<P>
<PRE>
void *check_pool_memory_size(void *buffer, size_t new-size);
</PRE>
<P>
where <B>new-size</B> is the buffer length needed. Note, if the buffer is
already equal to or larger than <B>new-size</B> no buffer size change will
occur. However, if a buffer size change is needed, the original contents of
the buffer will be preserved, but the buffer address may change. Many of the
low level Bacula subroutines expect to be passed a pool memory buffer and use
this call to ensure the buffer they use is sufficiently large.
<P>
<H4><A NAME="SECTION000171370000000000000">
Releasing All Pooled Memory:</A>
</H4>
<P>
In order to avoid orphaned buffer error messages when terminating the program,
use:
<P>
<PRE>
void close_memory_pool();
</PRE>
<P>
to free all unused memory retained in the Bacula memory pool. Note, any memory
not returned to the pool via free_pool_memory() will not be released by this
call.
<P>
<H4><A NAME="SECTION000171380000000000000">
Pooled Memory Statistics:</A>
</H4>
<P>
For debugging purposes and performance tuning, the following call will print
the current memory pool statistics:
<P>
<PRE>
void print_memory_pool_stats();
</PRE>
<P>
an example output is:
<P>
<PRE>
Pool Maxsize Maxused Inuse
0 256 0 0
1 256 1 0
2 256 1 0
</PRE>
<P>
<H1><A NAME="SECTION000180000000000000000"></A>
<A NAME="_ChapterStart5"></A><A NAME="5100"></A>
<A NAME="5101"></A>
<BR>
TCP/IP Network Protocol
</H1>
<A NAME="5104"></A>
<P>
<H2><A NAME="SECTION000181000000000000000"></A>
<A NAME="5106"></A>
<BR>
General
</H2>
<A NAME="5109"></A>
<P>
This document describes the TCP/IP protocol used by Bacula to communicate
between the various daemons and services. The definitive definition of the
protocol can be found in src/lib/bsock.h, src/lib/bnet.c and
src/lib/bnet_server.c.
<P>
Bacula's network protocol is basically a ``packet oriented'' protocol built on
a standard TCP/IP streams. At the lowest level all packet transfers are done
with read() and write() requests on system sockets. Pipes are not used as they
are considered unreliable for large serial data transfers between various
hosts.
<P>
Using the routines described below (bnet_open, bnet_write, bnet_recv, and
bnet_close) guarantees that the number of bytes you write into the socket
will be received as a single record on the other end regardless of how many
low level write() and read() calls are needed. All data transferred are
considered to be binary data.
<P>
<H2><A NAME="SECTION000182000000000000000"></A>
<A NAME="5111"></A>
<A NAME="5112"></A>
<BR>
bnet and Threads
</H2>
<A NAME="5115"></A>
<P>
These bnet routines work fine in a threaded environment. However, they assume
that there is only one reader or writer on the socket at any time. It is
highly recommended that only a single thread access any BSOCK packet. The
exception to this rule is when the socket is first opened and it is waiting
for a job to start. The wait in the Storage daemon is done in one thread and
then passed to another thread for subsequent handling.
<P>
If you envision having two threads using the same BSOCK, think twice, then you
must implement some locking mechanism. However, it probably would not be
appropriate to put locks inside the bnet subroutines for efficiency reasons.
<P>
<H2><A NAME="SECTION000183000000000000000"></A>
<A NAME="5117"></A>
<BR>
bnet_open
</H2>
<A NAME="5120"></A>
<P>
To establish a connection to a server, use the subroutine:
<P>
BSOCK *bnet_open(void *jcr, char *host, char *service, int port, int *fatal)
bnet_open(), if successful, returns the Bacula sock descriptor pointer to be
used in subsequent bnet_send() and bnet_read() requests. If not successful,
bnet_open() returns a NULL. If fatal is set on return, it means that a fatal
error occurred and that you should not repeatedly call bnet_open(). Any error
message will generally be sent to the JCR.
<P>
<H2><A NAME="SECTION000184000000000000000"></A>
<A NAME="5122"></A>
<BR>
bnet_send
</H2>
<A NAME="5125"></A>
<P>
To send a packet, one uses the subroutine:
<P>
int bnet_send(BSOCK *sock) This routine is equivalent to a write() except
that it handles the low level details. The data to be sent is expected to be
in sock->msg and be sock->msglen bytes. To send a packet, bnet_send()
first writes four bytes in network byte order than indicate the size of the
following data packet. It returns:
<P>
<PRE>
Returns 0 on failure
Returns 1 on success
</PRE>
<P>
In the case of a failure, an error message will be sent to the JCR contained
within the bsock packet.
<P>
<H2><A NAME="SECTION000185000000000000000"></A>
<A NAME="5131"></A>
<BR>
bnet_fsend
</H2>
<A NAME="5134"></A>
<P>
This form uses:
<P>
int bnet_fsend(BSOCK *sock, char *format, ...) and it allows you to send a
formatted messages somewhat like fprintf(). The return status is the same as
bnet_send.
<P>
<H2><A NAME="SECTION000186000000000000000"></A>
<A NAME="5136"></A>
<A NAME="5137"></A>
<BR>
Additional Error information
</H2>
<A NAME="5140"></A>
<P>
Fro additional error information, you can call <B>is_bnet_error(BSOCK
*bsock)</B> which will return 0 if there is no error or non-zero if there is an
error on the last transmission. The <B>is_bnet_stop(BSOCK *bsock)</B>
function will return 0 if there no errors and you can continue sending. It
will return non-zero if there are errors or the line is closed (no more
transmissions should be sent).
<P>
<H2><A NAME="SECTION000187000000000000000"></A>
<A NAME="5144"></A>
<BR>
bnet_recv
</H2>
<A NAME="5147"></A>
<P>
To read a packet, one uses the subroutine:
<P>
int bnet_recv(BSOCK *sock) This routine is similar to a read() except that it
handles the low level details. bnet_read() first reads packet length that
follows as four bytes in network byte order. The data is read into
sock->msg and is sock->msglen bytes. If the sock->msg is not large
enough, bnet_recv() realloc() the buffer. It will return an error (-2) if
maxbytes is less than the record size sent. It returns:
<P>
<PRE>
* Returns number of bytes read
* Returns 0 on end of file
* Returns -1 on hard end of file (i.e. network connection close)
* Returns -2 on error
</PRE>
<P>
It should be noted that bnet_recv() is a blocking read.
<P>
<H2><A NAME="SECTION000188000000000000000"></A>
<A NAME="5154"></A>
<BR>
bnet_sig
</H2>
<A NAME="5157"></A>
<P>
To send a ``signal'' from one daemon to another, one uses the subroutine:
<P>
int bnet_sig(BSOCK *sock, SIGNAL) where SIGNAL is one of the following:
<P>
<OL>
<LI>BNET_EOF - deprecated use BNET_EOD
</LI>
<LI>BNET_EOD - End of data stream, new data may follow
</LI>
<LI>BNET_EOD_POLL - End of data and poll all in one
</LI>
<LI>BNET_STATUS - Request full status
</LI>
<LI>BNET_TERMINATE - Conversation terminated, doing close()
</LI>
<LI>BNET_POLL - Poll request, I'm hanging on a read
</LI>
<LI>BNET_HEARTBEAT - Heartbeat Response requested
</LI>
<LI>BNET_HB_RESPONSE - Only response permitted to HB
</LI>
<LI>BNET_PROMPT - Prompt for UA
</LI>
</OL>
<P>
<H2><A NAME="SECTION000189000000000000000"></A>
<A NAME="5161"></A>
<BR>
bnet_strerror
</H2>
<A NAME="5164"></A>
<P>
Returns a formated string corresponding to the last error that occurred.
<P>
<H2><A NAME="SECTION0001810000000000000000"></A>
<A NAME="5166"></A>
<BR>
bnet_close
</H2>
<A NAME="5169"></A>
<P>
The connection with the server remains open until closed by the subroutine:
<P>
void bnet_close(BSOCK *sock)
<P>
<H2><A NAME="SECTION0001811000000000000000"></A>
<A NAME="5171"></A>
<A NAME="5172"></A>
<BR>
Becoming a Server
</H2>
<A NAME="5175"></A>
<P>
The bnet_open() and bnet_close() routines described above are used on the
client side to establish a connection and terminate a connection with the
server. To become a server (i.e. wait for a connection from a client), use the
routine <B>bnet_thread_server</B>. The calling sequence is a bit complicated,
please refer to the code in bnet_server.c and the code at the beginning of
each daemon as examples of how to call it.
<P>
<H2><A NAME="SECTION0001812000000000000000"></A>
<A NAME="5178"></A>
<A NAME="5179"></A>
<BR>
Higher Level Conventions
</H2>
<A NAME="5182"></A>
<P>
Within Bacula, we have established the convention that any time a single
record is passed, it is sent with bnet_send() and read with bnet_recv().
Thus the normal exchange between the server (S) and the client (C) are:
<P>
<PRE>
S: wait for connection C: attempt connection
S: accept connection C: bnet_send() send request
S: bnet_recv() wait for request
S: act on request
S: bnet_send() send ack C: bnet_recv() wait for ack
</PRE>
<P>
Thus a single command is sent, acted upon by the server, and then
acknowledged.
<P>
In certain cases, such as the transfer of the data for a file, all the
information or data cannot be sent in a single packet. In this case, the
convention is that the client will send a command to the server, who knows
that more than one packet will be returned. In this case, the server will
enter a loop:
<P>
<PRE>
while ((n=bnet_recv(bsock)) > 0) {
act on request
}
if (n < 0)
error
</PRE>
<P>
The client will perform the following:
<P>
<PRE>
bnet_send(bsock);
bnet_send(bsock);
...
bnet_sig(bsock, BNET_EOD);
</PRE>
<P>
Thus the client will send multiple packets and signal to the server when all
the packets have been sent by sending a zero length record.
<P>
<A NAME="5309"></A>
<IMG
WIDTH="442" HEIGHT="67" ALIGN="BOTTOM" BORDER="0"
SRC="smartall.png"
ALT="\includegraphics{./smartall.eps}">
<P>
<H1><A NAME="SECTION000190000000000000000"></A>
<A NAME="_ChapterStart4"></A><A NAME="5313"></A>
<A NAME="5314"></A>
<BR>
Smart Memory Allocation With Orphaned Buffer Detection
</H1>
<A NAME="5317"></A>
<P>
Few things are as embarrassing as a program that leaks, yet few errors are so
easy to commit or as difficult to track down in a large, complicated program
as failure to release allocated memory. SMARTALLOC replaces the standard C
library memory allocation functions with versions which keep track of buffer
allocations and releases and report all orphaned buffers at the end of program
execution. By including this package in your program during development and
testing, you can identify code that loses buffers right when it's added and
most easily fixed, rather than as part of a crisis debugging push when the
problem is identified much later in the testing cycle (or even worse, when the
code is in the hands of a customer). When program testing is complete, simply
recompiling with different flags removes SMARTALLOC from your program,
permitting it to run without speed or storage penalties.
<P>
In addition to detecting orphaned buffers, SMARTALLOC also helps to find other
common problems in management of dynamic storage including storing before the
start or beyond the end of an allocated buffer, referencing data through a
pointer to a previously released buffer, attempting to release a buffer twice
or releasing storage not obtained from the allocator, and assuming the initial
contents of storage allocated by functions that do not guarantee a known
value. SMARTALLOC's checking does not usually add a large amount of overhead
to a program (except for programs which use <TT>realloc()</TT> extensively; see
below). SMARTALLOC focuses on proper storage management rather than internal
consistency of the heap as checked by the malloc_debug facility available on
some systems. SMARTALLOC does not conflict with malloc_debug and both may be
used together, if you wish. SMARTALLOC makes no assumptions regarding the
internal structure of the heap and thus should be compatible with any C
language implementation of the standard memory allocation functions.
<P>
<H3><A NAME="SECTION000190100000000000000"></A>
<A NAME="5320"></A>
<A NAME="5321"></A>
<BR>
Installing SMARTALLOC
</H3>
<A NAME="5324"></A>
<P>
SMARTALLOC is provided as a Zipped archive,
<A NAME="tex2html20"
HREF="http://www.fourmilab.ch/smartall/smartall.zip">smartall.zip</A>; see the
download instructions below.
<P>
To install SMARTALLOC in your program, simply add the statement:
<P>
to every C program file which calls any of the memory allocation functions
(<TT>malloc</TT>, <TT>calloc</TT>, <TT>free</TT>, etc.). SMARTALLOC must be used for
all memory allocation with a program, so include file for your entire program,
if you have such a thing. Next, define the symbol SMARTALLOC in the
compilation before the inclusion of smartall.h. I usually do this by having my
Makefile add the ``<TT>-DSMARTALLOC</TT>'' option to the C compiler for
non-production builds. You can define the symbol manually, if you prefer, by
adding the statement:
<P>
<TT>#define SMARTALLOC</TT>
<P>
At the point where your program is all done and ready to relinquish control to
the operating system, add the call:
<P>
<TT> sm_dump(</TT><I>datadump</I><TT>);</TT>
<P>
where <I>datadump</I> specifies whether the contents of orphaned buffers are to
be dumped in addition printing to their size and place of allocation. The data
are dumped only if <I>datadump</I> is nonzero, so most programs will normally
use ``<TT>sm_dump(0);</TT>''. If a mysterious orphaned buffer appears that can't
be identified from the information this prints about it, replace the statement
with ``<TT>sm_dump(1)</TT>;''. Usually the dump of the buffer's data will
furnish the additional clues you need to excavate and extirpate the elusive
error that left the buffer allocated.
<P>
Finally, add the files ``smartall.h'' and ``smartall.c'' from this release to
your source directory, make dependencies, and linker input. You needn't make
inclusion of smartall.c in your link optional; if compiled with SMARTALLOC not
defined it generates no code, so you may always include it knowing it will
waste no storage in production builds. Now when you run your program, if it
leaves any buffers around when it's done, each will be reported by <TT>sm_dump()</TT> on stderr as follows:
<P>
<PRE>
Orphaned buffer: 120 bytes allocated at line 50 of gutshot.c
</PRE>
<P>
<H3><A NAME="SECTION000190200000000000000"></A>
<A NAME="5343"></A>
<A NAME="5344"></A>
<BR>
Squelching a SMARTALLOC
</H3>
<A NAME="5347"></A>
<P>
Usually, when you first install SMARTALLOC in an existing program you'll find
it nattering about lots of orphaned buffers. Some of these turn out to be
legitimate errors, but some are storage allocated during program
initialisation that, while dynamically allocated, is logically static storage
not intended to be released. Of course, you can get rid of the complaints
about these buffers by adding code to release them, but by doing so you're
adding unnecessary complexity and code size to your program just to silence
the nattering of a SMARTALLOC, so an escape hatch is provided to eliminate the
need to release these buffers.
<P>
Normally all storage allocated with the functions <TT>malloc()</TT>, <TT>calloc()</TT>, and <TT>realloc()</TT> is monitored by SMARTALLOC. If you make the
function call:
<P>
<PRE>
sm_static(1);
</PRE>
<P>
you declare that subsequent storage allocated by <TT>malloc()</TT>, <TT>calloc()</TT>, and <TT>realloc()</TT> should not be considered orphaned if found to
be allocated when <TT>sm_dump()</TT> is called. I use a call on ``<TT>sm_static(1);</TT>'' before I allocate things like program configuration tables
so I don't have to add code to release them at end of program time. After
allocating unmonitored data this way, be sure to add a call to:
<P>
<PRE>
sm_static(0);
</PRE>
<P>
to resume normal monitoring of buffer allocations. Buffers allocated while
<TT>sm_static(1</TT>) is in effect are not checked for having been orphaned but
all the other safeguards provided by SMARTALLOC remain in effect. You may
release such buffers, if you like; but you don't have to.
<P>
<H3><A NAME="SECTION000190300000000000000"></A>
<A NAME="5362"></A>
<A NAME="5363"></A>
<BR>
Living with Libraries
</H3>
<A NAME="5366"></A>
<P>
Some library functions for which source code is unavailable may gratuitously
allocate and return buffers that contain their results, or require you to pass
them buffers which they subsequently release. If you have source code for the
library, by far the best approach is to simply install SMARTALLOC in it,
particularly since this kind of ill-structured dynamic storage management is
the source of so many storage leaks. Without source code, however, there's no
option but to provide a way to bypass SMARTALLOC for the buffers the library
allocates and/or releases with the standard system functions.
<P>
For each function <I>xxx</I> redefined by SMARTALLOC, a corresponding routine
named ``<TT>actually</TT><I>xxx</I>'' is furnished which provides direct access to
the underlying system function, as follows:
<P>
<P>
<BLOCKQUOTE><TABLE CELLPADDING=3>
<TR><TD ALIGN="LEFT" COLSPAN=1><B>Standard function </B></TD>
<TD ALIGN="LEFT" COLSPAN=1><B>Direct
access function </B></TD>
</TR>
<TR><TD ALIGN="LEFT"><TT>malloc(</TT><I>size</I><TT>)</TT></TD>
<TD ALIGN="LEFT"><TT>actuallymalloc(</TT><I>size</I><TT>)</TT></TD>
</TR>
<TR><TD ALIGN="LEFT"><TT>calloc(</TT><I>nelem</I><TT>,</TT> <I>elsize</I><TT>)</TT></TD>
<TD ALIGN="LEFT"><TT>actuallycalloc(</TT><I>nelem</I>, <I>elsize</I><TT>)</TT></TD>
</TR>
<TR><TD ALIGN="LEFT"><TT>realloc(</TT><I>ptr</I><TT>,</TT> <I>size</I><TT>)</TT></TD>
<TD ALIGN="LEFT"><TT>actuallyrealloc(</TT><I>ptr</I>, <I>size</I><TT>)</TT></TD>
</TR>
<TR><TD ALIGN="LEFT"><TT>free(</TT><I>ptr</I><TT>)</TT></TD>
<TD ALIGN="LEFT"><TT>actuallyfree(</TT><I>ptr</I><TT>)</TT>
<P></TD>
</TR>
</TABLE>
</BLOCKQUOTE>
<P>
<P>
For example, suppose there exists a system library function named ``<TT>getimage()</TT>'' which reads a raster image file and returns the address of a
buffer containing it. Since the library routine allocates the image directly
with <TT>malloc()</TT>, you can't use SMARTALLOC's <TT>free()</TT>, as that call
expects information placed in the buffer by SMARTALLOC's special version of
<TT>malloc()</TT>, and hence would report an error. To release the buffer you
should call <TT>actuallyfree()</TT>, as in this code fragment:
<P>
<PRE>
struct image *ibuf = getimage("ratpack.img");
display_on_screen(ibuf);
actuallyfree(ibuf);
</PRE>
<P>
Conversely, suppose we are to call a library function, ``<TT>putimage()</TT>'',
which writes an image buffer into a file and then releases the buffer with
<TT>free()</TT>. Since the system <TT>free()</TT> is being called, we can't pass a
buffer allocated by SMARTALLOC's allocation routines, as it contains special
information that the system <TT>free()</TT> doesn't expect to be there. The
following code uses <TT>actuallymalloc()</TT> to obtain the buffer passed to such
a routine.
<P>
<PRE>
struct image *obuf =
(struct image *) actuallymalloc(sizeof(struct image));
dump_screen_to_image(obuf);
putimage("scrdump.img", obuf); /* putimage() releases obuf */
</PRE>
<P>
It's unlikely you'll need any of the ``actually'' calls except under very odd
circumstances (in four products and three years, I've only needed them once),
but they're there for the rare occasions that demand them. Don't use them to
subvert the error checking of SMARTALLOC; if you want to disable orphaned
buffer detection, use the <TT>sm_static(1)</TT> mechanism described above. That
way you don't forfeit all the other advantages of SMARTALLOC as you do when
using <TT>actuallymalloc()</TT> and <TT>actuallyfree()</TT>.
<P>
<H3><A NAME="SECTION000190400000000000000"></A>
<A NAME="5429"></A>
<A NAME="5430"></A>
<BR>
SMARTALLOC Details
</H3>
<A NAME="5433"></A>
<P>
When you include ``smartall.h'' and define SMARTALLOC, the following standard
system library functions are redefined with the #define mechanism to call
corresponding functions within smartall.c instead. (For details of the
redefinitions, please refer to smartall.h.)
<P>
<PRE>
void *malloc(size_t size)
void *calloc(size_t nelem, size_t elsize)
void *realloc(void *ptr, size_t size)
void free(void *ptr)
void cfree(void *ptr)
</PRE>
<P>
<TT>cfree()</TT> is a historical artifact identical to <TT>free()</TT>.
<P>
In addition to allocating storage in the same way as the standard library
functions, the SMARTALLOC versions expand the buffers they allocate to include
information that identifies where each buffer was allocated and to chain all
allocated buffers together. When a buffer is released, it is removed from the
allocated buffer chain. A call on <TT>sm_dump()</TT> is able, by scanning the
chain of allocated buffers, to find all orphaned buffers. Buffers allocated
while <TT>sm_static(1)</TT> is in effect are specially flagged so that, despite
appearing on the allocated buffer chain, <TT>sm_dump()</TT> will not deem them
orphans.
<P>
When a buffer is allocated by <TT>malloc()</TT> or expanded with <TT>realloc()</TT>,
all bytes of newly allocated storage are set to the hexadecimal value 0x55
(alternating one and zero bits). Note that for <TT>realloc()</TT> this applies
only to the bytes added at the end of buffer; the original contents of the
buffer are not modified. Initializing allocated storage to a distinctive
nonzero pattern is intended to catch code that erroneously assumes newly
allocated buffers are cleared to zero; in fact their contents are random. The
<TT>calloc()</TT> function, defined as returning a buffer cleared to zero,
continues to zero its buffers under SMARTALLOC.
<P>
Buffers obtained with the SMARTALLOC functions contain a special sentinel byte
at the end of the user data area. This byte is set to a special key value
based upon the buffer's memory address. When the buffer is released, the key
is tested and if it has been overwritten an assertion in the <TT>free</TT>
function will fail. This catches incorrect program code that stores beyond the
storage allocated for the buffer. At <TT>free()</TT> time the queue links are
also validated and an assertion failure will occur if the program has
destroyed them by storing before the start of the allocated storage.
<P>
In addition, when a buffer is released with <TT>free()</TT>, its contents are
immediately destroyed by overwriting them with the hexadecimal pattern 0xAA
(alternating bits, the one's complement of the initial value pattern). This
will usually trip up code that keeps a pointer to a buffer that's been freed
and later attempts to reference data within the released buffer. Incredibly,
this is <I>legal</I> in the standard Unix memory allocation package, which
permits programs to free() buffers, then raise them from the grave with <TT>realloc()</TT>. Such program ``logic'' should be fixed, not accommodated, and
SMARTALLOC brooks no such Lazarus buffer`` nonsense.
<P>
Some C libraries allow a zero size argument in calls to <TT>malloc()</TT>. Since
this is far more likely to indicate a program error than a defensible
programming stratagem, SMARTALLOC disallows it with an assertion.
<P>
When the standard library <TT>realloc()</TT> function is called to expand a
buffer, it attempts to expand the buffer in place if possible, moving it only
if necessary. Because SMARTALLOC must place its own private storage in the
buffer and also to aid in error detection, its version of <TT>realloc()</TT>
always moves and copies the buffer except in the trivial case where the size
of the buffer is not being changed. By forcing the buffer to move on every
call and destroying the contents of the old buffer when it is released,
SMARTALLOC traps programs which keep pointers into a buffer across a call on
<TT>realloc()</TT> which may move it. This strategy may prove very costly to
programs which make extensive use of <TT>realloc()</TT>. If this proves to be a
problem, such programs may wish to use <TT>actuallymalloc()</TT>, <TT>actuallyrealloc()</TT>, and <TT>actuallyfree()</TT> for such frequently-adjusted
buffers, trading error detection for performance. Although not specified in
the System V Interface Definition, many C library implementations of <TT>realloc()</TT> permit an old buffer argument of NULL, causing <TT>realloc()</TT> to
allocate a new buffer. The SMARTALLOC version permits this.
<P>
<H3><A NAME="SECTION000190500000000000000"></A>
<A NAME="5461"></A>
<A NAME="5462"></A>
<BR>
When SMARTALLOC is Disabled
</H3>
<A NAME="5465"></A>
<P>
When SMARTALLOC is disabled by compiling a program with the symbol SMARTALLOC
not defined, calls on the functions otherwise redefined by SMARTALLOC go
directly to the system functions. In addition, compile-time definitions
translate calls on the ''<TT>actually</TT>...<TT>()</TT>`` functions into the
corresponding library calls; ''<TT>actuallymalloc(100)</TT>``, for example,
compiles into ''<TT>malloc(100)</TT>``. The two special SMARTALLOC functions,
<TT>sm_dump()</TT> and <TT>sm_static()</TT>, are defined to generate no code
(hence the null statement). Finally, if SMARTALLOC is not defined, compilation
of the file smartall.c generates no code or data at all, effectively removing
it from the program even if named in the link instructions.
<P>
Thus, except for unusual circumstances, a program that works with SMARTALLOC
defined for testing should require no changes when built without it for
production release.
<P>
<H3><A NAME="SECTION000190600000000000000"></A>
<A NAME="5473"></A>
<A NAME="5474"></A>
<BR>
The <TT>alloc()</TT> Function
</H3>
<A NAME="5477"></A>
<P>
Many programs I've worked on use very few direct calls to <TT>malloc()</TT>,
using the identically declared <TT>alloc()</TT> function instead. Alloc detects
out-of-memory conditions and aborts, removing the need for error checking on
every call of <TT>malloc()</TT> (and the temptation to skip checking for
out-of-memory).
<P>
As a convenience, SMARTALLOC supplies a compatible version of <TT>alloc()</TT> in
the file alloc.c, with its definition in the file alloc.h. This version of
<TT>alloc()</TT> is sensitive to the definition of SMARTALLOC and cooperates with
SMARTALLOC's orphaned buffer detection. In addition, when SMARTALLOC is
defined and <TT>alloc()</TT> detects an out of memory condition, it takes
advantage of the SMARTALLOC diagnostic information to identify the file and
line number of the call on <TT>alloc()</TT> that failed.
<P>
<H3><A NAME="SECTION000190700000000000000"></A>
<A NAME="5486"></A>
<A NAME="5487"></A>
<BR>
Overlays and Underhandedness
</H3>
<A NAME="5490"></A>
<P>
String constants in the C language are considered to be static arrays of
characters accessed through a pointer constant. The arrays are potentially
writable even though their pointer is a constant. SMARTALLOC uses the
compile-time definition <TT>./smartall.wml</TT> to obtain the name of the file in
which a call on buffer allocation was performed. Rather than reserve space in
a buffer to save this information, SMARTALLOC simply stores the pointer to the
compiled-in text of the file name. This works fine as long as the program does
not overlay its data among modules. If data are overlayed, the area of memory
which contained the file name at the time it was saved in the buffer may
contain something else entirely when <TT>sm_dump()</TT> gets around to using the
pointer to edit the file name which allocated the buffer.
<P>
If you want to use SMARTALLOC in a program with overlayed data, you'll have to
modify smartall.c to either copy the file name to a fixed-length field added
to the <TT>abufhead</TT> structure, or else allocate storage with <TT>malloc()</TT>,
copy the file name there, and set the <TT>abfname</TT> pointer to that buffer,
then remember to release the buffer in <TT>sm_free</TT>. Either of these
approaches are wasteful of storage and time, and should be considered only if
there is no alternative. Since most initial debugging is done in non-overlayed
environments, the restrictions on SMARTALLOC with data overlaying may never
prove a problem. Note that conventional overlaying of code, by far the most
common form of overlaying, poses no problems for SMARTALLOC; you need only be
concerned if you're using exotic tools for data overlaying on MS-DOS or other
address-space-challenged systems.
<P>
Since a C language ''constant`` string can actually be written into, most C
compilers generate a unique copy of each string used in a module, even if the
same constant string appears many times. In modules that contain many calls on
allocation functions, this results in substantial wasted storage for the
strings that identify the file name. If your compiler permits optimization of
multiple occurrences of constant strings, enabling this mode will eliminate
the overhead for these strings. Of course, it's up to you to make sure
choosing this compiler mode won't wreak havoc on some other part of your
program.
<P>
<H3><A NAME="SECTION000190800000000000000"></A>
<A NAME="5498"></A>
<A NAME="5499"></A>
<BR>
Test and Demonstration Program
</H3>
<A NAME="5502"></A>
<P>
A test and demonstration program, smtest.c, is supplied with SMARTALLOC. You
can build this program with the Makefile included. Please refer to the
comments in smtest.c and the Makefile for information on this program. If
you're attempting to use SMARTALLOC on a new machine or with a new compiler or
operating system, it's a wise first step to check it out with smtest first.
<P>
<H3><A NAME="SECTION000190900000000000000"></A>
<A NAME="5504"></A>
<A NAME="5505"></A>
<BR>
Invitation to the Hack
</H3>
<A NAME="5508"></A>
<P>
SMARTALLOC is not intended to be a panacea for storage management problems,
nor is it universally applicable or effective; it's another weapon in the
arsenal of the defensive professional programmer attempting to create reliable
products. It represents the current state of evolution of expedient debug code
which has been used in several commercial software products which have,
collectively, sold more than third of a million copies in the retail market,
and can be expected to continue to develop through time as it is applied to
ever more demanding projects.
<P>
The version of SMARTALLOC here has been tested on a Sun SPARCStation, Silicon
Graphics Indigo2, and on MS-DOS using both Borland and Microsoft C. Moving
from compiler to compiler requires the usual small changes to resolve disputes
about prototyping of functions, whether the type returned by buffer allocation
is <TT>char *</TT> or <TT>void *</TT>, and so forth, but following those changes
it works in a variety of environments. I hope you'll find SMARTALLOC as useful
for your projects as I've found it in mine.
<P>
<H2><A NAME="SECTION000191000000000000000"></A>
<A NAME="5515"></A>
<A NAME="5516"></A>
<BR>
<A NAME="tex2html22"
HREF="http://www.fourmilab.ch/smartall/smartall.zip">Download smartall.zip</A>
(Zipped archive)
</H2>
<A NAME="5519"></A>
<P>
SMARTALLOC is provided as
<A NAME="tex2html23"
HREF="http://www.fourmilab.ch/smartall/smartall.zip">smartall.zip</A>, a
<A NAME="tex2html24"
HREF="http://www.pkware.com/">Zipped</A>
archive containing source code,
documentation, and a <TT>Makefile</TT> to build the software under Unix.
<P>
<H3><A NAME="SECTION000191100000000000000"></A>
<A NAME="5526"></A>
<BR>
Copying
</H3>
<A NAME="5529"></A>
<P>
<BLOCKQUOTE>
SMARTALLOC is in the public domain. Permission to use, copy, modify, and
distribute this software and its documentation for any purpose and without fee
is hereby granted, without any conditions or restrictions. This software is
provided ''as is`` without express or implied warranty.
</BLOCKQUOTE>
<P>
<I><A NAME="tex2html25"
HREF="http://www.fourmilab.ch">by John Walker</A>
October 30th, 1998 </I>
<P>
<H1><A NAME="SECTION000200000000000000000">
GNU Free Documentation License</A>
</H1>
<A NAME="5934"></A>
<A NAME="5935"></A>
<A NAME="5938"></A>
<P>
<DIV ALIGN="CENTER">
</DIV>
<P>
<DIV ALIGN="CENTER">Version 1.2, November 2002
</DIV>
<P>
<DIV ALIGN="CENTER">Copyright ©2000,2001,2002 Free Software Foundation, Inc.
</DIV>
<P>
<DIV ALIGN="CENTER"></DIV>
<P><P>
<BR>
<DIV ALIGN="CENTER"></DIV>
<P>
<DIV ALIGN="CENTER">51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
</DIV>
<P>
<DIV ALIGN="CENTER"></DIV>
<P><P>
<BR>
<DIV ALIGN="CENTER"></DIV>
<P>
<DIV ALIGN="CENTER">Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
</DIV>
<P>
<DIV ALIGN="CENTER">
<B><FONT SIZE="+1">Preamble</FONT></B>
</DIV>
<P>
The purpose of this License is to make a manual, textbook, or other
functional and useful document "free" in the sense of freedom: to
assure everyone the effective freedom to copy and redistribute it,
with or without modifying it, either commercially or noncommercially.
Secondarily, this License preserves for the author and publisher a way
to get credit for their work, while not being considered responsible
for modifications made by others.
<P>
This License is a kind of "copyleft", which means that derivative
works of the document must themselves be free in the same sense. It
complements the GNU General Public License, which is a copyleft
license designed for free software.
<P>
We have designed this License in order to use it for manuals for free
software, because free software needs free documentation: a free
program should come with manuals providing the same freedoms that the
software does. But this License is not limited to software manuals;
it can be used for any textual work, regardless of subject matter or
whether it is published as a printed book. We recommend this License
principally for works whose purpose is instruction or reference.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>1. APPLICABILITY AND DEFINITIONS</B></FONT>
<A NAME="5948"></A>
</DIV>
<P>
This License applies to any manual or other work, in any medium, that
contains a notice placed by the copyright holder saying it can be
distributed under the terms of this License. Such a notice grants a
world-wide, royalty-free license, unlimited in duration, to use that
work under the conditions stated herein. The <B>"Document"</B>, below,
refers to any such manual or work. Any member of the public is a
licensee, and is addressed as <B>"you"</B>. You accept the license if you
copy, modify or distribute the work in a way requiring permission
under copyright law.
<P>
A <B>"Modified Version"</B> of the Document means any work containing the
Document or a portion of it, either copied verbatim, or with
modifications and/or translated into another language.
<P>
A <B>"Secondary Section"</B> is a named appendix or a front-matter section of
the Document that deals exclusively with the relationship of the
publishers or authors of the Document to the Document's overall subject
(or to related matters) and contains nothing that could fall directly
within that overall subject. (Thus, if the Document is in part a
textbook of mathematics, a Secondary Section may not explain any
mathematics.) The relationship could be a matter of historical
connection with the subject or with related matters, or of legal,
commercial, philosophical, ethical or political position regarding
them.
<P>
The <B>"Invariant Sections"</B> are certain Secondary Sections whose titles
are designated, as being those of Invariant Sections, in the notice
that says that the Document is released under this License. If a
section does not fit the above definition of Secondary then it is not
allowed to be designated as Invariant. The Document may contain zero
Invariant Sections. If the Document does not identify any Invariant
Sections then there are none.
<P>
The <B>"Cover Texts"</B> are certain short passages of text that are listed,
as Front-Cover Texts or Back-Cover Texts, in the notice that says that
the Document is released under this License. A Front-Cover Text may
be at most 5 words, and a Back-Cover Text may be at most 25 words.
<P>
A <B>"Transparent"</B> copy of the Document means a machine-readable copy,
represented in a format whose specification is available to the
general public, that is suitable for revising the document
straightforwardly with generic text editors or (for images composed of
pixels) generic paint programs or (for drawings) some widely available
drawing editor, and that is suitable for input to text formatters or
for automatic translation to a variety of formats suitable for input
to text formatters. A copy made in an otherwise Transparent file
format whose markup, or absence of markup, has been arranged to thwart
or discourage subsequent modification by readers is not Transparent.
An image format is not Transparent if used for any substantial amount
of text. A copy that is not "Transparent" is called <B>"Opaque"</B>.
<P>
Examples of suitable formats for Transparent copies include plain
ASCII without markup, Texinfo input format, LaTeX input format, SGML
or XML using a publicly available DTD, and standard-conforming simple
HTML, PostScript or PDF designed for human modification. Examples of
transparent image formats include PNG, XCF and JPG. Opaque formats
include proprietary formats that can be read and edited only by
proprietary word processors, SGML or XML for which the DTD and/or
processing tools are not generally available, and the
machine-generated HTML, PostScript or PDF produced by some word
processors for output purposes only.
<P>
The <B>"Title Page"</B> means, for a printed book, the title page itself,
plus such following pages as are needed to hold, legibly, the material
this License requires to appear in the title page. For works in
formats which do not have any title page as such, "Title Page" means
the text near the most prominent appearance of the work's title,
preceding the beginning of the body of the text.
<P>
A section <B>"Entitled XYZ"</B> means a named subunit of the Document whose
title either is precisely XYZ or contains XYZ in parentheses following
text that translates XYZ in another language. (Here XYZ stands for a
specific section name mentioned below, such as <B>"Acknowledgements"</B>,
<B>"Dedications"</B>, <B>"Endorsements"</B>, or <B>"History"</B>.)
To <B>"Preserve the Title"</B>
of such a section when you modify the Document means that it remains a
section "Entitled XYZ" according to this definition.
<P>
The Document may include Warranty Disclaimers next to the notice which
states that this License applies to the Document. These Warranty
Disclaimers are considered to be included by reference in this
License, but only as regards disclaiming warranties: any other
implication that these Warranty Disclaimers may have is void and has
no effect on the meaning of this License.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>2. VERBATIM COPYING</B></FONT>
<A NAME="5969"></A>
</DIV>
<P>
You may copy and distribute the Document in any medium, either
commercially or noncommercially, provided that this License, the
copyright notices, and the license notice saying this License applies
to the Document are reproduced in all copies, and that you add no other
conditions whatsoever to those of this License. You may not use
technical measures to obstruct or control the reading or further
copying of the copies you make or distribute. However, you may accept
compensation in exchange for copies. If you distribute a large enough
number of copies you must also follow the conditions in section 3.
<P>
You may also lend copies, under the same conditions stated above, and
you may publicly display copies.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>3. COPYING IN QUANTITY</B></FONT>
<A NAME="5975"></A>
</DIV>
<P>
If you publish printed copies (or copies in media that commonly have
printed covers) of the Document, numbering more than 100, and the
Document's license notice requires Cover Texts, you must enclose the
copies in covers that carry, clearly and legibly, all these Cover
Texts: Front-Cover Texts on the front cover, and Back-Cover Texts on
the back cover. Both covers must also clearly and legibly identify
you as the publisher of these copies. The front cover must present
the full title with all words of the title equally prominent and
visible. You may add other material on the covers in addition.
Copying with changes limited to the covers, as long as they preserve
the title of the Document and satisfy these conditions, can be treated
as verbatim copying in other respects.
<P>
If the required texts for either cover are too voluminous to fit
legibly, you should put the first ones listed (as many as fit
reasonably) on the actual cover, and continue the rest onto adjacent
pages.
<P>
If you publish or distribute Opaque copies of the Document numbering
more than 100, you must either include a machine-readable Transparent
copy along with each Opaque copy, or state in or with each Opaque copy
a computer-network location from which the general network-using
public has access to download using public-standard network protocols
a complete Transparent copy of the Document, free of added material.
If you use the latter option, you must take reasonably prudent steps,
when you begin distribution of Opaque copies in quantity, to ensure
that this Transparent copy will remain thus accessible at the stated
location until at least one year after the last time you distribute an
Opaque copy (directly or through your agents or retailers) of that
edition to the public.
<P>
It is requested, but not required, that you contact the authors of the
Document well before redistributing any large number of copies, to give
them a chance to provide you with an updated version of the Document.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>4. MODIFICATIONS</B></FONT>
<A NAME="5981"></A>
</DIV>
<P>
You may copy and distribute a Modified Version of the Document under
the conditions of sections 2 and 3 above, provided that you release
the Modified Version under precisely this License, with the Modified
Version filling the role of the Document, thus licensing distribution
and modification of the Modified Version to whoever possesses a copy
of it. In addition, you must do these things in the Modified Version:
<P>
<DL COMPACT>
<DT>A.</DT>
<DD>Use in the Title Page (and on the covers, if any) a title distinct
from that of the Document, and from those of previous versions
(which should, if there were any, be listed in the History section
of the Document). You may use the same title as a previous version
if the original publisher of that version gives permission.
<P>
</DD>
<DT>B.</DT>
<DD>List on the Title Page, as authors, one or more persons or entities
responsible for authorship of the modifications in the Modified
Version, together with at least five of the principal authors of the
Document (all of its principal authors, if it has fewer than five),
unless they release you from this requirement.
<P>
</DD>
<DT>C.</DT>
<DD>State on the Title page the name of the publisher of the
Modified Version, as the publisher.
<P>
</DD>
<DT>D.</DT>
<DD>Preserve all the copyright notices of the Document.
<P>
</DD>
<DT>E.</DT>
<DD>Add an appropriate copyright notice for your modifications
adjacent to the other copyright notices.
<P>
</DD>
<DT>F.</DT>
<DD>Include, immediately after the copyright notices, a license notice
giving the public permission to use the Modified Version under the
terms of this License, in the form shown in the Addendum below.
<P>
</DD>
<DT>G.</DT>
<DD>Preserve in that license notice the full lists of Invariant Sections
and required Cover Texts given in the Document's license notice.
<P>
</DD>
<DT>H.</DT>
<DD>Include an unaltered copy of this License.
<P>
</DD>
<DT>I.</DT>
<DD>Preserve the section Entitled "History", Preserve its Title, and add
to it an item stating at least the title, year, new authors, and
publisher of the Modified Version as given on the Title Page. If
there is no section Entitled "History" in the Document, create one
stating the title, year, authors, and publisher of the Document as
given on its Title Page, then add an item describing the Modified
Version as stated in the previous sentence.
<P>
</DD>
<DT>J.</DT>
<DD>Preserve the network location, if any, given in the Document for
public access to a Transparent copy of the Document, and likewise
the network locations given in the Document for previous versions
it was based on. These may be placed in the "History" section.
You may omit a network location for a work that was published at
least four years before the Document itself, or if the original
publisher of the version it refers to gives permission.
<P>
</DD>
<DT>K.</DT>
<DD>For any section Entitled "Acknowledgements" or "Dedications",
Preserve the Title of the section, and preserve in the section all
the substance and tone of each of the contributor acknowledgements
and/or dedications given therein.
<P>
</DD>
<DT>L.</DT>
<DD>Preserve all the Invariant Sections of the Document,
unaltered in their text and in their titles. Section numbers
or the equivalent are not considered part of the section titles.
<P>
</DD>
<DT>M.</DT>
<DD>Delete any section Entitled "Endorsements". Such a section
may not be included in the Modified Version.
<P>
</DD>
<DT>N.</DT>
<DD>Do not retitle any existing section to be Entitled "Endorsements"
or to conflict in title with any Invariant Section.
<P>
</DD>
<DT>O.</DT>
<DD>Preserve any Warranty Disclaimers.
</DD>
</DL>
<P>
If the Modified Version includes new front-matter sections or
appendices that qualify as Secondary Sections and contain no material
copied from the Document, you may at your option designate some or all
of these sections as invariant. To do this, add their titles to the
list of Invariant Sections in the Modified Version's license notice.
These titles must be distinct from any other section titles.
<P>
You may add a section Entitled "Endorsements", provided it contains
nothing but endorsements of your Modified Version by various
parties--for example, statements of peer review or that the text has
been approved by an organization as the authoritative definition of a
standard.
<P>
You may add a passage of up to five words as a Front-Cover Text, and a
passage of up to 25 words as a Back-Cover Text, to the end of the list
of Cover Texts in the Modified Version. Only one passage of
Front-Cover Text and one of Back-Cover Text may be added by (or
through arrangements made by) any one entity. If the Document already
includes a cover text for the same cover, previously added by you or
by arrangement made by the same entity you are acting on behalf of,
you may not add another; but you may replace the old one, on explicit
permission from the previous publisher that added the old one.
<P>
The author(s) and publisher(s) of the Document do not by this License
give permission to use their names for publicity for or to assert or
imply endorsement of any Modified Version.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>5. COMBINING DOCUMENTS</B></FONT>
<A NAME="5989"></A>
</DIV>
<P>
You may combine the Document with other documents released under this
License, under the terms defined in section 4 above for modified
versions, provided that you include in the combination all of the
Invariant Sections of all of the original documents, unmodified, and
list them all as Invariant Sections of your combined work in its
license notice, and that you preserve all their Warranty Disclaimers.
<P>
The combined work need only contain one copy of this License, and
multiple identical Invariant Sections may be replaced with a single
copy. If there are multiple Invariant Sections with the same name but
different contents, make the title of each such section unique by
adding at the end of it, in parentheses, the name of the original
author or publisher of that section if known, or else a unique number.
Make the same adjustment to the section titles in the list of
Invariant Sections in the license notice of the combined work.
<P>
In the combination, you must combine any sections Entitled "History"
in the various original documents, forming one section Entitled
"History"; likewise combine any sections Entitled "Acknowledgements",
and any sections Entitled "Dedications". You must delete all sections
Entitled "Endorsements".
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>6. COLLECTIONS OF DOCUMENTS</B></FONT>
<A NAME="5995"></A>
</DIV>
<P>
You may make a collection consisting of the Document and other documents
released under this License, and replace the individual copies of this
License in the various documents with a single copy that is included in
the collection, provided that you follow the rules of this License for
verbatim copying of each of the documents in all other respects.
<P>
You may extract a single document from such a collection, and distribute
it individually under this License, provided you insert a copy of this
License into the extracted document, and follow this License in all
other respects regarding verbatim copying of that document.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>7. AGGREGATION WITH INDEPENDENT WORKS</B></FONT>
<A NAME="6001"></A>
</DIV>
<P>
A compilation of the Document or its derivatives with other separate
and independent documents or works, in or on a volume of a storage or
distribution medium, is called an "aggregate" if the copyright
resulting from the compilation is not used to limit the legal rights
of the compilation's users beyond what the individual works permit.
When the Document is included in an aggregate, this License does not
apply to the other works in the aggregate which are not themselves
derivative works of the Document.
<P>
If the Cover Text requirement of section 3 is applicable to these
copies of the Document, then if the Document is less than one half of
the entire aggregate, the Document's Cover Texts may be placed on
covers that bracket the Document within the aggregate, or the
electronic equivalent of covers if the Document is in electronic form.
Otherwise they must appear on printed covers that bracket the whole
aggregate.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>8. TRANSLATION</B></FONT>
<A NAME="6007"></A>
</DIV>
<P>
Translation is considered a kind of modification, so you may
distribute translations of the Document under the terms of section 4.
Replacing Invariant Sections with translations requires special
permission from their copyright holders, but you may include
translations of some or all Invariant Sections in addition to the
original versions of these Invariant Sections. You may include a
translation of this License, and all the license notices in the
Document, and any Warranty Disclaimers, provided that you also include
the original English version of this License and the original versions
of those notices and disclaimers. In case of a disagreement between
the translation and the original version of this License or a notice
or disclaimer, the original version will prevail.
<P>
If a section in the Document is Entitled "Acknowledgements",
"Dedications", or "History", the requirement (section 4) to Preserve
its Title (section 1) will typically require changing the actual
title.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>9. TERMINATION</B></FONT>
<A NAME="6013"></A>
</DIV>
<P>
You may not copy, modify, sublicense, or distribute the Document except
as expressly provided for under this License. Any other attempt to
copy, modify, sublicense or distribute the Document is void, and will
automatically terminate your rights under this License. However,
parties who have received copies, or rights, from you under this
License will not have their licenses terminated so long as such
parties remain in full compliance.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>10. FUTURE REVISIONS OF THIS LICENSE</B></FONT>
<A NAME="6019"></A>
</DIV>
<P>
The Free Software Foundation may publish new, revised versions
of the GNU Free Documentation License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns. See
http://www.gnu.org/copyleft/.
<P>
Each version of the License is given a distinguishing version number.
If the Document specifies that a particular numbered version of this
License "or any later version" applies to it, you have the option of
following the terms and conditions either of that specified version or
of any later version that has been published (not as a draft) by the
Free Software Foundation. If the Document does not specify a version
number of this License, you may choose any version ever published (not
as a draft) by the Free Software Foundation.
<P>
<DIV ALIGN="CENTER">
<FONT SIZE="+2"><B>ADDENDUM: How to use this License for your documents</B></FONT>
<A NAME="6025"></A>
</DIV>
<P>
To use this License in a document you have written, include a copy of
the License in the document and put the following copyright and
license notices just after the title page:
<P>
<P><P>
<BR>
<BLOCKQUOTE>
Copyright ©YEAR YOUR NAME.
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts.
A copy of the license is included in the section entitled "GNU
Free Documentation License".
</BLOCKQUOTE>
<P><P>
<BR>
<P>
If you have Invariant Sections, Front-Cover Texts and Back-Cover Texts,
replace the "with...Texts." line with this:
<P>
<P><P>
<BR>
<BLOCKQUOTE>
with the Invariant Sections being LIST THEIR TITLES, with the
Front-Cover Texts being LIST, and with the Back-Cover Texts being LIST.
</BLOCKQUOTE>
<P><P>
<BR>
<P>
If you have Invariant Sections without Cover Texts, or some other
combination of the three, merge those two alternatives to suit the
situation.
<P>
If your document contains nontrivial examples of program code, we
recommend releasing these examples in parallel under your choice of
free software license, such as the GNU General Public License,
to permit their use in free software.
<P>
<P>
<BR>
<H2><A NAME="SECTION000210000000000000000">
General Index</A>
</H2><DL COMPACT>
<DT><STRONG>Catalog Services </STRONG>
<DD><A HREF="developers.html#1796">Catalog Services</A>
<DT><STRONG>Catalog, Internal Bacula </STRONG>
<DD><A HREF="developers.html#1844">Internal Bacula Catalog</A>
<DT><STRONG>Database Table Design </STRONG>
<DD><A HREF="developers.html#1853">Database Table Design</A>
<DT><STRONG>Database Tables </STRONG>
<DD><A HREF="developers.html#1866">Database Tables</A>
<DT><STRONG>Definition, MySQL Table </STRONG>
<DD><A HREF="developers.html#2519">MySQL Table Definition</A>
<DT><STRONG>Design, Database Table </STRONG>
<DD><A HREF="developers.html#1852">Database Table Design</A>
<DT><STRONG>Difficult, GUI Interface is </STRONG>
<DD><A HREF="developers.html#3620">GUI Interface is Difficult</A>
<DT><STRONG>Filenames and Maximum Filename Length </STRONG>
<DD><A HREF="developers.html#1814">Filenames and Maximum Filename Length</A>
<DT><STRONG>General </STRONG>
<DD><A HREF="developers.html#1801">General</A>
| <A HREF="developers.html#3607">General</A>
<DT><STRONG>GNU ree Documentation License</STRONG>
<DD><A HREF="developers.html#5934">GNU Free Documentation License</A>
<DT><STRONG>GUI Interface is Difficult </STRONG>
<DD><A HREF="developers.html#3619">GUI Interface is Difficult</A>
<DT><STRONG>Implementing a Bacula GUI Interface </STRONG>
<DD><A HREF="developers.html#3602">Implementing a Bacula GUI Interface</A>
<DT><STRONG>Installing and Configuring MySQL </STRONG>
<DD><A HREF="developers.html#1821">Installing and Configuring MySQL</A>
<DT><STRONG>Installing and Configuring PostgreSQL </STRONG>
<DD><A HREF="developers.html#1829">Installing and Configuring PostgreSQL</A>
<DT><STRONG>Installing and Configuring SQLite </STRONG>
<DD><A HREF="developers.html#1836">Installing and Configuring SQLite</A>
<DT><STRONG>Interface, Implementing a Bacula GUI </STRONG>
<DD><A HREF="developers.html#3601">Implementing a Bacula GUI Interface</A>
<DT><STRONG>Internal Bacula Catalog </STRONG>
<DD><A HREF="developers.html#1845">Internal Bacula Catalog</A>
<DT><STRONG>Job, Sequence of Creation of Records for a Save </STRONG>
<DD><A HREF="developers.html#1859">Sequence of Creation of Records for a Save Job</A>
<DT><STRONG>Length, Filenames and Maximum Filename </STRONG>
<DD><A HREF="developers.html#1815">Filenames and Maximum Filename Length</A>
<DT><STRONG>License, GNU ree Documentation</STRONG>
<DD><A HREF="developers.html#5935">GNU Free Documentation License</A>
<DT><STRONG>Minimal Code in Console Program </STRONG>
<DD><A HREF="developers.html#3614">Minimal Code in Console Program</A>
<DT><STRONG>MySQL Table Definition </STRONG>
<DD><A HREF="developers.html#2518">MySQL Table Definition</A>
<DT><STRONG>MySQL, Installing and Configuring </STRONG>
<DD><A HREF="developers.html#1820">Installing and Configuring MySQL</A>
<DT><STRONG>PostgreSQL, Installing and Configuring </STRONG>
<DD><A HREF="developers.html#1828">Installing and Configuring PostgreSQL</A>
<DT><STRONG>Program, Minimal Code in Console </STRONG>
<DD><A HREF="developers.html#3613">Minimal Code in Console Program</A>
<DT><STRONG>Sequence of Creation of Records for a Save Job </STRONG>
<DD><A HREF="developers.html#1858">Sequence of Creation of Records for a Save Job</A>
<DT><STRONG>Services, Catalog </STRONG>
<DD><A HREF="developers.html#1795">Catalog Services</A>
<DT><STRONG>SQLite, Installing and Configuring </STRONG>
<DD><A HREF="developers.html#1837">Installing and Configuring SQLite</A>
<DT><STRONG>Tables, Database </STRONG>
<DD><A HREF="developers.html#1867">Database Tables</A>
</DL>
<P>
<H1><A NAME="SECTION000220000000000000000">
About this document ...</A>
</H1>
<STRONG><IMG
WIDTH="457" HEIGHT="99" ALIGN="BOTTOM" BORDER="0"
SRC="bacula-logo.png"
ALT="\includegraphics{./bacula-logo.eps}">
<BR><P><P>
<BR>
<DIV ALIGN="CENTER">
<FONT SIZE="+1">It comes in the night and sucks
the essence from your computers.
</FONT></DIV></STRONG><P>
This document was generated using the
<A HREF="http://www.latex2html.org/"><STRONG>LaTeX</STRONG>2<tt>HTML</tt></A> translator Version 2008 (1.71)
<P>
Copyright © 1993, 1994, 1995, 1996,
<A HREF="http://cbl.leeds.ac.uk/nikos/personal.html">Nikos Drakos</A>,
Computer Based Learning Unit, University of Leeds.
<BR>
Copyright © 1997, 1998, 1999,
<A HREF="http://www.maths.mq.edu.au/~ross/">Ross Moore</A>,
Mathematics Department, Macquarie University, Sydney.
<P>
The command line arguments were: <BR>
<STRONG>latex2html</STRONG> <TT>-white -no_subdir -split 0 -toc_stars -white -notransparent developers</TT>
<P>
The translation was initiated by on 2009-09-10<HR>
<!--Navigation Panel-->
<IMG WIDTH="81" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="next_inactive"
SRC="file:/usr/lib/latex2html/icons/nx_grp_g.png">
<IMG WIDTH="26" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="up"
SRC="file:/usr/lib/latex2html/icons/up_g.png">
<IMG WIDTH="63" HEIGHT="24" ALIGN="BOTTOM" BORDER="0" ALT="previous"
SRC="file:/usr/lib/latex2html/icons/prev_g.png">
<BR>
<!--End of Navigation Panel-->
<ADDRESS>
2009-09-10
</ADDRESS>
</BODY>
</HTML>
distrib
>
Mandriva
>
2010.0
>
i586
>
media
>
contrib-release
>
by-pkgid
>
73b2f9d46bafc15680e1ea470cc9cd24
>
files
>
3
