\input texinfo @c -*-texinfo-*-
@c 19980817: TeX-based systems (texi2dvi, texi2dvi --pdf) require texinfo.tex
@c Hyper-text systems (texi2html, makeinfo) do not require texinfo.tex
@ignore
$Header: /cvsroot/nco/nco/doc/nco.texi,v 1.480 2008/05/11 20:30:37 zender Exp $
Purpose: TeXInfo documentation for NCO suite
URL: http://nco.sf.net/nco.texi
Copyright (C) 1995--2008 Charlie Zender
Permission is granted to copy, distribute and/or modify this document
under the terms of the GNU Free Documentation License, Version 1.2
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at
http://www.gnu.org/copyleft/fdl.html
The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)
Department of Earth System Science
3200 Croul Hall
University of California, Irvine
Irvine, CA 92697-3100
After editing any hyperlink locations, use
C-c C-u C-a texinfo-all-menus-update
C-c C-u C-e texinfo-every-node-update
Multiple files:
M-x texinfo-multiple-files-update
Usage:
export TEX='tex --src-specials'
cd ~/nco/doc;texi2dvi nco.texi; makeinfo --html --ifinfo --no-split --output=nco.html nco.texi; makeinfo nco.texi; dvips -o nco.ps nco.dvi;texi2dvi --pdf nco.texi;makeinfo --xml --ifinfo --no-split --output=nco.xml nco.texi;makeinfo --no-headers --output=nco.txt nco.texi
dvips -o nco.ps nco.dvi
dvips -Ppdf -G0 -o nco.ps nco.dvi
makeinfo --html --ifinfo --no-split --output=nco.html nco.texi
makeinfo --no-split --output=nco.info nco.texi
makeinfo --no-headers --no-split --output=nco.txt nco.texi
makeinfo --xml --no-split --output=nco.xml nco.texi
makeinfo --docbook --no-split --output=nco.xml nco.texi
pdftotext nco.pdf nco.txt
ps2pdf -dMaxSubsetPct=100 -dCompatibilityLevel=1.2 -dSubsetFonts=true -dEmbedAllFonts=true nco.ps nco.pdf
texi2dvi --output=nco.dvi nco.texi
texi2dvi --pdf --output=nco.pdf nco.texi
texi2html -monolithic -verbose nco.texi
texi2html -l2h -l2h_tmp=./l2h_tmp -monolithic -verbose nco.texi # Invoke latex2html
cd ~/nco/doc;/usr/bin/scp index.shtml nco_news.shtml ChangeLog TODO README VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml ../data/ncap.in ../data/ncap2.in nco.sf.net:/home/groups/n/nc/nco/htdocs;cd -
cd ~/nco/doc; scp -p index.shtml nco_news.shtml ChangeLog TODO README VERSION nco.dvi nco.html nco.info* nco.pdf nco.ps nco.texi nco.txt nco.xml ../data/ncap.in ../data/ncap2.in dust.ess.uci.edu:/var/www/html/nco;cd -
NB: @cindex references in footnotes propagate to PDF file, but not to
HTML files, bug-report sent to makeinfo people 20040229
Producing HTML, makeinfo vs. texi2html:
makeinfo: Better format overall
Uses node names for cross references and index
Acronyms look better
Excludes @ifinfo sections by default (override with --ifinfo)
texi2html: Index sub-divided by first character
More fancy options (e.g., latex2html), though none very useful
Misprints title
Includes @ifinfo sections by default
Legend (defined in "highlighting" section of TeXInfo manual):
@code{}: Program text, e.g., @code{if(foo) x=y;}
@command{}: Commands, e.g., @command{ncra}
@dfn{}: Define use of term, e.g., @dfn{supercalifragilisticexpialidocious}
@email{}: E-mail address, e.g., @email{surname at uci dot edu}
@env{}: Environment variable, e.g., @env{HOME}
@file{}: Filename, e.g., @file{in.nc}
@html: Text until @end html passed without translation
@ifhtml: Text until @end ifhtml passed with translation
@kbd{}: Keyboard input, e.g., @kbd{ncra in.nc out.nc}
@key{}: Key name, e.g., @key{ESC} (rarely needed)
@option{}: Command-line option, e.g., @option{--dbg}
@samp{}: Extended commands, character sequences, e.g., @samp{ncra in.nc out.nc}
@uref{}: URL with optional alternate text, e.g., @uref{http://nco.sf.net,NCO homepage}
@url{}: URL, synonym for @uref
@var{}: Metasyntactic variable, e.g., @var{}
@w{}: Unbreakable text, e.g., @w{of 1}
Use '@*' to force hard carriage return
Use '@:', after periods, questions marks, exclamation marks, or colons
that do not end sentences, e.g., 'vs.@:'
Use '@.', '@!', and `@?' to end sentences that end with single capital letters (e.g., initials)
Resources:
Octave TeXInfo manual shows clean TeXInfo structure
/usr/share/doc/octave-2.1.34/interpreter/octave.texi
@end ignore
@c Start of header
@c No variables may be defined before TeXInfo @setfilename header
@setfilename nco.info
@c Define edition, date, ...
@set nco-edition 3.9.5
@set doc-edition 3.9.5
@set copyright-years 1995--2008
@set update-year 2008
@set update-date 11 May 2008
@set update-month May 2008
@settitle @acronym{NCO} @value{nco-edition} User's Guide
@c Uncomment following line to produce guide in smallbook format
@c @smallbook
@c Merge function index into concept index
@syncodeindex fn cp
@c end of header
@c TeXInfo macros may not appear before TeXInfo @setfilename header
@c [idx] Index
@macro idx {}
i
@end macro
@c [m s-1] Meridional wind speed
@macro wndmrd {}
v
@end macro
@c [m s-1] Zonal wind speed
@macro wndznl {}
u
@end macro
@macro xxx {}
x
@end macro
@c TeX macros may appear anywhere after line 1
@tex
% Define TeX macros to roughly correspond to LaTeX style files
% Use \gdef instead of \def to make definition persistent across TeX blocks
% These should be consistent with any TeXInfo macros
% 1. Primary commands
\gdef\dmn{D} % [dmn] Variable dimension
\gdef\rdr{R} % [dmn] Re-order dimension
\gdef\shr{S} % [dmn] Share dimension
\gdef\dmnidx{n} % [idx] Dimension index
\gdef\dmnnbr{N} % [nbr] Dimension number
\gdef\rdridx{r} % [idx] Re-order index
\gdef\rdrnbr{R} % [nbr] Re-order number
\gdef\shridx{s} % [idx] Share index
\gdef\shrnbr{S} % [nbr] Share number
\gdef\dfr{{\rm d}} % [frc] Math differential fxm: upright
\gdef\idx{i} % [idx] Index
\gdef\iii{i} % [idx] i
\gdef\jjj{j} % [idx] j
\gdef\kkk{k} % [idx] k
\gdef\lmnidx{i} % [idx] Element index
\gdef\outnbr{J} % [nbr] Number of elements in output hyperslab
\gdef\lmnnbr{N} % [nbr] Number of elements in input hyperslab
\gdef\tllnbr{M} % [nbr] Tally (number of valid elements in input hyperslab)
\gdef\me{{\rm e}} % [frc] Math e fxm: upright
\gdef\mi{{\rm i}} % [frc] Math i fxm: upright
\gdef\mpi{\pi} % [frc] Math pi
\gdef\mpp{\cal{M}} % [map] Map
\gdef\mskflg{m} % [flg] Mask flag
\gdef\mssflg{\mu} % [flg] Missing value flag
\gdef\tm{t} % [s] Time
\gdef\prmsbs{\prime} % [sbs] Prime subscript
\gdef\wgt{w} % [frc] Weight
\gdef\wndmrd{v} % [m s-1] Meridional wind speed
\gdef\wndznl{u} % [m s-1] Zonal wind speed
\gdef\xxx{x} % [ltr] x
\gdef\yyy{y} % [ltr] y
% 2. Derived commands
\gdef\dmnvct{{\bf \dmn}} % [vct] Dimension vector
\gdef\dmnprm{\dmn^{\prmsbs}} % [vct] Dimension prime
\gdef\dmnsubnnn{\dmn_{\dmnidx}} % [dmn] Dimension sub nnn
\gdef\shrsubnnn{\shr_{\dmnidx}} % [dmn] Share dimension sub nnn
\gdef\shrsubsss{\shr_{\shridx}} % [dmn] Share dimension sub sss
\gdef\dmnsubnnnprm{\dmn_{\dmnidx}^{\prmsbs}} % [vct] Dimension prime sub nnn
\gdef\dmnvctprm{{\bf \dmn}^{\prmsbs}} % [vct] Dimension vector prime
\gdef\rdrvct{{\bf \rdr}} % [vct] Re-order vector
\gdef\shrvct{{\bf \shr}} % [vct] Share vector
\gdef\xxxprm{\xxx^{\prmsbs}} % [ltr] x prime
% 3. Doubly derived commands
@end tex
@c install-info installs NCO info into this category
@dircategory netCDF
@direntry
* NCO:: User's Guide for the netCDF Operator suite
@end direntry
@iftex
@tolerance 10000
@end iftex
@c Set smallbook if printing in smallbook format
@c Example of smallbook font is actually written using smallbook
@c In bigbook, a kludge is used for TeX output
@c set smallbook
@clear smallbook
@tex
% fxm: Try to get thumbnails working with texinfo --pdf -generated PDF files
% \input thumbpdf.sty
% Experiment with smaller amounts of whitespace between chapters and sections
\global\chapheadingskip = 15pt plus 4pt minus 2pt
\global\secheadingskip = 12pt plus 3pt minus 2pt
\global\subsecheadingskip = 9pt plus 2pt minus 2pt
@end tex
@c Experiment with smaller amounts of whitespace between paragraphs in the 8.5 by 11 inch format
@ifclear smallbook
@tex
\global\parskip 6pt plus 1pt
@end tex
@end ifclear
@c Uncomment next line to remove ugly TeX warning blocks from overfull hboxes
@finalout
@ifinfo
This file documents @acronym{NCO}, a collection of utilities to
manipulate and analyze netCDF files.
Copyright @copyright{} @value{copyright-years} Charlie Zender
This is the first edition of the @cite{NCO User's Guide},@*
and is consistent with @w{version 2} of @file{texinfo.tex}.
Permission is granted to copy, distribute and/or modify this document
under the terms of the @acronym{GNU} Free Documentation License, @w{Version 1.2}
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at
@uref{http://www.gnu.org/copyleft/fdl.html}
The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.@*
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)@*
3200 Croul Hall@*
Department of Earth System Science@*
University of California, Irvine@*
Irvine, CA 92697-3100@*
@ignore
Permission is granted to process this file through TeX and print the
results, provided the printed document carries copying permission
notice identical to this one except for the removal of this paragraph
(this paragraph not being relevant to the printed manual).
@end ignore
@end ifinfo
@setchapternewpage odd
@titlepage
@html
<meta name="Author" content="Charlie Zender">
<meta name="Keywords" content="NCO documentation, NCO User's Guide,
netCDF, operator, GCM, CCM, scientific data, ncbo, ncea, ncecat,
ncflint, ncks, ncra, ncrcat, ncrename, ncwa">
@end html
@html
<body text="#000000" link="#0000EF" vlink="#008080" alink="#FF0000">
<font face="Arial">
@end html
@ignore
@end ignore
@title NCO User's Guide
@subtitle A suite of netCDF operators
@subtitle Edition @value{doc-edition}, for @acronym{NCO} Version @value{nco-edition}
@subtitle @value{update-month}
@author by Charlie Zender
@author Department of Earth System Science
@author University of California, Irvine
@html
<p>WWW readers: Having trouble finding the section you want?</p>
<p>Search for keywords in the <a href="#index">(hyper) index</a> at the end</p>
@end html
@c Include Distribution inside titlepage so that headings are turned off
@page
@vskip 0pt plus 1filll
Copyright @copyright{} @value{copyright-years} Charlie Zender.
@sp 2
This is the first edition of the @cite{NCO User's Guide},@*
and is consistent with @w{version 2} of @file{texinfo.tex}.
@sp 2
Published by Charlie Zender@*
Department of Earth System Science@*
3200 Croul Hall@*
University of California, Irvine@*
Irvine, CA 92697-3100 USA@*
Permission is granted to copy, distribute and/or modify this document
under the terms of the @acronym{GNU} Free Documentation License, @w{Version 1.2}
or any later version published by the Free Software Foundation;
with no Invariant Sections, no Front-Cover Texts, and no Back-Cover
Texts. The license is available online at
@uref{http://www.gnu.org/copyleft/fdl.html}
@sp 2
The original author of this software, Charlie Zender, wants to improve it
with the help of your suggestions, improvements, bug-reports, and patches.@*
Charlie Zender <surname at uci dot edu> (yes, my surname is zender)@*
Department of Earth System Science@*
3200 Croul Hall@*
University of California, Irvine@*
Irvine, CA 92697-3100@*
@sp 2
@c Cover art by Robynn Rudel
@end titlepage
@node Top, Foreword, (dir), (dir)
@ifinfo
@top NCO User's Guide
@c node format: node-name, next, previous, up
@ifhtml
@cartouche
@html
<p><b>Note to readers of the NCO User's Guide in HTML format</b>:
<b>The <a href="./nco.pdf">NCO User's Guide in PDF format</a>
(also on <a href="http://nco.sf.net/nco.pdf">SourceForge</a>)
contains the complete NCO documentation.</b>
<br>This HTML documentation is equivalent except it refers you to the
printed (i.e., DVI, PostScript, and PDF) documentation for description
of complex mathematical expressions.
@end html
@end cartouche
@end ifhtml
@ifnothtml
@emph{Note to readers of the NCO User's Guide in Info format}:
@emph{The @uref{./nco.pdf,NCO User's Guide in PDF format}
(also on @uref{http://nco.sf.net/nco.pdf,SourceForge})
contains the complete @acronym{NCO} documentation.}
This Info documentation is equivalent except it refers you to the
printed (i.e., DVI, PostScript, and PDF) documentation for description
of complex mathematical expressions.
@end ifnothtml
The netCDF Operators, or @acronym{NCO}, are a suite of programs known as
operators.
The operators facilitate manipulation and analysis of data stored in the
self-describing netCDF format, available from
(@uref{http://www.unidata.ucar.edu/packages/netcdf}).
Each @acronym{NCO} operator (e.g., ncks) takes netCDF input
file(s), performs an operation (e.g., averaging, hyperslabbing, or
renaming), and outputs a processed netCDF file.
Although most users of netCDF data are involved in scientific research,
these data formats, and thus @acronym{NCO}, are generic and are equally
useful in fields from agriculture to zoology.
The @acronym{NCO} User's Guide illustrates @acronym{NCO} use with
examples from the field of climate modeling and analysis.
The @acronym{NCO} homepage is @uref{http://nco.sf.net}, and
there is a mirror at @uref{http://dust.ess.uci.edu/nco}.
This documentation is for @acronym{NCO} version @value{nco-edition}.
It was last updated @value{update-date}.
Corrections, additions, and rewrites of this documentation are very
welcome.
Enjoy,@*
Charlie Zender
@end ifinfo
@menu
* Foreword::
* Summary::
* Introduction::
* Strategies::
* Common features::
* Operator Reference Manual::
* Contributing::
* CCSM Example::
* General Index::
@end menu
@html
<a name="fwd"></a> <!-- http://nco.sf.net/nco.html#fwd -->
@end html
@node Foreword, Summary, Top, Top
@unnumbered Foreword
@cindex foreword
@cindex Charlie Zender
@acronym{NCO} is the result of software needs that arose while I worked
on projects funded by @acronym{NCAR}, @acronym{NASA}, and @acronym{ARM}.
Thinking they might prove useful as tools or templates to others,
it is my pleasure to provide them freely to the scientific community.
Many users (most of whom I have never met) have encouraged the
development of @acronym{NCO}.
Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo @w{da Silva},
John Sheldon, and William Weibel for stimulating suggestions and
correspondence.
Your encouragment motivated me to complete the @cite{NCO User's Guide}.
So if you like @acronym{NCO}, send me a note!
@w{I should} mention that @acronym{NCO} is not connected to or
officially endorsed by Unidata, @acronym{ACD}, @acronym{ASP},
@acronym{CGD}, or Nike.@*
@sp 1
@noindent
Charlie Zender@*
May 1997@*
Boulder, Colorado@*
@sp 2
Major feature improvements entitle me to write another Foreword.
In the last five years a lot of work has been done to refine
@acronym{NCO}.
@cindex open source
@acronym{NCO} is now an open source project and appears to be much
healthier for it.
The list of illustrious institutions that do not endorse @acronym{NCO}
continues to grow, and now includes @acronym{UCI}.@*
@sp 1
@noindent
Charlie Zender@*
October 2000@*
Irvine, California@*
@sp 2
The most remarkable advances in @acronym{NCO} capabilities in the last
few years are due to contributions from the Open Source community.
Especially noteworthy are the contributions of Henry Butowsky and Rorik
Peterson.@*
@sp 1
@noindent
Charlie Zender@*
January 2003@*
Irvine, California@*
@sp 2
@acronym{NCO} has been generously supported from 2004--2008 by US
National Science Foundation (@acronym{NSF})grant
@uref{http://www.nsf.gov/awardsearch/showAward.do?AwardNumber=0431203,IIS-0431203}.
This support allowed me to maintain and extend core @acronym{NCO} code,
and others to advance @acronym{NCO} in new directions:
Gayathri Venkitachalam helped implement @acronym{MPI};
Harry Mangalam improved regression testing and benchmarking;
Daniel Wang developed the server-side capability, @acronym{SWAMP};
and Henry Butowsky, a long-time contributor, developed @command{ncap2}.
This support also led @acronym{NCO} to debut in professional journals
and meetings.
The personal and professional contacts made during this evolution have
been immensely rewarding.@*
@sp 1
@noindent
Charlie Zender@*
March 2008@*
Grenoble, France@*
@html
<a name="smr"></a> <!-- http://nco.sf.net/nco.html#smr -->
@end html
@node Summary, Introduction, Foreword, Top
@unnumbered Summary
@cindex operators
@cindex summary
This manual describes @acronym{NCO}, which stands for netCDF Operators.
@acronym{NCO} is a suite of programs known as @dfn{operators}.
Each operator is a standalone, command line program executed at the
shell-level like, e.g., @command{ls} or @command{mkdir}.
The operators take netCDF files (including @acronym{HDF5} files
constructed using the netCDF @acronym{API}) as input, perform an
operation (e.g., averaging or hyperslabbing), and produce a netCDF file
as output.
The operators are primarily designed to aid manipulation and analysis of
data.
The examples in this documentation are typical applications of the
operators for processing climate model output.
This stems from their origin, though the operators are as general as
netCDF itself.
@html
<a name="ntr"></a> <!-- http://nco.sf.net/nco.html#ntr -->
@end html
@node Introduction, Strategies, Summary, Top
@chapter Introduction
@cindex introduction
@menu
* Availability::
* Compatability::
* Libraries::
* netCDF2/3/4 and HDF4/5 Support::
* Help Requests and Bug Reports::
@end menu
@node Availability, Compatability, Introduction, Introduction
@section Availability
@cindex @acronym{NCO} availability
@cindex source code
The complete @acronym{NCO} source distribution is currently distributed
as a @dfn{compressed tarfile} from
@uref{http://sf.net/projects/nco}
and from
@uref{http://dust.ess.uci.edu/nco/nco.tar.gz}.
The compressed tarfile must be uncompressed and untarred before building
@acronym{NCO}.
Uncompress the file with @samp{gunzip nco.tar.gz}.
Extract the source files from the resulting tarfile with @samp{tar -xvf
nco.tar}.
@acronym{GNU} @code{tar} lets you perform both operations in one step
with @samp{tar -xvzf nco.tar.gz}.
@cindex documentation
@cindex WWW documentation
@cindex on-line documentation
@cindex @acronym{HTML}
@cindex @TeX{}info
@cindex Info
@cindex @cite{User's Guide}
@cindex @cite{NCO User's Guide}
The documentation for @acronym{NCO} is called the
@cite{NCO User's Guide}.
The @cite{User's Guide} is available in Postscript, @acronym{HTML},
@acronym{DVI}, @TeX{}info, and Info formats.
These formats are included in the source distribution in the files
@file{nco.ps}, @file{nco.html}, @file{nco.dvi}, @file{nco.texi}, and
@file{nco.info*}, respectively.
All the documentation descends from a single source file,
@file{nco.texi}
@footnote{
To produce these formats, @file{nco.texi} was simply run through the
freely available programs @code{texi2dvi}, @code{dvips},
@code{texi2html}, and @code{makeinfo}.
Due to a bug in @TeX{}, the resulting Postscript file, @file{nco.ps},
contains the Table of Contents as the final pages.
Thus if you print @file{nco.ps}, remember to insert the Table of
Contents after the cover sheet before you staple the manual.
}.
Hence the documentation in every format is very similar.
However, some of the complex mathematical expressions needed to describe
@command{ncwa} can only be displayed in @acronym{DVI}, Postscript, and
@acronym{PDF} formats.
@cindex @acronym{NCO} homepage
If you want to quickly see what the latest improvements in @acronym{NCO}
are (without downloading the entire source distribution), visit the
@acronym{NCO} homepage at
@uref{http://nco.sf.net}.
The @acronym{HTML} version of the @cite{User's Guide} is also available
online through the World Wide Web at @acronym{URL}
@uref{http://nco.sf.net/nco.html}.
@cindex netCDF
To build and use @acronym{NCO}, you must have netCDF installed.
The netCDF homepage is
@uref{http://www.unidata.ucar.edu/packages/netcdf}.
New @acronym{NCO} releases are announced on the netCDF list
and on the @code{nco-announce} mailing list
@uref{http://lists.sf.net/mailman/listinfo/nco-announce}.
@ignore
This tests incorporates an image using the @code{@@image} command.
@image{/data/zender/ps/odxc,6in,}
@end ignore
@node Compatability, Libraries, Availability, Introduction
@section Operating systems compatible with @acronym{NCO}
@cindex @acronym{OS}
@cindex @acronym{IBM}
@cindex @acronym{NEC}
@cindex @acronym{SGI}
@cindex @acronym{HP}
@cindex @acronym{DEC}
@cindex @acronym{PGI}
@cindex Cray
@cindex Digital
@cindex Sun
@cindex Intel
@cindex Comeau
@cindex Compaq
@cindex Macintosh
@cindex Microsoft
@cindex Windows
@cindex PathScale
@cindex QLogic
@cindex compatability
@cindex portability
@cindex installation
@acronym{NCO} has been successfully ported and tested and is known to
work on the following 32- and 64-bit platforms:
@c alphabetize by OS name
@acronym{IBM AIX} 4.x, 5.x,
FreeBSD 4.x,
@acronym{GNU}/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxARM, LinuxSparc64,
@acronym{SGI IRIX} 5.x and 6.x,
@w{MacOS X} 10.x,
@acronym{NEC} Super-UX 10.x,
@acronym{DEC OSF},
Sun SunOS 4.1.x, Solaris 2.x,
@acronym{Cray UNICOS} 8.x--10.x,
and MS Windows95 and all later versions.
If you port the code to a new operating system, please send me a note
and any patches you required.
@cindex @acronym{UNIX}
@cindex Unidata
@cindex UDUnits
The major prerequisite for installing @acronym{NCO} on a particular
platform is the successful, prior installation of the netCDF library
(and, as of 2003, the UDUnits library).
Unidata has shown a commitment to maintaining netCDF and UDUnits on all
popular @acronym{UNIX} platforms, and is moving towards full support for
the Microsoft Windows operating system (@acronym{OS}).
Given this, the only difficulty in implementing @acronym{NCO} on a
particular platform is standardization of various @w{C and} Fortran
interface and system calls.
@acronym{NCO} code is tested for @acronym{ANSI} compliance by
compiling with @w{C compilers} including those from
@cindex @command{CC}
@cindex @command{c++}
@cindex @command{cc}
@cindex @command{como}
@cindex @command{cxx}
@cindex @command{gcc}
@cindex @command{icc}
@cindex @command{pgcc}
@cindex @command{pgCC}
@cindex @command{pathcc}
@cindex @command{pathCC}
@cindex @command{xlC}
@cindex @command{xlc}
@acronym{GNU} (@samp{gcc -std=c99 -pedantic -D_BSD_SOURCE -D_POSIX_SOURCE} -Wall)
@footnote{
The @samp{_BSD_SOURCE} token is required on some Linux platforms where
@command{gcc} dislikes the network header files like
@file{netinet/in.h}).},
Comeau Computing (@samp{como --c99}),
Cray (@samp{cc}),
@acronym{HP}/Compaq/@acronym{DEC} (@samp{cc}),
@acronym{IBM} (@samp{xlc -c -qlanglvl=extc99}),
Intel (@samp{icc -std=c99}),
@acronym{NEC} (@samp{cc}),
PathScale (QLogic) (@samp{pathcc -std=c99}),
@acronym{PGI} (@samp{pgcc -c9x}),
@acronym{SGI} (@samp{cc -c99}),
and
Sun (@samp{cc}).
@cindex C++
@cindex @acronym{ISO}
@cindex @command{libnco}
@acronym{NCO} (all commands and the @command{libnco} library) and
the C++ interface to netCDF (called @command{libnco_c++}) comply with
the @acronym{ISO} C++ standards as implemented by
Comeau Computing (@samp{como}),
Cray (@samp{CC}),
@acronym{GNU} (@samp{g++ -Wall}),
@acronym{HP}/Compaq/@acronym{DEC} (@samp{cxx}),
@acronym{IBM} (@samp{xlC}),
Intel (@samp{icc}),
@acronym{NEC} (@samp{c++}),
PathScale (Qlogic) (@samp{pathCC}),
@acronym{PGI} (@samp{pgCC}),
@acronym{SGI} (@samp{CC -LANG:std}),
and
Sun (@samp{CC -LANG:std}).
@cindex @file{Makefile}
See @file{nco/bld/Makefile} and @file{nco/src/nco_c++/Makefile.old} for
more details and exact settings.
@cindex @acronym{ANSI}
@cindex C89
@cindex @code{printf}
Until recently (and not even yet), @acronym{ANSI}-compliant has meant
compliance with the 1989 @acronym{ISO} C-standard, usually called C89 (with
minor revisions made in 1994 and 1995).
C89 lacks variable-size arrays, restricted pointers, some useful
@code{printf} formats, and many mathematical special functions.
@cindex C99
These are valuable features of C99, the 1999 @acronym{ISO} C-standard.
@acronym{NCO} is C99-compliant where possible and C89-compliant where
necessary.
Certain branches in the code are required to satisfy the native
@acronym{SGI} and SunOS @w{C compilers}, which are strictly @acronym{ANSI}
C89 compliant, and cannot benefit from C99 features.
However, C99 features are fully supported by modern @acronym{AIX},
@acronym{GNU}, Intel, @acronym{NEC}, Solaris, and @acronym{UNICOS}
compilers.
@acronym{NCO} requires a C99-compliant compiler as of @acronym{NCO}
@w{version 2.9.8}, released in August, 2004.
The most time-intensive portion of @acronym{NCO} execution is spent in
arithmetic operations, e.g., multiplication, averaging, subtraction.
These operations were performed in Fortran by default until August,
1999.
This was a design decision based on the relative speed of Fortran-based
object code vs.@: C-based object code in late 1994.
@w{C compiler} vectorization capabilities have dramatically improved
since 1994.
We have accordingly replaced all Fortran subroutines with @w{C functions}.
This greatly simplifies the task of building @acronym{NCO} on nominally
unsupported platforms.
@cindex C language
As of August 1999, @acronym{NCO} built entirely @w{in C} by default.
This allowed @acronym{NCO} to compile on any machine with an
@acronym{ANSI} @w{C compiler}.
@cindex C99
@cindex C89
@cindex @code{restrict}
In August 2004, the first C99 feature, the @code{restrict} type
qualifier, entered @acronym{NCO} in version 2.9.8.
@w{C compilers} can obtain better performance with C99 restricted
pointers since they inform the compiler when it may make Fortran-like
assumptions regarding pointer contents alteration.
Subsequently, @acronym{NCO} requires a C99 compiler to build correctly
@footnote{@acronym{NCO} may still build with an
@acronym{ANSI} or @acronym{ISO} C89 or C94/95-compliant compiler if the
@w{C pre-processor} undefines the @code{restrict} type qualifier, e.g.,
by invoking the compiler with @samp{-Drestrict=''}.}.
@cindex @var{gamma}
In June 2005, @acronym{NCO} version 3.0.1 began to take advantage
of C99 mathematical special functions.
These include the standarized gamma function (called @code{tgamma()}
for ``true gamma'').
@cindex automagic
@acronym{NCO} automagically takes advantage of some @acronym{GNU}
Compiler Collection (@acronym{GCC}) extensions to @w{@acronym{ANSI} C}.
As of July 2000 and @acronym{NCO} @w{version 1.2}, @acronym{NCO} no
longer performs arithmetic operations in Fortran.
We decided to sacrifice executable speed for code maintainability.
Since no objective statistics were ever performed to quantify
the difference in speed between the Fortran and @w{C code},
the performance penalty incurred by this decision is unknown.
Supporting Fortran involves maintaining two sets of routines for every
arithmetic operation.
The @code{USE_FORTRAN_ARITHMETIC} flag is still retained in the
@file{Makefile}.
The file containing the Fortran code, @file{nco_fortran.F}, has been
deprecated but a volunteer (@w{Dr.@: Frankenstein}?) could resurrect it.
If you would like to volunteer to maintain @file{nco_fortran.F} please
contact me.
@c Following section is obsolete
@ignore
It is still possible to request Fortran routines to perform arithmetic
operations, however.
@cindex preprocessor tokens
@cindex @code{USE_FORTRAN_ARITHMETIC}
This can be accomplished by defining the preprocessor token
@code{USE_FORTRAN_ARITHMETIC} and rebuilding @acronym{NCO}.
@cindex performance
As its name suggests, the @code{USE_FORTRAN_ARITHMETIC} token instructs
@acronym{NCO} to attempts to interface the @w{C routines} with Fortran
arithmetic.
Although using Fortran calls instead @w{of C} reduces the portability and
and increases the maintenance of the @acronym{NCO} operators, it may
also increase the performance of the numeric operators.
Presumably this will depend on your machine type, the quality of @w{the C}
and Fortran compilers, and the size of the data files
@footnote{If you decide to test the efficiency of the averagers compiled
with @code{USE_FORTRAN_ARITHMETIC} versus the default @w{C averagers} I
would be most interested to hear the results.
Please E-mail me the results including the size of the datasets, the
platform, and the change in the wallclock time for execution.}.
@end ignore
@menu
* Windows Operating System::
@end menu
@html
<a name="wnd"></a> <!-- http://nco.sf.net/nco.html#wnd -->
@end html
@node Windows Operating System, , Compatability, Compatability
@subsection Compiling @acronym{NCO} for Microsoft Windows @acronym{OS}
@cindex Windows
@cindex Microsoft
@cindex XP (Microsoft operating system)
@cindex NT (Microsoft operating system)
@acronym{NCO} has been successfully ported and tested on the Microsoft
Windows (95/98/NT/2000/XP) operating systems.
The switches necessary to accomplish this are included in the standard
distribution of @acronym{NCO}.
Using the freely available Cygwin (formerly gnu-win32) development
environment
@footnote{The Cygwin package is available from@*
@code{http://sourceware.redhat.com/cygwin}@*
@cindex @code{gcc}
@cindex @code{g++}
@cindex @code{g77}
Currently, @w{Cygwin 20.x} comes with the @acronym{GNU} C/C++/Fortran
compilers (@command{gcc}, @command{g++}, @command{g77}).
These @acronym{GNU} compilers may be used to build the netCDF
distribution itself.}, the compilation process is very similar to
installing @acronym{NCO} on a @acronym{UNIX} system.
@cindex preprocessor tokens
@cindex Cygwin
@cindex @code{gnu-win32}
@cindex @code{WIN32}
@cindex @file{GNUmakefile}
@cindex @file{Makefile}
@cindex @code{f90}
Set the @code{PVM_ARCH} preprocessor token to @code{WIN32}.
Note that defining @code{WIN32} has the side effect of disabling
Internet features of @acronym{NCO} (see below).
@acronym{NCO} should now build like it does on @acronym{UNIX}.
@cindex @acronym{UNIX}
@cindex @code{getuid}
@cindex @code{gethostname}
@findex @file{<arpa/nameser.h>}
@findex @file{<resolv.h>}
The least portable section of the code is the use of standard
@acronym{UNIX} and Internet protocols (e.g., @code{ftp}, @code{rcp},
@code{scp}, @code{sftp}, @code{getuid}, @code{gethostname}, and header
files @file{<arpa/nameser.h>} and
@file{<resolv.h>}).
@cindex @code{ftp}
@cindex @code{sftp}
@cindex @code{rcp}
@cindex @code{scp}
@cindex @acronym{SSH}
@cindex remote files
Fortunately, these @acronym{UNIX}-y calls are only invoked by the single
@acronym{NCO} subroutine which is responsible for retrieving files
stored on remote systems (@pxref{Remote storage}).
In order to support @acronym{NCO} on the Microsoft Windows platforms,
this single feature was disabled (on Windows @acronym{OS} only).
This was required by @w{Cygwin 18.x}---newer versions of Cygwin may
support these protocols (let me know if this is the case).
The @acronym{NCO} operators should behave identically on Windows and
@acronym{UNIX} platforms in all other respects.
@html
<a name="lbr"></a> <!-- http://nco.sf.net/nco.html#lbr -->
@end html
@node Libraries, netCDF2/3/4 and HDF4/5 Support, Compatability, Introduction
@section Libraries
@cindex libraries
@cindex @code{LD_LIBRARY_PATH}
@cindex dynamic linking
@cindex static linking
Like all executables, the @acronym{NCO} operators can be built using dynamic
linking.
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
This reduces the size of the executable and can result in significant
performance enhancements on multiuser systems.
Unfortunately, if your library search path (usually the
@env{LD_LIBRARY_PATH} environment variable) is not set correctly, or if
the system libraries have been moved, renamed, or deleted since
@acronym{NCO} was installed, it is possible @acronym{NCO} operators
will fail with a message that they cannot find a dynamically loaded (aka
@dfn{shared object} or @samp{.so}) library.
This will produce a distinctive error message, such as
@samp{ld.so.1:@- /usr/local/bin/ncea:@- fatal:@- libsunmath.@-so.1:@- can't
open@- file:@- errno@-=2}.
If you received an error message like this, ask your system
administrator to diagnose whether the library is truly missing
@footnote{The @command{ldd} command, if it is available on your system,
will tell you where the executable is looking for each dynamically
loaded library. Use, e.g., @code{ldd `which ncea`}.}, or whether you
simply need to alter your library search path.
As a final remedy, you may re-compile and install @acronym{NCO} with all
operators statically linked.
@node netCDF2/3/4 and HDF4/5 Support, Help Requests and Bug Reports, Libraries, Introduction
@section netCDF2/3/4 and HDF4/5 Support
@cindex netCDF2
@cindex netCDF3
netCDF @w{version 2} was released in 1993.
@acronym{NCO} (specifically @command{ncks}) began soon after this in 1994.
@w{netCDF 3.0} was released in 1996, and we were eager to reap the
performance advantages of the newer netCDF implementation.
One @w{netCDF3} interface call (@code{nc_inq_libvers}) was added to
@acronym{NCO} in January, 1998, to aid in maintainance and debugging.
In March, 2001, the final conversion of @acronym{NCO} to @w{netCDF3}
was completed (coincidentally on the same day @w{netCDF 3.5} was
released).
@acronym{NCO} @w{versions 2.0} and higher are built with the
@code{-DNO_NETCDF_2} flag to ensure no @w{netCDF2} interface calls
are used.
@cindex @code{NO_NETCDF_2}
@cindex @acronym{HDF}
@cindex Hierarchical Data Format
@cindex Mike Folk
However, the ability to compile @acronym{NCO} with only @w{netCDF2}
calls is worth maintaining because @acronym{HDF} @w{version 4}
@footnote{The Hierarchical Data Format, or @acronym{HDF}, is another
self-describing data format similar to, but more elaborate than,
netCDF.}
(available from @uref{http://hdf.ncsa.uiuc.edu, HDF})
supports only the @w{netCDF2} library calls
(see @uref{http://hdf.ncsa.uiuc.edu/UG41r3_html/SDS_SD.fm12.html#47784}).
Note that there are multiple versions of @acronym{HDF}.
Currently @acronym{HDF} @w{version 4.x} supports @w{netCDF2} and thus
@acronym{NCO} @w{version 1.2.x}.
If @acronym{NCO} @w{version 1.2.x} (or earlier) is built with only
@w{netCDF2} calls then all @acronym{NCO} operators should work with
@acronym{HDF4} files as well as netCDF files
@footnote{One must link the @acronym{NCO} code to the @acronym{HDF4}
@acronym{MFHDF} library instead of the usual netCDF library.
Does @samp{MF} stands for Mike Folk?
Perhaps.
In any case, the @acronym{MFHDF} library only supports @w{netCDF2}
calls.
Thus I will try to keep this capability in @acronym{NCO} as long as it
is not too much trouble.}.
@cindex @code{NETCDF2_ONLY}
The preprocessor token @code{NETCDF2_ONLY} exists
in @acronym{NCO} @w{version 1.2.x} to eliminate all @w{netCDF3}
calls.
Only versions of @acronym{NCO} numbered 1.2.x and earlier have this
capability.
The @w{@acronym{NCO} 1.2.x} branch will be maintained with bugfixes only
(no new features) until @acronym{HDF} begins to fully support the
@w{netCDF3} interface (which is employed by @w{@acronym{NCO} 2.x}).
If, at compilation time, @code{NETCDF2_ONLY} is defined, then
@acronym{NCO} @w{version 1.2.x} will not use any @w{netCDF3} calls
and, if linked properly, the resulting @acronym{NCO} operators will work
with @acronym{HDF4} files.
@cindex @file{Makefile}
The @file{Makefile} supplied with @w{@acronym{NCO} 1.2.x} is written
to simplify building in this @acronym{HDF} capability.
When @acronym{NCO} is built with @code{make HDF4=Y}, the @file{Makefile}
sets all required preprocessor flags and library links to build
with the @acronym{HDF4} libraries (which are assumed to reside under
@code{/usr/local/hdf4}, edit the @file{Makefile} to suit your
installation).
@cindex Unidata
@cindex @acronym{NCSA}
@cindex netCDF4
@cindex @acronym{HDF5}
@acronym{HDF} @w{version 5} became available in 1999, but did not
support netCDF (or, for that matter, Fortran) as of December 1999.
By early 2001, @acronym{HDF5} did support Fortran90.
However, support for netCDF4 in @acronym{HDF5} is incomplete.
Much of the HDF5-netCDF interface is complete, however, and it may be
separately downloaded from the
@uref{http://my.unidata.ucar.edu/content/software/netcdf/netcdf-4,netCDF4}
website.
We are eager for @acronym{HDF5} to complete netCDF support.
This is scheduled to occur sometime in 2007, with the releases of
HDF @w{version 1.8} and netCDF @w{version 4}, which are collaborations
between Unidata and @acronym{NCSA}.
@acronym{NCO} version 3.0.3 added support for reading/writing
netCDF4-formatted @acronym{HDF5} files in October, 2005.
See @ref{Selecting Output File Format} for more details.
@html
<a name="nco4"></a> <!-- http://nco.sf.net/nco.html#nco4 -->
@end html
@acronym{NCO} version 3.9.0 added full support for all netCDF4
atomic data types in May, 2007.
Support for netCDF4 features will be incremental, i.e.,
we will add one netCDF4 feature at a time.
You must build @acronym{NCO} with netCDF4 to obtain this support.
@cindex @code{NC_UBYTE}
@cindex @code{NC_USHORT}
@cindex @code{NC_UINT}
@cindex @code{NC_INT64}
@cindex @code{NC_UINT64}
The main netCDF4 features that NCO currently supports are the new
atomic data types and Lempel-Ziv compression.
The new atomic data types are @code{NC_UBYTE}, @code{NC_USHORT},
@code{NC_UINT}, @code{NC_INT64}, and @code{NC_UINT64}.
Eight-byte integer support is especially useful improvement from
netCDF3.
All @acronym{NCO} operators support these types, e.g., @command{ncks}
copies and prints them, @command{ncra} averages them, and
@command{ncap2} processes algebraic scripts with them.
@command{ncks} prints compression information, if any, to screen.
Lempel-Ziv deflation is a lossless compression technique.
See @ref{Deflation} for more details.
@cindex @acronym{HDF5}
@cindex @code{-4}
@cindex @code{-3}
netCDF4-enabled @acronym{NCO} handles netCDF3 files without change.
In addition, it automagically handles netCDF4 (@acronym{HDF5}) files:
If you feed @acronym{NCO} netCDF3 files, it produces netCDF3 output.
If you feed @acronym{NCO} netCDF4 files, it produces netCDF4 output.
Use the handy-dandy @samp{-4} switch to request netCDF4 output from
netCDF3 input, i.e., to convert netCDF3 to netCDF4.
See @ref{Selecting Output File Format} for more details.
@cindex @acronym{RPM}
@cindex Debian
Use appropriate caution while netCDF4 is beta software.
Problems with netCDF4 and @acronym{HDF} libraries are still being fixed.
@acronym{NCO} support for netCDF4 atomic types is relatively untested.
Binary @acronym{NCO} distributions (@acronym{RPM}s and debs) still use
netCDF3.
@cindex @code{NETCDF4_ROOT}
For now you must build @acronym{NCO} from source to get netCDF4 support.
Typically, one specifies the root of the netCDF4-beta
installation directory. Do this with the @code{NETCDF4_ROOT} variable.
Then use your preferred @acronym{NCO} build mechanism, e.g.,
@example
export NETCDF4_ROOT=/usr/local/netcdf4 # Set netCDF4 location
cd ~/nco;./configure --enable-netcdf4 # Configure mechanism -or-
cd ~/nco/bld;./make NETCDF4=Y allinone # Old Makefile mechanism
@end example
Our short term goal is to track the netCDF4-beta releases, keep the
new netCDF4 atomic type support working, and iron out any problems.
Our long term goal is to utilize more of the extensive new netCDF4
feature set. The next major netCDF4 feature we are likely to utilize
is parallel I/O. We will enable this in the @acronym{MPI} netCDF operators.
@html
<a name="help"></a> <!-- http://nco.sf.net/nco.html#help -->
<a name="hlp"></a> <!-- http://nco.sf.net/nco.html#hlp -->
<a name="bug"></a> <!-- http://nco.sf.net/nco.html#bug -->
@end html
@node Help Requests and Bug Reports, , netCDF2/3/4 and HDF4/5 Support, Introduction
@section Help Requests and Bug Reports
@cindex reporting bugs
@cindex bugs, reporting
@cindex core dump
@cindex help
@cindex features, requesting
We generally receive three categories of mail from users: help requests,
bug reports, and feature requests.
Notes saying the equivalent of "Hey, @acronym{NCO} continues to work
great and it saves me more time everyday than it took to write this
note" are a distant fourth.
There is a different protocol for each type of request.
The preferred etiquette for all communications is via @acronym{NCO}
Project Forums.
Do not contact project members via personal e-mail unless your request
comes with money or you have damaging information about our personal
lives.
@emph{Please use the Forums}---they preserve a record of the questions
and answers so that others can learn from our exchange.
Also, since @acronym{NCO} is government-funded, this record helps us
provide program officers with information they need to evaluate our
project.
Before posting to the @acronym{NCO} forums described below, you might
first @uref{https://sf.net/account/register.php, register}
your name and email address with SourceForge.net or else all of your
postings will be attributed to "nobody".
Once registered you may choose to "monitor" any forum and to receive
(or not) email when there are any postings including responses to your
questions.
We usually reply to the forum message, not to the original poster.
If you want us to include a new feature in @acronym{NCO}, check first to
see if that feature is already on the @uref{file:./TODO,TODO} list.
If it is, why not implement that feature yourself and send us the patch?
If the feature is not yet on the list, then send a note to the
@uref{http://sf.net/forum/forum.php?forum_id=9829, NCO Discussion forum}.
Read the manual before reporting a bug or posting a help request.
Sending questions whose answers are not in the manual is the best
way to motivate us to write more documentation.
We would also like to accentuate the contrapositive of this statement.
If you think you have found a real bug @emph{the most helpful thing you
can do is simplify the problem to a manageable size and then report it}.
The first thing to do is to make sure you are running the latest
publicly released version of @acronym{NCO}.
Once you have read the manual, if you are still unable to get
@acronym{NCO} to perform a documented function, submit a help request.
Follow the same procedure as described below for reporting bugs
(after all, it might be a bug).
@cindex debugging
@cindex @code{-r}
@cindex @code{-D}
That is, describe what you are trying to do, and include the complete
commands (run with @samp{-D 5}), error messages, and version of
@acronym{NCO} (with @samp{-r}).
Post your help request to the
@uref{http://sf.net/forum/forum.php?forum_id=9830, NCO Help forum}.
If you think you used the right command when @acronym{NCO} misbehaves,
then you might have found a bug.
Incorrect numerical answers are the highest priority.
We usually fix those within one or two days.
Core dumps and sementation violations receive lower priority.
They are always fixed, eventually.
How do you simplify a problem that reveal a bug?
Cut out extraneous variables, dimensions, and metadata from the
offending files and re-run the command until it no longer breaks.
Then back up one step and report the problem.
Usually the file(s) will be very small, i.e., one variable with one or
two small dimensions ought to suffice.
@html
<a name="dbg"></a> <!-- http://nco.sf.net/nco.html#dbg -->
<a name="-D"></a> <!-- http://nco.sf.net/nco.html#-D -->
@end html
@cindex @code{-r}
@cindex @code{--revision}
@cindex @code{--version}
@cindex @code{--vrs}
@cindex @code{-D @var{debug-level}}
@cindex @code{--debug-level @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
@cindex @var{debug-level}
@cindex @var{dbg_lvl}
Run the operator with @samp{-r} and then run the command with
@samp{-D 5} to increase the verbosity of the debugging output.
It is very important that your report contain the exact error messages
and compile-time environment.
Include a copy of your sample input file, or place one on a
publically accessible location, of the file(s).
Post the full bug report to the
@uref{http://sf.net/bugs/?group_id=3331, NCO Project buglist}.
@cindex installation
@cindex @command{autoconf}
@cindex @file{nco.configure.$@{GNU_TRP@}.foo}
@cindex @file{nco.config.log.$@{GNU_TRP@}.foo}
@cindex @file{nco.make.$@{GNU_TRP@}.foo}
@cindex @file{config.guess}
@cindex @file{configure.eg}
Build failures count as bugs.
Our limited machine access means we cannot fix all build failures.
The information we need to diagnose, and often fix, build failures
are the three files output by @acronym{GNU} build tools,
@file{nco.config.log.$@{GNU_TRP@}.foo},
@file{nco.configure.$@{GNU_TRP@}.foo},
and @file{nco.make.$@{GNU_TRP@}.foo}.
The file @file{configure.eg} shows how to produce these files.
Here @code{$@{GNU_TRP@}} is the "@acronym{GNU} architecture triplet",
the @var{chip-vendor-OS} string returned by @file{config.guess}.
Please send us your improvements to the examples supplied in
@file{configure.eg}.
@cindex regressions archive
The regressions archive at @url{http://dust.ess.uci.edu/nco/rgr}
contains the build output from our standard test systems.
You may find you can solve the build problem yourself by examining the
differences between these files and your own.
@html
<a name="str"></a> <!-- http://nco.sf.net/nco.html#str -->
@end html
@node Strategies, Common features, Introduction, Top
@chapter Operator Strategies
@menu
* Philosophy::
* Climate Model Paradigm::
* Temporary Output Files::
* Appending Variables::
* Simple Arithmetic and Interpolation::
* Averaging vs. Concatenating::
* Large Numbers of Files::
* Large Datasets::
* Memory Requirements::
* Performance Limitations::
@end menu
@html
<a name="phl"></a> <!-- http://nco.sf.net/nco.html#phl -->
@end html
@node Philosophy, Climate Model Paradigm, Strategies, Strategies
@section Philosophy
@cindex philosophy
@cindex climate model
The main design goal is command line operators which perform useful,
scriptable operations on netCDF files.
Many scientists work with models and observations which produce too much
data to analyze in tabular format.
Thus, it is often natural to reduce and massage this raw or primary
level data into summary, or second level data, e.g., temporal or spatial
averages.
These second level data may become the inputs to graphical and
statistical packages, and are often more suitable for archival and
dissemination to the scientific community.
@acronym{NCO} performs a suite of operations useful in manipulating data
from the primary to the second level state.
@cindex @acronym{NCL}
@cindex @acronym{IDL}
@cindex Perl
@cindex Yorick
Higher level interpretive languages (e.g., @acronym{IDL}, Yorick,
Matlab, @acronym{NCL}, Perl, Python),
and lower level compiled languages (e.g., C, Fortran) can always perform
any task performed by @acronym{NCO}, but often with more overhead.
NCO, on the other hand, is limited to a much smaller set of arithmetic
and metadata operations than these full blown languages.
@cindex command line switches
Another goal has been to implement enough command line switches so that
frequently used sequences of these operators can be executed from a
shell script or batch file.
Finally, @acronym{NCO} was written to consume the absolute minimum
amount of system memory required to perform a given job.
The arithmetic operators are extremely efficient; their exact memory
usage is detailed in @ref{Memory Requirements}.
@html
<a name="clm"></a> <!-- http://nco.sf.net/nco.html#clm -->
@end html
@node Climate Model Paradigm, Temporary Output Files, Philosophy, Strategies
@section Climate Model Paradigm
@cindex climate model
@cindex @acronym{NCAR}
@cindex @acronym{GCM}
@acronym{NCO} was developed at @acronym{NCAR} to aid analysis and
manipulation of datasets produced by General Circulation Models
(@acronym{GCM}s).
Datasets produced by @acronym{GCM}s share many features with all gridded
scientific datasets and so provide a useful paradigm for the explication
of the @acronym{NCO} operator set.
Examples in this manual use a @acronym{GCM} paradigm because latitude,
longitude, time, temperature and other fields related to our natural
environment are as easy to visualize for the layman as the expert.
@html
<a name="out"></a> <!-- http://nco.sf.net/nco.html#out -->
@end html
@node Temporary Output Files, Appending Variables, Climate Model Paradigm, Strategies
@section Temporary Output Files
@cindex data safety
@cindex error tolerance
@cindex safeguards
@cindex temporary output files
@acronym{NCO} operators are designed to be reasonably fault tolerant, so
that if there is a system failure or the user aborts the operation (e.g.,
with @kbd{C-c}), then no data are lost.
The user-specified @var{output-file} is only created upon successful
completion of the operation
@footnote{The @command{ncrename} operator is an exception to this rule.
@xref{ncrename netCDF Renamer}.}.
This is accomplished by performing all operations in a temporary copy
of @var{output-file}.
The name of the temporary output file is constructed by appending
@code{.pid@var{<process ID>}.@var{<operator name>}.tmp} to the
user-specified @var{output-file} name.
When the operator completes its task with no fatal errors, the temporary
output file is moved to the user-specified @var{output-file}.
Note the construction of a temporary output file uses more disk space
than just overwriting existing files ``in place'' (because there may be
two copies of the same file on disk until the @acronym{NCO} operation
successfully concludes and the temporary output file overwrites the
existing @var{output-file}).
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
Also, note this feature increases the execution time of the operator
by approximately the time it takes to copy the @var{output-file}.
Finally, note this feature allows the @var{output-file} to be the same
as the @var{input-file} without any danger of ``overlap''.
@html
<a name="-A"></a> <!-- http://nco.sf.net/nco.html#-A -->
<a name="-O"></a> <!-- http://nco.sf.net/nco.html#-O -->
@end html
@cindex @code{-A}
@cindex @code{-O}
@cindex @code{--apn}
@cindex @code{--append}
@cindex @code{--ovr}
@cindex @code{--overwrite}
@cindex overwriting files
@cindex appending to files
Other safeguards exist to protect the user from inadvertently
overwriting data.
If the @var{output-file} specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing @var{output-file}, attempt to append to it, or abort the
operation.
However, in processing large amounts of data, too many interactive
questions slows productivity.
Therefore @acronym{NCO} also implements two ways to override its own
safety features, the @samp{-O} and @samp{-A} switches.
Specifying @samp{-O} tells the operator to overwrite any existing
@var{output-file} without prompting the user interactively.
Specifying @samp{-A} tells the operator to attempt to append to any
existing @var{output-file} without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.
@html
<a name="apn"></a> <!-- http://nco.sf.net/nco.html#apn -->
@end html
@node Appending Variables, Simple Arithmetic and Interpolation, Temporary Output Files, Strategies
@section Appending Variables
Adding variables from one file to another is often desirable.
@cindex concatenation
@cindex appending variables
@cindex merging files
@cindex pasting variables
This is referred to as @dfn{appending}, although some prefer the
terminology @dfn{merging} @footnote{The terminology @dfn{merging} is
reserved for an (unwritten) operator which replaces hyperslabs of a
variable in one file with hyperslabs of the same variable from another
file} or @dfn{pasting}.
Appending is often confused with what @acronym{NCO} calls
@dfn{concatenation}.
@cindex record dimension
In @acronym{NCO}, concatenation refers to splicing a variable along the
record dimension.
Appending, on the other hand, refers to adding variables from one file
to another
@footnote{Yes, the terminology is confusing.
By all means mail me if you think of a better nomenclature.
Should @acronym{NCO} use @dfn{paste} instead of @dfn{append}?
}.
In this sense, @command{ncks} can append variables from one file to
another file.
This capability is invoked by naming two files on the command line,
@var{input-file} and @var{output-file}.
When @var{output-file} already exists, the user is prompted whether to
@dfn{overwrite}, @dfn{append/replace}, or @dfn{exit} from the command.
Selecting @dfn{overwrite} tells the operator to erase the existing
@var{output-file} and replace it with the results of the operation.
Selecting @dfn{exit} causes the operator to exit---the @var{output-file}
will not be touched in this case.
Selecting @dfn{append/replace} causes the operator to attempt to place
the results of the operation in the existing @var{output-file},
@xref{ncks netCDF Kitchen Sink}.
@html
<a name="unn"></a> <!-- http://nco.sf.net/nco.html#unn -->
@end html
@cindex union of two files
@cindex disjoint files
The simplest way to create the union of two files is
@example
ncks -A fl_1.nc fl_2.nc
@end example
This puts the contents of @file{fl_1.nc} into @file{fl_2.nc}.
The @samp{-A} is optional.
On output, @file{fl_2.nc} is the union of the input files,
regardless of whether they share dimensions and variables,
or are completely disjoint.
The append fails if the input files have differently named record
dimensions (since netCDF supports only one), or have dimensions of the
same name but different sizes.
@html
<a name="bnr"></a> <!-- http://nco.sf.net/nco.html#bnr -->
@end html
@node Simple Arithmetic and Interpolation, Averaging vs. Concatenating, Appending Variables, Strategies
@section Simple Arithmetic and Interpolation
Users comfortable with @acronym{NCO} semantics may find it easier to
perform some simple mathematical operations in @acronym{NCO} rather than
higher level languages.
@command{ncbo} (@pxref{ncbo netCDF Binary Operator}) does file
addition, subtraction, multiplication, division, and broadcasting.
@command{ncflint} (@pxref{ncflint netCDF File Interpolator}) does
file addition, subtraction, multiplication and interpolation.
Sequences of these commands can accomplish simple but powerful
operations from the command line.
@html
<a name="averagers"></a> <!-- http://nco.sf.net/nco.html#averagers -->
@end html
@node Averaging vs. Concatenating, Large Numbers of Files, Simple Arithmetic and Interpolation, Strategies
@section Averagers vs.@: Concatenators
@html
<a name="sym_ncea"></a> <!-- http://nco.sf.net/nco.html#sym_ncea -->
<a name="sym_ncrcat"></a> <!-- http://nco.sf.net/nco.html#sym_ncrcat -->
@end html
@cindex symbolic links
The most frequently used operators of @acronym{NCO} are probably the
averagers and concatenators.
Because there are so many permutations of averaging (e.g., across files,
within a file, over the record dimension, over other dimensions, with or
without weights and masks) and of concatenating (across files, along the
record dimension, along other dimensions), there are currently no fewer
than five operators which tackle these two purposes: @command{ncra},
@command{ncea}, @command{ncwa}, @command{ncrcat}, and @command{ncecat}.
These operators do share many capabilities @footnote{Currently
@command{ncea} and @command{ncrcat} are symbolically linked to the
@command{ncra} executable, which behaves slightly differently based on
its invocation name (i.e., @samp{argv[0]}).
These three operators share the same source code, but merely have
different inner loops.}, but each has its unique specialty.
Two of these operators, @command{ncrcat} and @command{ncecat}, are for
concatenating hyperslabs across files.
The other two operators, @command{ncra} and @command{ncea}, are for
averaging hyperslabs across files
@footnote{The third averaging operator, @command{ncwa}, is the most
sophisticated averager in @acronym{NCO}.
However, @command{ncwa} is in a different class than @command{ncra} and
@command{ncea} because it can only operate on a single file per
invocation (as opposed to multiple files).
On that single file, however, @command{ncwa} provides a richer set of
averaging options---including weighting, masking, and broadcasting.}.
First, let's describe the concatenators, then the averagers.
@menu
* Concatenation::
* Averaging::
* Interpolating::
@end menu
@html
<a name="cnc"></a> <!-- http://nco.sf.net/nco.html#cnc -->
@end html
@node Concatenation, Averaging, Averaging vs. Concatenating, Averaging vs. Concatenating
@subsection Concatenators @command{ncrcat} and @command{ncecat}
@cindex @command{ncecat}
@cindex @command{ncrcat}
Joining independent files together along a record dimension is called
@dfn{concatenation}.
@command{ncrcat} is designed for concatenating record variables, while
@command{ncecat} is designed for concatenating fixed length variables.
Consider five files, @file{85.nc}, @file{86.nc},
@w{@dots{} @file{89.nc}} each containing a year's worth of data.
Say you wish to create from them a single file, @file{8589.nc}
containing all the data, i.e., spanning all five years.
If the annual files make use of the same record variable, then
@command{ncrcat} will do the job nicely with, e.g.,
@code{ncrcat 8?.nc 8589.nc}.
The number of records in the input files is arbitrary and can vary from
file to file.
@xref{ncrcat netCDF Record Concatenator}, for a complete description of
@command{ncrcat}.
However, suppose the annual files have no record variable, and thus
their data are all fixed length.
@cindex ensemble
@cindex climate model
For example, the files may not be conceptually sequential, but rather
members of the same group, or @dfn{ensemble}.
Members of an ensemble may have no reason to contain a record dimension.
@command{ncecat} will create a new record dimension (named @var{record}
by default) with which to glue together the individual files into the
single ensemble file.
If @command{ncecat} is used on files which contain an existing record
dimension, that record dimension is converted to a fixed-length
dimension of the same name and a new record dimension (named
@code{record}) is created.
Consider five realizations, @file{85a.nc}, @file{85b.nc},
@w{@dots{} @file{85e.nc}} of 1985 predictions from the same climate
model.
Then @code{ncecat 85?.nc 85_ens.nc} glues the individual realizations
together into the single file, @file{85_ens.nc}.
If an input variable was dimensioned [@code{lat},@code{lon}], it will
have dimensions [@code{record},@code{lat},@code{lon}] in the output file.
@w{A restriction} of @command{ncecat} is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain
data on different realizations of the same variables.
@xref{ncecat netCDF Ensemble Concatenator}, for a complete description
of @command{ncecat}.
@cindex @command{ncpdq}
@html
<a name="dmn_cat"></a> <!-- http://nco.sf.net/nco.html#dmn_cat -->
@end html
@command{ncpdq} makes it possible to concatenate files along any
dimension, not just the record dimension.
First, use @command{ncpdq} to convert the dimension to be concatenated
(i.e., extended with data from other files) into the record dimension.
Second, use @command{ncrcat} to concatenate these files.
Finally, if desirable, use @command{ncpdq} to revert to the original
dimensionality.
As a concrete example, say that files @file{x_01.nc}, @file{x_02.nc},
@w{@dots{} @file{x_10.nc}} contain time-evolving datasets from spatially
adjacent regions.
The time and spatial coordinates are @code{time} and @code{x}, respectively.
Initially the record dimension is @code{time}.
Our goal is to create a single file that contains joins all the
spatially adjacent regions into one single time-evolving dataset.
@example
for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
ncpdq -a x,time x_$@{idx@}.nc foo_$@{idx@}.nc # Make x record dimension
done
ncrcat foo_??.nc out.nc # Concatenate along x
ncpdq -a time,x out.nc out.nc # Revert to time as record dimension
@end example
Note that @command{ncrcat} will not concatenate fixed-length variables,
whereas @command{ncecat} concatenates both fixed-length and record
variables along a new record variable.
To conserve system memory, use @command{ncrcat} where possible.
@html
<a name="avg"></a> <!-- http://nco.sf.net/nco.html#avg -->
@end html
@node Averaging, Interpolating, Concatenation, Averaging vs. Concatenating
@subsection Averagers @command{ncea}, @command{ncra}, and @command{ncwa}
@cindex @command{ncea}
@cindex @command{ncra}
@cindex @command{ncwa}
The differences between the averagers @command{ncra} and @command{ncea}
are analogous to the differences between the concatenators.
@command{ncra} is designed for averaging record variables from at least
one file, while @command{ncea} is designed for averaging fixed length
variables from multiple files.
@command{ncra} performs a simple arithmetic average over the record
dimension of all the input files, with each record having an equal
weight in the average.
@command{ncea} performs a simple arithmetic average of all the input
files, with each file having an equal weight in the average.
Note that @command{ncra} cannot average fixed-length variables,
but @command{ncea} can average both fixed-length and record variables.
To conserve system memory, use @command{ncra} rather than
@command{ncea} where possible (e.g., if each @var{input-file} is one
record long).
The file output from @command{ncea} will have the same dimensions
(meaning dimension names as well as sizes) as the input hyperslabs
(@pxref{ncea netCDF Ensemble Averager}, for a complete description of
@command{ncea}).
The file output from @command{ncra} will have the same dimensions as
the input hyperslabs except for the record dimension, which will have a
size @w{of 1} (@pxref{ncra netCDF Record Averager}, for a complete
description of @command{ncra}).
@html
<a name="ntp"></a> <!-- http://nco.sf.net/nco.html#ntp -->
@end html
@node Interpolating, , Averaging, Averaging vs. Concatenating
@subsection Interpolator @command{ncflint}
@cindex @command{ncflint}
@command{ncflint} can interpolate data between or two files.
Since no other operators have this ability, the description of
interpolation is given fully on the @command{ncflint} reference page
(@pxref{ncflint netCDF File Interpolator}).
Note that this capability also allows @command{ncflint} to linearly
rescale any data in a netCDF file, e.g., to convert between differing
units.
@html
<a name="lrg"></a> <!-- http://nco.sf.net/nco.html#lrg -->
@end html
@node Large Numbers of Files, Large Datasets, Averaging vs. Concatenating, Strategies
@section Large Numbers of Files
@cindex files, numerous input
@cindex @code{-n @var{loop}}
Occasionally one desires to digest (i.e., concatenate or average)
hundreds or thousands of input files.
@cindex automagic
@cindex @acronym{NASA EOSDIS}
Unfortunately, data archives (e.g., @acronym{NASA EOSDIS}) may not
name netCDF files in a format understood by the @samp{-n @var{loop}}
switch (@pxref{Specifying Input Files}) that automagically generates
arbitrary numbers of input filenames.
The @samp{-n @var{loop}} switch has the virtue of being concise,
and of minimizing the command line.
This helps keeps output file small since the command line is stored
as metadata in the @code{history} attribute
(@pxref{History Attribute}).
However, the @samp{-n @var{loop}} switch is useless when there is no
simple, arithmetic pattern to the input filenames (e.g.,
@file{h00001.nc}, @file{h00002.nc}, @w{@dots{} @file{h90210.nc}}).
Moreover, filename globbing does not work when the input files are too
numerous or their names are too lengthy (when strung together as a
single argument) to be passed by the calling shell to the @acronym{NCO}
operator
@footnote{The exact length which exceeds the operating system internal
limit for command line lengths varies from @acronym{OS} to @acronym{OS}
and from shell to shell.
@acronym{GNU} @code{bash} may not have any arbitrary fixed limits to the
size of command line arguments.
Many @acronym{OS}s cannot handle command line arguments (including
results of file globbing) exceeding 4096 characters.}.
When this occurs, the @acronym{ANSI} C-standard @code{argc}-@code{argv}
method of passing arguments from the calling shell to a C-program (i.e.,
an @acronym{NCO} operator) breaks down.
There are (at least) three alternative methods of specifying the input
filenames to @acronym{NCO} in environment-limited situations.
@html
<a name="stdin"></a> <!-- http://nco.sf.net/nco.html#stdin -->
@end html
@cindex standard input
@cindex @code{stdin}
The recommended method for sending very large numbers (hundreds or
more, typically) of input filenames to the multi-file operators is
to pass the filenames with the @acronym{UNIX} @dfn{standard input}
feature, aka @code{stdin}:
@example
# Pipe large numbers of filenames to stdin
/bin/ls | grep $@{CASEID@}_'......'.nc | ncecat -o foo.nc
@end example
This method avoids all constraints on command line size imposed by
the operating system.
A drawback to this method is that the @code{history} attribute
(@pxref{History Attribute}) does not record the name of any input
files since the names were not passed on the command line.
This makes determining the data provenance at a later date difficult.
@cindex @code{nco_input_file_number}
@cindex @code{nco_input_file_list}
@cindex global attributes
@cindex attributes, global
To remedy this situation, multi-file operators store the number of
input files in the @code{nco_input_file_number} global attribute and the
input file list itself in the @code{nco_input_file_list} global attribute
(@pxref{File List Attributes}).
Although this does not preserve the exact command used to generate the
file, it does retains all the information required to reconstruct the
command and determine the data provenance.
@cindex globbing
@cindex shell
@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex @command{xargs}
@cindex @acronym{UNIX}
A second option is to use the @acronym{UNIX} @command{xargs} command.
This simple example selects as input to @command{xargs} all the
filenames in the current directory that match a given pattern.
For illustration, consider a user trying to average millions of
files which each have a six character filename.
If the shell buffer can not hold the results of the corresponding
globbing operator, @file{??????.nc}, then the filename globbing
technique will fail.
Instead we express the filename pattern as an extended regular
expression, @file{......\.nc} (@pxref{Subsetting Variables}).
We use @command{grep} to filter the directory listing for this pattern
and to pipe the results to @command{xargs} which, in turn, passes the
matching filenames to an @acronym{NCO} multi-file operator, e.g.,
@command{ncecat}.
@example
# Use xargs to transfer filenames on the command line
/bin/ls | grep $@{CASEID@}_'......'.nc | xargs -x ncecat -o foo.nc
@end example
@cindex pipes
The single quotes protect the only sensitive parts of the extended
regular expression (the @command{grep} argument), and allow shell
interpolation (the @code{$@{CASEID@}} variable substitution) to
proceed unhindered on the rest of the command.
@command{xargs} uses the @acronym{UNIX} pipe feature to append the
suitably filtered input file list to the end of the @command{ncecat}
command options.
@cindex output file
@cindex input files
@cindex @code{-o @var{fl_out}}
The @code{-o foo.nc} switch ensures that the input files supplied by
@command{xargs} are not confused with the output file name.
@command{xargs} does, unfortunately, have its own limit (usually about
20,000 characters) on the size of command lines it can pass.
Give @command{xargs} the @samp{-x} switch to ensure it dies if it
reaches this internal limit.
When this occurs, use either the @code{stdin} method above, or the
symbolic link presented next.
@cindex symbolic links
Even when its internal limits have not been reached, the
@command{xargs} technique may not be sophisticated enough to handle
all situations.
A full scripting language like Perl can handle any level of complexity
of filtering input filenames, and any number of filenames.
The technique of last resort is to write a script that creates symbolic
links between the irregular input filenames and a set of regular,
arithmetic filenames that the @samp{-n @var{loop}} switch understands.
@cindex Perl
For example, the following Perl script a monotonically enumerated
symbolic link to up to one million @file{.nc} files in a directory.
If there are 999,999 netCDF files present, the links are named
@file{000001.nc} to @file{999999.nc}:
@cindex @code{-n @var{loop}}
@example
# Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(<STDIN>)@{chop;symlink $_,sprintf("%06d.nc",$idx++);@}'
ncecat -n 999999,6,1 000001.nc foo.nc
# Remove symbolic links when finished
/bin/rm ??????.nc
@end example
The @samp{-n @var{loop}} option tells the @acronym{NCO} operator to
automatically generate the filnames of the symbolic links.
This circumvents any @acronym{OS} and shell limits on command line size.
The symbolic links are easily removed once @acronym{NCO} is finished.
@cindex @code{history}
One drawback to this method is that the @code{history} attribute
(@pxref{History Attribute}) retains the filename list of the symbolic
links, rather than the data files themselves.
This makes it difficult to determine the data provenance at a later date.
@node Large Datasets, Memory Requirements, Large Numbers of Files, Strategies
@section Large Datasets
@cindex large datasets
@cindex LFS
@cindex Large File Support
@dfn{Large datasets} are those files that are comparable in size to the
amount of random access memory (@acronym{RAM}) in your computer.
Many users of @acronym{NCO} work with files larger than @w{100 MB}.
Files this large not only push the current edge of storage technology,
they present special problems for programs which attempt to access the
entire file at once, such as @command{ncea} and @command{ncecat}.
@cindex swap space
If you work with a @w{300 MB} files on a machine with only @w{32 MB} of
memory then you will need large amounts of swap space (virtual memory on
disk) and @acronym{NCO} will work slowly, or even fail.
There is no easy solution for this.
The best strategy is to work on a machine with sufficient amounts of
memory and swap space.
Since about 2004, many users have begun to produce or analyze files
exceeding @w{2 GB} in size.
These users should familiarize themselves with @acronym{NCO}'s Large
File Support (@acronym{LFS}) capabilities (@pxref{Large File Support}).
The next section will increase your familiarity with @acronym{NCO}'s
memory requirements.
With this knowledge you may re-design your data reduction approach to
divide the problem into pieces solvable in memory-limited situations.
@cindex server
@cindex @acronym{UNICOS}
@cindex Cray
If your local machine has problems working with large files, try running
@acronym{NCO} from a more powerful machine, such as a network server.
Certain machine architectures, e.g., Cray @acronym{UNICOS}, have special
commands which allow one to increase the amount of interactive memory.
@cindex @code{ilimit}
On Cray systems, try to increase the available memory with the
@code{ilimit} command.
@cindex @acronym{GNU}/Linux
@cindex @code{ulimit}
@cindex @code{core dump}
If you get a memory-related core dump
(e.g., @samp{Error exit (core dumped)}) on a @acronym{GNU}/Linux system,
try increasing the process-available memory with @code{ulimit}.
@cindex speed
The speed of the @acronym{NCO} operators also depends on file size.
When processing large files the operators may appear to hang, or do
nothing, for large periods of time.
In order to see what the operator is actually doing, it is useful to
activate a more verbose output mode.
This is accomplished by supplying a number greater @w{than 0} to the
@samp{-D @var{debug-level}} (or @samp{--debug-level}, or
@samp{--dbg_lvl}) switch.
@cindex @code{-D @var{debug-level}}
@cindex @code{--debug-level @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
@cindex @var{debug-level}
@cindex @var{dbg_lvl}
@cindex debugging
When the @var{debug-level} is nonzero, the operators report their
current status to the terminal through the @var{stderr} facility.
Using @samp{-D} does not slow the operators down.
Choose a @var{debug-level} @w{between 1} @w{and 3} for most situations,
e.g., @code{ncea -D 2 85.nc 86.nc 8586.nc}.
@w{A full} description of how to estimate the actual amount of memory the
multi-file @acronym{NCO} operators consume is given in
@ref{Memory Requirements}.
@html
<a name="mmr"></a> <!-- http://nco.sf.net/nco.html#mmr -->
@end html
@node Memory Requirements, Performance Limitations, Large Datasets, Strategies
@section Memory Requirements
@cindex memory requirements
@cindex memory available
@cindex RAM
@cindex swap space
@cindex peak memory usage
Many people use @acronym{NCO} on gargantuan files which dwarf the
memory available (free @acronym{RAM} plus swap space) even on today's powerful
machines.
These users want @acronym{NCO} to consume the least memory possible
so that their scripts do not have to tediously cut files into smaller
pieces that fit into memory.
We commend these greedy users for pushing @acronym{NCO} to its limits!
@cindex threads
@cindex OpenMP
@cindex shared memory machines
This section describes the memory @acronym{NCO} requires during
operation.
The required memory is based on the underlying algorithms.
The description below is the memory usage per thread.
Users with shared memory machines may use the threaded @acronym{NCO}
operators (@pxref{OpenMP Threading}).
The peak and sustained memory usage will scale accordingly,
i.e., by the number of threads.
Memory consumption patterns of all operators are similar, with
the exception of @command{ncap2}.
@menu
* Single and Multi-file Operators::
* Memory for ncap2::
@end menu
@node Single and Multi-file Operators, Memory for ncap2, Memory Requirements, Memory Requirements
@subsection Single and Multi-file Operators
@cindex multi-file operators
The multi-file operators currently comprise the record operators,
@command{ncra} and @command{ncrcat}, and the ensemble operators,
@command{ncea} and @command{ncecat}.
The record operators require @emph{much less} memory than the ensemble
operators.
This is because the record operators operate on one single record (i.e.,
time-slice) at a time, wherease the ensemble operators retrieve the
entire variable into memory.
Let @math{MS} be the peak sustained memory demand of an operator,
@math{FT} be the memory required to store the entire contents of all the
variables to be processed in an input file,
@math{FR} be the memory required to store the entire contents of a
single record of each of the variables to be processed in an input file,
@math{VR} be the memory required to store a single record of the
largest record variable to be processed in an input file,
@math{VT} be the memory required to store the largest variable
to be processed in an input file,
@math{VI} be the memory required to store the largest variable
which is not processed, but is copied from the initial file to the
output file.
All operators require @math{MI = VI} during the initial copying of
variables from the first input file to the output file.
This is the @emph{initial} (and transient) memory demand.
The @emph{sustained} memory demand is that memory required by the
operators during the processing (i.e., averaging, concatenation)
phase which lasts until all the input files have been processed.
The operators have the following memory requirements:
@command{ncrcat} requires @math{MS <= VR}.
@command{ncecat} requires @math{MS <= VT}.
@command{ncra} requires @math{MS = 2FR + VR}.
@command{ncea} requires @math{MS = 2FT + VT}.
@command{ncbo} requires @math{MS <= 3VT}
(both input variables and the output variable).
@command{ncflint} requires @math{MS <= 3VT}
(both input variables and the output variable).
@command{ncpdq} requires @math{MS <= 2VT}
(one input variable and the output variable).
@command{ncwa} requires @math{MS <= 8VT} (see below).
Note that only variables that are processed, e.g., averaged,
concatenated, or differenced, contribute to @math{MS}.
Variables which do not appear in the output file
(@pxref{Subsetting Variables}) are never read and contribute nothing
to the memory requirements.
@html
<a name="mmr_ncwa"></a> <!-- http://nco.sf.net/nco.html#mmr_ncwa -->
@end html
@command{ncwa} consumes between two and seven times the memory of a
variable in order to process it.
Peak consumption occurs when storing simultaneously in memory
one input variable, one tally array,
one input weight, one conformed/working weight, one weight tally,
one input mask, one conformed/working mask, and
one output variable.
When invoked, the weighting and masking features contribute up to
three-sevenths and two-sevenths of these requirements apiece.
If weights and masks are @emph{not} specified
(i.e., no @samp{-w} or @samp{-a} options)
then @command{ncwa} requirements drop to @math{MS <= 3VT}
(one input variable, one tally array, and the output variable).
@cindex OpenMP
@cindex threads
The above memory requirements must be multiplied by the number of
threads @var{thr_nbr} (@pxref{OpenMP Threading}).
@cindex @code{-t @var{thr_nbr}}
If this causes problems then reduce (with @samp{-t @var{thr_nbr}}) the
number of threads.
@node Memory for ncap2, , Single and Multi-file Operators, Memory Requirements
@subsection Memory for @command{ncap2}
@cindex @command{ncap2}
@cindex binary operations
@cindex unary operations
@cindex memory leaks
@cindex left hand casting
@command{ncap2} has unique memory requirements due its ability to process
arbitrarily long scripts of any complexity.
All scripts acceptable to @command{ncap2} are ultimately processed as a
sequence of binary or unary operations.
@command{ncap2} requires @math{MS <= 2VT} under most conditions.
An exception to this is when left hand casting (@pxref{Left hand
casting}) is used to stretch the size of derived variables beyond the
size of any input variables.
Let @math{VC} be the memory required to store the largest variable
defined by left hand casting.
In this case, @math{MS <= 2VC}.
@findex malloc()
@command{ncap2} scripts are complete dynamic and may be of arbitrary
length.
A script that contains many thousands of operations, may uncover a
slow memory leak even though each single operation consumes little
additional memory.
Memory leaks are usually identifiable by their memory usage signature.
Leaks cause peak memory usage to increase monotonically with time
regardless of script complexity.
Slow leaks are very difficult to find.
Sometimes a @command{malloc()} (or @command{new[]}) failure is the
only noticeable clue to their existance.
If you have good reasons to believe that a memory allocation failure
is ultimately due to an @acronym{NCO} memory leak (rather than
inadequate @acronym{RAM} on your system), then we would be very
interested in receiving a detailed bug report.
@node Performance Limitations, , Memory Requirements, Strategies
@section Performance Limitations
@enumerate
@item
@cindex buffering
No data buffering is performed during @command{nc_get_var} and
@command{nc_put_var} operations.
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
Hyperslabs too large too hold in core memory will suffer substantial
performance penalties because of this.
@item
@cindex monotonic coordinates
Since coordinate variables are assumed to be monotonic, the search for
bracketing the user-specified limits should employ a quicker algorithm,
like bisection, than the two-sided incremental search currently
implemented.
@item
@cindex @var{C_format}
@cindex @var{FORTRAN_format}
@cindex @var{signedness}
@cindex @var{scale_format}
@cindex @var{add_offset}
@var{C_format}, @var{FORTRAN_format}, @var{signedness},
@var{scale_format} and @var{add_offset} attributes are ignored by
@command{ncks} when printing variables to screen.
@item
@cindex Yorick
In the late 1990s it was discovered that some random access operations
on large files on certain architectures (e.g., @acronym{UNICOS}) were
much slower with @acronym{NCO} than with similar operations performed
using languages that bypass the netCDF interface (e.g., Yorick).
This may be a penalty of unnecessary byte-swapping in the netCDF
interface.
It is unclear whether such problems exist in present day (2007)
netCDF/@acronym{NCO} environments.
@end enumerate
@html
<a name="ftr"></a> <!-- http://nco.sf.net/nco.html#ftr -->
@end html
@node Common features, Operator Reference Manual, Strategies, Top
@chapter NCO Features
Many features have been implemented in more than one operator and are
described here for brevity.
The description of each feature is preceded by a box listing the
operators for which the feature is implemented.
@cindex command line switches
Command line switches for a given feature are consistent across all
operators wherever possible.
If no ``key switches'' are listed for a feature, then that particular
feature is automatic and cannot be controlled by the user.
@menu
* Internationalization::
* Metadata Optimization::
* OpenMP Threading::
* Command Line Options::
* Specifying Input Files::
* Specifying Output Files::
* Remote storage::
* Retaining Retrieved Files::
* Selecting Output File Format::
* Large File Support::
* Subsetting Variables::
* Subsetting Coordinate Variables::
* C and Fortran Index Conventions::
* Hyperslabs::
* Stride::
* Multislabs::
* Wrapped Coordinates::
* Auxiliary Coordinates::
* UDUnits Support::
* Missing Values::
* Deflation::
* Packed data::
* Operation Types::
* Type Conversion::
* Batch Mode::
* History Attribute::
* File List Attributes::
* CF Conventions::
* ARM Conventions::
* Operator Version::
@end menu
@html
<a name="i18n"></a> <!-- http://nco.sf.net/nco.html#i18n -->
@end html
@node Internationalization, Metadata Optimization, Common features, Common features
@section Internationalization
@cindex Internationalization
@cindex I18N
@cartouche
Availability: All operators@*
@end cartouche
@cindex L10N
@acronym{NCO} support for @dfn{internationalization} of textual input
and output (e.g., Warning messages) is nascent.
We hope to produce foreign language string catalogues in 2004.
@c fxm: Work on this section
@html
<a name="hdr"></a> <!-- http://nco.sf.net/nco.html#hdr -->
@end html
@node Metadata Optimization, OpenMP Threading, Internationalization, Common features
@section Metadata Optimization
@cindex metadata optimization
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
@cindex @code{nc__enddef()}
@cindex @code{--hdr_pad @var{hdr_pad}}
@cindex @code{--header_pad @var{hdr_pad}}
@cartouche
Availability: @command{ncatted}, @command{ncks}, @command{ncrename}@*
Short options: None@*
Long options: @samp{--hdr_pad}, @samp{--header_pad}@*
@end cartouche
@acronym{NCO} supports padding headers to improve the speed of future
metadata operations.
Use the @samp{--hdr_pad} and @samp{--header_pad} switches to request
that @var{hdr_pad} bytes be inserted into the metadata section of the
output file.
Future metadata expansions will not incur the performance penalty of
copying the entire output file unless the expansion exceeds the amount
of header padding exceeded.
This can be beneficial when it is known that some metadata will be added
at a future date.
This optimization exploits the netCDF library @code{nc__enddef()}
function, which behaves differently with different versions of netCDF.
It will improve speed of future metadata expansion with @code{CLASSIC}
and @code{64bit} netCDF files, but not necessarily with @code{NETCDF4}
files, i.e., those created by the netCDF interface to the @acronym{HDF5}
library (@pxref{Selecting Output File Format}).
@html
<a name="omp"></a> <!-- http://nco.sf.net/nco.html#omp -->
<a name="openmp"></a> <!-- http://nco.sf.net/nco.html#openmp -->
@end html
@node OpenMP Threading, Command Line Options, Metadata Optimization, Common features
@section OpenMP Threading
@cindex OpenMP
@cindex threads
@cindex @acronym{SMP}
@cindex shared memory parallelism
@cindex parallelism
@cindex @code{nco_openmp_thread_number}
@cindex @code{--thr_nbr @var{thr_nbr}}
@cindex @code{--threads @var{thr_nbr}}
@cindex @code{--omp_num_threads @var{thr_nbr}}
@cindex @code{-t @var{thr_nbr}}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncpdq}, @command{ncra}, @command{ncrcat},
@command{ncwa}@*
Short options: @samp{-t}@*
Long options: @samp{--thr_nbr}, @samp{--threads},
@samp{--omp_num_threads}@*
@end cartouche
@acronym{NCO} supports shared memory parallelism (@acronym{SMP}) when
compiled with an OpenMP-enabled compiler.
Threads requests and allocations occur in two stages.
First, users may request a specific number of threads @var{thr_nbr} with
the @samp{-t} switch (or its long option equivalents, @samp{--thr_nbr},
@samp{--threads}, and @samp{--omp_num_threads}).
If not user-specified, OpenMP obtains @var{thr_nbr} from the
@code{OMP_NUM_THREADS} environment variable, if present, or from the
@acronym{OS}, if not.
@cindex @var{thr_nbr}
@cindex @code{OMP_NUM_THREADS}
@cindex @command{ncrcat}
@cindex @command{ncwa}
@cindex @command{ncpdq}
@cindex large datasets
@acronym{NCO} may modify @var{thr_nbr} according to its own internal
settings before it requests any threads from the system.
Certain operators contain hard-code limits to the number of threads they
request.
We base these limits on our experience and common sense, and to reduce
potentially wasteful system usage by inexperienced users.
For example, @code{ncrcat} is extremely I/O-intensive so we restrict
@math{@var{thr_nbr} <= 2} for @code{ncrcat}.
This is based on the notion that the best performance that can be
expected from an operator which does no arithmetic is to have one thread
reading and one thread writing simultaneously.
In the future (perhaps with netCDF4), we hope to
demonstrate significant threading improvements with operators
like @code{ncrcat} by performing multiple simultaneous writes.
Compute-intensive operators (@code{ncwa} and @code{ncpdq})
are expected to benefit the most from threading.
The greatest increases in throughput due to threading will occur on
large dataset where each thread performs millions or more floating
point operations.
Otherwise, the system overhead of setting up threads may outweigh
the theoretical speed enhancements due to @acronym{SMP} parallelism.
However, we have not yet demonstrated that the @acronym{SMP} parallelism
scales well beyone four threads for these operators.
Hence we restrict @math{@var{thr_nbr} <= 4} for all operators.
We encourage users to play with these limits (edit file
@file{nco_omp.c}) and send us their feedback.
@cindex debugging
@cindex @var{dbg_lvl}
Once the initial @var{thr_nbr} has been modified for any
operator-specific limits, @acronym{NCO} requests the system to allocate
a team of @var{thr_nbr} threads for the body of the code.
The operating system then decides how many threads to allocate
based on this request.
Users may keep track of this information by running the operator with
@math{@var{dbg_lvl} > 0}.
By default, operators with thread attach one global attribute to any
file they create or modify.
The @code{nco_openmp_thread_number} global attribute contains the
number of threads the operator used to process the input files.
This information helps to verify that the answers with threaded and
non-threaded operators are equal to within machine precision.
@cindex benchmarks
This information is also useful for benchmarking.
@html
<a name="cmd_ln"></a> <!-- http://nco.sf.net/nco.html#cmd_ln -->
@end html
@node Command Line Options, Specifying Input Files, OpenMP Threading, Common features
@section Command Line Options
@cindex command line options
@cartouche
Availability: All operators@*
@end cartouche
@cindex @acronym{POSIX}
@cindex @acronym{UNIX}
@cindex @acronym{GNU}
@cindex switches
@acronym{NCO} achieves flexibility by using @dfn{command line options}.
These options are implemented in all traditional @acronym{UNIX} commands
as single letter @dfn{switches}, e.g., @samp{ls -l}.
For many years @acronym{NCO} used only single letter option names.
In late 2002, we implemented @acronym{GNU}/@acronym{POSIX} extended
or long option names for all options.
This was done in a backward compatible way such that the full
functionality of @acronym{NCO} is still available through the familiar
single letter options.
In the future, however, some features of @acronym{NCO} may require the
use of long options, simply because we have nearly run out of single
letter options.
More importantly, mnemonics for single letter options are often
non-intuitive so that long options provide a more natural way of
expressing intent.
@cindex long options
Extended options, also called long options, are implemented using the
system-supplied @file{getopt.h} header file, if possible.
@cindex @code{BSD}
@cindex @code{getopt}
@cindex @code{getopt_long}
@cindex @file{getopt.h}
This provides the @command{getopt_long} function to @acronym{NCO}
@footnote{
If a @command{getopt_long} function cannot be found on the system,
@acronym{NCO} will use the @command{getopt_long} from the
@command{my_getopt} package by Benjamin Sittler
@email{bsittler@@iname.com}.
This is @acronym{BSD}-licensed software available from
@uref{http://www.geocities.com/ResearchTriangle/Node/9405/#my_getopt}.}.
@cindex @code{-D @var{debug-level}}
@cindex @code{--dbg_lvl @var{debug-level}}
The syntax of @dfn{short options} (single letter options) is
@kbd{-@var{key} @var{value}} (dash-key-space-value).
Here, @var{key} is the single letter option name, e.g.,
@samp{-D 2}.
The syntax of @dfn{long options} (multi-letter options) is
@kbd{--@var{long_name} @var{value}}
(dash-dash-key-space-value), e.g., @samp{--dbg_lvl 2} or
@kbd{--@var{long_name}=@var{value}}
(dash-dash-key-equal-value), e.g., @samp{--dbg_lvl=2}.
Thus the following are all valid for the @samp{-D} (short version)
or @samp{--dbg_lvl} (long version) command line option.
@example
ncks -D 3 in.nc # Short option
ncks --dbg_lvl=3 in.nc # Long option, preferred form
ncks --dbg_lvl 3 in.nc # Long option, alternate form
@end example
@noindent
The last example is preferred for two reasons.
First, @samp{--dbg_lvl} is more specific and less ambiguous than
@samp{-D}.
The long option form makes scripts more self documenting and less error
prone.
Often long options are named after the source code variable whose value
they carry.
Second, the equals sign @kbd{=} joins the key (i.e., @var{long_name}) to
the value in an uninterruptible text block.
Experience shows that users are less likely to mis-parse commands when
restricted to this form.
@acronym{GNU} implements a superset of the @acronym{POSIX} standard
which allows any unambiguous truncation of a valid option to be used.
@example
ncks -D 3 in.nc # Short option
ncks --dbg_lvl=3 in.nc # Long option, full form
ncks --dbg=3 in.nc # Long option, unambiguous truncation
ncks --db=3 in.nc # Long option, unambiguous truncation
ncks --d=3 in.nc # Long option, ambiguous truncation
@end example
@noindent
The first four examples are equivalent and will work as expected.
The final example will exit with an error since @command{ncks} cannot
disambiguate whether @samp{--d} is intended as a truncation of
@samp{--dbg_lvl}, of @samp{--dimension}, or of some other long option.
@acronym{NCO} provides many long options for common switches.
For example, the debugging level may be set in all operators with any
of the switches @samp{-D}, @samp{--debug-level}, or @samp{--dbg_lvl}.
This flexibility allows users to choose their favorite mnemonic.
For some, it will be @samp{--debug} (an unambiguous truncation of
@samp{--debug-level}, and other will prefer @samp{--dbg}.
Interactive users usually prefer the minimal amount of typing, i.e.,
@samp{-D}.
We recommend that scripts which are re-usable employ some form of
the long options for future maintainability.
This manual generally uses the short option syntax.
This is for historical reasons and to conserve space.
The remainder of this manual specifies the full @var{long_name} of
each option.
Users are expected to pick the unambiguous truncation of each option
name that most suits their taste.
@html
<a name="fl_in"></a> <!-- http://nco.sf.net/nco.html#fl_in -->
<a name="in"></a> <!-- http://nco.sf.net/nco.html#in -->
<a name="input"></a> <!-- http://nco.sf.net/nco.html#input -->
@end html
@node Specifying Input Files, Specifying Output Files, Command Line Options, Common features
@section Specifying Input Files
@cindex input files
@cindex globbing
@cindex regular expressions
@cindex wildcards
@cindex @code{NINTAP}
@cindex Processor, @acronym{CCM}
@cindex @acronym{CCM} Processor
@cindex @code{-n @var{loop}}
@cindex @code{--nintap @var{loop}}
@cindex @code{-p @var{input-path}}
@cindex @code{--pth @var{input-path}}
@cindex @code{--path @var{input-path}}
@cindex @var{input-path}
@cartouche
Availability (@code{-n}): @command{ncea}, @command{ncecat}, @command{ncra}, @command{ncrcat}@*
Availability (@code{-p}): All operators@*
Short options: @samp{-n}, @samp{-p}@*
Long options: @samp{--nintap}, @samp{--pth}, @samp{--path}@*
@end cartouche
It is important that users be able to specify multiple input files
without typing every filename in full, often a tedious task even
by graduate student standards.
@cindex @acronym{UNIX}
There are four different ways of specifying input files to @acronym{NCO}:
explicitly typing each, using @acronym{UNIX} shell wildcards, and using
the @acronym{NCO} @samp{-n} and @samp{-p} switches (or their long option
equivalents, @samp{--nintap} or @samp{--pth} and @samp{--path},
respectively).
To illustrate these methods, consider the simple problem of using
@command{ncra} to average five input files, @file{85.nc}, @file{86.nc},
@w{@dots{} @file{89.nc}}, and store the results in @file{8589.nc}.
Here are the four methods in order.
They produce identical answers.
@example
ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -p @var{input-path} 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
@end example
The first method (explicitly specifying all filenames) works by brute
force.
The second method relies on the operating system shell to @dfn{glob}
(expand) the @dfn{regular expression} @code{8[56789].nc}.
The shell passes valid filenames which match the expansion to
@command{ncra}.
The third method uses the @samp{-p @var{input-path}} argument to specify
the directory where all the input files reside.
@acronym{NCO} prepends @var{input-path} (e.g.,
@file{/data/usrname/model}) to all @var{input-files} (but not to
@var{output-file}).
Thus, using @samp{-p}, the path to any number of input files need only
be specified once.
Note @var{input-path} need not end with @samp{/}; the @samp{/} is
automatically generated if necessary.
The last method passes (with @samp{-n}) syntax concisely describing
the entire set of filenames
@footnote{The @samp{-n} option is a backward compatible superset of the
@code{NINTAP} option from the @acronym{NCAR} @acronym{CCM} Processor.}.
@cindex multi-file operators
@cindex files, multiple
This option is only available with the @dfn{multi-file operators}:
@command{ncra}, @command{ncrcat}, @command{ncea}, and @command{ncecat}.
By definition, multi-file operators are able to process an arbitrary
number of @var{input-files}.
This option is very useful for abbreviating lists of filenames
representable as
@var{alphanumeric_prefix}+@var{numeric_suffix}+@file{.}+@var{filetype}
where @var{alphanumeric_prefix} is a string of arbitrary length and
composition, @var{numeric_suffix} is a fixed width field of digits, and
@var{filetype} is a standard filetype indicator.
For example, in the file @file{ccm3_h0001.nc}, we have
@var{alphanumeric_prefix} = @file{ccm3_h}, @var{numeric_suffix} =
@file{0001}, and @var{filetype} = @file{nc}.
@acronym{NCO} is able to decode lists of such filenames encoded using the
@samp{-n} option.
The simpler (3-argument) @samp{-n} usage takes the form
@code{-n @var{file_number},@var{digit_number},@var{numeric_increment}}
where @var{file_number} is the number of files, @var{digit_number} is
the fixed number of numeric digits comprising the @var{numeric_suffix},
and @var{numeric_increment} is the constant, integer-valued difference
between the @var{numeric_suffix} of any two consecutive files.
The value of @var{alphanumeric_prefix} is taken from the input file,
which serves as a template for decoding the filenames.
In the example above, the encoding @code{-n 5,2,1} along with the input
file name @file{85.nc} tells @acronym{NCO} to
construct five (5) filenames identical to the template @file{85.nc}
except that the final two (2) digits are a numeric suffix to be
incremented by one (1) for each successive file.
Currently @var{filetype} may be either be empty, @file{nc},
@file{cdf}, @file{hdf}, or @file{hd5}.
If present, these @var{filetype} suffixes (and the preceding @file{.})
are ignored by @acronym{NCO} as it uses the @samp{-n} arguments to
locate, evaluate, and compute the @var{numeric_suffix} component of
filenames.
@cindex wrapped filenames
@cindex climate model
Recently the @samp{-n} option has been extended to allow convenient
specification of filenames with ``circular'' characteristics.
This means it is now possible for @acronym{NCO} to automatically
generate filenames which increment regularly until a specified maximum
value, and then wrap back to begin again at a specified minimum value.
The corresponding @samp{-n} usage becomes more complex, taking one or
two additional arguments for a total of four or five, respectively:
@code{-n
@var{file_number},@var{digit_number},@var{numeric_increment}[,@var{numeric_max}[,@var{numeric_min}]]}
where @var{numeric_max}, if present, is the maximum integer-value of
@var{numeric_suffix} and @var{numeric_min}, if present, is the minimum
integer-value of @var{numeric_suffix}.
Consider, for example, the problem of specifying non-consecutive input
files where the filename suffixes end with the month index.
In climate modeling it is common to create summertime and wintertime
averages which contain the averages of the months June--July--August,
and December--January--February, respectively:
@example
ncra -n 3,2,1 85_06.nc 85_0608.nc
ncra -n 3,2,1,12 85_12.nc 85_1202.nc
ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc
@end example
The first example shows that three arguments to the @samp{-n} option
suffice to specify consecutive months (@code{06, 07, 08}) which do not
``wrap'' back to a minimum value.
The second example shows how to use the optional fourth and fifth
elements of the @samp{-n} option to specify a wrap value to @acronym{NCO}.
The fourth argument to @samp{-n}, if present, specifies the maximum
integer value of @var{numeric_suffix}.
In this case the maximum value @w{is 12,} and will be formatted as
@file{12} in the filename string.
The fifth argument to @samp{-n}, if present, specifies the minimum
integer value of @var{numeric_suffix}.
The default minimum filename suffix @w{is 1,} which is formatted as
@file{01} in this case.
Thus the second and third examples have the same effect, that is, they
automatically generate, in order, the filenames @file{85_12.nc},
@file{85_01.nc}, and @file{85_02.nc} as input to @acronym{NCO}.
@html
<a name="fl_out"></a> <!-- http://nco.sf.net/nco.html#fl_out -->
<a name="out"></a> <!-- http://nco.sf.net/nco.html#out -->
<a name="output"></a> <!-- http://nco.sf.net/nco.html#output -->
@end html
@node Specifying Output Files, Remote storage, Specifying Input Files, Common features
@section Specifying Output Files
@cindex output file
@cindex input files
@cindex positional arguments
@cindex command line switches
@cindex @code{-o @var{fl_out}}
@cindex @code{--output @var{fl_out}}
@cindex @code{--fl_out @var{fl_out}}
@cartouche
Availability: All operators@*
Short options: @samp{-o}@*
Long options: @samp{--fl_out}, @samp{--output}@*
@end cartouche
@acronym{NCO} commands produce no more than one output file, @var{fl_out}.
Traditionally, users specify @var{fl_out} as the final argument to the
operator, following all input file names.
This is the @dfn{positional argument} method of specifying input and
ouput file names.
The positional argument method works well in most applications.
@acronym{NCO} also supports specifying @var{fl_out} using the command
line switch argument method, @samp{-o @var{fl_out}}.
Specifying @var{fl_out} with a switch, rather than as a positional
argument, allows @var{fl_out} to precede input files in the argument
list.
@cindex multi-file operators
This is particularly useful with multi-file operators for three reasons.
Multi-file operators may be invoked with hundreds (or more) filenames.
Visual or automatic location of @var{fl_out} in such a list is
difficult when the only syntactic distinction between input and output
files is their position.
@cindex @command{xargs}
@cindex input files
Second, specification of a long list of input files may be difficult
(@pxref{Large Numbers of Files}).
Making the input file list the final argument to an operator facilitates
using @command{xargs} for this purpose.
Some alternatives to @command{xargs} are very ugly and undesirable.
Finally, many users are more comfortable specifying output files
with @samp{-o @var{fl_out}} near the beginning of an argument list.
@cindex compilers
@cindex linkers
Compilers and linkers are usually invoked this way.
@html
<a name="rmt"></a> <!-- http://nco.sf.net/nco.html#rmt -->
@end html
@node Remote storage, Retaining Retrieved Files, Specifying Output Files, Common features
@section Accessing Remote Files
@cindex @code{rcp}
@cindex @code{scp}
@cindex @file{.rhosts}
@cindex @acronym{NCAR MSS}
@cindex @acronym{MSS}
@cindex Mass Store System
@cindex @acronym{URL}
@cindex @code{ftp}
@cindex @code{sftp}
@cindex remote files
@cindex synchronous file access
@cindex asynchronous file access
@cindex @code{--pth @var{input-path}}
@cindex @code{--path @var{input-path}}
@cindex @code{--lcl @var{output-path}}
@cindex @code{--local @var{output-path}}
@cindex @code{-l @var{output-path}}
@cindex @file{.netrc}
@cindex @code{history}
@cartouche
Availability: All operators@*
Short options: @samp{-p}, @samp{-l}@*
Long options: @samp{--pth}, @samp{--path}, @samp{--lcl}, @samp{--local}@*
@end cartouche
All @acronym{NCO} operators can retrieve files from remote sites as well
as from the local file system.
@w{A remote} site can be an anonymous @acronym{FTP} server, a machine on
which the user has @command{rcp}, @command{scp}, or @command{sftp}
privileges, or @acronym{NCAR}'s Mass Storage System (@acronym{MSS}), or
an @acronym{OPeNDAP} server.
Examples of each are given below, following a brief description of the
particular access protocol.
@html
<a name="ftp"></a> <!-- http://nco.sf.net/nco.html#ftp -->
@end html
To access a file via an anonymous @acronym{FTP} server, supply the
remote file's @acronym{URL}.
@acronym{FTP} is an intrinsically insecure protocol because it transfers
passwords in plain text format.
Users should access sites using anonymous @acronym{FTP} when possible.
Some @acronym{FTP} servers require a login/password combination for a
valid user account.
@acronym{NCO} allows these transactions so long as the required
information is stored in the @file{.netrc} file.
Usually this information is the remote machine name, login, and
password, in plain text, separated by those very keywords, e.g.,
@example
machine dust.ess.uci.edu login zender password bushlied
@end example
Eschew using valuable passwords for @acronym{FTP} transactions, since
@file{.netrc} passwords are potentially exposed to eavesdropping
software
@footnote{@acronym{NCO} does not implement command line options to
specify @acronym{FTP} logins and passwords because copying those data
into the @code{history} global attribute in the output file (done by
default) poses an unacceptable security risk.
}.
@html
<a name="sftp"></a> <!-- http://nco.sf.net/nco.html#sftp -->
@end html
@acronym{SFTP}, i.e., secure @acronym{FTP}, uses @acronym{SSH}-based
security protocols that solve the security issues associated with
plain @acronym{FTP}.
@acronym{NCO} supports @acronym{SFTP} protocol access to files
specified with a homebrew syntax of the form
@example
sftp://machine.domain.tld:/path/to/filename
@end example
Note the second colon following the top-level-domain (tld).
This syntax is a hybrid between an @acronym{FTP URL} and a standard
remote file syntax.
@html
<a name="rcp"></a> <!-- http://nco.sf.net/nco.html#rcp -->
<a name="scp"></a> <!-- http://nco.sf.net/nco.html#scp -->
@end html
To access a file using @command{rcp} or @command{scp}, specify the
Internet address of the remote file.
Of course in this case you must have @command{rcp} or @command{scp}
privileges which allow transparent (no password entry required) access
to the remote machine.
This means that @file{~/.rhosts} or @file{~/ssh/authorized_keys} must
be set accordingly on both local and remote machines.
@cindex @command{msrcp}
@cindex @command{msread}
@cindex @command{nrnet}
@html
<a name="mss"></a> <!-- http://nco.sf.net/nco.html#mss -->
<a name="msrcp"></a> <!-- http://nco.sf.net/nco.html#msrcp -->
<a name="msread"></a> <!-- http://nco.sf.net/nco.html#msread -->
<a name="nrnet"></a> <!-- http://nco.sf.net/nco.html#nrnet -->
@end html
To access a file on @acronym{NCAR}'s @acronym{MSS}, specify the full
@acronym{MSS} pathname of the remote file.
@acronym{NCO} will attempt to detect whether the local machine has direct
(synchronous) @acronym{MSS} access.
In this case, @acronym{NCO} attempts to use the @acronym{NCAR}
@command{msrcp} command
@footnote{The @command{msrcp} command must be in the user's path and
located in one of the following directories: @code{/usr/local/bin},
@code{/usr/bin}, @code{/opt/local/bin}, or @code{/usr/local/dcs/bin}.
}, or, failing that, @code{/usr/local/bin/msread}.
Otherwise @acronym{NCO} attempts to retrieve the @acronym{MSS} file
through the (asynchronous) Masnet Interface Gateway System
(@acronym{MIGS}) using the @command{nrnet} command.
The following examples show how one might analyze files stored on
remote systems.
@example
ncks -l . ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
ncks -l . sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco/in.nc
ncks -l . dust.ess.uci.edu:/home/zender/nco/data/in.nc
ncks -l . /ZENDER/nco/in.nc
ncks -l . mss:/ZENDER/nco/in.nc
ncks -l . http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata/in.nc
@end example
@noindent
The first example works verbatim if your system is connected to the
Internet and is not behind a firewall.
The second example works if you have @command{sftp} access to the
machine @code{dust.ess.uci.edu}.
The third example works if you have @command{rcp} or @command{scp}
access to the machine @code{dust.ess.uci.edu}.
The fourth and fifth examples work on @acronym{NCAR} computers with
local access to the @command{msrcp}, @command{msread}, or
@command{nrnet} commands.
The sixth command works if your local version of @acronym{NCO} is
@acronym{OPeNDAP}-enabled (this is fully described in @ref{OPeNDAP}).
The above commands can be rewritten using the @samp{-p @var{input-path}}
option as follows:
@cindex @code{-p @var{input-path}}
@cindex @var{input-path}
@cindex @code{-l @var{output-path}}
@cindex @var{output-path}
@example
ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l . in.nc
ncks -p sftp://dust.ess.uci.edu:/home/ftp/pub/zender/nco -l . in.nc
ncks -p dust.ess.uci.edu:/home/zender/nco -l . in.nc
ncks -p /ZENDER/nco -l . in.nc
ncks -p mss:/ZENDER/nco -l . in.nc
ncks -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
-l . in.nc
@end example
@noindent
Using @samp{-p} is recommended because it clearly separates the
@var{input-path} from the filename itself, sometimes called the
@dfn{stub}.
@cindex stub
When @var{input-path} is not explicitly specified using @samp{-p},
@acronym{NCO} internally generates an @var{input-path} from the first
input filename.
The automatically generated @var{input-path} is constructed by stripping
the input filename of everything following the final @samp{/} character
(i.e., removing the stub).
The @samp{-l @var{output-path}} option tells @acronym{NCO} where to
store the remotely retrieved file and the output file.
Often the path to a remotely retrieved file is quite different than the
path on the local machine where you would like to store the file.
If @samp{-l} is not specified then @acronym{NCO} internally generates an
@var{output-path} by simply setting @var{output-path} equal to
@var{input-path} stripped of any machine names.
If @samp{-l} is not specified and the remote file resides on the
@acronym{NCAR} @acronym{MSS} system, then the leading character of
@var{input-path}, @samp{/}, is also stripped from @var{output-path}.
Specifying @var{output-path} as @samp{-l ./} tells @acronym{NCO} to
store the remotely retrieved file and the output file in the current
directory.
Note that @samp{-l .} is equivalent to @samp{-l ./} though the latter is
recommended as it is syntactically more clear.
@menu
* OPeNDAP::
@end menu
@html
<a name="dap"></a> <!-- http://nco.sf.net/nco.html#dap -->
<a name="DAP"></a> <!-- http://nco.sf.net/nco.html#DAP -->
<a name="DODS"></a> <!-- http://nco.sf.net/nco.html#DODS -->
<a name="OPeNDAP"></a> <!-- http://nco.sf.net/nco.html#OPeNDAP -->
<a name="dods"></a> <!-- http://nco.sf.net/nco.html#dods -->
<a name="opendap"></a> <!-- http://nco.sf.net/nco.html#opendap -->
@end html
@node OPeNDAP, , Remote storage, Remote storage
@subsection @acronym{OPeNDAP}
@cindex @acronym{DAP}
@cindex @acronym{DODS}
@cindex @acronym{HTTP} protocol
@cindex @env{DODS_ROOT}
@cindex Distributed Oceanographic Data System
@cindex oceanography
@cindex data access protocol
@cindex Open-source Project for a Network Data Access Protocol
@cindex @acronym{OPeNDAP}.
@cindex server
@cindex client-server
The Distributed Oceanographic Data System (@acronym{DODS}) provides
useful replacements for common data interface libraries like netCDF.
The @acronym{DODS} versions of these libraries implement network
transparent access to data via a client-server data access protocol
that uses the @acronym{HTTP} protocol for communication.
Although @acronym{DODS}-technology originated with oceanography data,
it applyies to virtually all scientific data.
In recognition of this, the data access protocol underlying
@acronym{DODS} (which is what @acronym{NCO} cares about) has been
renamed the Open-source Project for a Network Data Access Protocol,
@acronym{OPeNDAP}.
We use the terms @acronym{DODS} and @acronym{OPeNDAP} interchangeably,
and often write @acronym{OPeNDAP}/@acronym{DODS} for now.
In the future we will deprecate @acronym{DODS} in favor of
@acronym{DAP} or @acronym{OPeNDAP}, as appropriate
@footnote{
@cindex @acronym{NVODS}
@cindex National Virtual Ocean Data System
@cindex open source
@acronym{DODS} is being deprecated because it is ambiguous, referring
both to a protocol and to a collection of (oceanography) data.
It is superceded by two terms.
@acronym{DAP} is the discipline-neutral Data Access Protocol at the
heart of @acronym{DODS}.
The National Virtual Ocean Data System (@acronym{NVODS}) refers to the
collection of oceanography data and oceanographic extensions to
@acronym{DAP}.
In other words, @acronym{NVODS} is implemented with @acronym{OPeNDAP}.
@acronym{OPeNDAP} is @emph{also} the open source project which
maintains, develops, and promulgates the @acronym{DAP} standard.
@acronym{OPeNDAP} and @acronym{DAP} really are interchangeable.
Got it yet?}.
@acronym{NCO} may be @acronym{DAP}-enabled by linking
@acronym{NCO} to the @acronym{OPeNDAP} libraries.
@cindex @file{Makefile}
This is described in the @acronym{OPeNDAP} documentation and
automagically implemented in @acronym{NCO} build mechanisms
@footnote{
Automagic support for @acronym{DODS} version 3.2.x was deprecated in
December, 2003 after @acronym{NCO} version 2.8.4.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.4.x commenced in
December, 2003, with @acronym{NCO} version 2.8.5.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.5.x commenced in
June, 2005, with @acronym{NCO} version 3.0.1.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.6.x commenced in
June, 2006, with @acronym{NCO} version 3.1.3.
@acronym{NCO} support for @acronym{OPeNDAP} versions 3.7.x commenced in
January, 2007, with @acronym{NCO} version 3.1.9.}.
The @file{./configure} mechanism automatically enables @acronym{NCO} as
@acronym{OPeNDAP} clients if it can find the required
@acronym{OPeNDAP} libraries
@footnote{
The minimal set of libraries required to build @acronym{NCO} as
@acronym{OPeNDAP} clients are, in link order,
@file{libnc-dap.a}, @file{libdap.a}, and
@file{libxml2} and @file{libcurl.a}.}.
in the usual locations.
The @env{$DODS_ROOT} environment variable may be used to override the
default @acronym{OPeNDAP} library location at @acronym{NCO}
compile-time.
Building @acronym{NCO} with @file{bld/Makefile} and the command
@code{make DODS=Y} adds the (non-intuitive) commands to link to the
@acronym{OPeNDAP} libraries installed in the @env{$DODS_ROOT}
directory.
The file @file{doc/opendap.sh} contains a generic script intended to help
users install @acronym{OPeNDAP} before building @acronym{NCO}.
The documentation at the
@uref{http://www.opendap.org, OPeNDAP Homepage}
is voluminous.
Check there and on the
@uref{http://www.unidata.ucar.edu/packages/dods/home/mailLists/, DODS mail lists}.
to learn more about the extensive capabilities of @acronym{OPeNDAP}
@footnote{
We are most familiar with the @acronym{OPeNDAP} ability to enable
network-transparent data access.
@cindex constraint expressions
@cindex server-side processing
@acronym{OPeNDAP} has many other features, including sophisticated
hyperslabbing and server-side processing via @dfn{constraint expressions}.
If you know more about this, please consider writing a section
on "@acronym{OPeNDAP} Capabilities of Interest to @acronym{NCO} Users"
for incorporation in the @cite{NCO User's Guide}.}.
Once @acronym{NCO} is @acronym{DAP}-enabled the operators are
@acronym{OPeNDAP} clients.
All @acronym{OPeNDAP} clients have network transparent access to
any files controlled by a @acronym{OPeNDAP} server.
Simply specify the input file path(s) in @acronym{URL} notation and all
@acronym{NCO} operations may be performed on remote files made
accessible by a @acronym{OPeNDAP} server.
This command tests the basic functionality of @acronym{OPeNDAP}-enabled
@acronym{NCO} clients:
@example
% ncks -o ~/foo.nc -C -H -v one -l /tmp \
-p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata in.nc
one = 1
% ncks -H -v one ~/foo.nc
one = 1
@end example
The @code{one = 1} outputs confirm (first) that @command{ncks} correctly
retrieved data via the @acronym{OPeNDAP} protocol and (second) that
@command{ncks} created a valid local copy of the subsetted remote file.
The next command is a more advanced example which demonstrates the real
power of @acronym{OPeNDAP}-enabled @acronym{NCO} clients.
The @command{ncwa} client requests an equatorial hyperslab from remotely
stored @acronym{NCEP reanalyses data} of the @w{year 1969}.
The @acronym{NOAA} @acronym{OPeNDAP} server (hopefully!) serves these data.
The local @command{ncwa} client then computes and stores (locally) the
regional mean surface pressure @w{(in Pa)}.
@example
ncwa -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc ~/foo.nc
@end example
@noindent
@cindex packing
@cindex unpacking
All with one command!
The data in this particular input file also happen to be packed
(@pxref{Intrinsic functions}), although this is completely transparent
to the user since @acronym{NCO} automatically unpacks data before
attempting arithmetic.
@acronym{NCO} obtains remote files from the @acronym{OPeNDAP} server
(e.g., @file{www.cdc.noaa.gov}) rather than the local machine.
Input files are first copied to the local machine, then processed.
The @acronym{OPeNDAP} server performs data access, hyperslabbing,
and transfer to the local machine.
@cindex I/O
This allows the I/O to appear to @acronym{NCO} as if the input files
were local.
The local machine performs all arithmetic operations.
Only the hyperslabbed output data are transferred over the network (to
the local machine) for the number-crunching to begin.
The advantages of this are obvious if you are examining small parts of
large files stored at remote locations.
@html
<a name="rtn"></a> <!-- http://nco.sf.net/nco.html#rtn -->
@end html
@node Retaining Retrieved Files, Selecting Output File Format, Remote storage, Common features
@section Retaining Retrieved Files
@cindex file deletion
@cindex file removal
@cindex file retention
@cindex @code{-R}
@cindex @code{--rtn}
@cindex @code{--retain}
@cartouche
Availability: All operators@*
Short options: @samp{-R}@*
Long options: @samp{--rtn}, @samp{--retain}@*
@end cartouche
In order to conserve local file system space, files retrieved from
remote locations are automatically deleted from the local file system
once they have been processed.
Many @acronym{NCO} operators were constructed to work with numerous
large (e.g., @w{200 MB}) files.
Retrieval of multiple files from remote locations is done serially.
Each file is retrieved, processed, then deleted before the cycle
repeats.
In cases where it is useful to keep the remotely-retrieved files on the
local file system after processing, the automatic removal feature may be
disabled by specifying @samp{-R} on the command line.
Invoking @code{-R} disables the default printing behavior of
@command{ncks}.
This allows @command{ncks} to retrieve remote files without
automatically trying to print them.
See @ref{ncks netCDF Kitchen Sink}, for more details.
@cindex @acronym{FTP}
@cindex @acronym{SSH}
@cindex @acronym{msrcp}
Note that the remote retrieval features of @acronym{NCO} can always be
used to retrieve @emph{any} file, including non-netCDF files, via
@command{SSH}, anonymous @acronym{FTP}, or @command{msrcp}.
Often this method is quicker than using a browser, or running an
@acronym{FTP} session from a shell window yourself.
@cindex server
For example, say you want to obtain a @acronym{JPEG} file from a weather
server.
@example
ncks -R -p ftp://weather.edu/pub/pix/jpeg -l . storm.jpg
@end example
@noindent
In this example, @command{ncks} automatically performs an anonymous
@acronym{FTP} login to the remote machine and retrieves the specified
file.
When @command{ncks} attempts to read the local copy of @file{storm.jpg}
as a netCDF file, it fails and exits, leaving @file{storm.jpg} in
the current directory.
@cindex @acronym{DODS}
@cindex server
If your @acronym{NCO} is @acronym{DAP}-enabled (@pxref{OPeNDAP}),
then you may use @acronym{NCO} to retrieve any files (including netCDF,
@acronym{HDF}, etc.) served by an @acronym{OPeNDAP} server to your local
machine.
For example,
@example
ncks -R -l . -p \
http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc
@end example
Note that @acronym{NCO} is never the preffered way to transport files
from remote machines.
For large jobs, that is best handled by @acronym{FTP}, @acronym{SSH},
or @command{wget}.
It may occasionally be useful to use @acronym{NCO} to transfer files
when your other preferred methods are not available locally.
@html
<a name="fl_fmt"></a> <!-- http://nco.sf.net/nco.html#fl_fmt -->
<a name="hdf"></a> <!-- http://nco.sf.net/nco.html#hdf -->
<a name="64bit"></a> <!-- http://nco.sf.net/nco.html#64bit -->
<a name="netcdf4"></a> <!-- http://nco.sf.net/nco.html#netcdf4 -->
@end html
@node Selecting Output File Format, Large File Support, Retaining Retrieved Files, Common features
@section Selecting Output File Format
@cindex @acronym{HDF}
@cindex netCDF2
@cindex netCDF3
@cindex netCDF4
@cindex @code{NETCDF4_CLASSIC} files
@cindex @code{NETCDF4} files
@cindex @code{CLASSIC} files
@cindex @code{64BIT} files
@cindex @code{--3}
@cindex @code{-3}
@cindex @code{-4}
@cindex @code{--4}
@cindex @code{--netcdf4}
@cindex @code{--fl_fmt}
@cindex @code{--file_format}
@cindex @code{--64bit}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-3}, @samp{-4}@*
Long options: @samp{--3}, @samp{--4}, @samp{--64bit}, @samp{--fl_fmt},
@samp{--netcdf4}@*
@end cartouche
All @acronym{NCO} operators support (read and write) all three (or four,
depending on how one counts) file formats supported by netCDF4.
The default output file format for all operators is the input file
format.
The operators listed under ``Availability'' above allow the user to
specify the output file format independent of the input file format.
These operators allow the user to convert between the various file
formats.
(The operators @command{ncatted} and @command{ncrename} do not support
these switches so they always write the output netCDF file in the same
format as the input netCDF file.)
netCDF supports four types of files: @code{CLASSIC}, @code{64BIT},
@code{NETCDF4}, and @code{NETCDF4_CLASSIC},
The @code{CLASSIC} format is the traditional 32-bit offset written by
netCDF2 and netCDF3.
As of 2005, most netCDF datasets are in @code{CLASSIC} format.
The @code{64BIT} format was added in Fall, 2004.
The @code{NETCDF4} format uses @acronym{HDF5} as the file storage layer.
The files are (usually) created, accessed, and manipulated using the
traditional netCDF3 @acronym{API} (with numerous extensions).
The @code{NETCDF4_CLASSIC} format refers to netCDF4 files created with
the @code{NC_CLASSIC_MODEL} mask.
Such files use @acronym{HDF5} as the back-end storage format (unlike
netCDF3), though they incorporate only netCDF3 features.
Hence @code{NETCDF4_CLASSIC} files are perfectly readable by
applications which use only the netCDF3 @acronym{API} and library.
@acronym{NCO} must be built with @w{netCDF4} to write files in the new
@code{NETCDF4} and @code{NETCDF4_CLASSIC} formats, and to read files in
the new @code{NETCDF4} format.
Users are advised to use the default @code{CLASSIC} format or the
@code{NETCDF4_CLASSIC} format until netCDF4 is more widespread.
Widespread support for @code{NETCDF4} format files is not expected for
many years, 2009--2010, say.
If performance or coolness are issues, then use @code{NETCDF4_CLASSIC}
instead of @code{CLASSIC} format files.
As mentioned above, all operators write use the input file format for
output files unless told otherwise.
Toggling the long option @samp{--64bit} switch (or its
@var{key}-@var{value} equivalent @samp{--fl_fmt=64bit}) produces the
netCDF3 64-bit offset format named @code{64BIT}.
@acronym{NCO} must be built with @w{netCDF 3.6} or higher to produce
a @code{64BIT} file.
Using the @samp{-4} switch (or its long option equivalents
@samp{--4} or @samp{--netcdf4}), or setting its @var{key}-@var{value}
equivalent @samp{--fl_fmt=netcdf4} produces a @code{NETCDF4} file
(i.e., @acronym{HDF}).
Casual users are advised to use the default (netCDF3) @code{CLASSIC}
format until @w{netCDF 3.6} and @w{netCDF 4.0} are more widespread.
Conversely, operators given the @samp{-3} (or @samp{--3}) switch
without arguments will (attempt to) produce netCDF3 @code{CLASSIC}
output, even from netCDF4 input files.
These examples demonstrate converting a file from any netCDF format
into any other netCDF format (subject to limits of the format):
@example
ncks --fl_fmt=classic in.nc foo_3c.nc # netCDF3 classic
ncks --fl_fmt=64bit in.nc foo_364.nc # netCDF3 64bit
ncks --fl_fmt=netcdf4_classic in.nc foo_4c.nc # netCDF4 classic
ncks --fl_fmt=netcdf4 in.nc foo_4.nc # netCDF4
ncks -3 in.nc foo_3c.nc # netCDF3 classic
ncks --3 in.nc foo_3c.nc # netCDF3 classic
ncks -4 in.nc foo_4.nc # netCDF4
ncks --4 in.nc foo_4.nc # netCDF4
ncks --64 in.nc foo364.nc # netCDF3 64bit
@end example
Of course since most operators support these switches, the
``conversions'' can be done at the output stage of arithmetic
or metadata processing rather than requiring a separate step.
Producing (netCDF3) @code{CLASSIC} or @code{64BIT} files from
@code{NETCDF4_CLASSIC} files will work.
However, producing netCDF3 files from @code{NETCDF4} files will only
work if the output files are not required to contain netCDF4-specific
features.
To discover whether a netCDF file is a classic (32-bit offset) or newer
64-bit offset netCDF3 format, or is netCDF4 format, examine it with the
@command{od} (octal dump) command:
@cindex @command{od}
@cindex octal dump
@cindex netCDF3 classic file format
@cindex netCDF4 classic file format
@cindex netCDF4 file format
@cindex 32-bit offset file format
@cindex 64-bit offset file format
@example
% od -An -c -N4 foo_3c.nc
C D F 001
% od -An -c -N4 foo_364.nc
C D F 002
% od -An -c -N4 foo_4.nc
211 H D F
% od -An -c -N4 foo_4c.nc
211 H D F
@end example
Values of @samp{C D F 001} and @samp{C D F 002} indicate 32-bit
(classic) and 64-bit netCDF3 formats, respectively, while values of
@samp{211 H D F} indicate the newer netCDF4 file format.
Note that @code{NETCDF4} and @code{NETCDF4_CLASSIC} are the same
formats.
The latter simply causes an application to fail if it attempts to
write a @code{NETCDF4} file that cannot be completely read by the
netCDF3 library.
As of October, 2005, @acronym{NCO} writes no netCDF4-specific data
structures and so always succeeds at writing @code{NETCDF4_CLASSIC}
files.
@html
<a name="lfs"></a> <!-- http://nco.sf.net/nco.html#lfs -->
@end html
@node Large File Support, Subsetting Variables, Selecting Output File Format, Common features
@section Large File Support
@cindex LFS
@cindex Large File Support
@cartouche
Availability: All operators@*
Short options: none@*
Long options: none@*
@end cartouche
@acronym{NCO} has Large File Support (@acronym{LFS}), meaning that
@acronym{NCO} can write files larger than @w{2 GB} on some 32-bit
operating systems with netCDF libraries earlier than @w{version 3.6}.
If desired, LFS support must be configured when both netCDF and
@acronym{NCO} are installed.
netCDF @w{versions 3.6} and higher support 64-bit file addresses as part
of the netCDF standard.
We recommend that users ignore LFS support which is difficult to
configure and is implemented in @acronym{NCO} only to support netCDF
versions prior @w{to 3.6}.
This obviates the need for configuring explicit LFS support in
applications (such as @acronym{NCO}) which now support 64-bit files
directly through the netCDF interface.
See @ref{Selecting Output File Format} for instructions on accessing
the different file formats, including 64-bit files, supported by the
modern netCDF interface.
If you are still interesting in explicit LFS support for netCDF versions
prior @w{to 3.6}, know that LFS support depends on a complex,
interlocking set of operating system
@footnote{
Linux and @acronym{AIX} are known to support @acronym{LFS}.}
and netCDF suppport issues.
The netCDF LFS FAQ at
@uref{http://my.unidata.ucar.edu/content/software/netcdf/faq-lfs.html}
describes the various file size limitations imposed by different
versions of the netCDF standard.
@acronym{NCO} and netCDF automatically attempt to configure LFS at
build time.
@html
<a name="var"></a> <!-- http://nco.sf.net/nco.html#var -->
<a name="xcl"></a> <!-- http://nco.sf.net/nco.html#xcl -->
<a name="sbs"></a> <!-- http://nco.sf.net/nco.html#sbs -->
@end html
@node Subsetting Variables, Subsetting Coordinate Variables, Large File Support, Common features
@section Subsetting Variables
@cindex subsetting
@cindex exclusion
@cindex extraction
@cindex @code{-v @var{var}}
@cindex @code{--variable @var{var}}
@cindex @code{-x}
@cindex @code{--exclude}
@cindex @code{--xcl}
@cartouche
Availability: (@command{ncap2}), @command{ncbo}, @command{ncea},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-v}, @samp{-x}@*
Long options: @samp{--variable}, @samp{--exclude} or @samp{--xcl}@*
@end cartouche
Subsetting variables refers to explicitly specifying variables to be
included or excluded from operator actions.
Subsetting is implemented with the @samp{-v @var{var}[,@dots{}]} and
@samp{-x} options.
@w{A list} of variables to extract is specified following the @samp{-v}
option, e.g., @samp{-v time,lat,lon}.
Not using the @samp{-v} option is equivalent to specifying all
variables.
The @samp{-x} option causes the list of variables specified with
@samp{-v} to be @emph{excluded} rather than @emph{extracted}.
Thus @samp{-x} saves typing when you only want to extract fewer than
half of the variables in a file.
Variables explicitly specified for extraction with
@samp{-v @var{var}[,@dots{}]} @emph{must} be present in the input file
or an error will result.
Variables explicitly specified for @emph{exclusion} with
@samp{-x -v @var{var}[,@dots{}]} need not be present in the input
file.
@cindex memory requirements
Remember, if averaging or concatenating large files stresses your
systems memory or disk resources, then the easiest solution is often to
use the @samp{-v} option to retain only the most important variables
(@pxref{Memory Requirements}).
Due to its special capabilities, @command{ncap2} interprets the
@samp{-v} switch differently
(@pxref{ncap2 netCDF Arithmetic Processor}).
For @command{ncap2}, the @samp{-v} switch takes no arguments and
indicates that @emph{only} user-defined variables should be output.
@command{ncap2} neither accepts nor understands the @var{-x} switch.
@html
<a name="rx"></a> <!-- http://nco.sf.net/nco.html#rx -->
<a name="wildcarding"></a> <!-- http://nco.sf.net/nco.html#wildcarding -->
@end html
@cindex extended regular expressions
@cindex regular expressions
@cindex pattern matching
@cindex wildcards
@cindex @command{egrep}
@cindex @acronym{GNU}
As of @acronym{NCO} 2.8.1 (August, 2003), variable name arguments
of the @samp{-v} switch may contain @dfn{extended regular expressions}.
For example, @samp{-v '^DST'} selects all variables beginning with the
string @samp{DST}.
Extended regular expressions are defined by the @acronym{GNU}
@command{egrep} command.
The meta-characters used to express pattern matching operations are
@samp{^$+?.*[]@{@}|}.
If the regular expression pattern matches @emph{any} part of a variable
name then that variable is selected.
This capability is called @dfn{wildcarding}, and is very useful for
sub-setting large data files.
@cindex @acronym{POSIX}
@cindex @code{regex}
Because of its wide availability, @acronym{NCO} uses the @acronym{POSIX}
regular expression library @code{regex}.
Regular expressions of arbitary complexity may be used.
Since netCDF variable names are relatively simple constructs, only a
few varieties of variable wildcards are likely to be useful.
For convenience, we define the most useful pattern matching operators
here:
@cindex @code{.} (wildcard character)
@cindex @code{$} (wildcard character)
@cindex @code{^} (wildcard character)
@cindex @code{?} (filename expansion)
@cindex @code{*} (filename expansion)
@table @samp
@item ^
Matches the beginning of a string
@item $
Matches the end of a string
@item .
Matches any single character
@end table
@noindent
The most useful repetition and combination operators are
@cindex @code{?} (wildcard character)
@cindex @code{*} (wildcard character)
@cindex @code{+} (wildcard character)
@cindex @code{|} (wildcard character)
@table @samp
@item ?
The preceding regular expression is optional and matched at most once
@item *
The preceding regular expression will be matched zero or more times
@item +
The preceding regular expression will be matched one or more times
@item |
The preceding regular expression will be joined to the following regular
expression.
The resulting regular expression matches any string matching either
subexpression.
@end table
@noindent
To illustrate the use of these operators in extracting variables,
consider a file with variables @code{Q}, @code{Q01}--@code{Q99},
@code{Q100}, @code{QAA}--@code{QZZ}, @code{Q_H2O}, @code{X_H2O},
@code{Q_CO2}, @code{X_CO2}.
@example
ncks -v 'Q.?' in.nc # Variables that contain Q
ncks -v '^Q.?' in.nc # Variables that start with Q
ncks -v '^Q+.?.' in.nc # Q, Q0--Q9, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..' in.nc # Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]' in.nc # Q01--Q99, Q100
ncks -v '^Q[[:digit:]]@{2@}' in.nc # Q01--Q99
ncks -v 'H2O$' in.nc # Q_H2O, X_H2O
ncks -v 'H2O$|CO2$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2
ncks -v '^Q[0-9][0-9]$' in.nc # Q01--Q99
ncks -v '^Q[0-6][0-9]|7[0-3]' in.nc # Q01--Q73, Q100
ncks -v '(Q[0-6][0-9]|7[0-3])$' in.nc # Q01--Q73
ncks -v '^[a-z]_[a-z]@{3@}$' in.nc # Q_H2O, X_H2O, Q_CO2, X_CO2
@end example
Beware---two of the most frequently used repetition pattern matching
operators, @samp{*} and @samp{?}, are also valid pattern matching
operators for filename expansion (globbing) at the shell-level.
Confusingly, their meanings in extended regular expressions and in
shell-level filename expansion are significantly different.
In an extended regular expression, @samp{*} matches zero or more
occurences of the preceding regular expression.
Thus @samp{Q*} selects all variables, and @samp{Q+.*} selects all
variables containing @samp{Q} (the @samp{+} ensures the preceding item
matches at least once).
To match zero or one occurence of the preceding regular expression,
use @samp{?}.
Documentation for the @acronym{UNIX} @command{egrep} command details the
extended regular expressions which @acronym{NCO} supports.
@cindex globbing
@cindex shell
@cindex @command{bash}
@cindex @command{csh}
@cindex quotes
One must be careful to protect any special characters in the regular
expression specification from being interpreted (globbed) by the shell.
This is accomplish by enclosing special characters within single or
double quotes
@example
ncra -v Q?? in.nc out.nc # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc out.nc # Correct: NCO interprets, Shell globs
@end example
The final example shows that commands may use a combination of variable
wildcarding and shell filename expansion (globbing).
For globbing, @samp{*} and @samp{?} @emph{have nothing to do} with the
preceding regular expression!
In shell-level filename expansion, @samp{*} matches any string,
including the null string and @samp{?} matches any single character.
Documentation for @command{bash} and @command{csh} describe the rules of
filename expansion (globbing).
@html
<a name="crd"></a> <!-- http://nco.sf.net/nco.html#crd -->
<a name="-C"></a> <!-- http://nco.sf.net/nco.html#-C -->
<a name="-c"></a> <!-- http://nco.sf.net/nco.html#-c -->
@end html
@node Subsetting Coordinate Variables, C and Fortran Index Conventions, Subsetting Variables, Common features
@section Subsetting Coordinate Variables
@cindex subsetting
@cindex @code{-C}
@cindex @code{-c}
@cindex @code{--no-coords}
@cindex @code{--no-crd}
@cindex @code{--coords}
@cindex @code{--crd}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-C}, @samp{-c}@*
Long options: @samp{--no-coords}, @samp{--no-crd}, @samp{--crd}, @samp{--coords}@*
@end cartouche
By default, coordinates variables associated with any variable appearing
in the @var{input-file} will also appear in the @var{output-file}, even
if they are not explicitly specified, e.g., with the @samp{-v} switch.
Thus variables with a latitude coordinate @code{lat} always carry the
values of @code{lat} with them into the @var{output-file}.
This feature can be disabled with @samp{-C}, which causes @acronym{NCO}
to not automatically add coordinates to the variables appearing in the
@var{output-file}.
However, using @samp{-C} does not preclude the user from including some
coordinates in the output files simply by explicitly selecting the
coordinates with the @var{-v} option.
The @samp{-c} option, on the other hand, is a shorthand way of
automatically specifying that @emph{all} coordinate variables in the
@var{input-files} should appear in the @var{output-file}.
Thus @samp{-c} allows the user to select all the coordinate variables
without having to know their names.
@cindex @acronym{CF} conventions
Both @samp{-c} and @samp{-C} honor the @acronym{CF} @code{coordinates}
convention described in @ref{CF Conventions}.
@html
<a name="ftn"></a> <!-- http://nco.sf.net/nco.html#ftn -->
<a name="-F"></a> <!-- http://nco.sf.net/nco.html#-F -->
@end html
@node C and Fortran Index Conventions, Hyperslabs, Subsetting Coordinate Variables, Common features
@section C and Fortran Index conventions
@cindex index convention
@cindex Fortran index convention
@cindex C index convention
@cindex @code{-F}
@cindex @code{--fortran}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@*
Short options: @samp{-F}@*
Long options: @samp{--fortran}@*
@end cartouche
@cindex I/O
The @samp{-F} switch changes @acronym{NCO} to read and write with
the Fortran index convention.
By default, @acronym{NCO} uses C-style (0-based) indices for all I/O.
@w{In C}, indices count @w{from 0} (rather @w{than 1}), and
dimensions are ordered from slowest (inner-most) to fastest
(outer-most) varying.
In Fortran, indices count @w{from 1} (rather @w{than 0}), and
dimensions are ordered from fastest (inner-most) to slowest
(outer-most) varying.
@cindex transpose
Hence @w{C and} Fortran data storage conventions represent mathematical
transposes of eachother.
@cindex record variable
Note that record variables contain the record dimension as the most
slowly varying dimension.
See @ref{ncpdq netCDF Permute Dimensions Quickly} for techniques
to re-order (including transpose) dimensions and to reverse data
storage order.
@cindex record dimension
Consider a file @file{85.nc} containing @w{12 months} of data in the
record dimension @code{time}.
The following hyperslab operations produce identical results, a
June-July-August average of the data:
@example
ncra -d time,5,7 85.nc 85_JJA.nc
ncra -F -d time,6,8 85.nc 85_JJA.nc
@end example
Printing variable @var{three_dmn_var} in file @file{in.nc} first with
the @w{C indexing} convention, then with Fortran indexing convention
results in the following output formats:
@example
% ncks -v three_dmn_var in.nc
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0
...
% ncks -F -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
...
@end example
@html
<a name="-d"></a> <!-- http://nco.sf.net/nco.html#-d -->
<a name="dmn"></a> <!-- http://nco.sf.net/nco.html#dmn -->
<a name="hyp"></a> <!-- http://nco.sf.net/nco.html#hyp -->
@end html
@node Hyperslabs, Stride, C and Fortran Index Conventions, Common features
@section Hyperslabs
@cindex hyperslab
@cindex dimension limits
@cindex coordinate limits
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options:
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
@w{A @dfn{hyperslab}} is a subset of a variable's data.
The coordinates of a hyperslab are specified with the
@code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]} short
option (or with the same arguments to the @samp{--dimension} or
@samp{--dmn} long options).
At least one hyperslab argument (@var{min}, @var{max}, or @var{stride})
must be present.
The bounds of the hyperslab to be extracted are specified by the
associated @var{min} and @var{max} values.
@w{A half}-open range is specified by omitting either the @var{min} or
@var{max} parameter.
The separating comma must be present to indicate the omission of one of
these arguments.
The unspecified limit is interpreted as the maximum or minimum value in
the unspecified direction.
@w{A cross}-section at a specific coordinate is extracted by specifying only
the @var{min} limit and omitting a trailing comma.
Dimensions not mentioned are passed with no reduction in range.
The dimensionality of variables is not reduced (in the case of a
cross-section, the size of the constant dimension will be one).
If values of a coordinate-variable are used to specify a range or
cross-section, then the coordinate variable must be monotonic (values
either increasing or decreasing).
In this case, command-line values need not exactly match coordinate
values for the specified dimension.
Ranges are determined by seeking the first coordinate value to occur in
the closed range [@var{min},@var{max}] and including all subsequent
values until one falls outside the range.
The coordinate value for a cross-section is the coordinate-variable
value closest to the specified value and must lie within the range or
coordinate-variable values.
@cindex stride
Coordinate values should be specified using real notation with a decimal
point required in the value, whereas dimension indices are specified
using integer notation without a decimal point.
This convention serves only to differentiate coordinate values from
dimension indices.
It is independent of the type of any netCDF coordinate variables.
For a given dimension, the specified limits must both be coordinate
values (with decimal points) or dimension indices (no decimal points).
The @var{stride} option, if any, must be a dimension index, not a
coordinate value.
@xref{Stride}, for more information on the @var{stride} option.
@cindex @code{NC_BYTE}
@cindex @code{NC_CHAR}
User-specified coordinate limits are promoted to double precision values
while searching for the indices which bracket the range.
Thus, hyperslabs on coordinates of type @code{NC_BYTE} and
@code{NC_CHAR} are computed numerically rather than lexically, so the
results are unpredictable.
@cindex wrapped coordinates
The relative magnitude of @var{min} and @var{max} indicate to the
operator whether to expect a @dfn{wrapped coordinate}
(@pxref{Wrapped Coordinates}), such as longitude.
If @math{@var{min} > @var{max}}, the @acronym{NCO} expects the
coordinate to be wrapped, and a warning message will be printed.
When this occurs, @acronym{NCO} selects all values outside the domain
[@math{@var{max} < @var{min}}], i.e., all the values exclusive of the
values which would have been selected if @var{min} and @var{max} were
swapped.
If this seems confusing, test your command on just the coordinate
variables with @command{ncks}, and then examine the output to ensure
@acronym{NCO} selected the hyperslab you expected (coordinate wrapping
is currently only supported by @command{ncks}).
Because of the way wrapped coordinates are interpreted, it is very
important to make sure you always specify hyperslabs in the
monotonically increasing sense, i.e., @math{@var{min} < @var{max}}
(even if the underlying coordinate variable is monotonically
decreasing).
The only exception to this is when you are indeed specifying a wrapped
coordinate.
The distinction is crucial to understand because the points selected by,
e.g., @code{-d longitude,50.,340.}, are exactly the complement of the
points selected by @code{-d longitude,340.,50.}.
Not specifying any hyperslab option is equivalent to specifying full
ranges of all dimensions.
This option may be specified more than once in a single command
(each hyperslabbed dimension requires its own @code{-d} option).
@html
<a name="srd"></a> <!-- http://nco.sf.net/nco.html#srd -->
<a name="stride"></a> <!-- http://nco.sf.net/nco.html#stride -->
@end html
@node Stride, Multislabs, Hyperslabs, Common features
@section Stride
@cindex stride
@cindex @code{-d @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cindex @code{--dimension @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cindex @code{--dmn @var{dim},[@var{min}],[@var{max}],@var{stride}}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options:
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
All data operators support specifying a @dfn{stride} for any and all
dimensions at the same time.
The @var{stride} is the spacing between consecutive points in a
hyperslab.
@w{A @var{stride}} @w{of 1} picks all the elements of the hyperslab, and
a @var{stride} @w{of 2} skips every other element, etc.@.
@command{ncks} multislabs support strides, and are more powerful than
the regular hyperslabs supported by the other operators
(@pxref{Multislabs}).
Using the @var{stride} option for the record dimension with
@command{ncra} and @command{ncrcat} makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data sets.
The @var{stride} is specified as the optional fourth argument to the
@samp{-d} hyperslab specification:
@code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}.
Specify @var{stride} as an integer (i.e., no decimal point) following
the third comma in the @samp{-d} argument.
There is no default value for @var{stride}.
Thus using @samp{-d time,,,2} is valid but @samp{-d time,,,2.0} and
@samp{-d time,,,} are not.
When @var{stride} is specified but @var{min} is not, there is an
ambiguity as to whether the extracted hyperslab should begin with (using
C-style, 0-based indexes) @w{element 0} or element @samp{stride-1}.
@acronym{NCO} must resolve this ambiguity and it chooses @w{element 0}
as the first element of the hyperslab when @var{min} is not specified.
Thus @samp{-d time,,,@var{stride}} is syntactically equivalent to
@samp{-d time,0,,@var{stride}}.
This means, for example, that specifying the operation
@samp{-d time,,,2} on the array @samp{1,2,3,4,5} selects the hyperslab
@samp{1,3,5}.
To obtain the hyperslab @samp{2,4} instead, simply explicitly specify
the starting index @w{as 1,} i.e., @samp{-d time,1,,2}.
For example, consider a file @file{8501_8912.nc} which contains 60
consecutive months of data.
Say you wish to obtain just the March data from this file.
Using 0-based subscripts (@pxref{C and Fortran Index Conventions}) these
data are stored in records @w{2, 14, @dots{} 50} so the desired
@var{stride} @w{is 12.}
Without the @var{stride} option, the procedure is very awkward.
One could use @command{ncks} five times and then use @command{ncrcat} to
concatenate the resulting files together:
@cindex Bourne Shell
@cindex C Shell
@example
for idx in 02 14 26 38 50; do # Bourne Shell
ncks -d time,$@{idx@} 8501_8912.nc foo.$@{idx@}
done
foreach idx (02 14 26 38 50) # C Shell
ncks -d time,$@{idx@} 8501_8912.nc foo.$@{idx@}
end
ncrcat foo.?? 8589_03.nc
rm foo.??
@end example
With the @var{stride} option, @command{ncks} performs this hyperslab
extraction in one operation:
@example
ncks -d time,2,,12 8501_8912.nc 8589_03.nc
@end example
@xref{ncks netCDF Kitchen Sink}, for more information on @command{ncks}.
Applying the @var{stride} option to the record dimension in
@command{ncra} and @command{ncrcat} makes it possible, for instance, to
average or concatenate regular intervals across multi-file input data
sets.
@example
ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc
@end example
@html
<a name="msa"></a> <!-- http://nco.sf.net/nco.html#msa -->
<a name="mlt"></a> <!-- http://nco.sf.net/nco.html#mlt -->
@end html
@node Multislabs, Wrapped Coordinates, Stride, Common features
@section Multislabs
@cindex multislab
@cindex multi-hyperslab
@cindex @acronym{MSA}
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options:
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
A @w{multislab} is a union of one or more hyperslabs.
One defines multislabs by chaining together hyperslab commands, i.e.,
@kbd{-d} options (@pxref{Hyperslabs}).
Support for specifying a @dfn{multi-hyperslab} or @dfn{multislab} for
any variable was first added to @command{ncks} in late 2002.
The other operators received @acronym{MSA} capabilities in April 2008.
Sometimes multi-slabbing is referred to by the acronym @acronym{MSA},
which stands for ``Multi-Slabbing Algorithm''.
Multislabs overcome some restraints that limit hyperslabs.
@w{A single} @kbd{-d} option can only specify a contiguous and/or
a regularly spaced multi-dimensional data array.
Multislabs are constructed from multiple @kbd{-d} options and may
therefore have non-regularly spaced arrays.
For example, suppose it is desired to operate on all longitudes
from 10.0 to 20.0 and from 80.0 to @w{90.0 degrees}.
The combined range of longitudes is not selectable in a single
hyperslab specfication of the form
@samp{-d @var{dimension},@var{min},@var{max}} or
@samp{-d @var{dimension},@var{min},@var{max},@var{stride}} because its
elements are irregularly spaced in coordinate space (and presumably
in index space too).
The multislab specification for obtaining these values is simply
the union of the hyperslabs specifications that comprise the multislab,
i.e.,
@example
ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc
@end example
@noindent
Any number of hyperslabs specifications may be chained together
to specify the multislab.
@cindex stride
Users may specify redundant ranges of indices in a multislab, e.g.,
@example
ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc
@end example
@noindent
This command retrieves the first five longitudes, and then every other
longitude value up to the tenth.
Elements 0, 2, @w{and 4} are specified by both hyperslab arguments (hence
this is redundant) but will count only once if an arithmetic operation
is being performed.
This example uses index-based (not coordinate-based) multislabs because
the @var{stride} option only supports index-based hyper-slabbing.
@xref{Stride}, for more information on the @var{stride} option.
Multislabs are more efficient than the alternative of sequentially
performing hyperslab operations and concatenating the results.
@cindex I/O
This is because @acronym{NCO} employs a novel multislab algorithm to
minimize the number of I/O operations when retrieving irregularly spaced
data from disk.
The @acronym{NCO} multislab algorithm retrieves each element from disk
once and only once.
Thus users may take some shortcuts in specifying multislabs and the
algorithm will obtain the intended values.
Specifying redundant ranges is not encouraged, but may be useful on
occasion and will not result in unintended consequences.
@w{A final} example shows the real power of multislabs.
Suppose the @var{Q} variable contains three dimensional arrays of
distinct chemical constituents in no particular order.
We are interested in the NOy species in a certain geographic range.
Say that NO, NO2, and N2O5 are @w{elements 0}, 1, @w{and 5} of the
@var{species} dimension of @var{Q}.
The multislab specification might look something like
@example
ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc
@end example
@noindent
Multislabs are powerful because they may be specified for every
dimension at the same time.
Thus multislabs obsolete the need to execute multiple @command{ncks}
commands to gather the desired range of data.
@html
<a name="wrp"></a> <!-- http://nco.sf.net/nco.html#wrp -->
@end html
@node Wrapped Coordinates, Auxiliary Coordinates, Multislabs, Common features
@section Wrapped Coordinates
@cindex wrapped coordinates
@cindex longitude
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncks}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options:
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
@w{A @dfn{wrapped coordinate}} is a coordinate whose values increase or
decrease monotonically (nothing unusual so far), but which represents a
dimension that ends where it begins (i.e., wraps around on itself).
Longitude (i.e., degrees on a circle) is a familiar example of a wrapped
coordinate.
Longitude increases to the East of Greenwich, England, where it is
defined to be zero.
Halfway around the globe, the longitude is @w{180 degrees} East (or West).
Continuing eastward, longitude increases to @w{360 degrees} East at
Greenwich.
The longitude values of most geophysical data are either in the range
[0,360), or [@minus{}180,180).
In either case, the Westernmost and Easternmost longitudes are
numerically separated by @w{360 degrees}, but represent contiguous
regions on the globe.
For example, the Saharan desert stretches from roughly 340 to
@w{50 degrees} East.
Extracting the hyperslab of data representing the Sahara from a global
dataset presents special problems when the global dataset is stored
consecutively in longitude from 0 to @w{360 degrees}.
This is because the data for the Sahara will not be contiguous in the
@var{input-file} but is expected by the user to be contiguous in the
@var{output-file}.
In this case, @command{ncks} must invoke special software routines to
assemble the desired output hyperslab from multiple reads of the
@var{input-file}.
Assume the domain of the monotonically increasing longitude coordinate
@code{lon} is @math{0 < @var{lon} < 360}.
@command{ncks} will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as @var{min} and
the easternmost longitude as @var{max}.
The following commands extract a hyperslab containing the Saharan desert:
@example
ncks -d lon,340.,50. in.nc out.nc
ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc
@end example
@noindent
The first example selects data in the same longitude range as the Sahara.
The second example further constrains the data to having the same
latitude as the Sahara.
The coordinate @code{lon} in the @var{output-file}, @file{out.nc}, will
no longer be monotonic!
The values of @code{lon} will be, e.g., @samp{340, 350, 0, 10, 20, 30,
40, 50}.
This can have serious implications should you run @file{out.nc} through
another operation which expects the @code{lon} coordinate to be
monotonically increasing.
Fortunately, the chances of this happening are slim, since @code{lon}
has already been hyperslabbed, there should be no reason to hyperslab
@code{lon} again.
Should you need to hyperslab @code{lon} again, be sure to give
dimensional indices as the hyperslab arguments, rather than coordinate
values (@pxref{Hyperslabs}).
@html
<a name="aux"></a> <!-- http://nco.sf.net/nco.html#aux -->
<a name="auxiliary"></a> <!-- http://nco.sf.net/nco.html#auxiliary -->
<a name="-X"></a> <!-- http://nco.sf.net/nco.html#-X -->
<a name="std_nm"></a> <!-- http://nco.sf.net/nco.html#std_nm -->
<a name="standard_name"></a> <!-- http://nco.sf.net/nco.html#standard_name -->
@end html
@node Auxiliary Coordinates, UDUnits Support, Wrapped Coordinates, Common features
@section Auxiliary Coordinates
@cindex @code{-X}
@cindex @code{--auxiliary}
@cindex @code{standard_name}
@cindex @code{coordinates}
@cindex @acronym{CF} conventions
@cindex @code{-X @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}
@cindex @code{--auxiliary @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}@*
Short options: @samp{-X @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}@*
Long options:
@samp{--auxiliary @var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}}@*
@end cartouche
Utilize auxiliary coordinates specified in values of coordinate
variable's @code{standard_name} attributes, if any, when interpreting
hyperslab and multi-slab options.
Also @samp{--auxiliary}.
This switch supports hyperslabbing cell-based grids over coordinate
ranges.
This works on datasets that associate coordinate variables to
grid-mappings using the @acronym{CF}-convention (@pxref{CF Conventions})
@code{coordinates} and @code{standard_name} attributes described
@uref{http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.1/cf-conventions.html#coordinate-system, here}.
Currently, @acronym{NCO} understands auxiliary coordinate variables
pointed to by the @code{standard_name} attributes for @var{latitude} and
@var{longitude}.
Cells that contain a value within the user-specified range
[@var{lon_min},@var{lon_max},@var{lat_min},@var{lat_max}] are
included in the output hyperslab.
A cell-based grid collapses the horizontal spatial information
(latitude and longitude) and stores it along a one-dimensional
coordinate that has a one-to-one mapping to both latitude and longitude
coordinates.
Rectangular (in longitude and latitude) horizontal hyperslabs cannot
be selected using the typical procedure (@pxref{Hyperslabs}) of
separately specifying @samp{-d} arguments for longitude and latitude.
Instead, when the @samp{-X} is used, @acronym{NCO} learns the names of
the latitude and longitude coordinates from the @code{coordinates}
attribute of each variable.
The @code{standard_name} attribute of the latitude and longitude
coordinates specifies, using the @acronym{CF}-convention
(@pxref{CF Conventions}), whether the spatial dimension represents
latitude or longitude.
Putting it all together, consider a variable @var{gds_3dvar} output from
simulations on a cell-based geodesic grid.
Although variable contains three dimensions of data (time, latitude, and
longitue), it is stored in the netCDF file with only two coordinate
dimensions, @code{time} and @code{gds_crd}.
@example
% ncks -m -C -v gds_3dvar ~/nco/data/in.nc
gds_3dvar: # dim. = 2, NC_FLOAT, # att. = 4, ID = 38
gds_3dvar dimension 0: time, size = 10 NC_DOUBLE, dim. ID = 18 (CRD)(REC)
gds_3dvar dimension 1: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
gds_3dvar memory size is 10*8*nco_typ_lng(NC_FLOAT) = 80*4 = 320 bytes
gds_3dvar attribute 0: long_name, size = 17 NC_CHAR, value = Geodesic variable
gds_3dvar attribute 1: units, size = 5 NC_CHAR, value = meter
gds_3dvar attribute 2: coordinates, size = 15 NC_CHAR, value = lat_gds lon_gds
@end example
The @code{coordinates} attribute lists the names of the latitude and
longitude coordinates, @code{lat_gds} and @code{lon_gds}, respectively.
The @code{standard_name} attribute of @code{lat_gds} and @code{lon_gds}
define whether the spatial dimension represents latitude or longitude:
@example
% ncks -m -C -v lat_gds,lon_gds ~/nco/data/in.nc
lat_gds: # dim. = 1, NC_DOUBLE, # att. = 4, ID = 34
lat_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lat_gds memory size is 8*nco_typ_lng(NC_DOUBLE) = 8*8 = 64 bytes
lat_gds attribute 0: long_name, size = 8 NC_CHAR, value = Latitude
lat_gds attribute 1: standard_name, size = 8 NC_CHAR, value = latitude
lat_gds attribute 2: units, size = 6 NC_CHAR, value = degree
lon_gds: # dim. = 1, NC_DOUBLE, # att. = 4, ID = 35
lon_gds dimension 0: gds_crd, size = 8 NC_FLOAT, dim. ID = 17 (CRD)
lon_gds memory size is 8*nco_typ_lng(NC_DOUBLE) = 8*8 = 64 bytes
lon_gds attribute 0: long_name, size = 9 NC_CHAR, value = Longitude
lon_gds attribute 1: standard_name, size = 9 NC_CHAR, value = longitude
lon_gds attribute 2: units, size = 6 NC_CHAR, value = degree
@end example
To time-average all the values between zero and @w{180 degrees}
longitude and between plus and minus @w{30 degress} latitude, we use
@example
ncra -O -X 0.,180.,-30.,30. -v gds_3dvar in.nc out.nc
@end example
@html
<a name="UDUnits"></a> <!-- http://nco.sf.net/nco.html#UDUnits -->
<a name="UDUnits2"></a> <!-- http://nco.sf.net/nco.html#UDUnits2 -->
@end html
@node UDUnits Support, Missing Values, Auxiliary Coordinates, Common features
@section UDUnits Support
@cindex UDUnits
@cindex Unidata
@cindex @code{units}
@cindex attribute, @code{units}
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cindex @code{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncks}, @command{ncpdq}, @command{ncra},
@command{ncrcat}, @command{ncwa}@*
Short options: @samp{-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
Long options:
@samp{--dimension @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]},@*
@samp{--dmn @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]}@*
@end cartouche
There is more than one way to hyperskin a cat.
The @uref{http://www.unidata.ucar.edu/packages/udunits, UDUnits} package
provides a library which, if present, @acronym{NCO} uses to translate
user-specified physical dimensions into the physical dimensions of data
stored in netCDF files.
Unidata provides UDUnits under the same terms as netCDF, so sites should
install both.
Compiling @acronym{NCO} with UDUnits support is currently optional but
may become required in a future version of @acronym{NCO}.
Two examples suffice to demonstrate the power and convenience of UDUnits
support.
@cindex MKS units
First, consider extraction of a variable containing non-record
coordinates with physical dimensions stored in MKS units.
In the following example, the user extracts all wavelengths
in the visible portion of the spectrum in terms of the units
very frequently used in visible spectroscopy, microns:
@example
% ncks -C -H -v wvl -d wvl,"0.4 micron","0.7 micron" in.nc
wvl[0]=5e-07 meter
@end example
@noindent
@cindex @code{units}
The hyperslab returns the correct values because the @var{wvl} variable
is stored on disk with a length dimension that UDUnits recognizes in the
@code{units} attribute.
The automagical algorithm that implements this functionality is worth
describing since understanding it helps one avoid some potential
pitfalls.
First, the user includes the physical units of the hyperslab dimensions
she supplies, separated by a simple space from the numerical values of
the hyperslab limits.
She encloses each coordinate specifications in quotes so that the shell
does not break the @emph{value-space-unit} string into separate
arguments before passing them to @acronym{NCO}.
Double quotes (@kbd{"foo"}) or single quotes (@kbd{'foo'}) are equally
valid for this purpose.
Second, @acronym{NCO} recognizes that units translation is requested
because each hyperslab argument contains text characters and non-initial
spaces.
Third, @acronym{NCO} determines whether the @var{wvl} is dimensioned
with a coordinate variable that has a @code{units} attribute.
@cindex coordinate variable
In this case, @var{wvl} itself is a coordinate variable.
The value of its @code{units} attribute is @code{meter}.
Thus @var{wvl} passes this test so UDUnits conversion is attempted.
If the coordinate associated with the variable does not contain a
@code{units} attribute, then @acronym{NCO} aborts.
Fourth, @acronym{NCO} passes the specified and desired dimension strings
(microns are specified by the user, meters are required by
@acronym{NCO}) to the UDUnits library.
Fifth, the UDUnits library that these dimension are commensurate
and it returns the appropriate linear scaling factors to convert from
microns to meters to @acronym{NCO}.
If the units are incommensurate (i.e., not expressible in the same
fundamental MKS units), or are not listed in the UDUnits database, then
NCO aborts since it cannot determine the user's intent.
Finally, @acronym{NCO} uses the scaling information to convert the
user-specified hyperslab limits into the same physical dimensions as
those of the corresponding cooridinate variable on disk.
At this point, @acronym{NCO} can perform a coordinate hyperslab using
the same algorithm as if the user had specified the hyperslab without
requesting units conversion.
@cindex @code{units}
@cindex @code{time}
The translation and dimensional innterpretation of time coordinates
shows a more powerful, and probably more common, UDUnits application.
In this example, the user prints all data between the eighth and ninth
of December, 1999, from a variable whose time dimension is hours
since the year 1900:
@example
% ncks -H -C -v time_udunits -d time_udunits,"1999-12-08 \
12:00:0.0","1999-12-09 00:00:0.0",2 in.nc foo2.nc
time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0
@end example
@noindent
@cindex stride
@cindex whitespace
Here, the user invokes the stride (@pxref{Stride}) capability to obtain
every other timeslice.
This is possible because the UDUnits feature is additive, not
exclusive---it works in conjunction with all other hyperslabbing
(@pxref{Hyperslabs}) options and in all operators which support
hyperslabbing.
The following example shows how one might average data in a
time period spread across multiple input files
@example
ncra -d time,"1939-09-09 12:00:0.0","1945-05-08 00:00:0.0" \
in1.nc in2.nc in3.nc out.nc
@end example
@noindent
Note that there is no excess whitespace before or after the individual
elements of the @samp{-d} argument.
@cindex shell
This is important since, as far as the shell knows, @samp{-d} takes
only @emph{one} command-line argument.
Parsing this argument into its component
@code{@var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]} elements
(@pxref{Hyperslabs}) is the job of @acronym{NCO}.
When unquoted whitespace is present between these elements, the shell
passes @acronym{NCO} arugment fragments which will not parse as
intended.
@acronym{NCO} implemented support for the UDUnits2 library with version
3.9.2 (August, 2007).
The @uref{http://www.unidata.ucar.edu/software/udunits/udunits-2/udunits2.html, UDUnits2} package supports non-ASCII characters and logarithmic units.
We are interested in user-feedback on these features,
which are relatively un-tested with @acronym{NCO}.
@cindex Climate and Forecast Metadata Convention
@cindex CF conventions
The @uref{http://www.unidata.ucar.edu/packages/udunits, UDUnits}
package documentation describes the supported formats of time
dimensions.
Among the metadata conventions which adhere to these formats are the
@uref{http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html,
Climate and Forecast (CF) Conventions} and the
@uref{http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html,
Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions}.
The following @samp{-d arguments} extract the same data using
commonly encountered time dimension formats:
@c fxm add more formats here
@example
-d time,"1918-11-11 11:00:0.0","1939-09-09 00:00:0.0"
@end example
@noindent
All of these formats include at least one dash @kbd{-} in a
non-leading character position (a dash in a leading character position
is a negative sign).
@acronym{NCO} assumes that a non-leading dash in a limit string
indicates that a UDUnits date conversion is requested.
@cindex MKS units
@cindex God
netCDF variables should always be stored with MKS (i.e., God's) units,
so that application programs may assume MKS dimensions apply to all
input variables.
The UDUnits feature is intended to alleviate some of the @acronym{NCO}
user's pain when handling MKS units.
It connects users who think in human-friendly units (e.g.,
miles, millibars, days) to extract data which are always stored in God's
units, MKS (e.g., meters, Pascals, seconds).
The feature is not intended to encourage writers to store data in
esoteric units (e.g., furlongs, pounds per square inch, fortnights).
@html
<a name="missing_value"></a> <!-- http://nco.sf.net/nco.html#missing_value -->
<a name="_FillValue"></a> <!-- http://nco.sf.net/nco.html#_FillValue -->
<a name="fll_val"></a> <!-- http://nco.sf.net/nco.html#fll_val -->
<a name="mss_val"></a> <!-- http://nco.sf.net/nco.html#mss_val -->
<a name="mss"></a> <!-- http://nco.sf.net/nco.html#mss -->
@end html
@node Missing Values, Deflation, UDUnits Support, Common features
@section Missing values
@cindex missing values
@cindex data, missing
@cindex averaging data
@cindex @code{missing_value}
@cindex @code{_FillValue}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncflint}, @command{ncpdq}, @command{ncra}, @command{ncwa}@*
Short options: None@*
@end cartouche
The phrase @dfn{missing data} refers to data points that are missing,
invalid, or for any reason not intended to be arithmetically processed
in the same fashion as valid data.
@cindex arithmetic operators
The @acronym{NCO} arithmetic operators attempt to handle missing data in
an intelligent fashion.
There are four steps in the @acronym{NCO} treatment of missing data:
@enumerate
@item
Identifying variables that may contain missing data.
@acronym{NCO} follows the convention that missing data should be stored
with the @var{_FillValue} specified in the variable's @code{_FillValue}
attributes.
The @emph{only} way @acronym{NCO} recognizes that a variable @emph{may}
contain missing data is if the variable has a @code{_FillValue}
attribute.
In this case, any elements of the variable which are numerically equal
to the @var{_FillValue} are treated as missing data.
@acronym{NCO} adopted the behavior that the default attribute name, if
any, assumed to specify the value of data to ignore is @code{_FillValue}
with version 3.9.2 (August, 2007).
Prior to that, the @code{missing_value} attribute, if any, was assumed to
specify the value of data to ignore.
Supporting both of these attributes simultaneously is not practical.
Hence the behavior NCO once applied to @var{missing_value} it now applies
to any @var{_FillValue}.
@acronym{NCO} now treats any @var{missing_value} as normal data
@footnote{
The old functionality, i.e., where the ignored values are indicated by
@code{missing_value} not @code{_FillValue}, may still be selected
@emph{at @acronym{NCO} build time} by compiling @acronym{NCO}
with the token definition
@kbd{CPPFLAGS='-DNCO_MSS_VAL_SNG=missing_value'}.
}.
@findex ncrename
@findex ncatted
It has been and remains most advisable to create both @code{_FillValue}
and @code{missing_value} attributes with identical values in datasets.
Many legacy datasets contain only @code{missing_value} attributes.
@acronym{NCO} can help migrating datasets between these conventions.
One may use @command{ncrename} (@pxref{ncrename netCDF Renamer}) to
rename all @code{missing_value} attributes to @code{_FillValue}:
@example
ncrename -a .missing_value,_FillValue inout.nc
@end example
Alternatively, one may use
@command{ncatted} (@pxref{ncatted netCDF Attribute Editor}) to
add a @code{_FillValue} attribute to all variables
@example
ncatted -O -a _FillValue,,o,f,1.0e36 inout.nc
@end example
@item
Converting the @var{_FillValue} to the type of the variable, if
neccessary.
Consider a variable @var{var} of type @var{var_type} with a
@code{_FillValue} attribute of type @var{att_type} containing the
value @var{_FillValue}.
As a guideline, the type of the @code{_FillValue} attribute should be
the same as the type of the variable it is attached to.
If @var{var_type} equals @var{att_type} then @acronym{NCO}
straightforwardly compares each value of @var{var} to
@var{_FillValue} to determine which elements of @var{var} are to be
treated as missing data.
@cindex C language
If not, then @acronym{NCO} converts @var{_FillValue} from
@var{att_type} to @var{var_type} by using the implicit conversion rules
@w{of C}, or, if @var{att_type} is @code{NC_CHAR}
@footnote{For example, the @acronym{DOE} @acronym{ARM} program often
uses @var{att_type} = @code{NC_CHAR} and @var{_FillValue} =
@samp{-99999.}.
}, by typecasting the results of the @w{C function}
@code{strtod(@var{_FillValue})}.
@cindex @command{ncatted}
You may use the @acronym{NCO} operator @command{ncatted} to change the
@code{_FillValue} attribute and all data whose data is
@var{_FillValue} to a new value
(@pxref{ncatted netCDF Attribute Editor}).
@item
Identifying missing data during arithmetic operations.
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
@cindex arithmetic operators
When an @acronym{NCO} arithmetic operator processes a variable @var{var}
with a @code{_FillValue} attribute, it compares each value of
@var{var} to @var{_FillValue} before performing an operation.
Note the @var{_FillValue} comparison imposes a performance penalty
on the operator.
Arithmetic processing of variables which contain the
@code{_FillValue} attribute always incurs this penalty, even when
none of the data are missing.
Conversely, arithmetic processing of variables which do not contain the
@code{_FillValue} attribute never incurs this penalty.
In other words, do not attach a @code{_FillValue} attribute to a
variable which does not contain missing data.
This exhortation can usually be obeyed for model generated data, but it
may be harder to know in advance whether all observational data will be
valid or not.
@item
Treatment of any data identified as missing in arithmetic operators.
@cindex @command{ncea}
@cindex @command{ncra}
@cindex @command{ncwa}
@cindex @command{ncbo}
@cindex @command{ncflint}
@acronym{NCO} averagers (@command{ncra}, @command{ncea}, @command{ncwa})
do not count any element with the value @var{_FillValue} towards the
average.
@command{ncbo} and @command{ncflint} define a @var{_FillValue} result
when either of the input values is a @var{_FillValue}.
Sometimes the @var{_FillValue} may change from file to file in a
multi-file operator, e.g., @command{ncra}.
@acronym{NCO} is written to account for this (it always compares a
variable to the @var{_FillValue} assigned to that variable in the
current file).
Suffice it to say that, in all known cases, @acronym{NCO} does ``the
right thing''.
It is impossible to determine and store the correct result of a binary
operation in a single variable.
One such corner case occurs when both operands have differing
@var{_FillValue} attributes, i.e., attributes with different
numerical values.
Since the output (result) of the operation can only have one
@var{_FillValue}, some information may be lost.
In this case, @acronym{NCO} always defines the output variable to have
the same @var{_FillValue} as the first input variable.
Prior to performing the arithmetic operation, all values of the second
operand equal to the second @var{_FillValue} are replaced with the
first @var{_FillValue}.
Then the arithmetic operation proceeds as normal, comparing each element
of each operand to a single @var{_FillValue}.
Comparing each element to two distinct @var{_FillValue}'s would be
much slower and would be no likelier to yield a more satisfactory
answer.
In practice, judicious choice of @var{_FillValue} values prevents any
important information from being lost.
@end enumerate
@html
<a name="dfl_lvl"></a> <!-- http://nco.sf.net/nco.html#dfl_lvl -->
<a name="dfl"></a> <!-- http://nco.sf.net/nco.html#dfl -->
<a name="lz"></a> <!-- http://nco.sf.net/nco.html#lz -->
<a name="lz77"></a> <!-- http://nco.sf.net/nco.html#lz77 -->
<a name="deflate"></a> <!-- http://nco.sf.net/nco.html#deflate -->
<a name="deflation"></a> <!-- http://nco.sf.net/nco.html#deflation -->
@end html
@node Deflation, Packed data, Missing Values, Common features
@section Deflation
@cindex @code{-L}
@cindex @code{--deflate}
@cindex @code{--dfl_lvl}
@cindex Lempel-Ziv deflation
@cindex compression
@cindex deflation
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncecat}, @command{ncflint}, @command{ncks}, @command{ncpdq},
@command{ncra}, @command{ncrcat}, @command{ncwa}@*
Short options: @samp{-L}@*
Long options: @samp{--dfl_lvl}, @samp{--deflate}@*
@end cartouche
All @acronym{NCO} operators that define variables support
the netCDF4 feature of storing variables compressed with Lempel-Ziv
deflation.
Activate this deflation with the @code{-L @var{dfl_lvl}} short option
(or with the same argument to the @samp{--dfl_lvl} or @samp{--deflate}
long options).
Deflation uses lossless techniques to compress data.
Specify the deflation level @var{dfl_lvl} on a scale from
no deflation (@var{dfl_lvl = 0}) to maximum deflation
(@var{dfl_lvl = 9}).
Higher deflation levels require more time for compression.
File sizes resulting from minimal (@var{dfl_lvl = 1}) and maximal
(@var{dfl_lvl = 9}) deflation levels typically differ by a few
percent.
To compress an entire file using deflation, use
@example
ncks -4 -L 0 in.nc out.nc # No deflation (fast, no time penalty)
ncks -4 -L 1 in.nc out.nc # Minimal deflation (little time penalty)
ncks -4 -L 9 in.nc out.nc # Maximal deflation (much slower)
@end example
Unscientific testing shows that deflation compresses typical climate
datasets by 30-60%.
Packing, a lossy compression technique available for all netCDF files
(see @ref{Packed data}), can easily compress files by 50%.
Packed data may be deflated to squeeze datasets by about 80%.
@example
ncks -4 -L 1 in.nc out.nc # Minimal deflation (~30-60% compression)
ncks -4 -L 9 in.nc out.nc # Maximal deflation (~31-63% compression)
ncpdq in.nc out.nc # Standard packing (~50% compression)
ncpdq -4 -L 9 in.nc out.nc # Deflated packing (~80% compression)
@end example
@findex ncks
@command{ncks} prints deflation parameters, if any, to screen
(@pxref{ncks netCDF Kitchen Sink}).
@html
<a name="pck"></a> <!-- http://nco.sf.net/nco.html#pck -->
<a name="pack"></a> <!-- http://nco.sf.net/nco.html#pack -->
@end html
@node Packed data, Operation Types, Deflation, Common features
@section Packed data
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
@cindex @code{missing_value}
@cindex @code{_FillValue}
@cindex @command{pack(x)}
@cindex @command{unpack(x)}
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncflint}, @command{ncpdq}, @command{ncra}, @command{ncwa}@*
Short options: None@*
@end cartouche
The phrase @dfn{packed data} refers to data which are stored in the
standard netCDF3 packing format which employs a lossy algorithm.
See @ref{ncks netCDF Kitchen Sink} for a description of deflation, a
lossless compression technique available with netCDF4 only.
Packed data may be deflated to save additional space.
@unnumberedsubsec Packing Algorithm
@dfn{Packing}
The standard netCDF packing algorithm is lossy, and produces data with
the same dynamic range as the original but which requires no more than
half the space to store.
The packed variable is stored (usually) as type @code{NC_SHORT}
with the two attributes required to unpack the variable,
@code{scale_factor} and @code{add_offset}, stored at the original
(unpacked) precision of the variable
@footnote{Although not a part of the standard, @acronym{NCO} enforces
the policy that the @code{_FillValue} attribute, if any, of a packed
variable is also stored at the original precision.}.
Let @var{min} and @var{max} be the minimum and maximum values
@w{of @var{x}.}
@tex
$$
\rm
\eqalign{\hbox{scale\_factor} &= (\hbox{max}-\hbox{min})/\hbox{ndrv}\cr
\hbox{add\_offset} &= (\hbox{min}+\hbox{max})/2\cr
\hbox{pck} &= (\hbox{upk}-\hbox{add\_offset})/\hbox{scale\_factor}\cr
&= {\hbox{ndrv}\times[\hbox{upk}-(\hbox{min}+\hbox{max})/2]\over\hbox{max}-\hbox{min}}\cr}
$$
@end tex
@ifnottex
@sp 1
@var{scale_factor} = (@var{max}-@var{min})/@var{ndrv}@*
@var{add_offset} = 0.5*(@var{min}+@var{max})@*
@var{pck} = (@var{upk}-@var{add_offset})/@var{scale_factor} = (@var{upk}-0.5*(@var{min}+@var{max}))*@var{ndrv}/(@var{max}-@var{min})@*
@sp 1
@end ifnottex
where @var{ndrv} is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for @var{ndrv} is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type @code{NC_SHORT}, a two-byte
datatype, then there are at most @math{2^16 = 65536} distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for @code{NC_SHORT}, @math{ndrv = 65536 - 1 = 65535}.
Less often, the variable may be packed into type @code{NC_CHAR},
where @math{ndrv = 256 - 1 = 255}, or type @code{NC_INT} where
where @math{ndrv = 4294967295 - 1 = 4294967294}.
One useful feature of (lossy) netCDF packing algorithm is that
additional, loss-less packing algorithms perform well on top of it.
@unnumberedsubsec Unpacking Algorithm
@dfn{Unpacking}
The unpacking algorithm depends on the presence of two attributes,
@code{scale_factor} and @code{add_offset}.
If @code{scale_factor} is present for a variable, the data are
multiplied by the value @var{scale_factor} after the data are read.
If @code{add_offset} is present for a variable, then the
@var{add_offset} value is added to the data after the data are read.
If both @code{scale_factor} and @code{add_offset} attributes are
present, the data are first scaled by @var{scale_factor} before the
offset @var{add_offset} is added.
@tex
$$
\rm
\eqalign{\hbox{upk} &= \hbox{scale\_factor}\times\hbox{pck} + \hbox{add\_offset}\cr
&= {\hbox{pck}\times(\hbox{max}-\hbox{min})\over\hbox{ndrv}} + {\hbox{min}+\hbox{max}\over2}\cr}
$$
@end tex
@ifnottex
@sp 1
@var{upk} = @var{scale_factor}*@var{pck} + @var{add_offset} = (@var{max}-@var{min})*@var{pck}/@var{ndrv} + 0.5*(@var{min}+@var{max})@*
@sp 1
@end ifnottex
When @code{scale_factor} and @code{add_offset} are used for packing, the
associated variable (containing the packed data) is typically of type
@code{byte} or @code{short}, whereas the unpacked values are intended to
be of type @code{int}, @code{float}, or @code{double}.
An attribute's @code{scale_factor} and @code{add_offset} and
@code{_FillValue}, if any, should all be of the type intended for the
unpacked data, i.e., @code{int}, @code{float} or @code{double}.
@unnumberedsubsec Default Handling of Packed Data
All @acronym{NCO} arithmetic operators understand packed data.
The operators automatically unpack any packed variable in the input
file which will be arithmetically processed.
For example, @command{ncra} unpacks all record variables,
and @command{ncwa} unpacks all variable which contain a dimension to
be averaged.
These variables are stored unpacked in the output file.
On the other hand, arithmetic operators do not unpack non-processed
variables.
For example, @command{ncra} leaves all non-record variables packed,
and @command{ncwa} leaves packed all variables lacking an averaged
dimension.
These variables (called fixed variables) are passed unaltered from the
input to the output file.
Hence fixed variables which are packed in input files remain packed in
output files.
Completely packing and unpacking files is easily accomplished with
@command{ncpdq} (@pxref{ncpdq netCDF Permute Dimensions Quickly}).
Packing and unpacking individual variables may be done with
@command{ncpdq} and the @command{ncap2}
@command{pack()} and @command{unpack()} functions
(@pxref{Intrinsic functions}).
@html
<a name="op_typ"></a> <!-- http://nco.sf.net/nco.html#op_typ -->
@end html
@node Operation Types, Type Conversion, Packed data, Common features
@section Operation Types
@cindex operation types
@cindex @code{avg}
@cindex @code{sqravg}
@cindex @code{avgsqr}
@cindex @code{min}
@cindex @code{max}
@cindex @code{rmssdn}
@cindex @code{rms}
@cindex @code{ttl}
@cindex @code{sqrt}
@cindex average
@cindex mean
@cindex total
@cindex minimum
@cindex maximum
@cindex root-mean-square
@cindex standard deviation
@cindex variance
@cindex @code{-y @var{op_typ}}
@cindex @code{--operation @var{op_typ}}
@cindex @code{--op_typ @var{op_typ}}
@cartouche
Availability: @command{ncap2}, @command{ncra}, @command{ncea}, @command{ncwa}@*
Short options: @samp{-y}@*
Long options: @samp{--operation}, @samp{--op_typ}@*
@end cartouche
@noindent
The @samp{-y @var{op_typ}} switch allows specification of many different
types of operations
Set @var{op_typ} to the abbreviated key for the corresponding operation:
@table @code
@item avg
Mean value (default)
@item sqravg
Square of the mean
@item avgsqr
Mean of sum of squares
@item max
Maximium value
@item min
Minimium value
@item rms
Root-mean-square (normalized by @var{N})
@item rmssdn
Root-mean square (normalized by @var{N-1})
@item sqrt
Square root of the mean
@item ttl
Sum of values
@end table
@noindent
@cindex coordinate variable
@acronym{NCO} assumes coordinate variables represent grid axes, e.g.,
longitude.
The only rank-reduction which makes sense for coordinate variables
is averaging.
Hence @acronym{NCO} implements the operation type requested with
@samp{-y} on all non-coordinate variables, but not on coorniate
variables.
When an operation requires a coordinate variable to be reduced in
rank, i.e., from one dimension to a scalar or from one dimension to
a degenerate (single value) array, then @acronym{NCO}
@emph{always averages} the coordinate variable regardless of the
arithmetic operation type performed on the non-coordinate variables.
The mathematical definition of each arithmetic operation is given below.
@xref{ncwa netCDF Weighted Averager}, for additional information on
masks and normalization.
If an operation type is not specified with @samp{-y} then the operator
performs an arithmetic average by default.
Averaging is described first so the terminology for the other operations
is familiar.
@ifhtml
@cartouche
@html
<p><b>Note for HTML users</b>:
<br>The definition of mathematical operations involving rank reduction
(e.g., averaging) relies heavily on mathematical expressions which
cannot easily be represented in HTML.
<b>See the <a href="./nco.pdf">printed manual</a> for much more detailed
and complete documentation of this subject.</b>
@end html
@end cartouche
@end ifhtml
@ifnothtml
@ifinfo
@emph{Note for Info users}:
The definition of mathematical operations involving rank reduction
(e.g., averaging) relies heavily on mathematical expressions which
cannot be easily represented in Info.
@emph{See the @uref{./nco.pdf, printed manual} for much more detailed and
complete documentation of this subject.}
@end ifinfo
@end ifnothtml
@tex
The masked, weighted average of a variable $\xxx$ can be generally
represented as
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}
$$
where $\bar \xxx_{\jjj}$ is the $\jjj$'th element of the output
hyperslab, $\xxx_{\idx}$ is the $\idx$'th element of the input
hyperslab, $\mssflg_{\idx}$ is~1 unless $\xxx_{\idx}$ equals the missing
value, $\mskflg_{\idx}$ is~1 unless $\xxx_{\idx}$ is masked, and
$\wgt_{\idx}$ is the weight.
This formiddable looking formula represents a simple weighted average
whose bells and whistles are all explained below.
It is not too early to note, however, that when
$\mssflg_{\idx} = \mskflg_{\idx} = \wgt_{\idx} = 1$, the
generic averaging expression above reduces to a simple arithmetic
average.
Furthermore, $\mskflg_{\idx} = \wgt_{\idx} = 1$ for all operators
except @command{ncwa}.
These variables are included in the discussion below for completeness,
and for possible future use in other operators.
The size $\outnbr$ of the output hyperslab for a given variable is the
product of all the dimensions of the input variable which are not
averaged over.
The size $\lmnnbr$ of the input hyperslab contributing to each
$\bar \xxx_{\jjj}$ is simply the product of the sizes of all dimensions
which are averaged over (i.e., dimensions specified with @samp{-a}).
Thus $\lmnnbr$ is the number of input elements which @emph{potentially}
contribute to each output element.
An input element $\xxx_{\idx}$ contributes to the output element
$\xxx_{\jjj}$ except in two conditions:
@cindex missing values
@enumerate
@item $\xxx_{\idx}$ equals the @var{missing value}
(@pxref{Missing Values}) for the variable.
@item $\xxx_{\idx}$ is located at a point where the mask condition
(@pxref{Mask condition}) is false.
@end enumerate
Points $\xxx_{\idx}$ in either of these two categories do not contribute
to $\xxx_{\jjj}$---they are ignored.
We now define these criteria more rigorously.
Each~$\xxx_{\idx}$ has an associated Boolean weight~$\mssflg_{\idx}$
whose value is~0 or~1 (false or true).
The value of~$\mssflg_{\idx}$ is~1 (true) unless $\xxx_{\idx}$ equals
the @var{missing value} (@pxref{Missing Values}) for the variable.
Thus, for a variable with no @code{_FillValue} attribute,
$\mssflg_{\idx}$~is always~1.
All @acronym{NCO} arithmetic operators (@command{ncbo},
@command{ncra}, @command{ncea}, @command{ncflint}, @command{ncwa}) treat
missing values analogously.
Besides (weighted) averaging, @command{ncwa}, @command{ncra}, and
@command{ncea} also compute some common non-linear operations which may
be specified with the @samp{-y} switch (@pxref{Operation Types}).
The other rank-reducing operations are simple variations of the generic
weighted mean described above.
The total value of~$\xxx$ (@code{-y ttl}) is
$$
\bar \xxx_{\jjj} = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx}
$$
@cindex @code{-N}
@cindex @code{numerator}
Note that the total is the same as the numerator of the mean
of~$\xxx$, and may also be obtained in @command{ncwa} by using the
@samp{-N} switch (@pxref{ncwa netCDF Weighted Averager}).
The minimum value of~$\xxx$ (@code{-y min}) is
$$
\bar \xxx_{\jjj} = \min [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr}
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ]
$$
Analogously, the maximum value of~$\xxx$ (@code{-y max}) is
$$
\bar \xxx_{\jjj} = \max [ \mssflg_{1} \mskflg_{1} \wgt_{1} \xxx_{1},
\mssflg_{2} \mskflg_{2} \wgt_{2} \xxx_{2}, \ldots, \mssflg_{\lmnnbr}
\mskflg_{\lmnnbr} \wgt_{\lmnnbr} \xxx_{\lmnnbr} ]
$$
Thus the minima and maxima are determined after any weights are applied.
The square of the mean value of~$\xxx$ (@code{-y sqravg}) is
$$
\bar \xxx_{\jjj} = \left( {\sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}
\right)^{2}
$$
The mean of the sum of squares of~$\xxx$ (@code{-y avgsqr}) is
$$
\bar \xxx_{\jjj} = {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}
$$
If $\xxx$ represents a deviation from the mean of another variable,
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$ (possibly created by
@command{ncbo} in a previous step), then applying @code{avgsqr} to
$\xxx$ computes the approximate variance of $\yyy$.
Computing the true variance of~$\yyy$ requires subtracting~1 from the
denominator, discussed below.
For a large sample size however, the two results will be nearly
indistinguishable.
The root mean square of~$\xxx$ (@code{-y rms}) is
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} x^{2}_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}}
$$
Thus @code{rms} simply computes the squareroot of the quantity computed
by @code{avgsqr}.
The root mean square of~$\xxx$ with standard-deviation-like
normalization (@code{-y rmssdn}) is implemented as follows.
When weights are not specified, this function is the same as the root
mean square of~$\xxx$ except one is subtracted from the sum in the
denominator
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} x^{2}_{\idx} \over -1 + \sum_{\idx=1}^{\idx=\lmnnbr}
\mssflg_{\idx} \mskflg_{\idx}} }
$$
If $\xxx$ represents the deviation from the mean of another variable,
$\xxx_{\idx} = \yyy_{\idx} - \bar{\yyy}$, then applying @code{rmssdn} to
$\xxx$ computes the standard deviation of~$\yyy$.
In this case the $-1$ in the denominator compensates for the degree of
freedom already used in computing $\bar{\yyy}$ in the numerator.
Consult a statistics book for more details.
When weights are specified it is unclear how to compensate for this
extra degree of freedom.
Weighting the numerator and denominator of the above by $\wgt_{\idx}$
and subtracting one from the denominator is only appropriate when all
the weights are~1.0.
When the weights are arbitrary (e.g., Gaussian weights), subtracting one
from the sum in the denominator does not necessarily remove one degree
of freedom.
Therefore when @code{-y rmssdn} is requested and weights are specified,
@command{ncwa} actually implements the @code{rms} procedure.
@command{ncea} and @command{ncra}, which do not allow weights to be
specified, always implement the @code{rmssdn} procedure when asked.
The square root of the mean of~$\xxx$ (@code{-y sqrt}) is
$$
\bar \xxx_{\jjj} = \sqrt{ {\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx}
\mskflg_{\idx} \wgt_{\idx} \xxx_{\idx} \over
\sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx} \wgt_{\idx}}}
$$
@end tex
The definitions of some of these operations are not universally useful.
Mostly they were chosen to facilitate standard statistical
computations within the @acronym{NCO} framework.
We are open to redefining and or adding to the above.
If you are interested in having other statistical quantities
defined in @acronym{NCO} please contact the @acronym{NCO} project
(@pxref{Help Requests and Bug Reports}).
@noindent
EXAMPLES
@html
<a name="min"></a> <!-- http://nco.sf.net/nco.html#min -->
<a name="max"></a> <!-- http://nco.sf.net/nco.html#max -->
<a name="rms"></a> <!-- http://nco.sf.net/nco.html#rms -->
@end html
@noindent
Suppose you wish to examine the variable @code{prs_sfc(time,lat,lon)}
which contains a time series of the surface pressure as a function of
latitude and longitude.
Find the minimium value of @code{prs_sfc} over all dimensions:
@example
ncwa -y min -v prs_sfc in.nc foo.nc
@end example
@noindent
Find the maximum value of @code{prs_sfc} at each time interval for each
latitude:
@example
ncwa -y max -v prs_sfc -a lon in.nc foo.nc
@end example
@noindent
Find the root-mean-square value of the time-series of @code{prs_sfc} at
every gridpoint:
@example
ncra -y rms -v prs_sfc in.nc foo.nc
ncwa -y rms -v prs_sfc -a time in.nc foo.nc
@end example
@noindent
The previous two commands give the same answer but @command{ncra} is
preferred because it has a smaller memory footprint.
@cindex degenerate dimension
Also, by default, @command{ncra} leaves the (degenerate) @code{time}
dimension in the output file (which is usually useful) whereas
@command{ncwa} removes the @code{time} dimension (unless @samp{-b} is
given).
@noindent
These operations work as expected in multi-file operators.
Suppose that @code{prs_sfc} is stored in multiple timesteps per file
across multiple files, say @file{jan.nc}, @file{feb.nc},
@file{march.nc}.
We can now find the three month maximium surface pressure at every point.
@example
ncea -y max -v prs_sfc jan.nc feb.nc march.nc out.nc
@end example
@html
<a name="stddvn"></a> <!-- http://nco.sf.net/nco.html#stddvn -->
<a name="sdn"></a> <!-- http://nco.sf.net/nco.html#sdn -->
<a name="xmp_sdn"></a> <!-- http://nco.sf.net/nco.html#xmp_sdn -->
@end html
@noindent
It is possible to use a combination of these operations to compute
the variance and standard deviation of a field stored in a single file
or across multiple files.
The procedure to compute the temporal standard deviation of the surface
pressure at all points in a single file @file{in.nc} involves three
steps.
@example
ncwa -O -v prs_sfc -a time in.nc out.nc
ncbo -O -v prs_sfc in.nc out.nc out.nc
ncra -O -y rmssdn out.nc out.nc
@end example
First construct the temporal mean of @code{prs_sfc} in the file
@file{out.nc}.
Next overwrite @file{out.nc} with the anomaly (deviation from the mean).
Finally overwrite @file{out.nc} with the root-mean-square of itself.
Note the use of @samp{-y rmssdn} (rather than @samp{-y rms}) in the
final step.
This ensures the standard deviation is correctly normalized by one fewer
than the number of time samples.
The procedure to compute the variance is identical except for the use of
@samp{-y var} instead of @samp{-y rmssdn} in the final step.
@command{ncap2} can also compute statistics like standard deviations.
Brute-force implementation of formulae is one option, e.g.,
@example
ncap2 -s 'prs_sfc_sdn=sqrt((prs_sfc-prs_sfc.avg($time)^2).total($time))/($time.size-1)'
in.nc out.nc
@end example
The operation may, of course, be broken into multiple steps in order
to archive intermediate quantities, such as the time-anomalies
@example
ncap2 -s 'prs_sfc_anm=prs_sfc-prs_sfc.avg($time)' \
-s 'prs_sfc_sdn=sqrt((prs_sfc_anm^2).total($time))/($time.size-1)' \
in.nc out.nc
@end example
@command{ncap2} supports intrinsic standard deviation functions
(@pxref{Operation Types}) which simplify the above expression to
@example
ncap2 -s 'prs_sfc_sdn=(prs_sfc-prs_sfc.avg($time)).rmssdn($time)' in.nc out.nc
@end example
These instrinsic functions compute the answer quickly and concisely.
The procedure to compute the spatial standard deviation of a field
in a single file @file{in.nc} involves three steps.
@example
ncwa -O -v prs_sfc,gw -a lat,lon -w gw in.nc out.nc
ncbo -O -v prs_sfc,gw in.nc out.nc out.nc
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw out.nc out.nc
@end example
First the appropriately weighted (with @samp{-w gw}) spatial mean values
are written to the output file.
This example includes the use of a weighted variable specified with
@samp{-w gw}.
When using weights to compute standard deviations one must remember to
include the weights in the initial output files so that they may be used
again in the final step.
The initial output file is then overwritten with the gridpoint
deviations from the spatial mean.
Finally the root-mean-square of the appropriately weighted spatial
deviations is taken.
The @command{ncap2} solution to the spatially-weighted standard
deviation problem is
@example
ncap2 -s 'prs_sfc_sdn=(prs_sfc*gw-prs_sfc*gw.avg($lat,$lon)).rmssdn($lat,$lon)' \
in.nc out.nc
@end example
Be sure to multiply the variable by the weight prior to computing the
the anomalies and the standard deviation.
The procedure to compute the standard deviation of a time-series across
multiple files involves one extra step since all the input must first be
collected into one file.
@example
ncrcat -O -v tpt in.nc in.nc foo1.nc
ncwa -O -a time foo1.nc foo2.nc
ncbo -O -v tpt foo1.nc foo2.nc foo2.nc
ncra -O -y rmssdn foo2.nc out.nc
@end example
The first step assembles all the data into a single file.
This may require a lot of temporary disk space, but is more or less
required by the @command{ncbo} operation in the third step.
@html
<a name="typ_cnv"></a> <!-- http://nco.sf.net/nco.html#typ_cnv -->
@end html
@node Type Conversion, Batch Mode, Operation Types, Common features
@section Type Conversion
@cindex type conversion
@cartouche
Availability: @command{ncap2}, @command{ncbo}, @command{ncea},
@command{ncra}, @command{ncwa}@*
Short options: None@*
@end cartouche
@cindex promotion
@cindex demotion
@cindex automatic type conversion
@cindex manual type conversion
Type conversion (often called @dfn{promotion} or @dfn{demotion}) refers
to the casting of one fundamental data type to another, e.g., converting
@code{NC_SHORT} (two bytes) to @code{NC_DOUBLE} (eight bytes).
Type conversion is automatic when the language carries out this
promotion according to an internal set of rules without explicit user
intervention.
In contrast, manual type conversion refers to explicit user commands to
change the type of a variable or attribute.
Most type conversion happens automatically, yet there are situations in
which manual type conversion is advantageous.
@menu
* Automatic type conversion::
* Manual type conversion::
@end menu
@node Automatic type conversion, Manual type conversion, Type Conversion, Type Conversion
@subsection Automatic type conversion
As a general rule, automatic type conversions should be avoided for at
least two reasons.
First, type conversions are expensive since they require creating
(temporary) buffers and casting each element of a variable from
the type it was stored at to some other type.
Second, the dataset's creator probably had a good reason
for storing data as, say, @code{NC_FLOAT} rather than @code{NC_DOUBLE}.
In a scientific framework there is no reason to store data with more
precision than the observations were made.
Thus @acronym{NCO} tries to avoid performing automatic type conversions
when performing arithmetic.
@cindex C language
@cindex Fortran
Automatic type conversion during arithmetic in the @w{languages C} and
Fortran is performed only when necessary.
All operands in an operation are converted to the most precise type
before the operation takes place.
However, following this parsimonious conversion rule dogmatically
results in numerous headaches.
For example, the average of the two @code{NC_SHORT}s @code{17000s} and
@code{17000s} results in garbage since the intermediate value which
holds their sum is also of type @code{NC_SHORT} and thus cannot
represent values greater than 32,767
@footnote{
@set flg
@tex
$32767 = 2^{15}-1$
@clear flg
@end tex
@ifinfo
@math{32767 = 2^15-1}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
32767 = 2^15@minus{}1
@clear flg
@end ifset
}.
There are valid reasons for expecting this operation to succeed and
the @acronym{NCO} philosophy is to make operators do what you want, not
what is most pure.
Thus, unlike C and Fortran, but like many other higher level interpreted
languages, @acronym{NCO} arithmetic operators will perform automatic type
conversion when all the following conditions are met
@footnote{Operators began performing type conversions before arithmetic
in @acronym{NCO} @w{version 1.2}, August, 2000.
Previous versions never performed unnecessary type conversion for
arithmetic.}:
@enumerate
@item The operator is @command{ncea}, @command{ncra}, or @command{ncwa}.
@command{ncbo} is not yet included in this list because subtraction did
not benefit from type conversion.
This will change in the future
@c fxm TODO #265
@item The arithmetic operation could benefit from type conversion.
Operations that could benefit (e.g., from larger representable sums)
include averaging, summation, or any "hard" arithmetic.
Type conversion does not benefit searching for minima and maxima
(@samp{-y min}, or @samp{-y max}).
@item The variable on disk is of type @code{NC_BYTE}, @code{NC_CHAR},
@code{NC_SHORT}, or @code{NC_INT}.
Type @code{NC_DOUBLE} is not type converted because there is no type of
higher precision to convert to.
Type @code{NC_FLOAT} is not type converted because, in our judgement,
the performance penalty of always doing so would outweigh the (extremely
rare) potential benefits.
@end enumerate
When these criteria are all met, the operator promotes the variable in
question to type @code{NC_DOUBLE}, performs all the arithmetic
operations, casts the @code{NC_DOUBLE} type back to the original type,
and finally writes the result to disk.
The result written to disk may not be what you expect, because of
incommensurate ranges represented by different types, and because of
(lack of) rounding.
First, continuing the above example, the average (e.g., @samp{-y avg})
of @code{17000s} and @code{17000s} is written to disk as @code{17000s}.
The type conversion feature of @acronym{NCO} makes this possible since
the arithmetic and intermediate values are stored as @code{NC_DOUBLE}s,
i.e., @code{34000.0d} and only the final result must be represented
as an @code{NC_SHORT}.
Without the type conversion feature of @acronym{NCO}, the average would
have been garbage (albeit predictable garbage near @code{-15768s}).
Similarly, the total (e.g., @samp{-y ttl}) of @code{17000s} and
@code{17000s} written to disk is garbage (actually @code{-31536s}) since
the final result (the true total) of @math{34000} is outside the range
of type @code{NC_SHORT}.
@cindex @code{floor}
Type conversions use the @code{floor} function to convert floating point
number to integers.
Type conversions do not attempt to round floating point numbers to the
nearest integer.
Thus the average of @code{1s} and @code{2s} is computed in double
precisions arithmetic as
@math{(@code{1.0d} + @code{1.5d})/2) = @code{1.5d}}.
This result is converted to @code{NC_SHORT} and stored on disk as
@math{@code{floor(1.5d)} = @code{1s}}
@footnote{
@cindex C language
The actual type conversions are handled by intrinsic C-language type
conversion, so the @code{floor()} function is not explicitly called,
though the results would be the same if it were.}.
Thus no "rounding up" is performed.
The type conversion rules @w{of C} can be stated as follows:
If @var{n} is an integer then any floating point value @var{x}
satisfying
@set flg
@tex
$n \le x < n+1$
@clear flg
@end tex
@ifinfo
@math{n <= x < n+1}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{n} <= @var{x} < @var{n+1}
@clear flg
@end ifset
will have the value @var{n} when converted to an integer.
@node Manual type conversion, , Automatic type conversion, Type Conversion
@subsection Manual type conversion
@cindex @command{ncap2}
@command{ncap2} provides intrinsic functions for performing manual type
conversions.
This, for example, converts variable @code{tpt} to external type
@code{NC_SHORT} (a C-type @code{short}), and variable @code{prs} to
external type @code{NC_DOUBLE} (a C-type @code{double}).
@example
ncap2 -s 'tpt=short(tpt);prs=double(prs)' in.nc out.nc
@end example
@xref{ncap2 netCDF Arithmetic Processor}, for more details.
@html
<a name="ovr"></a> <!-- http://nco.sf.net/nco.html#ovr -->
<a name="-O"></a> <!-- http://nco.sf.net/nco.html#-O -->
@end html
@node Batch Mode, History Attribute, Type Conversion, Common features
@section Batch Mode
@cindex batch mode
@cindex overwriting files
@cindex appending to files
@cindex force overwrite
@cindex force append
@cindex @code{-O}
@cindex @code{-A}
@cindex @code{--overwrite}
@cindex @code{--ovr}
@cindex @code{--apn}
@cindex @code{--append}
@cartouche
Availability: All operators@*
Short options: @samp{-O}, @samp{-A}@*
Long options: @samp{--ovr}, @samp{--overwrite}, @samp{--apn}, @samp{--append}@*
@end cartouche
If the @var{output-file} specified for a command is a pre-existing file,
then the operator will prompt the user whether to overwrite (erase) the
existing @var{output-file}, attempt to append to it, or abort the
operation.
However, interactive questions reduce productivity when processing large
amounts of data.
Therefore @acronym{NCO} also implements two ways to override its own safety
features, the @samp{-O} and @samp{-A} switches.
Specifying @samp{-O} tells the operator to overwrite any existing
@var{output-file} without prompting the user interactively.
Specifying @samp{-A} tells the operator to attempt to append to any
existing @var{output-file} without prompting the user interactively.
These switches are useful in batch environments because they suppress
interactive keyboard input.
@html
<a name="hst"></a> <!-- http://nco.sf.net/nco.html#hst -->
<a name="history"></a> <!-- http://nco.sf.net/nco.html#history -->
<a name="-h"></a> <!-- http://nco.sf.net/nco.html#-h -->
@end html
@node History Attribute, File List Attributes, Batch Mode, Common features
@section History Attribute
@cindex @code{history}
@cindex timestamp
@cindex global attributes
@cindex attributes, global
@cindex @code{-h}
@cindex @code{--hst}
@cindex @code{--history}
@cartouche
Availability: All operators@*
Short options: @samp{-h}@*
Long options: @samp{--hst}, @samp{--history}@*
@end cartouche
All operators automatically append a @code{history} global attribute to
any file they create or modify.
The @code{history} attribute consists of a timestamp and the full string
of the invocation command to the operator, e.g., @samp{Mon May 26 20:10:24
1997: ncks in.nc foo.nc}.
The full contents of an existing @code{history} attribute are copied
from the first @var{input-file} to the @var{output-file}.
The timestamps appear in reverse chronological order, with the most
recent timestamp appearing first in the @code{history} attribute.
Since @acronym{NCO} and many other netCDF operators adhere to the
@code{history} convention, the entire data processing path of a given
netCDF file may often be deduced from examination of its @code{history}
attribute.
As of May, 2002, @acronym{NCO} is case-insensitive to the spelling
of the @code{history} attribute name.
Thus attributes named @code{History} or @code{HISTORY} (which are
non-standard and not recommended) will be treated as valid history
attributes.
When more than one global attribute fits the case-insensitive search
for "history", the first one found will be used.
@code{history} attribute
@cindex @command{ncatted}
To avoid information overkill, all operators have an optional switch
(@samp{-h}, @samp{--hst}, or @samp{--history}) to override
automatically appending the @code{history} attribute
(@pxref{ncatted netCDF Attribute Editor}).
Note that the @samp{-h} switch also turns off writing the
@code{nco_input_file_list} attribute for multi-file operators
(@pxref{File List Attributes}).
@html
<a name="fl_lst_in_att"></a> <!-- http://nco.sf.net/nco.html#fl_lst_in_att -->
@end html
@node File List Attributes, CF Conventions, History Attribute, Common features
@section File List Attributes
@cindex @code{nco_input_file_list}
@cindex @code{nco_input_file_number}
@cindex @code{stdin}
@cindex global attributes
@cindex attributes, global
@cindex @code{-H}
@cindex @code{--fl_lst_in}
@cindex @code{--file_list}
@cartouche
Availability: @command{ncea}, @command{ncecat}, @command{ncra}, @command{ncrcat}@*
Short options: @samp{-H}@*
Long options: @samp{--fl_lst_in}, @samp{--file_list}@*
@end cartouche
Many methods of specifying large numbers of input file names pass
these names via pipes, encodings, or argument transfer programs
(@pxref{Large Numbers of Files}).
When these methods are used, the input file list is not explicitly
passed on the command line.
This results in a loss of information since the @code{history}
attribute no longer contains the exact command by which the file
was created.
@acronym{NCO} solves this dilemma by archiving input file list
attributes.
When the input file list to a multi-file operator is specified
via @code{stdin}, the operator, by default, attaches two global
attributes to any file they create or modify.
The @code{nco_input_file_number} global attribute contains the number of
input files, and @code{nco_input_file_list} contains the file names,
specified as standard input to the multi-file operator.
This information helps to verify that all input files the user thinks
were piped through @code{stdin} actually arrived.
Without the @code{nco_input_file_list} attribute, the information is lost
forever and the ``chain of evidence'' would be broken.
The @samp{-H} switch overrides (turns off) the default behavior of
writing the input file list global attributes when input is from
@code{stdin}.
The @samp{-h} switch does this too, and turns off the @code{history}
attribute as well (@pxref{History Attribute}).
Hence both switches allows space-conscious users to avoid storing what
may amount to many thousands of filenames in a metadata attribute.
@html
<a name="CF"></a> <!-- http://nco.sf.net/nco.html#CF -->
<a name="cnv_CF"></a> <!-- http://nco.sf.net/nco.html#cnv_CF -->
<a name="cnv_CCSM"></a> <!-- http://nco.sf.net/nco.html#cnv_CCSM -->
@end html
@node CF Conventions, ARM Conventions, File List Attributes, Common features
@section @acronym{CF} Conventions
@cindex @acronym{CF} conventions
@cindex @acronym{CCSM} conventions
@cindex UDUnits
@cindex @code{ORO}
@cindex @code{area}
@cindex @code{datesec}
@cindex @code{date}
@cindex @code{gw}
@cindex @code{hyai}
@cindex @code{hyam}
@cindex @code{hybi}
@cindex @code{hybm}
@cindex @code{lat_bnds}
@cindex @code{lon_bnds}
@cindex @code{msk_*}
@cartouche
Availability: @command{ncbo}, @command{ncea}, @command{ncecat},
@command{ncflint}, @command{ncra}, @command{ncwa}@*
Short options: None@*
@end cartouche
@acronym{NCO} recognizes the Climate and Forecast (@acronym{CF})
metadata conventions, and treats such data (often called history tapes),
specially.
@acronym{NCO} handles older @acronym{NCAR} model datasets, such as
@acronym{CCM} and early @acronym{CCSM} datasets, with its @acronym{CF}
rules even though the earlier data may not contain an explicit
@code{Conventions} attribute (e.g., @samp{CF-1.0}).
We refer to all such data collectively as @acronym{CF} data.
Skip this section if you never work with @acronym{CF} data.
The @acronym{CF} netCDF conventions are described at
@uref{http://www.cgd.ucar.edu/cms/eaton/cf-metadata/CF-1.0.html}.
Most @acronym{CF} netCDF conventions are transparent to @acronym{NCO}
@footnote{
The exception is appending/altering the attributes @code{x_op},
@code{y_op}, @code{z_op}, and @code{t_op} for variables which have been
averaged across space and time dimensions.
This feature is scheduled for future inclusion in @acronym{NCO}.
}.
There are no known pitfalls associated with using any @acronym{NCO}
operator on files adhering to these conventions
@footnote{
The @acronym{CF} conventions recommend @code{time} be stored in the
format @var{time} since @var{base_time}, e.g., the @code{units}
attribute of @code{time} might be
@samp{days since 1992-10-8 15:15:42.5 -6:00}.
@w{A problem} with this format occurs when using @command{ncrcat} to
concatenate multiple files together, each with a different
@var{base_time}.
That is, any @code{time} values from files following the first file to
be concatenated should be corrected to the @var{base_time} offset
specified in the @code{units} attribute of @code{time} from the first
file.
The analogous problem has been fixed in @acronym{ARM} files
(@pxref{ARM Conventions}) and could be fixed for @acronym{CF} files if
there is sufficient lobbying.
}.
However, to facilitate maximum user friendliness, @acronym{NCO} does
treat certain variables in some @acronym{CF} files specially.
The special functions are not required by the @acronym{CF} netCDF
conventions, but experience shows they simplify data analysis.
Currently, @acronym{NCO} determines whether a datafile is a
@acronym{CF} output datafile simply by checking whether value of the
global attribute @code{Conventions} (if it exists) equals
@samp{CF-1.0} or @samp{NCAR-CSM}.
Should @code{Conventions} equal either of these in the (first)
@var{input-file}, @acronym{NCO} will attempt to treat certain variables
specially, because of their meaning in @acronym{CF} files.
@acronym{NCO} will not average the following variables often found in
@acronym{CF} files:
@code{ntrm}, @code{ntrn}, @code{ntrk}, @code{ndbase}, @code{nsbase},
@code{nbdate}, @code{nbsec}, @code{mdt}, @code{mhisf}.
These variables contain scalar metadata such as the resolution of the
host geophysical model and it makes no sense to change their values.
@cindex non-coordinate grid properties
Furthermore, the @command{ncbo} operator does not operate on (i.e., add,
subtract, etc.) the following variables:
@code{ORO},
@code{area},
@code{datesec},
@code{date},
@code{gw},
@code{hyai},
@code{hyam},
@code{hybi}.
@code{hybm},
@code{lat_bnds},
@code{lon_bnds},
@code{msk_*}.
These variables represent the Gaussian weights, the orography field,
time fields, hybrid pressure coefficients, and latititude/longitude
boundaries.
We call these fields non-coordinate @dfn{grid properties}.
Coordinate grid properties are easy to identify because they are
coordinate variables such as @code{latitude} and @code{longitude}.
Users usually want @emph{all} grid properties to remain unaltered in the
output file.
To be treated as a grid property, the variable name must @emph{exactly}
match a name in the above list, or be a coordinate variable.
The handling of @code{msk_*} is exceptional in that @emph{any} variable
name beginning with the string @code{msk_} is considered to be a
``mask'' and is thus preserved (not operated on arithmetically).
You must spoof @acronym{NCO} if you would like any grid properties
or other special @acronym{CF} fields processed normally.
For example rename the variables first with @command{ncrename},
or alter the @code{Conventions} attribute.
@html
<a name="cnv_CF_crd"></a> <!-- http://nco.sf.net/nco.html#cnv_CF_crd -->
@end html
@cindex @code{coordinates}
@cindex coordinates convention
@cindex coordinate variable
@cindex auxiliary coordinates
@cindex subsetting
@cindex @code{-C}
@cindex @code{-c}
@cindex @code{--no-coords}
@cindex @code{--no-crd}
@cindex @code{--coords}
@cindex @code{--crd}
NCO supports the @acronym{CF} @code{coordinates} convention described
@uref{http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.1/cf-conventions.html#coordinate-system, here}.
This convention allows variables to specify additional coordinates
(including multidimensional coordinates) in a space-separated string
attribute named @code{coordinates}.
NCO attaches any such coordinates to the extraction list along with
variable and its usual (one-dimensional) coordinates, if any.
These auxiliary coordinates are subject to the user-specified overrides
described in @ref{Subsetting Coordinate Variables}.
@html
<a name="cnv_ARM"></a> <!-- http://nco.sf.net/nco.html#cnv_ARM -->
@end html
@node ARM Conventions, Operator Version, CF Conventions, Common features
@section @acronym{ARM} Conventions
@cindex @acronym{ARM} conventions
@cindex @code{time_offset}
@cindex @code{base_time}
@cindex @code{time}
@cartouche
Availability: @command{ncrcat}@*
Short options: None@*
@end cartouche
@command{ncrcat} has been programmed to correctly handle data files
which utilize the Atmospheric Radiation Measurement (@acronym{ARM})
Program @uref{http://www.arm.gov/data/time.stm,convention} for
time and time offsets.
If you do not work with @acronym{ARM} data then you may skip this
section.
@acronym{ARM} data files store time information in two variables, a
scalar, @code{base_time}, and a record variable, @code{time_offset}.
Subtle but serious problems can arise when these type of files are
just blindly concatenated.
Therefore @command{ncrcat} has been specially programmed to be able to
chain together consecutive @acronym{ARM} @var{input-files} and produce
and an @var{output-file} which contains the correct time information.
Currently, @command{ncrcat} determines whether a datafile is an
@acronym{ARM} datafile simply by testing for the existence of the
variables @code{base_time}, @code{time_offset}, and the dimension
@code{time}.
If these are found in the @var{input-file} then @command{ncrcat} will
automatically perform two non-standard, but hopefully useful,
procedures.
First, @command{ncrcat} will ensure that values of @code{time_offset}
appearing in the @var{output-file} are relative to the @code{base_time}
appearing in the first @var{input-file} (and presumably, though not
necessarily, also appearing in the @var{output-file}).
Second, if a coordinate variable named @code{time} is not found in the
@var{input-files}, then @command{ncrcat} automatically creates the
@code{time} coordinate in the @var{output-file}.
The values of @code{time} are defined by the @acronym{ARM} conventions
@math{@var{time} = @var{base_time} + @var{time_offset}}.
Thus, if @var{output-file} contains the @code{time_offset}
variable, it will also contain the @code{time} coordinate.
@cindex @code{history}
@cindex global attributes
@cindex attributes, global
@w{A short} message is added to the @code{history} global attribute
whenever these @acronym{ARM}-specific procedures are executed.
@html
<a name="vrs"></a> <!-- http://nco.sf.net/nco.html#vrs -->
<a name="version"></a> <!-- http://nco.sf.net/nco.html#version -->
@end html
@node Operator Version, , ARM Conventions, Common features
@section Operator Version
@cindex version
@cindex @acronym{RCS}
@cindex @code{-r}
@cindex @code{--revision}
@cindex @code{--version}
@cindex @code{--vrs}
@cartouche
Availability: All operators@*
Short options: @samp{-r}@*
Long options: @samp{--revision}, @samp{--version}, or @samp{--vrs}@*
@end cartouche
All operators can be told to print their internal version number and
copyright notice and then quit with the @samp{-r} switch.
The internal version number varies between operators, and indicates the
most recent change to a particular operator's source code.
This is useful in making sure you are working with the most recent
operators.
The version of @acronym{NCO} you are using might be, e.g., @code{1.2}.
However using @samp{-r} on, say, @command{ncks}, will produce something
like @samp{NCO netCDF Operators version 1.2
Copyright (C) 1995--2000 Charlie Zender
ncks version 1.30 (2000/07/31) "Bolivia"}.
This tells you @command{ncks} contains all patches up to version
@code{1.30}, which dates from @w{July 31}, 2000.
@html
<a name="rfr"></a> <!-- http://nco.sf.net/nco.html#rfr -->
@end html
@node Operator Reference Manual, Contributing, Common features, Top
@chapter Operator Reference Manual
This chapter presents reference pages for each of the operators
individually.
The operators are presented in alphabetical order.
@cindex command line switches
All valid command line switches are included in the syntax statement.
Recall that descriptions of many of these command line switches are
provided only in @ref{Common features}, to avoid redundancy.
Only options specific to, or most useful with, a particular operator are
described in any detail in the sections below.
@menu
* ncap2 netCDF Arithmetic Processor::
* ncatted netCDF Attribute Editor::
* ncbo netCDF Binary Operator::
* ncea netCDF Ensemble Averager::
* ncecat netCDF Ensemble Concatenator::
* ncflint netCDF File Interpolator::
* ncks netCDF Kitchen Sink::
* ncpdq netCDF Permute Dimensions Quickly::
* ncra netCDF Record Averager::
* ncrcat netCDF Record Concatenator::
* ncrename netCDF Renamer::
* ncwa netCDF Weighted Averager::
@end menu
@page
@html
<a name="ncap"></a> <!-- http://nco.sf.net/nco.html#ncap -->
<a name="ncap2"></a> <!-- http://nco.sf.net/nco.html#ncap2 -->
@end html
@node ncap2 netCDF Arithmetic Processor, ncatted netCDF Attribute Editor, Operator Reference Manual, Operator Reference Manual
@section @command{ncap2} netCDF Arithmetic Processor
@cindex parser
@cindex lexer
@cindex arithmetic processor
@findex ncap
@findex ncap2
@cartouche
@command{ncap2} understands a relatively full-featured set of
language features (e.g., loops, if-then, array indices).
Unfortunately, the @command{ncap2} documentation is woefully
out-of-date.
Please see the distribution file @file{data/ncap2_tst.nco}
for an up-to-date overview of its syntax and capabilities.
The @file{data/*.nco} distribution files (especially
@file{bin_cnt.nco}, @file{psd_wrf.nco}, and @file{rgr.nco}) contain
in-depth examples of @command{ncap2} solutions to complex problems.
@end cartouche
@c fxm: TODO nco549 hyper-link all switches to explanatory sections?
@c Problem is that only works well in HTML mode
@c TeXInfo has no native mode for concise hyperlinks in text mode
@c Currently in TeX/PDF mode, TeXInfo opens browser to find link,
@c rather than jumping to internal link within document
@noindent
SYNTAX
@example
ncap2 [-3] [-4] [@uref{http://nco.sf.net/nco.html#-A,,-A}] [-C] [-c] [-D @var{dbg}] [-F] [-f] [-L @var{dfl_lvl}]
[-l @var{path}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-s @var{algebra}] [-S @var{fl.nco}] [-v]
@var{input-file} [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncap2} arithmetically processes netCDF files.
@footnote{@command{ncap2} is the successor to @command{ncap} which was
put into maintenance mode in November, 2006.
This documentation refers to @command{ncap2}, which has a superset of
the @command{ncap} functionality.
Eventually @command{ncap} will be deprecated in favor @command{ncap2}.
@command{ncap2} will be renamed @command{ncap} in 2007.}.
@cindex script file
@cindex @code{--script-file}
@cindex @code{--fl_spt}
@cindex @code{--script}
@cindex @code{--spt}
The processing instructions are contained either in the @acronym{NCO}
script file @file{fl.nco} or in a sequence of command line arguments.
The options @samp{-s} (or long options @samp{--spt} or @samp{--script})
are used for in-line scripts and @samp{-S} (or long options
@samp{--fl_spt} or @samp{--script-file}) are used to provide the
filename where (usually multiple) scripting commands are pre-stored.
@command{ncap2} was written to perform arbitrary albebraic
transformations of data and archive the results as easily as possible.
@cindex derived fields
@xref{Missing Values}, for treatment of missing values.
The results of the algebraic manipulations are called
@dfn{derived fields}.
Unlike the other operators, @command{ncap2} does not accept a list of
variables to be operated on as an argument to @samp{-v}
(@pxref{Subsetting Variables}).
Rather, the @samp{-v} switch takes no arguments and indicates
that @command{ncap2} should output @emph{only} user-defined variables.
@command{ncap2} does not accept or understand the @var{-x} switch.
@c @subsection Scripting Mathematical Processing with @command{ncap2}
@menu
* Left hand casting::
* Syntax of ncap2 statements::
* Irregular grids::
* Intrinsic functions::
* Intrinsic mathematical functions::
@end menu
@html
<a name="att_prp"></a> <!-- http://nco.sf.net/nco.html#att_prp -->
@end html
@cindex attribute propagation
Defining new variables in terms of existing variables is one of
@command{ncap2}'s most powerful features.
@cindex derived fields
Derived fields inherit the metadata (i.e., attributes) of their
identically named ancestors, if any, in the script or input file.
When the derived field is completely new (no identically-named ancestors
exist), then it inherits the metadata (if any) of the left-most variable
on the right hand side of the defining expression.
This metadata inheritance is called @dfn{attribute propagation}.
Attribute propagation is intended to facilitate well-documented
data analysis, and we welcome suggestions to improve this feature.
@html
<a name="lhc"></a> <!-- http://nco.sf.net/nco.html#lhc -->
<a name="lhs"></a> <!-- http://nco.sf.net/nco.html#lhs -->
@end html
@node Left hand casting, Syntax of ncap2 statements, ncap2 netCDF Arithmetic Processor, ncap2 netCDF Arithmetic Processor
@subsection Left hand casting
@cindex hybrid coordinate system
@cindex left hand casting
@cindex @acronym{LHS}
The following examples demonstrate the utility of the
@dfn{left hand casting} ability of @command{ncap2}.
Consider first this simple, artificial, example.
If @var{lat} and @var{lon} are one dimensional coordinates of
dimensions @var{lat} and @var{lon}, respectively, then addition
of these two one-dimensional arrays is intrinsically ill-defined because
whether @var{lat_lon} should be dimensioned @var{lat} by @var{lon}
or @var{lon} by @var{lat} is ambiguous (assuming that addition is to
remain a @dfn{commutative} procedure, i.e., one that does not depend on
the order of its arguments).
Differing dimensions are said to be @dfn{orthogonal} to one another,
and sets of dimensions which are mutually exclusive are orthogonal
as a set and any arithmetic operation between variables in orthogonal
dimensional spaces is ambiguous without further information.
The ambiguity may be resolved by enumerating the desired dimension
ordering of the output expression inside square brackets on the
left hand side (@acronym{LHS}) of the equals sign.
This is called @dfn{left hand casting} because the user resolves the
dimensional ordering of the @acronym{RHS} of the expression by
specifying the desired ordering on the @acronym{LHS}.
@example
ncap2 -s 'lat_lon[lat,lon]=lat+lon' in.nc out.nc
ncap2 -s 'lon_lat[lon,lat]=lat+lon' in.nc out.nc
@end example
The explicit list of dimensions on the @acronym{LHS}, @code{[lat,lon]}
resolves the otherwise ambiguous ordering of dimensions in
@var{lat_lon}.
In effect, the @acronym{LHS} @dfn{casts} its rank properties onto the
@acronym{RHS}.
Without @acronym{LHS} casting, the dimensional ordering of @var{lat_lon}
would be undefined and, hopefully, @command{ncap2} would print an error
message.
Consider now a slightly more complex example.
In geophysical models, a coordinate system based on
a blend of terrain-following and density-following surfaces is
called a @dfn{hybrid coordinate system}.
In this coordinate system, four variables must be manipulated to
obtain the pressure of the vertical coordinate:
@var{PO} is the domain-mean surface pressure offset (a scalar),
@var{PS} is the local (time-varying) surface pressure (usually two
horizontal spatial dimensions, i.e, latitude by longitude), @var{hyam}
is the weight given to surfaces of constant density (one spatial
dimension, pressure, which is orthogonal to the horizontal
dimensions), and @var{hybm} is the weight given to surfaces of
constant elevation (also one spatial dimension).
This command constructs a four-dimensional pressure @code{prs_mdp}
from the four input variables of mixed rank and orthogonality:
@example
ncap2 -s 'prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm' in.nc out.nc
@end example
Manipulating the four fields which define the pressure in a hybrid
coordinate system is easy with left hand casting.
@html
<a name="syn"></a> <!-- http://nco.sf.net/nco.html#syn -->
@end html
@node Syntax of ncap2 statements, Irregular grids, Left hand casting, ncap2 netCDF Arithmetic Processor
@subsection Syntax of @command{ncap2} statements
@cindex statement
@cindex syntax
Mastering @command{ncap2} is relatively simple.
Each valid statement @var{statement} consists of standard forward
algebraic expression.
The @file{fl.nco}, if present, is simply a list of such statements,
whitespace, and comments.
@cindex C language
The syntax of statements is most like the computer @w{language C}.
The following characteristics @w{of C} are preserved:
@table @asis
@item Array syntax
@cindex array syntax
@cindex @code{[]} (array delimiters)
Arrays elements are placed within @code{[]} characters;
@item Array indexing
@cindex array indexing
Arrays are 0-based;
@item Array storage
@cindex array storage
Last dimension is most rapidly varying;
@item Assignment statements
@cindex assignment statement
@cindex semi-colon
@cindex @code{;} (end of statement)
@w{A semi}-colon @samp{;} indicates the end of an assignment statement.
@item Comments
@cindex comments
@cindex @code{/*...*/} (comment)
@cindex @code{//} (comment)
Multi-line comments are enclosed within @code{/* */} characters.
Single line comments are preceded by @code{//} characters.
@item Nesting
@cindex including files
@cindex nesting
@cindex @code{#include}
Files may be nested in scripts using @code{#include @var{script}}.
Note that the @code{#include} command is not followed by a semi-colon
because it is a pre-processor directive, not an assignment statement.
The filename @file{script} is interpreted relative to the run directory.
@item Attribute syntax
@cindex attribute syntax
@cindex @code{@@} (attribute)
The at-sign @code{@@} is used to delineate an attribute name from a
variable name.
@end table
@html
<a name="grd"></a> <!-- http://nco.sf.net/nco.html#grd -->
<a name="rrg"></a> <!-- http://nco.sf.net/nco.html#rrg -->
<a name="rct"></a> <!-- http://nco.sf.net/nco.html#rct -->
@end html
@node Irregular grids, Intrinsic functions, Syntax of ncap2 statements, ncap2 netCDF Arithmetic Processor
@subsection Irregular Grids
@cindex irregular grids
@cindex rectangular grids
@cindex non-rectangular grids
@cindex non-standard grids
@cindex mask
@c fxm need to edit rrg sxn beginning here
@acronym{NCO} is capable of analyzing datasets for many different
underlying coordinate grid types.
netCDF was developed for and initially used with grids comprised of
orthogonal dimensions forming a rectangular coordinate system.
We call such grids @emph{standard} grids.
It is increasingly common for datasets to use metadata to describe
much more complex grids.
Let us first define three important coordinate grid properties:
rectangularity, regularity, and fxm.
Grids are @emph{regular} if the spacing between adjacent is constant.
For example, a 4-by-5 degree latitude-longitude grid is regular
because the spacings between adjacent latitudes (@w{4 degrees}) are
constant as are the (@w{5 degrees}) spacings between adjacent
longitudes.
Spacing in @emph{irregular} grids depends on the location along the
coordinate.
Grids such as Gaussian grids have uneven spacing in latitude (points
cluster near the equator) and so are irregular.
Grids are @emph{rectangular} if the number of elements in any
dimension is not a function of any other dimension.
For example, a T42 Gaussian latitude-longitude grid is rectangular
because there are the same number of longitudes (128) for each of the
(64) latitudes.
Grids are @emph{non-rectangular} if the elements in any dimension
depend on another dimension.
Non-rectangular grids present many special challenges to
analysis software like @acronym{NCO}.
Wrapped coordinates (@pxref{Wrapped Coordinates}), such as longitude,
are independent of these grid properties (regularity,
rectangularity).
@cindex wrapped coordinates
The preferred @acronym{NCO} technique to analyze data on non-standard
coordinate grids is to create a region mask with @command{ncap2}, and
then to use the mask within @command{ncap2} for variable-specific
processing, and/or with other operators (e.g., @command{ncwa},
@command{ncdiff}) for entire file processing.
Before describing the construction of masks, let us review how
irregularly gridded geoscience data are described.
Say that latitude and longitude are stored as @var{R}-dimensional
arrays and the product of the dimension sizes is the total number of
elements N in the other variables.
Geoscience applications tend to use @math{@var{R}=1},
@math{@var{R}=2}, and @math{@var{R}=3}.
If the grid is has no simple representation (e.g., discontinuous) then
it makes sense to store all coordinates as 1D arrays with the same
size as the number of grid points.
These gridpoints can be completely independent of all the other (own
weight, area, etc.).
@var{R}=1: lat(number_of_gridpoints) and lon(number_of_gridpoints)
If the horizontal grid is time-invariant then @var{R}=2 is common:
@var{R}=2: lat(south_north,east_west) and lon(south_north,east_west)
The WRF (Weather and Research Forecast) model uses @var{R}=3
@var{R}=3: lat(time,south_north,east_west), lon(time,south_north,east_west)
and so supports grids that change with time.
Grids with @var{R} > 1 often use missing values to indicated empty points.
For example, so-called "staggered grids" will use fewer east_west
points near the poles and more near the equator. netCDF only accepts
rectangular arrays so space must be allocated for the maximum number
of east_west points at all latitudes. Then the application writes
missing values into the unused points near the poles.
Let's demonstrate the recommended @command{ncap2} analysis technique by
constructing a region mask for an @var{R}=2 grid.
We wish to find, say, the mean temperature within
[@var{lat_min},@var{lat_max}] and [@var{lon_min},@var{lon_max}]:
@example
ncap2 -s 'mask= (lat >= lat_min && lat <= lat_max) && \
(lon >= lon_min && lon <= lon_max);' in.nc out.nc
@end example
Once you have a mask, you can use it on specific variables:
@example
ncap2 -s 'temperature_avg=(temperature*mask).avg()' in.nc out.nc
@end example
and you can apply it to entire files:
@example
ncwa -a lat,lon -m mask -w area in.nc out.nc
@end example
You can put this altogether on the command line or in a script, e.g.,
cleaner.
@example
cat > ncap2.in << EOF
mask = (lat >= lat_min && lat <= lat_max) && (lon >= lon_min && > lon <= lon_max);
if(mask.total() > 0)@{ // Check that mask contains some valid values
temperature_avg=(temperature*mask).avg(); // Average temperature
temperature_max=(temperature*mask).max(); // Maximum temperature
@}
EOF
ncap2 -S ncap2.in in.nc out.nc
@end example
For the WRF file creating the mask looks like
@example
mask = (XLAT >= lat_min && XLAT <= lat_max) && (XLONG >= lon_min && > XLONG <= lon_max);
@end example
In practice with WRF it's a bit more complicated because you must
use the global metadata to determine the grid staggering and offsets
to translate XLAT and XLONG into real latitudes and longitudes and
missing points. The WRF grid documentation should describe this.
A few notes:
Irregular regions are the union of arrays lat/lon_min/max's.
The mask procedure is identical for all @var{R}.
@c fxm need to edit rrg sxn down to here
@html
<a name="fnc"></a> <!-- http://nco.sf.net/nco.html#fnc -->
@end html
@node Intrinsic functions, Intrinsic mathematical functions, Irregular grids, ncap2 netCDF Arithmetic Processor
@subsection Intrinsic functions
@command{ncap2} contains a small (and growing) library of intrinsic
functions.
In addition to the standard mathematical functions
(@pxref{Intrinsic mathematical functions}), @command{ncap2} currently
supports packing and unpacking.
@cindex packing
@cindex unpacking
@table @code
@item pack(x)
The standard packing algorithm is applied to variable @var{x}.
@item unpack(x)
The standard unpacking algorithm is applied to variable @var{x}.
@end table
@unnumberedsubsec Type Conversion Functions
@html
<a name="typ_cnv"></a> <!-- http://nco.sf.net/nco.html#typ_cnv -->
<a name="typ_cst"></a> <!-- http://nco.sf.net/nco.html#typ_cst -->
<a name="cst"></a> <!-- http://nco.sf.net/nco.html#cst -->
@end html
These intrinsic functions allow @command{ncap2} to convert variables on
disk among the available types supported by netCDF.
@cindex type conversion
@cindex promotion
@cindex demotion
@cindex @code{byte(x)}
@cindex @code{char(x)}
@cindex @code{double(x)}
@cindex @code{float(x)}
@cindex @code{int(x)}
@cindex @code{short(x)}
@table @code
@item byte(x)
@dfn{Convert to @code{NC_BYTE}}
Converts @var{x} to external type @code{NC_BYTE}, a C-type @code{signed char}.
@item char(x)
@dfn{Convert to @code{NC_CHAR}}
Converts @var{x} to external type @code{NC_CHAR}, a C-type @code{unsigned char}.
@item double(x)
@dfn{Convert to @code{NC_DOUBLE}}
Converts @var{x} to external type @code{NC_DOUBLE}, a C-type @code{double}.
@item float(x)
@dfn{Convert to @code{NC_FLOAT}}
Converts @var{x} to external type @code{NC_FLOAT}, a C-type @code{float}.
@item int(x)
@dfn{Convert to @code{NC_INT}}
Converts @var{x} to external type @code{NC_INT}, a C-type @code{int}.
@item short(x)
@dfn{Convert to @code{NC_SHORT}}
Converts @var{x} to external type @code{NC_SHORT}, a C-type @code{short}.
@end table
@xref{Type Conversion}, for more details on automatic and manual type conversion.
@html
<a name="mth"></a> <!-- http://nco.sf.net/nco.html#mth -->
@end html
@node Intrinsic mathematical functions, , Intrinsic functions, ncap2 netCDF Arithmetic Processor
@subsection Intrinsic mathematical functions
@command{ncap2} supports the standard mathematical functions supplied with
most operating systems.
@cindex addition
@cindex subtraction
@cindex multiplication
@cindex division
@cindex exponentiation
@cindex power
@cindex modulus
@cindex @code{+} (addition)
@cindex @code{-} (subtraction)
@cindex @code{*} (multiplication)
@cindex @code{/} (division)
@cindex @code{^} (power)
@cindex @code{%} (modulus)
Standard calculator notation is used for addition @kbd{+}, subtraction
@kbd{-}, multiplication @kbd{*}, division @kbd{/}, exponentiation
@kbd{^}, and modulus @kbd{%}.
The available elementary mathematical functions are:
@cindex @var{abs}
@cindex @var{acosh}
@cindex @var{acos}
@cindex @var{asinh}
@cindex @var{asin}
@cindex @var{atanh}
@cindex @var{atan}
@cindex @var{ceil}
@cindex @var{cosh}
@cindex @var{cos}
@cindex @var{erfc}
@cindex @var{erf}
@cindex @var{exp}
@cindex @var{floor}
@cindex @var{gamma}
@cindex @var{ln}
@cindex @var{log10}
@cindex @var{log}
@cindex @var{nearbyint}
@cindex @var{pow}
@cindex @var{rint}
@cindex @var{round}
@cindex @var{sinh}
@cindex @var{sin}
@cindex @var{sqrt}
@cindex @var{tanh}
@cindex @var{tan}
@cindex @var{trunc}
@cindex mathematical functions
@cindex nearest integer function (inexact)
@cindex nearest integer function (exact)
@cindex rounding functions
@cindex truncation function
@cindex absolute value
@cindex arccosine function
@cindex arcsine function
@cindex arctangent function
@cindex ceiling function
@cindex complementary error function
@cindex cosine function
@cindex error function
@cindex exponentiation function
@cindex floor function
@cindex gamma function
@cindex hyperbolic arccosine function
@cindex hyperbolic arcsine function
@cindex hyperbolic arctangent function
@cindex hyperbolic cosine function
@cindex hyperbolic sine function
@cindex hyperbolic tangent
@cindex logarithm, base 10
@cindex logarithm, natural
@cindex power function
@cindex sine function
@cindex square root function
@table @code
@item abs(x)
@dfn{Absolute value}
@tex
Absolute value of $x$, $|x|$.
@end tex
@ifinfo
Absolute value of @var{x}.
@end ifinfo
Example:
@tex
abs$(-1) = 1$
@end tex
@ifinfo
@math{abs(-1) = 1}
@end ifinfo
@item acos(x)
@dfn{Arc-cosine}
Arc-cosine of @var{x} where @var{x} is specified in radians.
Example:
@tex
acos$(1.0) = 0.0$
@end tex
@ifinfo
@math{acos(1.0) = 0.0}
@end ifinfo
@item acosh(x)
@dfn{Hyperbolic arc-cosine}
Hyperbolic arc-cosine of @var{x} where @var{x} is specified in radians.
Example:
@tex
acosh$(1.0) = 0.0$
@end tex
@ifinfo
@math{acosh(1.0) = 0.0}
@end ifinfo
@item asin(x)
@dfn{Arc-sine}
Arc-sine of @var{x} where @var{x} is specified in radians.
Example:
@tex
asin$(1.0) = 1.57079632679489661922$
@end tex
@ifinfo
@math{asin(1.0) = 1.57079632679489661922}
@end ifinfo
@item asinh(x)
@dfn{Hyperbolic arc-sine}
Hyperbolic arc-sine of @var{x} where @var{x} is specified in radians.
Example:
@tex
asinh$(1.0) = 0.88137358702$
@end tex
@ifinfo
@math{asinh(1.0) = 0.88137358702}
@end ifinfo
@item atan(x)
@dfn{Arc-tangent}
Arc-tangent of @var{x} where @var{x} is specified in radians between
@tex
$-\pi/2$ and $\pi/2$.
@end tex
@ifinfo
@math{-pi/2} and @math{pi/2}.
@end ifinfo
Example:
@tex
atan$(1.0) = 0.78539816339744830961$
@end tex
@ifinfo
@math{atan(1.0) = 0.78539816339744830961}
@end ifinfo
@item atanh(x)
@dfn{Hyperbolic arc-tangent}
Hyperbolic arc-tangent of @var{x} where @var{x} is specified in radians between
@tex
$-\pi/2$ and $\pi/2$.
@end tex
@ifinfo
@math{-pi/2} and @math{pi/2}.
@end ifinfo
Example:
@tex
atanh$(3.14159265358979323844) = 1.0$
@end tex
@ifinfo
@math{atanh(3.14159265358979323844) = 1.0}
@end ifinfo
@item ceil(x)
@dfn{Ceil}
Ceiling of @var{x}. Smallest integral value not less than argument.
Example:
@tex
ceil$(0.1) = 1.0$
@end tex
@ifinfo
@math{ceil(0.1) = 1.0}
@end ifinfo
@item cos(x)
@dfn{Cosine}
Cosine of @var{x} where @var{x} is specified in radians.
Example:
@tex
cos$(0.0) = 1.0$
@end tex
@ifinfo
@math{cos(0.0) = 1.0}
@end ifinfo
@item cosh(x)
@dfn{Hyperbolic cosine}
Hyperbolic cosine of @var{x} where @var{x} is specified in radians.
Example:
@tex
cosh$(0.0) = 1.0$
@end tex
@ifinfo
@math{cosh(0.0) = 1.0}
@end ifinfo
@item erf(x)
@dfn{Error function}
Error function of @var{x} where @var{x} is specified between
@tex
$-1$ and $1$.
@end tex
@ifinfo
@math{-1} and @math{1}.
@end ifinfo
Example:
@tex
erf$(1.0) = 0.842701$
@end tex
@ifinfo
@math{erf(1.0) = 0.842701}
@end ifinfo
@item erfc(x)
@dfn{Complementary error function}
Complementary error function of @var{x} where @var{x} is specified between
@tex
$-1$ and $1$.
@end tex
@ifinfo
@math{-1} and @math{1}.
@end ifinfo
Example:
@tex
erfc$(1.0) = 0.15729920705$
@end tex
@ifinfo
@math{erfc(1.0) = 0.15729920705}
@end ifinfo
@item exp(x)
@dfn{Exponential}
Exponential of @var{x},
@tex
$e^{x}$.
@end tex
@ifinfo
@math{e^x}.
@end ifinfo
Example:
@tex
exp$(1.0) = 2.71828182845904523536$
@end tex
@ifinfo
@math{exp(1.0) = 2.71828182845904523536}
@end ifinfo
@item floor(x)
@dfn{Floor}
Floor of @var{x}. Largest integral value not greater than argument.
Example:
@tex
floor$(1.9) = 1$
@end tex
@ifinfo
@math{floor(1.9) = 1}
@end ifinfo
@item gamma(x)
@dfn{Gamma function}
Gamma function of @var{x},
@tex
$\Gamma(x)$.
@end tex
@ifinfo
@math{Gamma(x)}.
@end ifinfo
The well-known and loved continuous factorial function.
Example:
@tex
gamma$(0.5) = \sqrt{\pi}$
@end tex
@ifinfo
@math{gamma(0.5) = sqrt(pi)}
@end ifinfo
@item ln(x)
@dfn{Natural Logarithm}
Natural logarithm of @var{x},
@tex
$\ln(x)$.
@end tex
@ifinfo
@math{ln(x)}.
@end ifinfo
Example:
@tex
ln$(2.71828182845904523536) = 1.0$
@end tex
@ifinfo
@math{ln(2.71828182845904523536) = 1.0}
@end ifinfo
@item log(x)
@dfn{Natural Logarithm}
Exact synonym for @code{ln(x)}.
@item log10(x)
@dfn{Base 10 Logarithm}
@w{Base 10} logarithm of @var{x},
@tex
$\log_{10}(x)$.
@end tex
@ifinfo
@math{log10(x)}.
@end ifinfo
Example:
@tex
log$(10.0) = 1.0$
@end tex
@ifinfo
@math{log(10.0) = 1.0}
@end ifinfo
@item nearbyint(x)
@dfn{Round inexactly}
Nearest integer to @var{x} is returned in floating point format.
@cindex inexact conversion
No exceptions are raised for @dfn{inexact conversions}.
Example:
@tex
nearbyint$(0.1) = 0.0$
@end tex
@ifinfo
@math{nearbyint(0.1) = 0.0}
@end ifinfo
@item pow(x,y)
@dfn{Power}
@cindex promotion
@cindex automatic type conversion
Value of @var{x} is raised to the power of @var{y}.
Exceptions are raised for @dfn{domain errors}.
Due to type-limitations in the @w{C language} @code{pow} function,
integer arguments are promoted (@pxref{Type Conversion}) to type
@code{NC_FLOAT} before evaluation.
Example:
@tex
pow$(2,3) = 8$
@end tex
@ifinfo
@math{pow(2,3) = 8}
@end ifinfo
@item rint(x)
@dfn{Round exactly}
Nearest integer to @var{x} is returned in floating point format.
Exceptions are raised for @dfn{inexact conversions}.
Example:
@tex
rint$(0.1) = 0$
@end tex
@ifinfo
@math{rint(0.1) = 0}
@end ifinfo
@item round(x)
@dfn{Round}
Nearest integer to @var{x} is returned in floating point format.
Round halfway cases away from zero, regardless of current IEEE rounding direction.
Example:
@tex
round$(0.5) = 1.0$
@end tex
@ifinfo
@math{round(0.5) = 1.0}
@end ifinfo
@item sin(x)
@dfn{Sine}
Sine of @var{x} where @var{x} is specified in radians.
Example:
@tex
sin$(1.57079632679489661922) = 1.0$
@end tex
@ifinfo
@math{sin(1.57079632679489661922) = 1.0}
@end ifinfo
@item sinh(x)
@dfn{Hyperbolic sine}
Hyperbolic sine of @var{x} where @var{x} is specified in radians.
Example:
@tex
sinh$(1.0) = 1.1752$
@end tex
@ifinfo
@math{sinh(1.0) = 1.1752}
@end ifinfo
@item sqrt(x)
@dfn{Square Root}
Square Root of @var{x},
@tex
$\sqrt{x}$.
@end tex
@ifinfo
@math{sqrt(x)}.
@end ifinfo
Example:
@tex
sqrt$(4.0) = 2.0$
@end tex
@ifinfo
@math{sqrt(4.0) = 2.0}
@end ifinfo
@item tan(x)
@dfn{Tangent}
Tangent of @var{x} where @var{x} is specified in radians.
Example:
@tex
tan$(0.78539816339744830961) = 1.0$
@end tex
@ifinfo
@math{tan(0.78539816339744830961) = 1.0}
@end ifinfo
@item tanh(x)
@dfn{Hyperbolic tangent}
Hyperbolic tangent of @var{x} where @var{x} is specified in radians.
Example:
@tex
tanh$(1.0) = 0.761594155956$
@end tex
@ifinfo
@math{tanh(1.0) = 0.761594155956}
@end ifinfo
@item trunc(x)
@dfn{Truncate}
Nearest integer to @var{x} is returned in floating point format.
Round halfway cases toward zero, regardless of current IEEE rounding direction.
Example:
@tex
trunc$(0.5) = 0.0$
@end tex
@ifinfo
@math{trunc(0.5) = 0.0}
@end ifinfo
@end table
@noindent
The complete list of mathematical functions supported is
platform-specific.
Functions mandated by @w{ANSI C} are @emph{guaranteed} to be present
and are indicated with an asterisk
@c fxm No they're not, not yet
@cindex @code{ANSI C}
@cindex @code{float}
@cindex precision
@cindex quadruple precision
@cindex single precision
@cindex double precision
@cindex @code{long double}
@cindex @code{NC_DOUBLE}
@footnote{
@w{ANSI C} compilers are guaranteed to support double precision versions
of these functions.
These functions normally operate on netCDF variables of type @code{NC_DOUBLE}
without having to perform intrinsic conversions.
For example, @acronym{ANSI} compilers provide @code{sin} for the sine of C-type
@code{double} variables.
The @acronym{ANSI} standard does not require, but many compilers provide,
an extended set of mathematical functions that apply to single
(@code{float}) and quadruple (@code{long double}) precision variables.
Using these functions (e.g., @code{sinf} for @code{float},
@code{sinl} for @code{long double}), when available, is (presumably)
more efficient than casting variables to type @code{double},
performing the operation, and then re-casting.
@acronym{NCO} uses the faster intrinsic functions when they are
available, and uses the casting method when they are not.
}.
and are indicated with an asterisk.
@cindex @code{-f}
@cindex @code{--prn_fnc_tbl}
@cindex @code{--fnc_tbl}
Use the @samp{-f} (or @samp{fnc_tbl} or @samp{prn_fnc_tbl}) switch
to print a complete list of functions supported on your platform.
This prints a list of functions and whether they are supported
for netCDF variables of intrinsic type @code{NC_FLOAT} and @code{NC_DOUBLE}.
@cindex Linux
@footnote{Linux supports more of these intrinsic functions than
other OSs.}
@noindent
@html
<a name="xmp_ncap"></a> <!-- http://nco.sf.net/nco.html#xmp_ncap -->
<a name="xmp_ncap2"></a> <!-- http://nco.sf.net/nco.html#xmp_ncap2 -->
@end html
EXAMPLES
See the @file{ncap.in} and @file{ncap2.in} scripts released with NCO
for more complete demonstrations of @command{ncap} and @command{ncap2}
functionality, respectively (these scripts are available on-line at
@url{http://nco.sf.net/ncap.in} and
@url{http://nco.sf.net/ncap2.in}).
Define new attribute @var{new} for existing variable @var{one}
as twice the existing attribute @var{double_att} of variable
@var{att_var}:
@example
ncap2 -s 'one@@new=2*att_var@@double_att' in.nc out.nc
@end example
Average variables of mixed types (result is of type @code{double}):
@example
ncap2 -s 'average=(var_float+var_double+var_int)/3' in.nc out.nc
@end example
Multiple commands may be given to @command{ncap2} in three ways.
First, the commands may be placed in a script which is executed, e.g.,
@file{tst.nco}.
Second, the commands may be individually specified with multiple
@samp{-s} arguments to the same @command{ncap2} invocation.
Third, the commands may be chained together into a single @samp{-s}
argument to @command{ncap2}.
Assuming the file @file{tst.nco} contains the commands
@code{a=3;b=4;c=sqrt(a^2+b^2);}, then the following @command{ncap2}
invocations produce identical results:
@example
ncap2 -v -S tst.nco in.nc out.nc
ncap2 -v -s 'a=3' -s 'b=4' -s 'c=sqrt(a^2+b^2)' in.nc out.nc
ncap2 -v -s 'a=3;b=4;c=sqrt(a^2+b^2)' in.nc out.nc
@end example
The second and third examples show that @command{ncap2} does not require
that a trailing semi-colon @samp{;} be placed at the end of a @samp{-s}
argument, although a trailing semi-colon @samp{;} is always allowed.
However, semi-colons are required to separate individual assignment
statements chained together as a single @samp{-s} argument.
@html
<a name="xmp_grw"></a> <!-- http://nco.sf.net/nco.html#xmp_grw -->
@end html
@cindex growing dimensions
@cindex dimensions, growing
@command{ncap2} may be used to ``grow'' dimensions, i.e., to increase
dimension sizes without altering existing data.
Say @file{in.nc} has @code{ORO(lat,lon)} and the user wishes a new
file with @code{new_ORO(new_lat,new_lon)} that contains zeros in the
undefined portions of the new grid.
@example
defdim("new_lat",$lat.size+1); // Define new dimension sizes
defdim("new_lon",$lon.size+1);
new_ORO[$new_lat,$new_lon]=0.0f; // Initialize to zero
new_ORO(0:$lat.size-1,0:$lon.size-1)=ORO; // Fill valid data
@end example
The commands to define new coordinate variables @code{new_lat}
and @code{new_lon} in the output file follow a similar pattern.
One would might store these commands in a script @file{grow.nco}
and then execute the script with
@example
ncap2 -v -S grow.nco in.nc out.nc
@end example
@html
<a name="flg"></a> <!-- http://nco.sf.net/nco.html#flg -->
@end html
@cindex flags
Imagine you wish to create a binary flag based on the value of
an array.
The flag should have @w{value 1.0} where the array @w{exceeds 1.0},
and @w{value 0.0} elsewhere.
This example creates the binary flag @code{ORO_flg} in @file{out.nc}
from the continuous array named @code{ORO} in @file{in.nc}.
@example
ncap2 -s 'ORO_flg=(ORO > 1.0)' in.nc out.nc
@end example
Suppose your task is to change all values of @code{ORO} which
@w{equal 2.0} to the new @w{value 3.0}:
@example
ncap2 -s 'ORO_msk=(ORO==2.0);ORO=ORO_msk*3.0+!ORO_msk*ORO' in.nc out.nc
@end example
@cindex mask
This creates and uses @code{ORO_msk} to mask the subsequent arithmetic
operation.
Values of @code{ORO} are only changed where @code{ORO_msk} is true,
i.e., where @code{ORO} @w{equals 2.0}.
@cindex Fortran90
In the future, @command{ncap2} will support the Fortran90 @code{where}
construct to further simplify this syntax.
@html
<a name="cvr"></a> <!-- http://nco.sf.net/nco.html#cvr -->
@end html
This example uses @command{ncap2} to compute the covariance of two
variables.
Let the variables @var{@wndznl{}} and @var{@wndmrd{}} be the horizontal
wind components.
@cindex covariance
@c fxm 20030423: texi2html 1.64 has problems with this legal syntax but makeinfo --html does not
The @dfn{covariance} of @var{@wndznl{}} and @var{@wndmrd{}} is defined
as the time mean product of the deviations of @var{@wndznl{}} and
@var{@wndmrd{}} from their respective time means.
Symbolically, the covariance
@set flg
@tex
$[\wndznl^{\prime} \wndmrd^{\prime}] =
[\wndznl\wndmrd]-[\wndznl][\wndmrd]$ where $[\xxx]$ denotes the
time-average of~$\xxx$,
$[\xxx] \equiv {1 \over \tau} \int_{\tm=0}^{\tm=\tau} \xxx(\tm) \,\dfr\tm$
and $\xxx^{\prime}$
@clear flg
@end tex
@ifinfo
@math{[@var{@wndznl{}'@wndmrd{}'}] =
[@var{@wndznl{}@wndmrd{}}]-[@var{@wndznl{}}][@var{@wndmrd{}}]}
where @math{[@var{@xxx{}}]} denotes the time-average of
@math{@var{@xxx{}}} and @math{@var{@xxx{}'}}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
[@var{@wndznl{}'@wndmrd{}'}] =
[@var{@wndznl{}@wndmrd{}}]-[@var{@wndznl{}}][@var{@wndmrd{}}] where
[@xxx{}] denotes the time-average of @var{@xxx{}} and @var{@xxx{}'}
@clear flg
@end ifset
denotes the deviation from the time-mean.
The covariance tells us how much of the correlation of two signals
arises from the signal fluctuations versus the mean signals.
@cindex eddy covariance
Sometimes this is called the @dfn{eddy covariance}.
We will store the covariance in the variable @code{uprmvprm}.
@example
ncwa -O -a time -v u,v in.nc foo.nc # Compute time mean of u,v
ncrename -O -v u,uavg -v v,vavg foo.nc # Rename to avoid conflict
ncks -A -v uavg,vavg foo.nc in.nc # Place time means with originals
ncap2 -O -s 'uprmvprm=u*v-uavg*vavg' in.nc in.nc # Covariance
ncra -O -v uprmvprm in.nc foo.nc # Time-mean covariance
@end example
The mathmatically inclined will note that the same covariance would be
obtained by replacing the step involving @command{ncap2} with
@example
ncap2 -O -s 'uprmvprm=(u-uavg)*(v-vavg)' foo.nc foo.nc # Covariance
@end example
As of @acronym{NCO} version 3.1.8 (December, 2006), @command{ncap2}
can compute averages, and thus covariances, by itself:
@example
ncap2 -s 'uavg=u.avg($time);vavg=v.avg($time);uprmvprm=u*v-uavg*vavg' \
-s 'uprmvrpmavg=uprmvprm.avg($time)' in.nc foo.nc
@end example
We have not seen a simpler method to script and execute powerful
arithmetic than @command{ncap2}.
@cindex globbing
@cindex shell
@cindex quotes
@cindex extended regular expressions
@cindex regular expressions
@command{ncap2} utilizes many meta-characters
(e.g., @samp{$}, @samp{?}, @samp{;}, @samp{()}, @samp{[]})
that can confuse the command-line shell if not quoted properly.
The issues are the same as those which arise in utilizing extended
regular expressions to subset variables (@pxref{Subsetting Variables}).
The example above will fail with no quotes and with double quotes.
This is because shell globbing tries to @dfn{interpolate} the value of
@code{$time} from the shell environment unless it is quoted:
@example
ncap2 -s 'uavg=u.avg($time)' in.nc foo.nc # Correct (recommended)
ncap2 -s uavg=u.avg('$time') in.nc foo.nc # Correct (and dangerous)
ncap2 -s uavg=u.avg($time) in.nc foo.nc # Fails ($time = '')
ncap2 -s "uavg=u.avg($time)" in.nc foo.nc # Fails ($time = '')
@end example
Without the single quotes, the shell replaces @code{$time} with an
empty string.
The command @command{ncap2} receives from the shell is
@code{uavg=u.avg()}.
This causes @command{ncap2} to average over all dimensions rather than
just the @var{time} dimension, and unintended consequence.
We recommend using single quotes to protect @command{ncap2}
command-line scripts from the shell, even when such protection is not
strictly necessary.
Expert users may violate this rule to exploit the ability to use shell
variables in @command{ncap2} command-line scripts
(@pxref{CCSM Example}).
In such cases it may be necessary to use the shell backslash character
@samp{\} to protect the @command{ncap2} meta-character.
@cindex appending data
@cindex time-averaging
@findex ncks
@findex ncwa
@findex ncra
@cindex degenerate dimension
@cindex @samp{-b}
Whether a degenerate record dimension is desirable or undesirable
depends on the application.
Often a degenerate @var{time} dimension is useful, e.g., for
concatentating, but it may cause problems with arithmetic.
Such is the case in the above example, where the first step employs
@command{ncwa} rather than @command{ncra} for the time-averaging.
Of course the numerical results are the same with both operators.
The difference is that, unless @samp{-b} is specified, @command{ncwa}
writes no @var{time} dimension to the output file, while @command{ncra}
defaults to keeping @var{time} as a degenerate (@w{size 1}) dimension.
Appending @code{u} and @code{v} to the output file would cause
@command{ncks} to try to expand the degenerate time axis of @code{uavg}
and @code{vavg} to the size of the non-degenerate @var{time} dimension
in the input file.
Thus the append (@command{ncks -A}) command would be undefined (and
should fail) in this case.
@cindex @code{-C}
Equally important is the @samp{-C} argument
(@pxref{Subsetting Coordinate Variables}) to @command{ncwa} to prevent
any scalar @var{time} variable from being written to the output file.
Knowing when to use @command{ncwa -a time} rather than the default
@command{ncra} for time-averaging takes, well, time.
@page
@html
<a name="ncatted"></a> <!-- http://nco.sf.net/nco.html#ncatted -->
@end html
@node ncatted netCDF Attribute Editor, ncbo netCDF Binary Operator, ncap2 netCDF Arithmetic Processor, Operator Reference Manual
@section @command{ncatted} netCDF Attribute Editor
@cindex attributes
@cindex attribute names
@cindex editing attributes
@findex ncatted
@noindent
SYNTAX
@example
ncatted [-a @var{att_dsc}] [-a @dots{}] [-D @var{dbg}] [-h] [--hdr_pad @var{nbr}]
[-l @var{path}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
@var{input-file} [[@var{output-file}]]
@end example
@noindent
DESCRIPTION
@command{ncatted} edits attributes in a netCDF file.
If you are editing attributes then you are spending too much time in the
world of metadata, and @command{ncatted} was written to get you back out as
quickly and painlessly as possible.
@command{ncatted} can @dfn{append}, @dfn{create}, @dfn{delete},
@dfn{modify}, and @dfn{overwrite} attributes (all explained below).
Furthermore, @command{ncatted} allows each editing operation to be applied
to every variable in a file.
This saves time when changing attribute conventions throughout a file.
Note that @command{ncatted} interprets character attributes
(i.e., attributes of type @code{NC_CHAR}) as strings.
@cindex @code{history}
@cindex @code{-h}
Because repeated use of @command{ncatted} can considerably increase the size
of the @code{history} global attribute (@pxref{History Attribute}), the
@samp{-h} switch is provided to override automatically appending the
command to the @code{history} global attribute in the @var{output-file}.
@cindex missing values
@cindex data, missing
@cindex @code{_FillValue}
When @command{ncatted} is used to change the @code{_FillValue} attribute,
it changes the associated missing data self-consistently.
If the internal floating point representation of a missing value,
e.g., 1.0e36, differs between two machines then netCDF files produced
on those machines will have incompatible missing values.
This allows @command{ncatted} to change the missing values in files from
different machines to a single value so that the files may then be
concatenated together, e.g., by @command{ncrcat}, without losing any
information.
@xref{Missing Values}, for more information.
The key to mastering @command{ncatted} is understanding the meaning of the
structure describing the attribute modification, @var{att_dsc} specified by the required option @samp{-a} or @samp{--attribute}.
Each @var{att_dsc} contains five elements, which makes using
@command{ncatted} somewhat complicated, but powerful.
The @var{att_dsc} argument structure contains five arguments in the
following order:@*
@var{att_dsc} = @var{att_nm}, @var{var_nm}, @var{mode}, @var{att_type},
@var{att_val}@*
@table @var
@item att_nm
Attribute name.
Example: @code{units}
@item var_nm
Variable name.
Example: @code{pressure}
@item mode
Edit mode abbreviation.
Example: @code{a}.
See below for complete listing of valid values of @var{mode}.
@item att_type
Attribute type abbreviation.
Example: @code{c}.
See below for complete listing of valid values of @var{att_type}.
@item att_val
Attribute value.
Example: @code{pascal}.
@end table
@noindent
There should be no empty space between these five consecutive
arguments.
The description of these arguments follows in their order of
appearance.
The value of @var{att_nm} is the name of the attribute you want to edit.
This meaning of this should be clear to all users of the @command{ncatted}
operator.
If @var{att_nm} is omitted (i.e., left blank) and @dfn{Delete} mode is
selected, then all attributes associated with the specified variable
will be deleted.
@cindex global attributes
@cindex attributes, global
The value of @var{var_nm} is the name of the variable containing the
attribute (named @var{att_nm}) that you want to edit.
There are two very important and useful exceptions to this rule.
The value of @var{var_nm} can also be used to direct @command{ncatted} to
edit global attributes, or to repeat the editing operation for every
variable in a file.
@w{A value} of @var{var_nm} of ``global'' indicates that @var{att_nm} refers
to a global attribute, rather than a particular variable's attribute.
This is the method @command{ncatted} supports for editing global
attributes.
If @var{var_nm} is left blank, on the other hand, then @command{ncatted}
attempts to perform the editing operation on every variable in the file.
This option may be convenient to use if you decide to change the
conventions you use for describing the data.
@html
<a name="mode"></a> <!-- http://nco.sf.net/nco.html#mode -->
<a name="att_mode"></a> <!-- http://nco.sf.net/nco.html#att_mode -->
@end html
The value of @var{mode} is a single character abbreviation (@code{a},
@code{c}, @code{d}, @code{m}, or @code{o}) standing for one of
five editing modes:@*
@cindex attributes, appending
@cindex attributes, creating
@cindex attributes, deleting
@cindex attributes, modifying
@cindex attributes, editing
@cindex attributes, overwriting
@table @code
@item a
@dfn{Append}.
Append value @var{att_val} to current @var{var_nm} attribute
@var{att_nm} value @var{att_val}, if any.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect.
@item c
@dfn{Create}.
Create variable @var{var_nm} attribute @var{att_nm} with @var{att_val}
if @var{att_nm} does not yet exist.
If @var{var_nm} already has an attribute @var{att_nm}, there is no
effect.
@item d
@dfn{Delete}.
Delete current @var{var_nm} attribute @var{att_nm}.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect.
If @var{att_nm} is omitted (left blank), then all attributes associated
with the specified variable are automatically deleted.
When @dfn{Delete} mode is selected, the @var{att_type} and @var{att_val}
arguments are superfluous and may be left blank.
@item m
@dfn{Modify}.
Change value of current @var{var_nm} attribute @var{att_nm} to value
@var{att_val}.
If @var{var_nm} does not have an attribute @var{att_nm}, there is no
effect.
@item o
@dfn{Overwrite}.
Write attribute @var{att_nm} with value @var{att_val} to variable
@var{var_nm}, overwriting existing attribute @var{att_nm}, if any.
This is the default mode.
@end table
@html
<a name="att_typ"></a> <!-- http://nco.sf.net/nco.html#att_typ -->
@end html
The value of @var{att_type} is a single character abbreviation
(@code{f}, @code{d}, @code{l}, @code{i}, @code{s}, @code{c},
@code{b}, @code{u}) or a short string standing for one of the twelve
primitive netCDF data types:@*
@table @code
@item f
@dfn{Float}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_FLOAT}.
@item d
@dfn{Double}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_DOUBLE}.
@item i, l
@dfn{Integer} or @dfn{Long}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_INT}.
@item s
@dfn{Short}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_SHORT}.
@item c
@dfn{Char}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_CHAR}.
@item b
@dfn{Byte}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_BYTE}.
@item ub
@dfn{Unsigned Byte}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UBYTE}.
@item us
@dfn{Unsigned Short}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_USHORT}.
@item u, ui, ul
@dfn{Unsigned Int}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UINT}.
@item ll, int64
@dfn{Int64}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_INT64}.
@item ull, uint64
@dfn{Uint64}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_UINT64}.
@item sng
@dfn{String}.
Value(s) specified in @var{att_val} will be stored as netCDF intrinsic
type @code{NC_STRING}.
@end table
@noindent
The specification of @var{att_type} is optional (and is ignored) in
@dfn{Delete} mode.
The value of @var{att_val} is what you want to change attribute
@var{att_nm} to contain.
The specification of @var{att_val} is optional in @dfn{Delete} (and is
ignored) mode.
Attribute values for all types besides @code{NC_CHAR} must have an
attribute length of at least one.
Thus @var{att_val} may be a single value or one-dimensional array of
elements of type @code{att_type}.
If the @var{att_val} is not set or is set to empty space,
and the @var{att_type} is @code{NC_CHAR}, e.g., @code{-a units,T,o,c,""}
or @code{-a units,T,o,c,}, then the corresponding attribute is set to
have zero length.
When specifying an array of values, it is safest to enclose
@var{att_val} in single or double quotes, e.g.,
@code{-a levels,T,o,s,"1,2,3,4"} or
@code{-a levels,T,o,s,'1,2,3,4'}.
The quotes are strictly unnecessary around @var{att_val} except
when @var{att_val} contains characters which would confuse the calling
shell, such as spaces, commas, and wildcard characters.
@cindex Perl
@cindex @acronym{ASCII}
@acronym{NCO} processing of @code{NC_CHAR} attributes is a bit like Perl in
that it attempts to do what you want by default (but this sometimes
causes unexpected results if you want unusual data storage).
@cindex @code{printf()}
@cindex @code{\n} (@acronym{ASCII} LF, linefeed)
@cindex characters, special
@cindex @code{\t} (@acronym{ASCII} HT, horizontal tab)
If the @var{att_type} is @code{NC_CHAR} then the argument is interpreted as a
string and it may contain C-language escape sequences, e.g., @code{\n},
which @acronym{NCO} will interpret before writing anything to disk.
@acronym{NCO} translates valid escape sequences and stores the
appropriate @acronym{ASCII} code instead.
Since two byte escape sequences, e.g., @code{\n}, represent one-byte
@acronym{ASCII} codes, e.g., @acronym{ASCII} 10 (decimal), the stored
string attribute is one byte shorter than the input string length for
each embedded escape sequence.
The most frequently used C-language escape sequences are @code{\n} (for
linefeed) and @code{\t} (for horizontal tab).
These sequences in particular allow convenient editing of formatted text
attributes.
@cindex @code{\a} (@acronym{ASCII} BEL, bell)
@cindex @code{\b} (@acronym{ASCII} BS, backspace)
@cindex @code{\f} (@acronym{ASCII} FF, formfeed)
@cindex @code{\r} (@acronym{ASCII} CR, carriage return)
@cindex @code{\v} (@acronym{ASCII} VT, vertical tab)
@cindex @code{\\} (@acronym{ASCII} \, backslash)
The other valid @acronym{ASCII} codes are @code{\a}, @code{\b}, @code{\f},
@code{\r}, @code{\v}, and @code{\\}.
@xref{ncks netCDF Kitchen Sink}, for more examples of string formatting
(with the @command{ncks} @samp{-s} option) with special characters.
@cindex @code{\'} (protected end quote)
@cindex @code{\"} (protected double quote)
@cindex @code{\?} (protected question mark)
@cindex @code{\\} (protected backslash)
@cindex @code{'} (end quote)
@cindex @code{"} (double quote)
@cindex @code{?} (question mark)
@cindex @code{\} (backslash)
@cindex special characters
@cindex @acronym{ASCII}
Analogous to @code{printf}, other special characters are also allowed by
@command{ncatted} if they are "protected" by a backslash.
The characters @code{"}, @code{'}, @code{?}, and @code{\} may be
input to the shell as @code{\"}, @code{\'}, @code{\?}, and @code{\\}.
@acronym{NCO} simply strips away the leading backslash from these
characters before editing the attribute.
No other characters require protection by a backslash.
Backslashes which precede any other character (e.g., @code{3}, @code{m},
@code{$}, @code{|}, @code{&}, @code{@@}, @code{%}, @code{@{}, and
@code{@}}) will not be filtered and will be included in the attribute.
@cindex strings
@cindex NUL-termination
@cindex NUL
@cindex C language
@cindex @code{0} (NUL)
Note that the NUL character @code{\0} which terminates @w{C language}
strings is assumed and need not be explicitly specified.
If @code{\0} is input, it will not be translated (because it would
terminate the string in an additional location).
Because of these context-sensitive rules, if wish to use an attribute of
type @code{NC_CHAR} to store data, rather than text strings, you should use
@command{ncatted} with care.
@noindent
@html
<a name="xmp_ncatted"></a> <!-- http://nco.sf.net/nco.html#xmp_ncatted -->
@end html
EXAMPLES
Append the string "Data version 2.0.\n" to the global attribute
@code{history}:
@example
ncatted -a history,global,a,c,"Data version 2.0\n" in.nc
@end example
Note the use of embedded @w{C language} @code{printf()}-style escape
sequences.
Change the value of the @code{long_name} attribute for variable @code{T}
from whatever it currently is to "temperature":
@example
ncatted -a long_name,T,o,c,temperature in.nc
@end example
Delete all existing @code{units} attributes:
@example
ncatted -a units,,d,, in.nc
@end example
@noindent
The value of @var{var_nm} was left blank in order to select all
variables in the file.
The values of @var{att_type} and @var{att_val} were left blank because
they are superfluous in @dfn{Delete} mode.
@cindex global attributes
@cindex attributes, global
Delete all attributes associated with the @code{tpt} variable:
@example
ncatted -a ,tpt,d,, in.nc
@end example
@noindent
The value of @var{att_nm} was left blank in order to select all
attributes associated with the variable.
To delete all global attributes, simply replace @code{tpt} with
@code{global} in the above.
@cindex @code{units}
Modify all existing @code{units} attributes to "meter second-1"
@example
ncatted -a units,,m,c,"meter second-1" in.nc
@end example
Overwrite the @code{quanta} attribute of variable
@code{energy} to an array of four integers.
@example
ncatted -O -a quanta,energy,o,s,"010,101,111,121" in.nc
@end example
Demonstrate input of C-language escape sequences (e.g., @code{\n}) and
other special characters (e.g., @code{\"})
@example
ncatted -h -a special,global,o,c,
'\nDouble quote: \"\nTwo consecutive double quotes: \"\"\n
Single quote: Beyond my shell abilities!\nBackslash: \\\n
Two consecutive backslashes: \\\\\nQuestion mark: \?\n' in.nc
@end example
Note that the entire attribute is protected from the shell by single
quotes.
These outer single quotes are necessary for interactive use, but may be
omitted in batch scripts.
@page
@html
<a name="ncbo"></a> <!-- http://nco.sf.net/nco.html#ncbo -->
<a name="ncdiff"></a> <!-- http://nco.sf.net/nco.html#ncdiff -->
<a name="ncadd"></a> <!-- http://nco.sf.net/nco.html#ncadd -->
<a name="ncsub"></a> <!-- http://nco.sf.net/nco.html#ncsub -->
<a name="ncsubtract"></a> <!-- http://nco.sf.net/nco.html#ncsubtract -->
<a name="ncmult"></a> <!-- http://nco.sf.net/nco.html#ncmult -->
<a name="ncmultiply"></a> <!-- http://nco.sf.net/nco.html#ncmultiply -->
<a name="ncdivide"></a> <!-- http://nco.sf.net/nco.html#ncdivide -->
@end html
@node ncbo netCDF Binary Operator, ncea netCDF Ensemble Averager, ncatted netCDF Attribute Editor, Operator Reference Manual
@section @command{ncbo} netCDF Binary Operator
@findex ncbo
@findex ncdiff
@findex ncadd
@findex ncsub
@findex ncsubtract
@findex ncmult
@findex ncmultiply
@findex ncdivide
@cindex binary operations
@cindex addition
@cindex subtraction
@cindex multiplication
@cindex adding data
@cindex subtracting data
@cindex multiplying data
@cindex dividing data
@noindent
SYNTAX
@example
ncbo [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h]
[-L @var{dfl_lvl}] [-l @var{path}] [-O] [-o @var{file_3}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-X ...] [-x] [-y @var{op_typ}]
@var{file_1} @var{file_2} [@var{file_3}]
@end example
@noindent
DESCRIPTION
@command{ncbo} performs binary operations on variables in @var{file_1}
and the corresponding variables (those with the same name) in
@var{file_2} and stores the results in @var{file_3}.
The binary operation operates on the entire files (modulo any excluded
variables).
@xref{Missing Values}, for treatment of missing values.
One of the four standard arithmetic binary operations currently
supported must be selected with the @samp{-y @var{op_typ}} switch (or
long options @samp{--op_typ} or @samp{--operation}).
@cindex @code{add}
@cindex @code{subtract}
@cindex @code{multiply}
@cindex @code{divide}
@cindex @code{+}
@cindex @code{-}
@cindex @code{*}
@cindex @code{/}
@cindex @code{-y @var{op_typ}}
@cindex @code{--operation @var{op_typ}}
@cindex @code{--op_typ @var{op_typ}}
@cindex alternate invocations
The valid binary operations for @command{ncbo}, their definitions,
corresponding values of the @var{op_typ} key, and alternate invocations
are:
@table @dfn
@item Addition
@c Internal operation code: @{nco_op_add}@*
Definition: @var{file_3} = @var{file_1} + @var{file_2}@*
Alternate invocation: @command{ncadd}@*
@var{op_typ} key values: @samp{add}, @samp{+}, @samp{addition}@*
Examples: @samp{ncbo --op_typ=add 1.nc 2.nc 3.nc}, @samp{ncadd 1.nc 2.nc 3.nc}@*
@item Subtraction
Definition: @var{file_3} = @var{file_1} - @var{file_2}@*
Alternate invocations: @command{ncdiff}, @command{ncsub}, @command{ncsubtract}@*
@var{op_typ} key values: @samp{sbt}, @samp{-}, @samp{dff}, @samp{diff}, @samp{sub}, @samp{subtract}, @samp{subtraction}@*
Examples: @samp{ncbo --op_typ=- 1.nc 2.nc 3.nc}, @samp{ncdiff 1.nc 2.nc 3.nc}@*
@item Multiplication
Definition: @var{file_3} = @var{file_1} * @var{file_2}@*
Alternate invocations: @command{ncmult}, @command{ncmultiply}@*
@var{op_typ} key values: @samp{mlt}, @samp{*}, @samp{mult}, @samp{multiply}, @samp{multiplication}@*
Examples: @samp{ncbo --op_typ=mlt 1.nc 2.nc 3.nc}, @samp{ncmult 1.nc 2.nc 3.nc}@*
@item Division
Definition: @var{file_3} = @var{file_1} / @var{file_2}@*
Alternate invocation: @command{ncdivide}@*
@var{op_typ} key values: @samp{dvd}, @samp{/}, @samp{divide}, @samp{division}@*
Examples: @samp{ncbo --op_typ=/ 1.nc 2.nc 3.nc}, @samp{ncdivide 1.nc 2.nc 3.nc}@*
@end table
@noindent
Care should be taken when using the shortest form of key values,
i.e., @samp{+}, @samp{-}, @samp{*}, @w{and @samp{/}}.
Some of these single characters may have special meanings to the shell
@cindex naked characters
@footnote{@w{A naked} (i.e., unprotected or unquoted) @samp{*} is a
wildcard character.
@w{A naked} @samp{-} may confuse the command line parser.
@w{A naked} @samp{+} and @samp{/} are relatively harmless.}.
@cindex Bash shell
Place these characters inside quotes to keep them from being interpreted
(globbed) by the shell
@footnote{The widely used shell Bash correctly interprets all these
special characters even when they are not quoted.
That is, Bash does not prevent @acronym{NCO} from correctly interpreting
the intended arithmetic operation when the following arguments are given
(without quotes) to @command{ncbo}:
@samp{--op_typ=+}, @samp{--op_typ=-}, @samp{--op_typ=*},
and @samp{--op_typ=/}}.
@cindex globbing
@cindex shell
@cindex quotes
For example, the following commands are equivalent
@example
ncbo --op_typ=* 1.nc 2.nc 3.nc # Dangerous (shell may try to glob)
ncbo --op_typ='*' 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ="*" 1.nc 2.nc 3.nc # Safe ('*' protected from shell)
ncbo --op_typ=mlt 1.nc 2.nc 3.nc
ncbo --op_typ=mult 1.nc 2.nc 3.nc
ncbo --op_typ=multiply 1.nc 2.nc 3.nc
ncbo --op_typ=multiplication 1.nc 2.nc 3.nc
ncmult 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmult'
ncmultiply 1.nc 2.nc 3.nc # First do 'ln -s ncbo ncmultiply'
@end example
No particular argument or invocation form is preferred.
Users are encouraged to use the forms which are most intuitive to them.
@cindex @command{alias}
@cindex @command{ln -s}
@cindex symbolic links
Normally, @command{ncbo} will fail unless an operation type is specified
with @samp{-y} (equivalent to @samp{--op_typ}).
You may create exceptions to this rule to suit your particular tastes,
in conformance with your site's policy on @dfn{symbolic links} to
executables (files of a different name point to the actual executable).
For many years, @command{ncdiff} was the main binary file operator.
As a result, many users prefer to continue invoking @command{ncdiff}
rather than memorizing a new command (@samp{ncbo -y @var{sbt}}) which
behaves identically to the original @command{ncdiff} command.
However, from a software maintenance standpoint, maintaining a distinct
executable for each binary operation (e.g., @command{ncadd}) is untenable,
and a single executable, @command{ncbo}, is desirable.
To maintain backward compatibility, therefore, @acronym{NCO}
automatically creates a symbolic link from @command{ncbo} to
@command{ncdiff}.
Thus @command{ncdiff} is called an @dfn{alternate invocation} of
@command{ncbo}.
@command{ncbo} supports many additional alternate invocations which must
be manually activated.
Should users or system adminitrators decide to activate them, the
procedure is simple.
For example, to use @samp{ncadd} instead of @samp{ncbo --op_typ=add},
simply create a symbolic link from @command{ncbo} to @command{ncadd}
@footnote{The command to do this is @samp{ln -s -f ncbo ncadd}}.
The alternatate invocations supported for each operation type are listed
above.
Alternatively, users may always define @samp{ncadd} as an @dfn{alias} to
@samp{ncbo --op_typ=add}
@footnote{The command to do this is @samp{alias ncadd='ncbo --op_typ=add'}}.
It is important to maintain portability in @acronym{NCO} scripts.
Therefore we recommend that site-specfic invocations (e.g.,
@samp{ncadd}) be used only in interactive sessions from the
command-line.
For scripts, we recommend using the full invocation (e.g.,
@samp{ncbo --op_typ=add}).
This ensures portability of scripts between users and sites.
@command{ncbo} operates (e.g., adds) variables in @var{file_2} with the
corresponding variables (those with the same name) in @var{file_1} and
stores the results in @var{file_3}.
@cindex broadcasting variables
Variables in @var{file_2} are @dfn{broadcast} to conform to the
corresponding variable in @var{file_1} if necessary, but the reverse is
not true.
Broadcasting a variable means creating data in non-existing dimensions
from the data in existing dimensions.
For example, a two dimensional variable in @var{file_2} can be
subtracted from a four, three, or two (but not one or zero)
dimensional variable (of the same name) in @code{file_1}.
@cindex anomalies
This functionality allows the user to compute anomalies from the mean.
Note that variables in @var{file_1} are @emph{not} broadcast to conform
to the dimensions in @var{file_2}.
In the future, we will broadcast variables in @var{file_1}, if necessary
to conform to their counterparts in @var{file_2}.
@c TODO #268
@cindex rank
Thus, presently, the number of dimensions, or @dfn{rank}, of any
processed variable in @var{file_1} must be greater than or equal to the
rank of the same variable in @var{file_2}.
Furthermore, the size of all dimensions common to both @var{file_1} and
@var{file_2} must be equal.
When computing anomalies from the mean it is often the case that
@var{file_2} was created by applying an averaging operator to a file
with initially the same dimensions as @var{file_1} (often @var{file_1}
itself).
In these cases, creating @var{file_2} with @command{ncra} rather than
@command{ncwa} will cause the @command{ncbo} operation to fail.
For concreteness say the record dimension in @code{file_1} is
@code{time}.
If @var{file_2} were created by averaging @var{file_1} over the
@code{time} dimension with the @command{ncra} operator rather than with
the @command{ncwa} operator, then @var{file_2} will have a @code{time}
dimension of @w{size 1} rather than having no @code{time} dimension at
all
@cindex degenerate dimension
@cindex @samp{-b}
@footnote{This is because @command{ncra} collapses the record dimension
to a size @w{of 1} (making it a @dfn{degenerate} dimension), but does
not remove it, while, unless @samp{-b} is given, @command{ncwa} removes
all averaged dimensions.
In other words, by default @command{ncra} changes variable size but not
rank, while, @command{ncwa} changes both variable size and rank.}.
In this case the input files to @command{ncbo}, @var{file_1} and
@var{file_2}, will have unequally sized @code{time} dimensions which
causes @command{ncbo} to fail.
To prevent this from occuring, use @command{ncwa} to remove the
@code{time} dimension from @var{file_2}.
See the example below.
@cindex coordinate variable
@cindex @code{NC_BYTE}
@cindex @code{NC_CHAR}
@command{ncbo} never operates on coordinate variables or variables
of type @code{NC_CHAR} or @code{NC_BYTE}.
This ensures that coordinates like (e.g., latitude and longitude) are
physically meaningful in the output file, @var{file_3}.
This behavior is hardcoded.
@cindex @acronym{CF} conventions
@command{ncbo} applies special rules to some
@acronym{CF}-defined (and/or @acronym{NCAR CCSM} or @acronym{NCAR CCM}
fields) such as @code{ORO}.
See @ref{CF Conventions} for a complete description.
Finally, we note that @command{ncflint} (@pxref{ncflint netCDF File
Interpolator}) is designed for file interpolation.
As such, it also performs file subtraction, addition, multiplication,
albeit in a more convoluted way than @command{ncbo}.
@noindent
@html
<a name="xmp_ncbo"></a> <!-- http://nco.sf.net/nco.html#xmp_ncbo -->
<a name="xmp_ncdiff"></a> <!-- http://nco.sf.net/nco.html#xmp_ncdiff -->
@end html
EXAMPLES
Say files @file{85_0112.nc} and @file{86_0112.nc} each contain 12 months
of data.
Compute the change in the monthly averages from 1985 to 1986:
@example
ncbo -op_typ=sub 86_0112.nc 85_0112.nc 86m85_0112.nc
ncdiff 86_0112.nc 85_0112.nc 86m85_0112.nc
@end example
The following examples demonstrate the broadcasting feature of
@command{ncbo}.
Say we wish to compute the monthly anomalies of @code{T} from the yearly
average of @code{T} for the year 1985.
First we create the 1985 average from the monthly data, which is stored
with the record dimension @code{time}.
@example
ncra 85_0112.nc 85.nc
ncwa -O -a time 85.nc 85.nc
@end example
@noindent
The second command, @command{ncwa}, gets rid of the @code{time} dimension
of @w{size 1} that @command{ncra} left in @file{85.nc}.
Now none of the variables in @file{85.nc} has a @code{time} dimension.
@w{A quicker} way to accomplish this is to use @command{ncwa} from the
beginning:
@example
ncwa -a time 85_0112.nc 85.nc
@end example
@noindent
We are now ready to use @command{ncbo} to compute the anomalies for 1985:
@example
ncdiff -v T 85_0112.nc 85.nc t_anm_85_0112.nc
@end example
@noindent
Each of the 12 records in @file{t_anm_85_0112.nc} now contains the
monthly deviation of @code{T} from the annual mean of @code{T} for each
gridpoint.
Say we wish to compute the monthly gridpoint anomalies from the zonal
annual mean.
@w{A @dfn{zonal mean}} is a quantity that has been averaged over the
longitudinal (or @var{x}) direction.
First we use @command{ncwa} to average over longitudinal direction
@code{lon}, creating @file{85_x.nc}, the zonal mean of @file{85.nc}.
Then we use @command{ncbo} to subtract the zonal annual means from the
monthly gridpoint data:
@example
ncwa -a lon 85.nc 85_x.nc
ncdiff 85_0112.nc 85_x.nc tx_anm_85_0112.nc
@end example
@noindent
This examples works assuming @file{85_0112.nc} has dimensions
@code{time} and @code{lon}, and that @file{85_x.nc} has no @code{time}
or @code{lon} dimension.
@html
<a name="csn_anm"></a> <!-- http://nco.sf.net/nco.html#csn_anm -->
@end html
As a final example, say we have five years of monthly data (i.e.,
@w{60 months}) stored in @file{8501_8912.nc} and we wish to create a
file which contains the twelve month seasonal cycle of the average
monthly anomaly from the five-year mean of this data.
The following method is just one permutation of many which will
accomplish the same result.
First use @command{ncwa} to create the five-year mean:
@example
ncwa -a time 8501_8912.nc 8589.nc
@end example
@noindent
Next use @command{ncbo} to create a file containing the difference of
each month's data from the five-year mean:
@example
ncbo 8501_8912.nc 8589.nc t_anm_8501_8912.nc
@end example
@noindent
Now use @command{ncks} to group the five January anomalies together in
one file, and use @command{ncra} to create the average anomaly for all
five Januarys.
These commands are embedded in a shell loop so they are repeated for all
twelve months:
@cindex Bash Shell
@cindex Bourne Shell
@cindex C Shell
@example
for idx in @{1..12@}; do # Bash Shell (version 3.0+)
idx=`printf "%02d" $@{idx@}` # Zero-pad to preserve order
ncks -F -d time,$@{idx@},,12 t_anm_8501_8912.nc foo.$@{idx@}
ncra foo.$@{idx@} t_anm_8589_$@{idx@}.nc
done
for idx in 01 02 03 04 05 06 07 08 09 10 11 12; do # Bourne Shell
ncks -F -d time,$@{idx@},,12 t_anm_8501_8912.nc foo.$@{idx@}
ncra foo.$@{idx@} t_anm_8589_$@{idx@}.nc
done
foreach idx (01 02 03 04 05 06 07 08 09 10 11 12) # C Shell
ncks -F -d time,$@{idx@},,12 t_anm_8501_8912.nc foo.$@{idx@}
ncra foo.$@{idx@} t_anm_8589_$@{idx@}.nc
end
@end example
@noindent
Note that @command{ncra} understands the @code{stride} argument so the
two commands inside the loop may be combined into the single command
@example
ncra -F -d time,$@{idx@},,12 t_anm_8501_8912.nc foo.$@{idx@}
@end example
@noindent
Finally, use @command{ncrcat} to concatenate the @w{12 average} monthly
anomaly files into one twelve-record file which contains the entire
seasonal cycle of the monthly anomalies:
@example
ncrcat t_anm_8589_??.nc t_anm_8589_0112.nc
@end example
@noindent
@page
@html
<a name="ncea"></a> <!-- http://nco.sf.net/nco.html#ncea -->
@end html
@node ncea netCDF Ensemble Averager, ncecat netCDF Ensemble Concatenator, ncbo netCDF Binary Operator, Operator Reference Manual
@section @command{ncea} netCDF Ensemble Averager
@cindex averaging data
@cindex ensemble average
@findex ncea
@noindent
SYNTAX
@example
ncea [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-L @var{dfl_lvl}] [-l @var{path}]
[-n @var{loop}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-X ...] [-x] [-y @var{op_typ}]
[@var{input-files}] [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncea} performs gridpoint averages of variables across an
arbitrary number (an @dfn{ensemble}) of @var{input-files}, with each
file receiving an equal weight in the average.
@cindex ensemble
@command{ncea} averages entire files, and weights each file evenly.
This is distinct from @command{ncra}, which only averages over the
record dimension (e.g., time), and weights each record in the record
dimension evenly,
Variables in the @var{output-file} are the same size as the variable
in each of the @var{input-files}, and all @var{input-files} must be
the same size.
@cindex record dimension
@cindex hyperslab
The only exception is that @command{ncea} allows files to differ in
the record dimension size if the requested record hyperslab
(@pxref{Hyperslabs}) resolves to the same size for all files.
@command{ncea} recomputes the record dimension hyperslab limits for
each input file so that coordinate limits may be used to select equal
length timeseries from unequal length files.
@cindex IPCC
This simplifies analysis of unequal length timeseries from simulation
ensembles (e.g., the IPCC AR4 archive).
@cindex operation types
@command{ncea} @emph{always averages} coordinate variables regardless of
the arithmetic operation type performed on the non-coordinate variables.
(@pxref{Operation Types}).
@cindex record dimension
All dimensions, including the record dimension, are treated identically
and preserved in the @var{output-file}.
@xref{Averaging vs. Concatenating}, for a description of the
distinctions between the various averagers and concatenators.
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncea} will read the list of
@var{input-files} from @code{stdin} if they are not specified
as positional arguments on the command line
(@pxref{Large Numbers of Files}).
The file is the logical unit of organization for the results of many
scientific studies.
Often one wishes to generate a file which is the gridpoint average of
many separate files.
This may be to reduce statistical noise by combining the results of a
large number of experiments, or it may simply be a step in a procedure
whose goal is to compute anomalies from a mean state.
In any case, when one desires to generate a file whose properties are
the mean of all the input files, then @command{ncea} is the operator to
use.
@command{ncea} assumes coordinate variable are properties common to all
of the experiments and so does not average them across files.
Instead, @command{ncea} copies the values of the coordinate variables
from the first input file to the output file.
@noindent
@html
<a name="xmp_ncea"></a> <!-- http://nco.sf.net/nco.html#xmp_ncea -->
@end html
EXAMPLES
Consider a model experiment which generated five realizations of one
year of data, say 1985.
You can imagine that the experimenter slightly perturbs the
initial conditions of the problem before generating each new solution.
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing the ensemble average
(mean) seasonal cycle.
Here the numeric filename suffix denotes the experiment number
(@emph{not} the month):
@example
ncea 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
ncea 85_0[1-5].nc 85.nc
ncea -n 5,2,1 85_01.nc 85.nc
@end example
@noindent
These three commands produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
The output file, @file{85.nc}, is the same size as the inputs files.
It contains 12 months of data (which might or might not be stored in the
record dimension, depending on the input files), but each value in the
output file is the average of the five values in the input files.
In the previous example, the user could have obtained the ensemble
average values in a particular spatio-temporal region by adding a
hyperslab argument to the command, e.g.,
@example
ncea -d time,0,2 -d lat,-23.5,23.5 85_??.nc 85.nc
@end example
@noindent
In this case the output file would contain only three slices of data in
the @var{time} dimension.
These three slices are the average of the first three slices from the
input files.
Additionally, only data inside the tropics is included.
@page
@html
<a name="ncecat"></a> <!-- http://nco.sf.net/nco.html#ncecat -->
@end html
@node ncecat netCDF Ensemble Concatenator, ncflint netCDF File Interpolator, ncea netCDF Ensemble Averager, Operator Reference Manual
@section @command{ncecat} netCDF Ensemble Concatenator
@cindex concatenation
@cindex ensemble concatenation
@findex ncecat
@noindent
SYNTAX
@example
ncecat [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-L @var{dfl_lvl}] [-l @var{path}]
[-M] [-n @var{loop}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-u @var{ulm_nm}] [-v @var{var}[,@dots{}]] [-X ...] [-x]
[@var{input-files}] [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncecat} concatenates an arbitrary number of input files into a
single output file.
The @var{input-files} are stored consecutively as records in
@var{output-file}.
Each variable in each input file becomes one record in the same variable
in the output file.
All @var{input-files} must contain all extracted variables (or else
there would be "gaps" in the output file).
A new record dimension is the glue which binds the input file data
together.
The new record dimension name is, by default, ``record''.
@cindex unlimited dimension
@cindex record dimension
@cindex @samp{-u @var{ulm_nm}}
@cindex @samp{--ulm_nm @var{ulm_nm}}
@cindex @samp{--rcd_nm @var{ulm_nm}}
Its name can be specified with the @samp{-u @var{ulm_nm}} short
option (or the @samp{--ulm_nm} or @samp{rcd_nm} long options).
Each extracted variable must be constant in size and rank across all
@var{input-files}.
@cindex record dimension
@cindex hyperslab
The only exception is that @command{ncecat} allows files to differ in
the record dimension size if the requested record hyperslab
(@pxref{Hyperslabs}) resolves to the same size for all files.
@cindex IPCC
This allows easier gluing/averaging of unequal length timeseries from
simulation ensembles (e.g., the IPCC AR4 archive).
Thus, the @var{output-file} size is the sum of the sizes of the
extracted variables in the input files.
@xref{Averaging vs. Concatenating}, for a description of the
distinctions between the various averagers and concatenators.
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncecat} will read the list of
@var{input-files} from @code{stdin} if they are not specified
as positional arguments on the command line
(@pxref{Large Numbers of Files}).
@cindex @code{-M}
@cindex @code{--glb_mtd_spr}
@cindex metadata, global
Turn off global metadata copying.
By default all @acronym{NCO} operators copy the global metadata of the
first input file into @var{output-file}.
This helps preserve the provenance of the output data.
However, the use of metadata is burgeoning and is not uncommon to
encounter files with excessive amounts of extraneous metadata.
Extracting small bits of data from such files leads to output files
which are much larger than necessary due to the automatically copied
metadata.
@command{ncecat} supports turning off the default copying of global
metadata via the @samp{-M} switch (or its long option equivalents,
@samp{--glb_mtd_spr} and @samp{--global_metadata_suppress}).
@cindex climate model
Consider five realizations, @file{85a.nc}, @file{85b.nc},
@w{@dots{} @file{85e.nc}} of 1985 predictions from the same climate
model.
Then @code{ncecat 85?.nc 85_ens.nc} glues the individual realizations
together into the single file, @file{85_ens.nc}.
If an input variable was dimensioned [@code{lat},@code{lon}], it will
by default have dimensions [@code{record},@code{lat},@code{lon}] in
the output file.
@w{A restriction} of @command{ncecat} is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain
data on different realizations of the same variables.
@findex ncpdq
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
Concatenating a variable packed with different scales multiple datasets
is beyond the capabilities of @command{ncecat} (and @command{ncrcat},
the other concatenator (@ref{Concatenation}).
@command{ncecat} does not unpack data, it simply @emph{copies} the data
from the @var{input-files}, and the metadata from the @emph{first}
@var{input-file}, to the @var{output-file}.
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., @code{scale_factor} and
@code{add_offset}) for a given variable across @emph{all} input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
@var{input-files} requires three steps:
First, unpack the data using @command{ncpdq}.
Second, concatenate the unpacked data using @command{ncecat},
Third, re-pack the result with @command{ncpdq}.
@noindent
@html
<a name="xmp_ncecat"></a> <!-- http://nco.sf.net/nco.html#xmp_ncecat -->
@end html
EXAMPLES
Consider a model experiment which generated five realizations of one
year of data, say 1985.
You can imagine that the experimenter slightly perturbs the
initial conditions of the problem before generating each new solution.
Assume each file contains all twelve months (a seasonal cycle) of data
and we want to produce a single file containing all the seasonal
cycles.
Here the numeric filename suffix denotes the experiment number
(@emph{not} the month):
@example
ncecat 85_01.nc 85_02.nc 85_03.nc 85_04.nc 85_05.nc 85.nc
ncecat 85_0[1-5].nc 85.nc
ncecat -n 5,2,1 85_01.nc 85.nc
@end example
@noindent
These three commands produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
The output file, @file{85.nc}, is five times the size as a single
@var{input-file}.
It contains @w{60 months} of data.
@html
<a name="ncecat_rnm"></a> <!-- http://nco.sf.net/nco.html#ncecat_rnm -->
@end html
One often prefers that the (new) record dimension have a more
descriptive, context-based name than simply ``record''.
This is easily accomplished with the @samp{-u @var{ulm_nm}} switch:
@example
ncecat -u realization 85_0[1-5].nc 85.nc
@end example
@noindent
Users are more likely to understand the data processing history when
such descriptive coordinates are used.
@html
<a name="dmn_rcd_rm"></a> <!-- http://nco.sf.net/nco.html#dmn_rcd_rm -->
@end html
@cindex record dimension
Consider a file with an existing record dimension named @code{time}.
and suppose the user wishes to convert @code{time} from a record
dimension to a non-record dimension.
This may be useful, for example, when the user has another use for the
record variable.
The procedure is to use @command{ncecat} followed by @command{ncwa}
@cindex degenerate dimension
@example
ncecat in.nc out.nc # Convert time to non-record dimension
ncwa -a record in.nc out.nc # Remove new degenerate record dimension
@end example
@noindent
The second step removes the degenerate record dimension.
See @ref{ncpdq netCDF Permute Dimensions Quickly} for other methods
of changing variable dimensionality, including the record dimension.
@page
@html
<a name="ncflint"></a> <!-- http://nco.sf.net/nco.html#ncflint -->
@end html
@node ncflint netCDF File Interpolator, ncks netCDF Kitchen Sink, ncecat netCDF Ensemble Concatenator, Operator Reference Manual
@section @command{ncflint} netCDF File Interpolator
@cindex interpolation
@cindex adding data
@cindex multiplying data
@cindex addition
@findex ncflint
@noindent
SYNTAX
@example
ncflint [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-i @var{var},@var{val3}]
[-L @var{dfl_lvl}] [-l @var{path}] [-O] [-o @var{file_3}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-w @var{wgt1}[,@var{wgt2}]] [-X ...] [-x]
@var{file_1} @var{file_2} [@var{file_3}]
@end example
@noindent
DESCRIPTION
@command{ncflint} creates an output file that is a linear combination of
the input files.
This linear combination is a weighted average, a normalized weighted
average, or an interpolation of the input files.
Coordinate variables are not acted upon in any case, they are simply
copied from @var{file_1}.
There are two conceptually distinct methods of using @command{ncflint}.
The first method is to specify the weight each input file contributes to
the output file.
In this method, the value @var{val3} of a variable in the output file
@var{file_3} is determined from its values @var{val1} and @var{val2} in
the two input files according to
@set flg
@tex
$val3 = wgt1 \times val1 + wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}
@clear flg
@end ifset
.
Here at least @var{wgt1}, and, optionally, @var{wgt2}, are specified on
the command line with the @samp{-w} (or @samp{--weight} or
@samp{--wgt_var}) switch.
@cindex @code{-w @var{wgt1}[,@var{wgt2}]}
@cindex @code{--weight @var{wgt1}[,@var{wgt2}]}
@cindex @code{--wgt_var @var{wgt1}[,@var{wgt2}]}
If only @var{wgt1} is specified then @var{wgt2} is automatically
computed as @math{@var{wgt2} = 1 @minus{} @var{wgt1}}.
Note that weights larger @w{than 1} are allowed.
Thus it is possible to specify @math{@var{wgt1} = 2} and
@math{@var{wgt2} = -3}.
One can use this functionality to multiply all the values in a given
file by a constant.
The second method of using @command{ncflint} is to specify the
interpolation option @w{with @samp{-i}} (or with the @samp{--ntp} or
@samp{--interpolate} long options).
This is really the inverse of the first method in the following sense.
When the user specifies the weights directly, @command{ncflint} has no
work to do besides multiplying the input values by their respective
weights and adding the results together to produce the output values.
It makes sense to use this when the weights are known
@emph{@w{a priori}}.
@cindex arrival value
Another class of problems has the @dfn{arrival value} (i.e., @var{val3})
of a particular variable @var{var} known @emph{@w{a priori}}.
In this case, the implied weights can always be inferred by examining
the values of @var{var} in the input files.
This results in one equation in two unknowns, @var{wgt1} and @var{wgt2}:
@set flg
@tex
$val3 = wgt1 \times val1 + wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt1}*@var{val1} + @var{wgt2}*@var{val2}
@clear flg
@end ifset
.
Unique determination of the weights requires imposing the additional
constraint of normalization on the weights:
@math{@var{wgt1} + @var{wgt2} = 1}.
Thus, to use the interpolation option, the user specifies @var{var}
and @var{val3} with the @samp{-i} option.
@command{ncflint} then computes @var{wgt1} and @var{wgt2}, and uses these
weights on all variables to generate the output file.
Although @var{var} may have any number of dimensions in the input
files, it must represent a single, scalar value.
@cindex degenerate dimension
Thus any dimensions associated with @var{var} must be @dfn{degenerate},
i.e., of size one.
If neither @samp{-i} nor @samp{-w} is specified on the command line,
@command{ncflint} defaults to weighting each input file equally in the
output file.
This is equivalent to specifying @samp{-w 0.5} or @samp{-w 0.5,0.5}.
Attempting to specify both @samp{-i} and @samp{-w} methods in the same
command is an error.
@command{ncflint} does not interpolate variables of type @code{NC_CHAR}
and @code{NC_BYTE}.
This behavior is hardcoded.
@cindex missing values
@cindex @code{_FillValue}
Depending on your intuition, @command{ncflint} may treat missing values
unexpectedly.
Consider a point where the value in one input file, say @var{val1},
equals the missing value @var{mss_val_1} and, at the same point,
the corresponding value in the other input file @var{val2} is not
misssing (i.e., does not equal @var{mss_val_2}).
There are three plausible answers, and this creates ambiguity.
@w{Option one} is to set @math{@var{val3} = @var{mss_val_1}}.
The rationale is that @command{ncflint} is, at heart, an interpolator
and interpolation involving a missing value is intrinsically undefined.
@command{ncflint} currently implements this behavior since it is the
most conservative and least likely to lead to misinterpretation.
@w{Option two} is to output the weighted valid data point, i.e.,
@set flg
@tex
$val3 = wgt2 \times val2$
@clear flg
@end tex
@ifinfo
@math{@var{val3} = @var{wgt2}*@var{val2}}
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{val3} = @var{wgt2}*@var{val2}
@clear flg
@end ifset
.
The rationale for this behavior is that interpolation is really a
weighted average of known points, so @command{ncflint} should weight the
valid point.
@w{Option three} is to return the @emph{unweighted} valid point, i.e.,
@math{@var{val3} = @var{val2}}.
This behavior would appeal to those who use @command{ncflint} to
estimate data using the closest available data.
When a point is not bracketed by valid data on both sides, it is better
to return the known datum than no datum at all.
The current implementation uses the first approach, @w{Option one}.
If you have strong opinions on this matter, let us know, since we are
willing to implement the other approaches as options if there is enough
interest.
@noindent
@html
<a name="xmp_ncflint"></a> <!-- http://nco.sf.net/nco.html#xmp_ncflint -->
@end html
EXAMPLES
Although it has other uses, the interpolation feature was designed
to interpolate @var{file_3} to a time between existing files.
Consider input files @file{85.nc} and @file{87.nc} containing variables
describing the state of a physical system at times @math{@code{time} =
85} and @math{@code{time} = 87}.
Assume each file contains its timestamp in the scalar variable
@code{time}.
Then, to linearly interpolate to a file @file{86.nc} which describes
the state of the system at time at @code{time} = 86, we would use
@example
ncflint -i time,86 85.nc 87.nc 86.nc
@end example
Say you have observational data covering January and April 1985 in two
files named @file{85_01.nc} and @file{85_04.nc}, respectively.
Then you can estimate the values for February and March by interpolating
the existing data as follows.
Combine @file{85_01.nc} and @file{85_04.nc} in a 2:1 ratio to make
@file{85_02.nc}:
@example
ncflint -w 0.667 85_01.nc 85_04.nc 85_02.nc
ncflint -w 0.667,0.333 85_01.nc 85_04.nc 85_02.nc
@end example
Multiply @file{85.nc} @w{by 3} and @w{by @minus{}2} and add them
together to make @file{tst.nc}:
@example
ncflint -w 3,-2 85.nc 85.nc tst.nc
@end example
@noindent
@cindex null operation
This is an example of a null operation, so @file{tst.nc} should be
identical (within machine precision) to @file{85.nc}.
Add @file{85.nc} to @file{86.nc} to obtain @file{85p86.nc},
then subtract @file{86.nc} from @file{85.nc} to obtain @file{85m86.nc}
@example
ncflint -w 1,1 85.nc 86.nc 85p86.nc
ncflint -w 1,-1 85.nc 86.nc 85m86.nc
ncdiff 85.nc 86.nc 85m86.nc
@end example
@noindent
Thus @command{ncflint} can be used to mimic some @command{ncbo}
operations.
@cindex broadcasting variables
However this is not a good idea in practice because @command{ncflint}
does not broadcast (@pxref{ncbo netCDF Binary Operator}) conforming
variables during arithmetic.
Thus the final two commands would produce identical results except that
@command{ncflint} would fail if any variables needed to be broadcast.
@cindex @code{units}
Rescale the dimensional units of the surface pressure @code{prs_sfc}
from Pascals to hectopascals (millibars)
@example
ncflint -C -v prs_sfc -w 0.01,0.0 in.nc in.nc out.nc
ncatted -a units,prs_sfc,o,c,millibar out.nc
@end example
@noindent
@page
@html
<a name="ncks"></a> <!-- http://nco.sf.net/nco.html#ncks -->
@end html
@node ncks netCDF Kitchen Sink, ncpdq netCDF Permute Dimensions Quickly, ncflint netCDF File Interpolator, Operator Reference Manual
@section @command{ncks} netCDF Kitchen Sink
@cindex kitchen sink
@cindex printing files contents
@cindex printing variables
@findex ncks
@noindent
SYNTAX
@example
ncks [-3] [-4] [-A] [-a] [-B] [-b @var{binary-file}] [-C] [-c]
[-D @var{dbg}] [-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]
[-F] [-H] [-h] [--hdr_pad @var{nbr}] [-L @var{dfl_lvl}] [-l @var{path}] [-M] [-m]
[-O] [-o @var{output-file}] [-P] [-p @var{path}] [-Q] [-q] [-R] [-r]
[-s @var{format}] [-u] [-v @var{var}[,@dots{}]] [-X ...] [-x]
@var{input-file} [[@var{output-file}]]
@end example
@noindent
DESCRIPTION
@cindex @command{ncextr}
@command{ncks} combines selected features of @command{ncdump},
@command{ncextr}, and the nccut and ncpaste specifications into one
versatile utility.
@command{ncks} extracts a subset of the data from @var{input-file} and
prints it as @acronym{ASCII} text to @file{stdout}, writes it in
flat binary format to @file{binary-file}, and writes (or pastes) it in
netCDF format to @var{output-file}.
@command{ncks} will print netCDF data in @acronym{ASCII} format to
@code{stdout}, like @command{ncdump}, but with these differences:
@command{ncks} prints data in a tabular format intended to be easy to
search for the data you want, one datum per screen line, with all
dimension subscripts and coordinate values (if any) preceding the datum.
Option @samp{-s} (or long options @samp{--sng_fmt} and @samp{--string})
lets the user format the data using C-style format strings.
Options @samp{-a}, @samp{-F} , @samp{-H}, @samp{-M}, @samp{-m},
@samp{-P}, @samp{-Q}, @samp{-q}, @samp{-s}, and @samp{-u} (and their
long option counterparts) control the formatted appearance of the data.
@cindex global attributes
@cindex attributes, global
@command{ncks} extracts (and optionally creates a new netCDF file
comprised of) only selected variables from the input file
(similar to the old @command{ncextr} specification).
Only variables and coordinates may be specifically included or
excluded---all global attributes and any attribute associated with an
extracted variable are copied to the screen and/or output netCDF file.
Options @samp{-c}, @samp{-C}, @samp{-v}, and @samp{-x} (and their long
option synonyms) control which variables are extracted.
@command{ncks} extracts hyperslabs from the specified variables
(@command{ncks} implements the original @command{nccut} specification).
Option @samp{-d} controls the hyperslab specification.
Input dimensions that are not associated with any output variable do
not appear in the output netCDF.
This feature removes superfluous dimensions from netCDF files.
@cindex appending data
@cindex merging files
@command{ncks} will append variables and attributes from the
@var{input-file} to @var{output-file} if @var{output-file} is a
pre-existing netCDF file whose relevant dimensions conform to dimension
sizes of @var{input-file}.
The append features of @command{ncks} are intended to provide a
rudimentary means of adding data from one netCDF file to another,
conforming, netCDF file.
If naming conflicts exist between the two files, data in
@var{output-file} is usually overwritten by the corresponding data from
@var{input-file}.
Thus, when appending, the user should backup @var{output-file} in case
valuable data are inadvertantly overwritten.
If @var{output-file} exists, the user will be queried whether to
@dfn{overwrite}, @dfn{append}, or @dfn{exit} the @command{ncks} call
completely.
Choosing @dfn{overwrite} destroys the existing @var{output-file} and
create an entirely new one from the output of the @command{ncks} call.
Append has differing effects depending on the uniqueness of the
variables and attributes output by @command{ncks}: If a variable or
attribute extracted from @var{input-file} does not have a name conflict
with the members of @var{output-file} then it will be added to
@var{output-file} without overwriting any of the existing contents of
@var{output-file}.
In this case the relevant dimensions must agree (conform) between the
two files; new dimensions are created in @var{output-file} as required.
@cindex global attributes
@cindex attributes, global
When a name conflict occurs, a global attribute from @var{input-file}
will overwrite the corresponding global attribute from
@var{output-file}.
If the name conflict occurs for a non-record variable, then the
dimensions and type of the variable (and of its coordinate dimensions,
if any) must agree (conform) in both files.
Then the variable values (and any coordinate dimension values)
from @var{input-file} will overwrite the corresponding variable values
(and coordinate dimension values, if any) in @var{output-file}
@footnote{
Those familiar with netCDF mechanics might wish to know what is
happening here: @command{ncks} does not attempt to redefine the variable
in @var{output-file} to match its definition in @var{input-file},
@command{ncks} merely copies the values of the variable and its
coordinate dimensions, if any, from @var{input-file} to
@var{output-file}.
}.
Since there can only be one record dimension in a file, the record
dimension must have the same name (but not necessarily the same size) in
both files if a record dimension variable is to be appended.
If the record dimensions are of differing sizes, the record dimension of
@var{output-file} will become the greater of the two record dimension
sizes, the record variable from @var{input-file} will overwrite any
counterpart in @var{output-file} and fill values will be written to any
gaps left in the rest of the record variables (I think).
In all cases variable attributes in @var{output-file} are superseded by
attributes of the same name from @var{input-file}, and left alone if
there is no name conflict.
Some users may wish to avoid interactive @command{ncks} queries about
whether to overwrite existing data.
For example, batch scripts will fail if @command{ncks} does not receive
responses to its queries.
Options @samp{-O} and @samp{-A} are available to force overwriting
existing files and variables, respectively.
@unnumberedsubsec Options specific to @command{ncks}
The following list provides a short summary of the features unique to
@command{ncks}.
Features common to many operators are described in
@ref{Common features}.
@table @samp
@cindex alphabetization
@cindex sort alphabetically
@cindex @code{-a}
@cindex @code{--abc}
@cindex @code{--alphabetize}
@item -a
Do not alphabetize extracted fields.
By default, the specified output variables are extracted, printed, and
written to disk in alphabetical order.
This tends to make long output lists easier to search for particular
variables.
Specifying @code{-a} results in the variables being extracted, printed,
and written to disk in the order in which they were saved in the input
file.
Thus @code{-a} retains the original ordering of the variables.
Also @samp{--abc} and @samp{--alphabetize}.
@cindex binary format
@cindex @code{-B}
@cindex @code{--bnr}
@cindex @code{--binary}
@item -B @file{file}
Activate native machine binary output writing to the default binary
file, @file{ncks.bnr}.
The @code{-B} switch is redundant when the @w{@code{-b} @file{file}}
option is specified, and native binary output will be directed to the
binary file @file{file}.
Also @samp{--bnr} and @samp{--binary}.
Writing packed variables in binary format is not supported.
@cindex @code{-b}
@cindex @code{--fl_bnr}
@item -b @file{file}
Activate native machine binary output writing to binary file
@file{file}.
Also @samp{--fl_bnr} and @samp{--binary-file}.
Writing packed variables in binary format is not supported.
@cindex stride
@item -d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]]
Add @dfn{stride} argument to hyperslabber.
For a complete description of the @var{stride} argument, @xref{Stride}.
@html
<a name="prn"></a> <!-- http://nco.sf.net/nco.html#prn -->
@end html
@cindex @code{-H}
@cindex @code{--hieronymus}
@item -H
Print data to screen.
Also activated using @samp{--print} or @samp{--prn}.
By default @command{ncks} prints all metadata and data to screen if
no netCDF output file is specified.
Use @samp{-H} to print data to screen if a netCDF output is specified,
or to restrict printing to data (no metadata) when no netCDF output is
specified.
Unless otherwise specified (with @code{-s}), each element of the data
hyperslab prints on a separate line containing the names, indices,
and, values, if any, of all of the variables dimensions.
The dimension and variable indices refer to the location of the
corresponding data element with respect to the variable as stored on
disk (i.e., not the hyperslab).
@example
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
@end example
Printing the same variable with the @samp{-F} option shows the same
variable indexed with Fortran conventions
@example
% ncks -F -C -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1
lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2
...
@end example
Printing a hyperslab does not affect the variable or dimension indices
since these indices are relative to the full variable (as stored in the
input file), and the input file has not changed.
However, if the hypserslab is saved to an output file and those values
are printed, the indices will change:
@c fxm: replace with new MSA output style
@example
% ncks -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
...
lat[1]=90 lev[2]=1000 lon[0]=0 three_dmn_var[20]=20
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
% ncks -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=0 three_dmn_var[0]=20
lat[0]=90 lev[0]=1000 lon[1]=90 three_dmn_var[1]=21
lat[0]=90 lev[0]=1000 lon[2]=180 three_dmn_var[2]=22
lat[0]=90 lev[0]=1000 lon[3]=270 three_dmn_var[3]=23
@end example
@cindex @code{-M}
@cindex @code{--Mtd}
@cindex @code{--Metadata}
@cindex metadata, global
@item -M
Print to screen the global metadata describing the file.
This includes file summary information and global attributes.
Also @samp{--Mtd} and @samp{--Metadata}.
By default @command{ncks} prints global metadata to screen if no netCDF
output file and no variable extraction list is specified (with @samp{-v}).
Use @samp{-M} to print global metadata to screen if a netCDF output is
specified, or if a variable extraction list is specified (with @samp{-v}).
The various combinations of printing switches can be confusing.
In an attempt to anticipate what most users want to do, @command{ncks}
uses context-sensitive defaults for printing.
Our goal is to minimize the use of switches required to accomplish the
common operations.
We assume that users creating a new file or overwriting (e.g., with
@samp{-O}) an existing file usually wish to copy all global and
variable-specific attributes to the new file.
In contrast, we assume that users appending (e.g., with @samp{-A} an
explicit variable list from one file to another usually wish to copy
only the variable-specific attributes to the output file.
The switches @samp{-H}, @samp{-M}, and @samp{-m} switches are
implemented as toggles which reverse the default behavior.
The most confusing aspect of this is that @samp{-M} inhibits copying
global metadata in overwrite mode and causes copying of global
metadata in append mode.
@example
ncks -O in.nc out.nc # Copy VAs and GAs
ncks -O -v one in.nc out.nc # Copy VAs and GAs
ncks -O -M -v one in.nc out.nc # Copy VAs not GAs
ncks -O -m -v one in.nc out.nc # Copy GAs not VAs
ncks -O -M -m -v one in.nc out.nc # Copy only data (no atts)
ncks -A in.nc out.nc # Append VAs and GAs
ncks -A -v one in.nc out.nc # Append VAs not GAs
ncks -A -M -v one in.nc out.nc # Append VAs and GAs
ncks -A -m -v one in.nc out.nc # Append only data (no atts)
ncks -A -M -m -v one in.nc out.nc # Append GAs not VAs
@end example
where @code{VAs} and @code{GAs} denote variable and global attributes,
respectively.
@cindex @command{ncdump}
@cindex @code{-m}
@cindex @code{--mtd}
@cindex @code{--metadata}
@cindex metadata
@item -m
Print variable metadata to screen (similar to @kbd{ncdump -h}).
This displays all metadata pertaining to each variable, one variable
at a time.
@cindex compression
@cindex deflation
This includes information on the compression level, if any
@xref{Deflation}.
Also activated using @samp{--mtd} and @samp{--metadata}.
The @command{ncks} default behavior is to print variable metadata to
screen if no netCDF output file is specified.
Use @samp{-m} to print variable metadata to screen if a netCDF output is
specified.
@cindex @code{-P}
@cindex @code{--print}
@cindex @code{--prn}
@item -P
Print data, metadata, and units to screen.
The @samp{-P} switch is a convenience abbreviation for
@samp{-C -H -M -m -u}.
Also activated using @samp{--print} or @samp{--prn}.
This set of switches is useful for exploring file contents.
@cindex @code{-Q}
@item -Q
Toggle printing of dimension indices and coordinate values when printing
arrays.
Each variable's name appears flush left in the output.
This helps locate specific variables in lists with many variables and
different dimensions.
@cindex @code{-q}
@cindex @code{--quiet}
@cindex quiet
@item -q
Turn off all printing to screen.
This overrides the setting of all print-related switches, equivalent to
@kbd{-H -M -m} when in single-file printing mode.
When invoked with @code{-R} (@pxref{Retaining Retrieved Files}), @command{ncks}
automatically sets @code{-q}.
This allows @command{ncks} to retrieve remote files without
automatically trying to print them.
Also @samp{--quiet}.
@cindex @code{-s}
@cindex @code{--string}
@cindex @code{--sng_fmt}
@cindex @code{printf()}
@cindex C language
@item -s @var{format}
String format for text output.
Accepts @w{C language} escape sequences and @code{printf()} formats.
Also @samp{--string} and @samp{--sng_fmt}.
@cindex @code{-u}
@cindex @code{--units}
@item -u
Toggle the printing of a variable's @code{units} attribute, if any,
with its values.
Also @samp{--units}.
@end table
@noindent
@html
<a name="xmp_ncks"></a> <!-- http://nco.sf.net/nco.html#xmp_ncks -->
@end html
EXAMPLES
View all data in netCDF @file{in.nc}, printed with Fortran indexing
conventions:
@example
ncks -F in.nc
@end example
Copy the netCDF file @file{in.nc} to file @file{out.nc}.
@example
ncks in.nc out.nc
@end example
Now the file @file{out.nc} contains all the data from @file{in.nc}.
There are, however, two differences between @file{in.nc} and
@file{out.nc}.
@cindex @code{history}
First, the @code{history} global attribute (@pxref{History Attribute})
will contain the command used to create @file{out.nc}.
@cindex alphabetize output
@cindex sort alphabetically
@cindex @code{-a}
Second, the variables in @file{out.nc} will be defined in alphabetical
order.
Of course the internal storage of variable in a netCDF file should be
transparent to the user, but there are cases when alphabetizing a file
is useful (see description of @code{-a} switch).
@html
<a name="xmp_att_glb_cpy"></a> <!-- http://nco.sf.net/nco.html#xmp_att_glb_cpy -->
@end html
@cindex global attributes
@cindex attributes, global
@cindex subsetting
@cindex exclusion
@cindex extraction
@cindex @code{-v @var{var}}
@cindex @code{--variable @var{var}}
@cindex @code{-x}
@cindex @code{--exclude}
@cindex @code{--xcl}
Copy all global attributes (and no variables) from @file{in.nc} to
@file{out.nc}:
@example
ncks -A -x ~/nco/data/in.nc ~/out.nc
@end example
The @samp{-x} switch tells NCO to use the complement of the extraction
list (@pxref{Subsetting Variables}).
Since no extraction list is explicitly specified (with @samp{-v}),
the default is to extract all variables.
The complement of all variables is no variables.
@cindex @code{-A}
@cindex @code{--apn}
@cindex @code{--append}
@cindex appending to files
Without any variables to extract, the append (@samp{-A}) command
(@pxref{Appending Variables}) has only to extract and copy
(i.e., append) global attributes to the output file.
@cindex @code{printf()}
@cindex @code{\n} (linefeed)
@cindex @code{\t} (horizontal tab)
Print variable @code{three_dmn_var} from file @file{in.nc} with
default notations.
Next print @code{three_dmn_var} as an un-annotated text column.
Then print @code{three_dmn_var} signed with very high precision.
Finally, print @code{three_dmn_var} as a comma-separated list.
@example
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
...
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
% ncks -s '%f\n' -C -v three_dmn_var in.nc
0.000000
1.000000
...
23.000000
% ncks -s '%+16.10f\n' -C -v three_dmn_var in.nc
+0.0000000000
+1.0000000000
...
+23.0000000000
% ncks -s '%f, ' -C -v three_dmn_var in.nc
0.000000, 1.000000, ..., 23.000000,
@end example
@noindent
The second and third options are useful when pasting data into text
files like reports or papers.
@xref{ncatted netCDF Attribute Editor}, for more details on string
formatting and special characters.
One dimensional arrays of characters stored as netCDF variables are
automatically printed as strings, whether or not they are
NUL-terminated, e.g.,
@example
ncks -v fl_nm in.nc
@end example
@noindent
The @code{%c} formatting code is useful for printing
multidimensional arrays of characters representing fixed length strings
@example
ncks -s '%c' -v fl_nm_arr in.nc
@end example
@noindent
@cindex @code{core dump}
Using the @code{%s} format code on strings which are not NUL-terminated
(and thus not technically strings) is likely to result in a core dump.
@cindex subsetting
@cindex exclusion
@cindex extraction
Create netCDF @file{out.nc} containing all variables, and any associated
coordinates, except variable @code{time}, from netCDF @file{in.nc}:
@example
ncks -x -v time in.nc out.nc
@end example
Extract variables @code{time} and @code{pressure} from netCDF
@file{in.nc}.
If @file{out.nc} does not exist it will be created.
Otherwise the you will be prompted whether to append to or to
overwrite @file{out.nc}:
@example
ncks -v time,pressure in.nc out.nc
ncks -C -v time,pressure in.nc out.nc
@end example
@noindent
The first version of the command creates an @file{out.nc} which contains
@code{time}, @code{pressure}, and any coordinate variables associated
with @var{pressure}.
The @file{out.nc} from the second version is guaranteed to contain only
two variables @code{time} and @code{pressure}.
Create netCDF @file{out.nc} containing all variables from file
@file{in.nc}.
Restrict the dimensions of these variables to a hyperslab.
Print (with @code{-H}) the hyperslabs to the screen for good measure.
The specified hyperslab is: the fifth value in dimension @code{time};
the
half-open range @math{@var{lat} > 0.} in coordinate @code{lat}; the
half-open range @math{@var{lon} < 330.} in coordinate @code{lon}; the
closed interval @math{0.3 < @var{band} < 0.5} in coordinate @code{band};
and cross-section closest to 1000.@: in coordinate @code{lev}.
Note that limits applied to coordinate values are specified with a
decimal point, and limits applied to dimension indices do not have a
decimal point @xref{Hyperslabs}.
@example
ncks -H -d time,5 -d lat,,0.0 -d lon,330.0, -d band,0.3,0.5
-d lev,1000.0 in.nc out.nc
@end example
@cindex wrapped coordinates
Assume the domain of the monotonically increasing longitude coordinate
@code{lon} is @math{0 < @var{lon} < 360}.
Here, @code{lon} is an example of a wrapped coordinate.
@command{ncks} will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as @var{min} and
the easternmost longitude as @var{max}, as follows:
@example
ncks -d lon,260.0,45.0 in.nc out.nc
@end example
For more details @xref{Wrapped Coordinates}.
@page
@html
<a name="ncpdq"></a> <!-- http://nco.sf.net/nco.html#ncpdq -->
<a name="ncpack"></a> <!-- http://nco.sf.net/nco.html#ncpack -->
<a name="ncunpack"></a> <!-- http://nco.sf.net/nco.html#ncunpack -->
@end html
@node ncpdq netCDF Permute Dimensions Quickly, ncra netCDF Record Averager, ncks netCDF Kitchen Sink, Operator Reference Manual
@section @command{ncpdq} netCDF Permute Dimensions Quickly
@findex ncpdq
@findex ncpack
@findex ncunpack
@cindex reshape variables
@cindex permute dimensions
@cindex reverse dimensions
@cindex re-order dimensions
@cindex re-dimension
@cindex packing
@cindex unpacking
@noindent
SYNTAX
@example
ncpdq [-3] [-4] [-A] [-a [-]@var{dim}[,@dots{}]] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-L @var{dfl_lvl}] [-l @var{path}]
[-M @var{pck_map}] [-O] [-o @var{output-file}] [-P @var{pck_plc}] [-p @var{path}]
[-R] [-r] [-t @var{thr_nbr}] [-U] [-v @var{var}[,@dots{}]] [-X ...] [-x]
@var{input-file} [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncpdq} performs one of two distinct functions, packing or
dimension permutation, but not both, when invoked.
@command{ncpdq} is optimized to perform these actions in a parallel
fashion with a minimum of time and memory.
The @dfn{pdq} may stand for ``Permute Dimensions Quickly'',
``Pack Data Quietly'', ``Pillory Dan Quayle'', or other silly uses.
@cindex @code{add_offset}
@cindex @code{scale_factor}
@cindex @command{ncap2}
@cindex packing policy
@unnumberedsubsec Packing and Unpacking Functions
The @command{ncpdq} packing (and unpacking) algorithms are described
in @ref{Intrinsic functions}, and are also implemented in
@command{ncap2}.
@command{ncpdq} extends the functionality of these algorithms by
providing high level control of the @dfn{packing policy} so that
users can pack (and unpack) entire files consistently with one command.
@cindex @var{pck_plc}
@cindex @code{-P @var{pck_plc}}
@cindex @code{--pck_plc @var{pck_plc}}
@cindex @code{--pack_policy @var{pck_plc}}
The user specifies the desired packing policy with the @samp{-P} switch
(or its long option equivalents, @samp{--pck_plc} and
@samp{--pack_policy}) and its @var{pck_plc} argument.
Four packing policies are currently implemented:@*
@table @dfn
@item Packing (and Re-Packing) Variables [@emph{default}]
Definition: Pack unpacked variables, re-pack packed variables@*
Alternate invocation: @code{ncpack}@*
@var{pck_plc} key values: @samp{all_new}, @samp{pck_all_new_att}@*
@item Packing (and not Re-Packing) Variables
Definition: Pack unpacked variables, copy packed variables@*
Alternate invocation: none@*
@var{pck_plc} key values: @samp{all_xst}, @samp{pck_all_xst_att}@*
@item Re-Packing Variables
Definition: Re-pack packed variables, copy unpacked variables@*
Alternate invocation: none@*
@var{pck_plc} key values: @samp{xst_new}, @samp{pck_xst_new_att}@*
@item Unpacking
Definition: Unpack packed variables, copy unpacked variables@*
Alternate invocation: @code{ncunpack}@*
@var{pck_plc} key values: @samp{upk}, @samp{unpack}, @samp{pck_upk}@*
@end table
@noindent
Equivalent key values are fully interchangeable.
Multiple equivalent options are provided to satisfy disparate needs
and tastes of @acronym{NCO} users working with scripts and from the
command line.
To reduce required memorization of these complex policy switches,
@command{ncpdq} may also be invoked via a synonym or with switches
that imply a particular policy.
@command{ncpack} is a synonym for @command{ncpdq} and behaves the same
in all respects.
Both @command{ncpdq} and @command{ncpack} assume a default packing
policy request of @samp{all_new}.
Hence @command{ncpack} may be invoked without any @samp{-P} switch,
unlike @command{ncpdq}.
Similarly, @command{ncunpack} is a synonym for @command{ncpdq}
except that @command{ncpack} implicitly assumes a request to unpack,
i.e., @samp{-P pck_upk}.
@cindex @code{-U}
@cindex @code{--upk}
@cindex @code{--unpack}
Finally, the @command{ncpdq} @samp{-U} switch (or its long option
equivalents, @samp{--upk} and @samp{--unpack}) requires no argument.
It simply requests unpacking.
Given the menagerie of synonyms, equivalent options, and implied
options, a short list of some equivalent commands is appropriate.
The following commands are equivalent for packing:
@code{ncpdq -P all_new}, @code{ncpdq --pck_plc=all_new}, and
@code{ncpack}.
The following commands are equivalent for unpacking:
@code{ncpdq -P upk}, @code{ncpdq -U}, @code{ncpdq --pck_plc=unpack},
and @code{ncunpack}.
Equivalent commands for other packing policies, e.g., @samp{all_xst},
follow by analogy.
@cindex @command{alias}
@cindex @command{ln -s}
@cindex symbolic links
Note that @command{ncpdq} synonyms are subject to the same constraints
and recommendations discussed in the secion on @command{ncbo} synonyms
(@pxref{ncbo netCDF Binary Operator}).
That is, symbolic links must exist from the synonym to @command{ncpdq},
or else the user must define an @command{alias}.
@cindex packing map
@cindex @var{pck_map}
@cindex @code{-M @var{pck_map}}
@cindex @code{--pck_map @var{pck_map}}
@cindex @code{--map @var{pck_map}}
The @command{ncpdq} packing algorithms must know to which type
particular types of input variables are to be packed.
The correspondence between the input variable type and the output,
packed type, is called the @dfn{packing map}.
The user specifies the desired packing map with the @samp{-M} switch
(or its long option equivalents, @samp{--pck_map} and
@samp{--map}) and its @var{pck_map} argument.
Five packing maps are currently implemented:@*
@cindex @samp{hgh_sht}
@cindex @samp{hgh_byt}
@cindex @samp{flt_sht}
@cindex @samp{flt_byt}
@cindex @samp{nxt_lsr}
@cindex @code{NC_DOUBLE}
@cindex @code{NC_FLOAT}
@cindex @code{NC_INT}
@cindex @code{NC_SHORT}
@cindex @code{NC_CHAR}
@cindex @code{NC_BYTE}
@table @dfn
@item Pack Floating Precisions to @code{NC_SHORT} [@emph{default}]
Definition: Pack floating precision types to @code{NC_SHORT}@*
Map: Pack [@code{NC_DOUBLE},@code{NC_FLOAT}] to @code{NC_SHORT}@*
Types copied instead of packed: [@code{NC_INT},@code{NC_SHORT},@code{NC_CHAR},@code{NC_BYTE}]@*
@var{pck_map} key values: @samp{flt_sht}, @samp{pck_map_flt_sht}@*
@item Pack Floating Precisions to @code{NC_BYTE}
Definition: Pack floating precision types to @code{NC_BYTE}@*
Map: Pack [@code{NC_DOUBLE},@code{NC_FLOAT}] to @code{NC_BYTE}@*
Types copied instead of packed: [@code{NC_INT},@code{NC_SHORT},@code{NC_CHAR},@code{NC_BYTE}]@*
@var{pck_map} key values: @samp{flt_byt}, @samp{pck_map_flt_byt}@*
@item Pack Higher Precisions to @code{NC_SHORT}
Definition: Pack higher precision types to @code{NC_SHORT}@*
Map:
Pack [@code{NC_DOUBLE},@code{NC_FLOAT},@code{NC_INT}] to @code{NC_SHORT}@*
Types copied instead of packed: [@code{NC_SHORT},@code{NC_CHAR},@code{NC_BYTE}]@*
@var{pck_map} key values: @samp{hgh_sht}, @samp{pck_map_hgh_sht}@*
@item Pack Higher Precisions to @code{NC_BYTE}
Definition: Pack higher precision types to @code{NC_BYTE}@*
Map:
Pack [@code{NC_DOUBLE},@code{NC_FLOAT},@code{NC_INT},@code{NC_SHORT}] to @code{NC_BYTE}@*
Types copied instead of packed: [@code{NC_CHAR},@code{NC_BYTE}]@*
@var{pck_map} key values: @samp{hgh_byt}, @samp{pck_map_hgh_byt}@*
@item Pack to Next Lesser Precision
Definition: Pack each type to type of next lesser size@*
Map: Pack @code{NC_DOUBLE} to @code{NC_INT}.
Pack [@code{NC_FLOAT},@code{NC_INT}] to @code{NC_SHORT}.
Pack @code{NC_SHORT} to @code{NC_BYTE}.@*
Types copied instead of packed: [@code{NC_CHAR},@code{NC_BYTE}]@*
@var{pck_map} key values: @samp{nxt_lsr}, @samp{pck_map_nxt_lsr}@*
@end table
@noindent
The default @samp{all_new} packing policy with the default
@samp{flt_sht} packing map reduces the typical @code{NC_FLOAT}-dominated
file size by @w{about 50%.}
@samp{flt_byt} packing reduces an @code{NC_DOUBLE}-dominated file by
@w{about 87%.}
@cindex @var{_FillValue}
@cindex @code{_FillValue}
@cindex @code{NUL}
The netCDF packing algorithm (@pxref{Intrinsic functions}) is
lossy---once packed, the exact original data cannot be recovered without
a full backup.
Hence users should be aware of some packing caveats:
First, the interaction of packing and data equal to the
@var{_FillValue} is complex.
Test the @code{_FillValue} behavior by performing a pack/unpack cycle
to ensure data that are missing @emph{stay} missing and data that are
not misssing do not join the Air National Guard and go missing.
This may lead you to elect a new @var{_FillValue}.
Second, @code{ncpdq} actually allows packing into @code{NC_CHAR} (with,
e.g., @samp{flt_chr}).
However, the intrinsic conversion of @code{signed char} to higher
precision types is tricky so for values equal to zero, i.e.,
@code{NUL}.
Hence packing to @code{NC_CHAR} is not documented or advertised.
Pack into @code{NC_BYTE} (with, e.g., @samp{flt_byt}) instead.
@unnumberedsubsec Dimension Permutation
@command{ncpdq} re-shapes variables in @var{input-file} by re-ordering
and/or reversing dimensions specified in the dimension list.
The dimension list is a whitespace-free, comma separated list of
dimension names, optionally prefixed by negative signs, that follows the
@samp{-a} (or long options @samp{--arrange}, @samp{--permute},
@samp{--re-order}, or @samp{--rdr}) switch.
To re-order variables by a subset of their dimensions, specify
these dimensions in a comma-separated list following @samp{-a}, e.g.,
@samp{-a lon,lat}.
To reverse a dimension, prefix its name with a negative sign in the
dimension list, e.g., @samp{-a -lat}.
Re-ordering and reversal may be performed simultaneously, e.g.,
@samp{-a lon,-lat,time,-lev}.
@cindex record dimension
Users may specify any permutation of dimensions, including
permutations which change the record dimension identity.
The record dimension is re-ordered like any other dimension.
@cindex concatenation
@cindex record dimension
This unique @command{ncpdq} capability makes it possible to concatenate
files along any dimension.
See @ref{Concatenation} for a detailed example.
@cindex record variable
The record dimension is always the most slowly varying dimension in a
record variable (@pxref{C and Fortran Index Conventions}).
The specified re-ordering fails if it requires creating more than
one record dimension amongst all the output variables
@footnote{This limitation, imposed by the netCDF storage layer,
may be relaxed in the future with netCDF4.}.
Two special cases of dimension re-ordering and reversal deserve special
mention.
First, it may be desirable to completely reverse the storage order of a
variable.
To do this, include all the variable's dimensions in the dimension
re-order list in their original order, and prefix each dimension name
with the negative sign.
@cindex transpose
Second, it may useful to transpose a variable's storage order, e.g.,
@w{from C} to Fortran data storage order
(@pxref{C and Fortran Index Conventions}).
To do this, include all the variable's dimensions in the dimension
re-order list in reversed order.
Explicit examples of these two techniques appear below.
@tex
NB: fxm ncpdq documentation will evolve through Fall 2004.
I will upload updates to documentation linked to by the NCO homepage.
ncpdq is a powerful operator, and
I am unfamiliar with the terminology needed to describe what ncpdq does.
Sequences, sets, sheesh!
I just know that it does "The right thing" according to my gut feelings.
Now do you feel more comfortable using it?
Let $\dmnvct(\xxx)$ represent the dimensionality of the variable $\xxx$.
Dimensionality describes the order and sizes of dimensions.
If $\xxx$ has rank $\dmnnbr$, then we may write $\dmnvct(\xxx)$ as the
$\dmnnbr$-element vector
$$
\dmnvct(\xxx) = [ \dmn_{1}, \dmn_{2}, \dmn_{3}, \ldots,
\dmn_{\dmnidx-1}, \dmn_{\dmnidx}, \dmn_{\dmnidx+1},
\ldots, \dmn_{\dmnnbr-2}, \dmn_{\dmnnbr-1}, \dmn_{\dmnnbr} ]
$$
where $\dmn_{\dmnidx}$ is the size of the $\dmnidx$'th dimension.
The dimension re-order list specified with @samp{-a} is the
$\rdrnbr$-element vector
$$
\rdrvct = [ \rdr_{1}, \rdr_{2}, \rdr_{3}, \ldots,
\rdr_{\rdridx-1}, \rdr_{\rdridx}, \rdr_{\rdridx+1},
\ldots, \rdr_{\rdrnbr-2}, \rdr_{\rdrnbr-1}, \rdr_{\rdrnbr} ]
$$
There need be no relation between $\dmnnbr$ and $\rdrnbr$.
Let the $\shrnbr$-element vector $\shrvct$ be the intersection
(i.e., the ordered set of unique shared dimensions) of $\dmnvct$ and
$\rdrvct$
Then
$$
\eqalign{{\shrvct} &= \rdrvct \cap \dmnvct \cr
&= [ \shr_{1}, \shr_{2}, \shr_{3}, \ldots,
\shr_{\shridx-1}, \shr_{\shridx}, \shr_{\shridx+1},
\ldots, \shr_{\shrnbr-2}, \shr_{\shrnbr-1}, \shr_{\shrnbr} ]}
$$
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.
Re-ordering (or re-shaping) a variable means mapping the input state
with dimensionality $\dmnvct(\xxx)$ to the output state with
dimensionality $\dmnvctprm(\xxxprm)$.
In practice, mapping occurs in three logically distinct steps.
First, we tranlate the user input to a one-to-one mapping $\mpp$
between input and output dimensions,
$\dmnvct \mapsto \dmnvctprm$.
This tentative map is final unless external constraints (typically
netCDF restrictions) impose themselves.
Second, we check and, if necessary, refine the tentative mapping so that
the re-shaped variables will co-exist in the same file without violating
netCDF-imposed storage restrictions.
This refined map specifies the final (output) dimensionality.
Third, we translate the output dimensionality into one-dimensional
memory offsets for each datum according to the @w{C language} convention
for multi-dimensional array storage.
Dimension reversal changes the ordering of data, but not the
dimensionality, and so is part of the third step.
Dimensions $\rdr$ disjoint from $\dmnvct$ play no role in re-ordering.
The first step taken to re-order a variable is to determine $\shrvct$.
$\rdrvct$ is constant for all variables, whereas $\dmnvct$, and hence
$\shrvct$, is variable-specific.
$\shrvct$ is empty if $\rdrvct \notin \dmnvct$.
This may be the case for some extracted variables.
The user may explicitly specify the one-to-one mapping of input
to output dimension order by supplying (with @samp{-a}) a re-order list
$\rdrvct$ such that $\shrnbr = \dmnnbr$.
In this case $\dmnsubnnnprm = \shrsubnnn$.
The degenerate case occurs when $\dmnvct = \shrvct$.
This produces the identity mapping $\dmnsubnnnprm = \dmnsubnnn$.
The mapping of input to output dimension order is more complex
when $\shrnbr \ne \dmnnbr$.
In this case $\dmnsubnnnprm = \dmnsubnnn$ for the $\dmnnbr-\shrnbr$
dimensions $\dmnsubnnnprm \notin \shrvct$.
For the $\shrnbr$ dimensions $\dmnsubnnnprm \in \shrvct$,
$\dmnsubnnnprm = \shrsubsss$.
@end tex
@c fxm: discuss netCDF-imposed constraints here
@noindent
@html
<a name="xmp_ncpdq"></a> <!-- http://nco.sf.net/nco.html#xmp_ncpdq -->
@end html
EXAMPLES
Pack and unpack all variables in file @file{in.nc} and store the results
in @file{out.nc}:
@example
ncpdq in.nc out.nc # Same as ncpack in.nc out.nc
ncpdq -P all_new -M flt_sht in.nc out.nc # Defaults
ncpdq -P all_xst in.nc out.nc
ncpdq -P upk in.nc out.nc # Same as ncunpack in.nc out.nc
ncpdq -U in.nc out.nc # Same as ncunpack in.nc out.nc
@end example
The first two commands pack any unpacked variable in the input file.
They also unpack and then re-pack every packed variable.
The third command only packs unpacked variables in the input file.
If a variable is already packed, the third command copies it unchanged
to the output file.
The fourth and fifth commands unpack any packed variables.
If a variable is not packed, the third command copies it unchanged.
The previous examples all utilized the default packing map.
Suppose you wish to archive all data that are currently unpacked
into a form which only preserves 256 distinct values.
Then you could specify the packing map @var{pck_map} as @samp{hgh_byt}
and the packing policy @var{pck_plc} as @samp{all_xst}:
@example
ncpdq -P all_xst -M hgh_byt in.nc out.nc
@end example
@cindex appending variables
@cindex @samp{-A}
@cindex @samp{-v}
Many different packing maps may be used to construct a given file
by performing the packing on subsets of variables (e.g., with @samp{-v})
and using the append feature with @samp{-A} (@pxref{Appending Variables}).
Re-order file @file{in.nc} so that the dimension @code{lon} always
precedes the dimension @code{lat} and store the results in
@file{out.nc}:
@example
ncpdq -a lon,lat in.nc out.nc
ncpdq -v three_dmn_var -a lon,lat in.nc out.nc
@end example
The first command re-orders every variable in the input file.
The second command extracts and re-orders only the variable
@code{three_dmn_var}.
@cindex reverse dimensions
Suppose the dimension @code{lat} represents latitude and monotonically
increases increases from south to north.
Reversing the @code{lat} dimension means re-ordering the data so that
latitude values decrease monotonically from north to south.
Accomplish this with
@example
% ncpdq -a -lat in.nc out.nc
% ncks -C -v lat in.nc
lat[0]=-90
lat[1]=90
% ncks -C -v lat out.nc
lat[0]=90
lat[1]=-90
@end example
This operation reversed the latitude dimension of all variables.
Whitespace immediately preceding the negative sign that specifies
dimension reversal may be dangerous.
@cindex long options
@cindex quotes
Quotes and long options can help protect negative signs that should
indicate dimension reversal from being interpreted by the shell as
dashes that indicate new command line switches.
@example
ncpdq -a -lat in.nc out.nc # Dangerous? Whitespace before "-lat"
ncpdq -a "-lat" in.nc out.nc # OK. Quotes protect "-" in "-lat"
ncpdq -a lon,-lat in.nc out.nc # OK. No whitespace before "-"
ncpdq --rdr=-lat in.nc out.nc # Preferred. Uses "=" not whitespace
@end example
@cindex reverse dimensions
To create the mathematical transpose of a variable, place all its
dimensions in the dimension re-order list in reversed order.
This example creates the transpose of @code{three_dmn_var}:
@example
% ncpdq -a lon,lev,lat -v three_dmn_var in.nc out.nc
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
% ncks -C -v three_dmn_var out.nc
lon[0]=0 lev[0]=100 lat[0]=-90 three_dmn_var[0]=0
lon[0]=0 lev[0]=100 lat[1]=90 three_dmn_var[1]=12
lon[0]=0 lev[1]=500 lat[0]=-90 three_dmn_var[2]=4
...
lon[3]=270 lev[1]=500 lat[1]=90 three_dmn_var[21]=19
lon[3]=270 lev[2]=1000 lat[0]=-90 three_dmn_var[22]=11
lon[3]=270 lev[2]=1000 lat[1]=90 three_dmn_var[23]=23
@end example
@cindex reverse data
To completely reverse the storage order of a variable, include
all its dimensions in the re-order list, each prefixed by a negative
sign.
This example reverses the storage order of @code{three_dmn_var}:
@example
% ncpdq -a -lat,-lev,-lon -v three_dmn_var in.nc out.nc
% ncks -C -v three_dmn_var in.nc
lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
...
lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
% ncks -C -v three_dmn_var out.nc
lat[0]=90 lev[0]=1000 lon[0]=270 three_dmn_var[0]=23
lat[0]=90 lev[0]=1000 lon[1]=180 three_dmn_var[1]=22
lat[0]=90 lev[0]=1000 lon[2]=90 three_dmn_var[2]=21
...
lat[1]=-90 lev[2]=100 lon[1]=180 three_dmn_var[21]=2
lat[1]=-90 lev[2]=100 lon[2]=90 three_dmn_var[22]=1
lat[1]=-90 lev[2]=100 lon[3]=0 three_dmn_var[23]=0
@end example
@html
<a name="dmn_rcd_mk"></a> <!-- http://nco.sf.net/nco.html#dmn_rcd_mk -->
@end html
@cindex record dimension
Consider a file with all dimensions, including @code{time}, fixed
(non-record).
Suppose the user wishes to convert @code{time} from a fixed dimension to
a record dimension.
This may be useful, for example, when the user wishes to append
additional time slices to the data.
The procedure is to use @command{ncecat} followed by @command{ncpdq}
and then @command{ncwa}:
@cindex degenerate dimension
@example
ncecat -O in.nc out.nc # Add degenerate record dimension named "record"
ncpdq -O -a time,record out.nc out.nc # Switch "record" and "time"
ncwa -O -a record out.nc out.nc # Remove (degenerate) "record"
@end example
@noindent
The first step creates a degenerate (size equals one) record dimension.
The second step swaps the ordering of the dimensions named @code{time}
and @code{record}.
Since @code{time} now occupies the position of the first (least rapidly
varying) dimension, it becomes the record dimension.
The dimension named @code{record} is no longer a record dimension.
The third step averages over the degenerate @code{record} dimension.
Averaging over a degenerate dimension does not alter the data.
The ordering of other dimensions in the file (@code{lat}, @code{lon},
etc.) is immaterial to this procedure.
See @ref{ncecat netCDF Ensemble Concatenator} for other methods of
changing variable dimensionality, including the record dimension.
@page
@html
<a name="ncra"></a> <!-- http://nco.sf.net/nco.html#ncra -->
@end html
@node ncra netCDF Record Averager, ncrcat netCDF Record Concatenator, ncpdq netCDF Permute Dimensions Quickly, Operator Reference Manual
@section @command{ncra} netCDF Record Averager
@cindex averaging data
@cindex record average
@cindex record dimension
@cindex running average
@findex ncra
@noindent
SYNTAX
@example
ncra [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-L @var{dfl_lvl}] [-l @var{path}]
[-n @var{loop}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-X ...] [-x] [-y @var{op_typ}]
[@var{input-files}] [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncra} averages record variables across an arbitrary number of
@var{input-files}.
@cindex degenerate dimension
@cindex record dimension
The record dimension is, by default, retained as a degenerate
@w{(size 1)} dimension in the output variables.
@xref{Averaging vs. Concatenating}, for a description of the
distinctions between the various averagers and concatenators.
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncra} will read the list of
@var{input-files} from @code{stdin} if they are not specified
as positional arguments on the command line
(@pxref{Large Numbers of Files}).
Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated).
@cindex hyperslab
Hyperslabs of the record dimension which include more than one file
work correctly.
@cindex stride
@command{ncra} supports the @var{stride} argument to the @samp{-d}
hyperslab option (@pxref{Hyperslabs}) for the record dimension only,
@var{stride} is not supported for non-record dimensions.
@command{ncra} weights each record (e.g., time slice) in the
@var{input-files} equally.
@command{ncra} does not attempt to see if, say, the @code{time}
coordinate is irregularly spaced and thus would require a weighted
average in order to be a true time average.
@cindex operation types
@command{ncra} @emph{always averages} coordinate variables regardless of
the arithmetic operation type performed on the non-coordinate variables.
(@pxref{Operation Types}).
@noindent
@html
<a name="xmp_ncra"></a> <!-- http://nco.sf.net/nco.html#xmp_ncra -->
@end html
EXAMPLES
Average files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
along the record dimension, and store the results in @file{8589.nc}:
@cindex globbing
@cindex @code{NINTAP}
@cindex Processor
@cindex @acronym{CCM} Processor
@example
ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
@end example
These three methods produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
@cindex Fortran
Assume the files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
each contain a record coordinate @var{time} of length 12 defined such
that the third record in @file{86.nc} contains data from March 1986,
etc.
@acronym{NCO} knows how to hyperslab the record dimension across files.
Thus, to average data from December, 1985 through February, 1986:
@example
ncra -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncra -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
@end example
@noindent
The file @file{87.nc} is superfluous, but does not cause an error.
The @samp{-F} turns on the Fortran (1-based) indexing convention.
@cindex stride
The following uses the @var{stride} option to average all the March
temperature data from multiple input files into a single output file
@example
ncra -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
@end example
@xref{Stride}, for a description of the @var{stride} argument.
Assume the @var{time} coordinate is incrementally numbered such that
January, @w{@math{1985 = 1}} and December, @w{@math{1989 = 60}}.
Assuming @samp{??} only expands to the five desired files, the following
averages June, 1985--June, 1989:
@example
ncra -d time,6.,54. ??.nc 8506_8906.nc
@end example
@page
@html
<a name="ncrcat"></a> <!-- http://nco.sf.net/nco.html#ncrcat -->
@end html
@node ncrcat netCDF Record Concatenator, ncrename netCDF Renamer, ncra netCDF Record Averager, Operator Reference Manual
@section @command{ncrcat} netCDF Record Concatenator
@cindex concatenation
@cindex record concatenation
@findex ncrcat
@noindent
SYNTAX
@example
ncrcat [-3] [-4] [-A] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-L @var{dfl_lvl}] [-l @var{path}]
[-n @var{loop}] [-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-X ...] [-x]
[@var{input-files}] [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncrcat} concatenates record variables across an arbitrary
number of @var{input-files}.
@cindex record dimension
The final record dimension is by default the sum of the lengths of the
record dimensions in the input files.
@xref{Averaging vs. Concatenating}, for a description of the
distinctions between the various averagers and concatenators.
@cindex multi-file operators
@cindex standard input
@cindex @code{stdin}
As a multi-file operator, @command{ncrcat} will read the list of
@var{input-files} from @code{stdin} if they are not specified
as positional arguments on the command line
(@pxref{Large Numbers of Files}).
Input files may vary in size, but each must have a record dimension.
The record coordinate, if any, should be monotonic (or else non-fatal
warnings may be generated).
@cindex hyperslab
Hyperslabs along the record dimension that span more than one file are
handled correctly.
@cindex stride
@command{ncra} supports the @var{stride} argument to the @samp{-d}
hyperslab option for the record dimension only, @var{stride} is not
supported for non-record dimensions.
@findex ncpdq
@cindex packing
@cindex unpacking
@cindex @code{add_offset}
@cindex @code{scale_factor}
Concatenating a variable packed with different scales multiple datasets
is beyond the capabilities of @command{ncrcat} (and @command{ncecat},
the other concatenator (@ref{Concatenation}).
@command{ncrcat} does not unpack data, it simply @emph{copies} the data
from the @var{input-files}, and the metadata from the @emph{first}
@var{input-file}, to the @var{output-file}.
This means that data compressed with a packing convention must use
the identical packing parameters (e.g., @code{scale_factor} and
@code{add_offset}) for a given variable across @emph{all} input files.
Otherwise the concatenated dataset will not unpack correctly.
The workaround for cases where the packing parameters differ across
@var{input-files} requires three steps:
First, unpack the data using @command{ncpdq}.
Second, concatenate the unpacked data using @command{ncrcat},
Third, re-pack the result with @command{ncpdq}.
@cindex ARM conventions
@command{ncrcat} applies special rules to @acronym{ARM} convention time
fields (e.g., @code{time_offset}).
See @ref{ARM Conventions} for a complete description.
@noindent
@html
<a name="xmp_ncrcat"></a> <!-- http://nco.sf.net/nco.html#xmp_ncrcat -->
@end html
EXAMPLES
Concatenate files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
along the record dimension, and store the results in @file{8589.nc}:
@cindex globbing
@cindex @code{NINTAP}
@cindex Processor
@cindex @acronym{CCM} Processor
@example
ncrcat 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncrcat 8[56789].nc 8589.nc
ncrcat -n 5,2,1 85.nc 8589.nc
@end example
@noindent
These three methods produce identical answers.
@xref{Specifying Input Files}, for an explanation of the distinctions
between these methods.
@cindex Fortran
Assume the files @file{85.nc}, @file{86.nc}, @w{@dots{} @file{89.nc}}
each contain a record coordinate @var{time} of @w{length 12} defined
such that the third record in @file{86.nc} contains data from March
1986, etc.
@acronym{NCO} knows how to hyperslab the record dimension across files.
Thus, to concatenate data from December, 1985--February, 1986:
@example
ncrcat -d time,11,13 85.nc 86.nc 87.nc 8512_8602.nc
ncrcat -F -d time,12,14 85.nc 86.nc 87.nc 8512_8602.nc
@end example
@noindent
The file @file{87.nc} is superfluous, but does not cause an error.
When @command{ncra} and @command{ncrcat} encounter a file which does
contain any records that meet the specified hyperslab criteria, they
disregard the file and proceed to the next file without failing.
The @samp{-F} turns on the Fortran (1-based) indexing convention.
@cindex stride
The following uses the @var{stride} option to concatenate all the March
temperature data from multiple input files into a single output file
@example
ncrcat -F -d time,3,,12 -v temperature 85.nc 86.nc 87.nc 858687_03.nc
@end example
@xref{Stride}, for a description of the @var{stride} argument.
Assume the @var{time} coordinate is incrementally numbered such that
January, @w{1985 = 1} and December, @w{1989 = 60.}
Assuming @code{??} only expands to the five desired files, the following
concatenates June, 1985--June, 1989:
@example
ncrcat -d time,6.,54. ??.nc 8506_8906.nc
@end example
@page
@html
<a name="ncrename"></a> <!-- http://nco.sf.net/nco.html#ncrename -->
@end html
@node ncrename netCDF Renamer, ncwa netCDF Weighted Averager, ncrcat netCDF Record Concatenator, Operator Reference Manual
@section @command{ncrename} netCDF Renamer
@cindex renaming variables
@cindex renaming dimensions
@cindex renaming attributes
@cindex variable names
@cindex dimension names
@cindex attribute names
@findex ncrename
@noindent
SYNTAX
@example
ncrename [-a @var{old_name},@var{new_name}] [-a @dots{}] [-D @var{dbg}]
[-d @var{old_name},@var{new_name}] [-d @dots{}] [-h] [--hdr_pad @var{nbr}] [-l @var{path}]
[-O] [-o @var{output-file}] [-p @var{path}] [-R] [-r]
[-v @var{old_name},@var{new_name}] [-v @dots{}]
@var{input-file} [[@var{output-file}]]
@end example
@noindent
DESCRIPTION
@command{ncrename} renames dimensions, variables, and attributes in a
netCDF file.
Each object that has a name in the list of old names is renamed using
the corresponding name in the list of new names.
All the new names must be unique.
Every old name must exist in the input file, unless the old name is
preceded by the character @samp{.}.
The validity of @var{old_name} is not checked prior to the renaming.
Thus, if @var{old_name} is specified without the the @samp{.} prefix and
is not present in @var{input-file}, @command{ncrename} will abort.
The @var{new_name} should never be prefixed by a @samp{.} (the period
will be included as part of the new name).
The OPTIONS and EXAMPLES show how to select specific variables
whose attributes are to be renamed.
@cindex data safety
@cindex safeguards
@cindex temporary output files
@command{ncrename} is the exception to the normal rules that the user will
be interactively prompted before an existing file is changed, and that a
temporary copy of an output file is constructed during the operation.
If only @var{input-file} is specified, then @command{ncrename} will change
the names of the @var{input-file} in place without prompting and without
creating a temporary copy of @code{input-file}.
This is because the renaming operation is considered reversible if the
user makes a mistake.
The @var{new_name} can easily be changed back to @var{old_name} by using
@command{ncrename} one more time.
Note that renaming a dimension to the name of a dependent variable can
be used to invert the relationship between an independent coordinate
variable and a dependent variable.
In this case, the named dependent variable must be one-dimensional and
should have no missing values.
Such a variable will become a coordinate variable.
@cindex performance
@cindex operator speed
@cindex speed
@cindex execution time
According to the @cite{netCDF User's Guide}, renaming properties in
netCDF files does not incur the penalty of recopying the entire file
when the @var{new_name} is shorter than the @var{old_name}.
@noindent
OPTIONS
@table @samp
@item -a @var{old_name},@var{new_name}
Attribute renaming.
The old and new names of the attribute are specified with @samp{-a}
(or @samp{--attribute}) by the associated @var{old_name} and
@var{new_name} values.
@cindex global attributes
@cindex attributes, global
Global attributes are treated no differently than variable attributes.
This option may be specified more than once.
As mentioned above, all occurrences of the attribute of a given name
will be renamed unless the @samp{.} form is used, with one exception.
To change the attribute name for a particular variable, specify
the @var{old_name} in the format @var{old_var_name@@old_att_name}.
The @samp{@@} symbol delimits the variable and attribute names.
If the attribute is uniquely named (no other variables contain the
attribute) then the @var{old_var_name@@old_att_name} syntax is
redundant.
The @var{var_name@@att_name} syntax is accepted, but not required,
for the @var{new_name}.
@item -d @var{old_name},@var{new_name}
Dimension renaming.
The old and new names of the dimension are specified with @samp{-d}
(or @samp{--dmn}, @samp{--dimension}) by the associated @var{old_name}
and @var{new_name} values.
This option may be specified more than once.
@item -v @var{old_name},@var{new_name}
Variable renaming.
The old and new names of the variable are specified with @samp{-v}
(or @samp{--variable}) by the associated @var{old_name} and
@var{new_name} values.
This option may be specified more than once.
@end table
@noindent
@html
<a name="xmp_ncrename"></a> <!-- http://nco.sf.net/nco.html#xmp_ncrename -->
@end html
EXAMPLES
Rename the variable @code{p} to @code{pressure} and @code{t} to
@code{temperature} in netCDF @file{in.nc}.
In this case @code{p} must exist in the input file (or
@command{ncrename} will abort), but the presence of @code{t} is optional:
@example
ncrename -v p,pressure -v .t,temperature in.nc
@end example
Rename the attribute @code{long_name} to @code{largo_nombre} in the
variable @code{u}, and no other variables in netCDF @file{in.nc}.
@example
ncrename -a u:long_name,largo_nombre in.nc
@end example
@cindex coordinate variables
@command{ncrename} does not automatically attach dimensions to variables of
the same name.
If you want to rename a coordinate variable so that it remains a
coordinate variable, you must separately rename both the dimension and
the variable:
@example
ncrename -d lon,longitude -v lon,longitude in.nc
@end example
@cindex global attributes
@cindex attributes, global
@cindex @code{_FillValue}
@cindex @code{missing_value}
Create netCDF @file{out.nc} identical to @file{in.nc} except the
attribute @code{_FillValue} is changed to @code{missing_value},
the attribute @code{units} is changed to @code{CGS_units} (but only in
those variables which possess it), the attribute @code{hieght} is
changed to @code{height} in the variable @code{tpt}, and in the
variable @code{prs_sfc}, if it exists.
@example
ncrename -a _FillValue,missing_value -a .units,CGS_units \
-a tpt@@hieght,height -a prs_sfc@@.hieght,height in.nc out.nc
@end example
The presence and absence of the @samp{.} and @samp{@@} features
cause this command to execute successfully only if a number of
conditions are met.
All variables @emph{must} have a @code{_FillValue} attribute @emph{and}
@code{_FillValue} must also be a global attribute.
The @code{units} attribute, on the other hand, will be renamed to
@code{CGS_units} wherever it is found but need not be present in
the file at all (either as a global or a variable attribute).
The variable @code{tpt} must contain the @code{hieght} attribute.
The variable @code{prs_sfc} need not exist, and need not contain the
@code{hieght} attribute.
@page
@html
<a name="ncwa"></a> <!-- http://nco.sf.net/nco.html#ncwa -->
@end html
@node ncwa netCDF Weighted Averager, , ncrename netCDF Renamer, Operator Reference Manual
@section @command{ncwa} netCDF Weighted Averager
@cindex averaging data
@cindex weighted average
@cindex masked average
@cindex broadcasting variables
@findex ncwa
@noindent
SYNTAX
@example
ncwa [-3] [-4] [-A] [-a @var{dim}[,@dots{}]] [-B @var{mask_cond}] [-b] [-C] [-c] [-D @var{dbg}]
[-d @var{dim},[@var{min}][,[@var{max}][,[@var{stride}]]] [-F] [-h] [-I] [-L @var{dfl_lvl}] [-l @var{path}]
[-M @var{mask_val}] [-m @var{mask_var}] [-N] [-O]
[-o @var{output-file}] [-p @var{path}] [-R] [-r] [-T @var{mask_comp}]
[-t @var{thr_nbr}] [-v @var{var}[,@dots{}]] [-w @var{weight}] [-X ...] [-x] [-y @var{op_typ}]
@var{input-file} [@var{output-file}]
@end example
@noindent
DESCRIPTION
@command{ncwa} averages variables in a single file over arbitrary
dimensions, with options to specify weights, masks, and normalization.
@xref{Averaging vs. Concatenating}, for a description of the
distinctions between the various averagers and concatenators.
The default behavior of @command{ncwa} is to arithmetically average
every numerical variable over all dimensions and to produce a scalar
result for each.
@cindex degenerate dimension
Averaged dimensions are, by default, eliminated as dimensions.
Their corresponding coordinates, if any, are output as scalars.
The @samp{-b} switch
(and its long option equivalents @samp{--rdd} and
@samp{--retain-degenerate-dimensions}) causes @command{ncwa} to retain
averaged dimensions as degenerate (@w{size 1}) dimensions.
This maintains the association between a dimension (or coordinate) and
variables after averaging and simplifies, for instance, later
concatenation along the degenerate dimension.
To average variables over only a subset of their dimensions, specify
these dimensions in a comma-separated list following @samp{-a}, e.g.,
@samp{-a time,lat,lon}.
@cindex arithmetic operators
@cindex hyperslab
@cindex @code{-d @var{dim},[@var{min}][,[@var{max}]]}
As with all arithmetic operators, the operation may be restricted to
an arbitrary hypserslab by employing the @samp{-d} option
(@pxref{Hyperslabs}).
@command{ncwa} also handles values matching the variable's
@code{_FillValue} attribute correctly.
Moreover, @command{ncwa} understands how to manipulate user-specified
weights, masks, and normalization options.
With these options, @command{ncwa} can compute sophisticated averages
(and integrals) from the command line.
@html
<a name="-w"></a> <!-- http://nco.sf.net/nco.html#-w -->
<a name="wgt"></a> <!-- http://nco.sf.net/nco.html#wgt -->
@end html
@cindex @code{-w @var{weight}}
@cindex @code{--weight @var{weight}}
@cindex @code{--wgt_var @var{weight}}
@cindex @code{-m @var{mask_var}}
@cindex @code{--mask-variable @var{mask_var}}
@cindex @code{--mask_variable @var{mask_var}}
@cindex @code{--msk_nm @var{mask_var}}
@cindex @code{--msk_var @var{mask_var}}
@cindex @code{-B @var{mask_cond}}
@cindex @code{--msk_cnd @var{mask_cond}}
@cindex @code{--mask_condition @var{mask_cond}}
@var{mask_var} and @var{weight}, if specified, are broadcast to conform
to the variables being averaged.
@cindex rank
The rank of variables is reduced by the number of dimensions which they
are averaged over.
Thus arrays which are one dimensional in the @var{input-file} and are
averaged by @command{ncwa} appear in the @var{output-file} as scalars.
This allows the user to infer which dimensions may have been averaged.
Note that that it is impossible for @command{ncwa} to make make a
@var{weight} or @var{mask_var} of rank @var{W} conform to a @var{var} of
rank @var{V} if @var{W > V}.
This situation often arises when coordinate variables (which, by
definition, are one dimensional) are weighted and averaged.
@command{ncwa} assumes you know this is impossible and so @command{ncwa}
does not attempt to broadcast @var{weight} or @var{mask_var} to conform
to @var{var} in this case, nor does @command{ncwa} print a warning
message telling you this, because it is so common.
Specifying @var{dbg > 2} does cause @command{ncwa} to emit warnings in
these situations, however.
Non-coordinate variables are always masked and weighted if specified.
Coordinate variables, however, may be treated specially.
By default, an averaged coordinate variable, e.g., @code{latitude},
appears in @var{output-file} averaged the same way as any other variable
containing an averaged dimension.
In other words, by default @command{ncwa} weights and masks
coordinate variables like all other variables.
This design decision was intended to be helpful but for some
applications it may be preferable not to weight or mask coordinate
variables just like all other variables.
Consider the following arguments to @command{ncwa}:
@code{-a latitude -w lat_wgt -d latitude,0.,90.} where @code{lat_wgt} is
a weight in the @code{latitude} dimension.
Since, by default @command{ncwa} weights coordinate variables, the
value of @code{latitude} in the @var{output-file} depends on the weights
in @var{lat_wgt} and is not likely to @w{be 45.0}, the midpoint latitude
of the hyperslab.
@cindex coordinate variable
@cindex @code{-I}
Option @samp{-I} overrides this default behavior and causes
@command{ncwa} not to weight or mask coordinate variables
@footnote{The default behavior of (@samp{-I}) changed on
1998/12/01---before this date the default was not to weight or mask
coordinate variables.}.
In the above case, this causes the value of @code{latitude} in the
@var{output-file} to @w{be 45.0}, an appealing result.
Thus, @samp{-I} specifies simple arithmetic averages for the coordinate
variables.
In the case of latitude, @samp{-I} specifies that you prefer to archive
the arithmetic mean latitude of the averaged hyperslabs rather than the
area-weighted mean latitude.
@footnote{If @code{lat_wgt} contains Gaussian weights then the value of
@code{latitude} in the @var{output-file} will be the area-weighted
centroid of the hyperslab.
For the example given, this is about @w{30 degrees.}}.
@cindex average
@cindex operation types
As explained in @xref{Operation Types}, @command{ncwa}
@emph{always averages} coordinate variables regardless of the arithmetic
operation type performed on the non-coordinate variables.
This is independent of the setting of the @samp{-I} option.
The mathematical definition of operations involving rank reduction
is given above (@pxref{Operation Types}).
@menu
* Mask condition::
* Normalization and Integration::
@end menu
@html
<a name="msk"></a> <!-- http://nco.sf.net/nco.html#msk -->
<a name="-m"></a> <!-- http://nco.sf.net/nco.html#-m -->
@end html
@node Mask condition, Normalization and Integration, ncwa netCDF Weighted Averager, ncwa netCDF Weighted Averager
@subsection Mask condition
@cindex mask condition
@cindex truth condition
@tex
Each $\xxx_{\idx}$ also has an associated masking
weight~$\mskflg_{\idx}$ whose value is~0 or~1 (false or true).
The value of~$\mskflg_{\idx}$ is always~1 unless a @var{mask\_var} is
specified (with
@samp{-m}).
As noted above, @var{mask\_var} is broadcast, if possible, to conform
to the variable being averaged.
In this case, the value of~$\mskflg_{\idx}$ depends on the
@dfn{mask condition} also known as the @dfn{truth condition}.
As expected, $\mskflg_{\idx} = 1$ when the mask condition is
@dfn{true} and $\mskflg_{\idx} = 0$ otherwise.
@end tex
@cindex @code{--op_rlt @var{mask_comp}}
@cindex @code{--mask_comparator @var{mask_comp}}
@cindex @code{--msk_cmp_typ @var{mask_comp}}
@cindex @code{--msk_cnd_sng @var{mask_cond}}
@cindex @code{--mask_condition @var{mask_cond}}
@cindex @code{-B @var{mask_cond}}
The mask condition has the syntax @math{@var{mask_var}}
@math{@var{mask_comp}} @math{@var{mask_val}}.
The preferred method to specify the mask condition is in one string with
the @samp{-B} or @samp{--mask_condition} switches.
The older method is to use the three switches @samp{-m}, @samp{-T}, and
@samp{-M} to specify the @var{mask_var}, @var{mask_comp}, and
@var{mask_val}, respectively.
@footnote{The three switches @samp{-m}, @samp{-T}, and @samp{-M} are
maintained for backward compatibility and may be deprecated in the
future.
It is safest to write scripts using @samp{--mask_condition}.}.
The @var{mask_condition} string is automatically parsed into its three
constituents @var{mask_var}, @var{mask_comp}, and @var{mask_val}.
@cindex comparator
Here @var{mask_var} is the name of the masking variable (specified with
@samp{-m}, @samp{--mask-variable}, @samp{--mask_variable},
@samp{--msk_nm}, or @samp{--msk_var}).
The truth @var{mask_comp} argument (specified with @samp{-T},
@samp{--mask_comparator}, @samp{--msk_cmp_typ}, or @samp{--op_rlt} may
be any one of the six arithmetic comparators: @kbd{eq}, @kbd{ne},
@kbd{gt}, @kbd{lt}, @kbd{ge}, @kbd{le}.
@set flg
@tex
These are the Fortran-style character abbreviations for the logical
comparisons $=$, $\neq$, $>$, $<$, $\ge$, $\le$.
@clear flg
@end tex
@ifinfo
These are the Fortran-style character abbreviations for the logical
comparisons @math{==}, @math{!=}, @math{>}, @math{<}, @math{>=},
@math{<=}.
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
These are the Fortran-style character abbreviations for the logical
comparisons @kbd{==}, @kbd{!=}, @kbd{>}, @kbd{<}, @kbd{>=},
@clear flg
@end ifset
The mask comparator defaults to @kbd{eq} (equality).
@cindex @code{--mask-value @var{mask_val}}
@cindex @code{--mask_value @var{mask_val}}
@cindex @code{--msk_val @var{mask_val}}
The @var{mask_val} argument to @samp{-M} (or @samp{--mask-value}, or
@samp{--msk_val}) is the right hand side of the
@dfn{mask condition}.
Thus for the @var{i}'th element of the hyperslab to be averaged,
the mask condition is
@set flg
@tex
{\it mask$_{\idx}$ mask\_comp mask\_val}.
@clear flg
@end tex
@ifinfo
@math{mask(i)} @var{mask_comp} @var{mask_val}.
@clear flg
@end ifinfo
@ifset flg
@c texi2html does not like @math{}
@var{mask}(@var{i}) @var{mask_comp} @var{mask_val}.
@clear flg
@end ifset
@tex
Each~$\xxx_{\idx}$ is also associated with an additional
weight~$\wgt_{\idx}$ whose value may be user-specified.
The value of~$\wgt_{\idx}$ is identically~1 unless the user specifies a
weighting variable @var{weight} (with @samp{-w}, @samp{--weight}, or
@samp{--wgt\_var}).
In this case, the value of~$\wgt_{\idx}$ is determined by the
@var{weight} variable in the @var{input-file}.
As noted above, @var{weight} is broadcast, if possible, to conform
to the variable being averaged.
$\tllnbr$ is the number of input elements $\xxx_{\idx}$ which actually
contribute to output element $\xxx_{\jjj}$.
$\tllnbr$ is also known as the @dfn{tally} and is defined as
$$
\tllnbr = \sum_{\idx=1}^{\idx=\lmnnbr} \mssflg_{\idx} \mskflg_{\idx}
$$
$\tllnbr$ is identical to the denominator of the generic averaging
expression except for the omission of the weight $\wgt_{\idx}$.
Thus $\tllnbr = \lmnnbr$ whenever no input points are missing values or
are masked.
Whether an element contributes to the output, and thus increments
$\tllnbr$ by one, has more to do with the above two criteria (missing
value and masking) than with the numeric value of the element per se.
For example, $\xxx_{\idx}=0.0$ does contribute to $\xxx_{\jjj}$
(assuming the @code{_FillValue} attribute is not~0.0 and location
$\idx$ is not masked).
The value $\xxx_{\idx}=0.0$ will not change the numerator of the generic
averaging expression, but it will change the denominator (unless its
weight $\wgt_{\idx}=0.0$ as well).
@end tex
@html
<a name="nrm"></a> <!-- http://nco.sf.net/nco.html#nrm -->
<a name="ntg"></a> <!-- http://nco.sf.net/nco.html#ntg -->
@end html
@node Normalization and Integration, , Mask condition, ncwa netCDF Weighted Averager
@subsection Normalization and Integration
@cindex normalization
@cindex @code{-N}
@cindex @code{numerator}
@cindex integration
@cindex dot product
@command{ncwa} has one switch which controls the normalization of the
averages appearing in the @var{output-file}.
Short option @samp{-N} (or long options @samp{--nmr} or
@samp{--numerator}) prevents @command{ncwa} from dividing the weighted
sum of the variable (the numerator in the averaging expression) by the
weighted sum of the weights (the denominator in the averaging
expression).
Thus @samp{-N} tells @command{ncwa} to return just the numerator of the
arithmetic expression defining the operation (@pxref{Operation Types}).
With this normalization option, @command{ncwa} can integrate variables.
Averages are first computed as sums, and then normalized to obtain the
average.
The original sum (i.e., the numerator of the expression in
@ref{Operation Types}) is output if default normalization is turned off
(@w{with @samp{-N}}).
This sum is the integral (not the average) over the specified
(@w{with @samp{-a}}, or all, if none are specified) dimensions.
The weighting variable, if specified (@w{with @samp{-w}}), plays the
role of the differential increment and thus permits more sophisticated
integrals (i.e., weighted sums) to be output.
For example, consider the variable
@code{lev} where @math{@var{lev} = [100,500,1000]} weighted by
the weight @code{lev_wgt} where @math{@var{lev_wgt} = [10,2,1]}.
@cindex dot product
The vertical integral of @code{lev}, weighted by @code{lev_wgt},
is the dot product of @var{lev} and @var{lev_wgt}.
That this is @w{is 3000.0} can be seen by inspection and verified with
the integration command
@example
ncwa -N -a lev -v lev -w lev_wgt in.nc foo.nc;ncks foo.nc
@end example
@ignore
fxm TODO nco702
As explained in @xref{Operation Types}, @command{ncwa}
@emph{always averages} coordinate variables regardless of the arithmetic
operation type performed on the non-coordinate variables.
The single exception is shown in the above example.
The @samp{-N} switch turns off normalization so variable which are to be
averaged, including coordinate variables, are not normalized.
This is equivalent to summation or integration.
@end ignore
@ignore
@c NB: these masking features are deprecated
The second normalization option tells @command{ncwa} to multiply the
weighted average the variable (given by the averaging expression)
by the @w{tally, @var{M}}.
Thus this option is similar to integration---multiplying the mean value
of a quantity by the number of gridpoints to which it applies.
The third normalization option is equivalent to specifying the first two
options simultaneously.
In other words this option causes @command{ncwa} to return
@w{@var{M} times} the numerator of the generic averaging expression.
With these normalization options, @command{ncwa} can compute
sophisticated averages (and integrals) from the command line.
@end ignore
@noindent
@html
<a name="xmp_ncwa"></a> <!-- http://nco.sf.net/nco.html#xmp_ncwa -->
@end html
EXAMPLES
Given file @file{85_0112.nc}:
@example
netcdf 85_0112 @{
dimensions:
lat = 64 ;
lev = 18 ;
lon = 128 ;
time = UNLIMITED ; // (12 currently)
variables:
float lat(lat) ;
float lev(lev) ;
float lon(lon) ;
float time(time) ;
float scalar_var ;
float three_dmn_var(lat, lev, lon) ;
float two_dmn_var(lat, lev) ;
float mask(lat, lon) ;
float gw(lat) ;
@}
@end example
Average all variables in @file{in.nc} over all dimensions and store
results in @file{out.nc}:
@example
ncwa in.nc out.nc
@end example
@noindent
All variables in @file{in.nc} are reduced to scalars in @file{out.nc}
since @command{ncwa} averages over all dimensions unless otherwise
specified (with @samp{-a}).
Store the zonal (longitudinal) mean of @file{in.nc} in @file{out.nc}:
@example
ncwa -a lon in.nc out1.nc
ncwa -a lon -b in.nc out2.nc
@end example
@noindent
@cindex degenerate dimension
The first command turns @code{lon} into a scalar and the second retains
@code{lon} as a degenerate dimension in all variables.
@example
% ncks -C -H -v lon out1.nc
lon = 135
% ncks -C -H -v lon out2.nc
lon[0] = 135
@end example
In either case the tally is simply the size of @code{lon}, i.e.,
for the @file{85_0112.nc} file described by the sample header above.
@cindex @code{gw}
@cindex Gaussian weights
@cindex climate model
Compute the meridional (latitudinal) mean, with values weighted by
the corresponding element of @var{gw}
@footnote{@code{gw} stands for @dfn{Gaussian weight} in many
climate models.}:
@example
ncwa -w gw -a lat in.nc out.nc
@end example
@noindent
Here the tally is simply the size of @code{lat}, @w{or 64.}
The sum of the Gaussian weights @w{is 2.0.}
Compute the area mean over the tropical Pacific:
@example
ncwa -w gw -a lat,lon -d lat,-20.,20. -d lon,120.,270. in.nc out.nc
@end example
@noindent
Here the tally is
@set flg
@tex
$64 \times 128 = 8192$.
@clear flg
@end tex
@ifset flg
64 times 128 = 8192.
@clear flg
@end ifset
@cindex @code{ORO}
@cindex climate model
Compute the area-mean over the globe using only points for which
@set flg
@tex
$ORO < 0.5$
@clear flg
@end tex
@ifset flg
@var{ORO} < 0.5
@clear flg
@end ifset
@footnote{@code{ORO} stands for @dfn{Orography} in some climate models.
@math{@var{ORO} < 0.5} selects ocean-covered gridpoints.}:
@example
ncwa -B "ORO < 0.5" -w gw -a lat,lon in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon in.nc out.nc
@end example
@noindent
It is considerably simpler to specify the complete @var{mask_cond} with
the single string argument to @samp{-B} than with the three separate
switches @samp{-m}, @samp{-T}, and @samp{-M}.
If in doubt, enclose the @var{mask_cond} with double quotes since some
of the comparators have special meanings to the shell.
Assuming 70% of the gridpoints are maritime, then here the tally is
@set flg
@tex
$0.70 \times 8192 \approx 5734$.
@clear flg
@end tex
@ifset flg
0.70 times 8192 = 5734.
@clear flg
@end ifset
Compute the global annual mean over the maritime tropical Pacific:
@example
ncwa -B "ORO < 0.5" -w gw -a lat,lon,time \
-d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
ncwa -m ORO -M 0.5 -T lt -w gw -a lat,lon,time \
-d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
@end example
Further examples will use the one-switch specification of
@var{mask_cond}.
Determine the total area of the maritime tropical Pacific, assuming
the variable @var{area} contains the area of each gridcell
@example
ncwa -N -v area -B "ORO < 0.5" -a lat,lon \
-d lat,-20.0,20.0 -d lon,120.0,270.0 in.nc out.nc
@end example
Weighting @var{area} (e.g., by @var{gw}) is not appropriate because
@var{area} is @emph{already} area-weighted by definition.
Thus the @samp{-N} switch, or, equivalently, the @samp{-y ttl} switch,
correctly integrate the cell areas into a total regional area.
@cindex mask condition
@cindex truth condition
Mask a file to contain @var{_FillValue} everywhere except where
@math{@var{thr_min} <= @var{msk_var} <= @var{thr_max}}:
@example
# Set masking variable and its scalar thresholds
export msk_var='three_dmn_var_dbl' # Masking variable
export thr_max='20' # Maximum allowed value
export thr_min='10' # Minimum allowed value
ncecat -O in.nc out.nc # Wrap out.nc in degenerate "record" dimension
ncwa -O -a record -B "$@{msk_var@} <= $@{thr_max@}" out.nc out.nc
ncecat -O out.nc out.nc # Wrap out.nc in degenerate "record" dimension
ncwa -O -a record -B "$@{msk_var@} >= $@{thr_min@}" out.nc out.nc
@end example
After the first use of @command{ncwa}, @file{out.nc} contains
@var{_FillValue} where @code{$@{msk_var@} >= $@{thr_max@}}.
The process is then repeated on the remaining data to filter out
points where @code{$@{msk_var@} <= $@{thr_min@}}.
The resulting @file{out.nc} contains valid data only
where @math{@var{thr_min} <= @var{msk_var} <= @var{thr_max}}.
@html
<a name="ctr"></a> <!-- http://nco.sf.net/nco.html#ctr -->
@end html
@node Contributing, CCSM Example, Operator Reference Manual, Top
@chapter Contributing
@cindex contributing
We welcome contributions from anyone.
The @acronym{NCO} project homepage at
@uref{https://sf.net/projects/nco}
contains more information on how to contribute.
@cindex PayPal
Financial contributions to @acronym{NCO} development may be made through
@uref{https://www.paypal.com/xclick/business=zender%40uci.edu&item_name=NCO+development&item_number=nco_dnt_dvl&no_note=1&tax=0¤cy_code=USD, PayPal}.
@acronym{NCO} has been shared for over @w{10 years} yet only two
users have contributed any money to the developers
@footnote{
@cindex chocolate
Happy users have sent me a few gifts, though.
This includes a box of imported chocolate.
Mmm.
Appreciation and gifts are definitely better than money.
Naturally, I'm too lazy to split and send gifts to the other developers.
However, unlike some @acronym{NCO} developers, I have a steady "real job".
My intent is to split monetary donations among the active developers
and to send them their shares via PayPal.}.
So you could be the third!
@html
<a name="dvl"></a> <!-- http://nco.sf.net/nco.html#dvl -->
<a name="cnt"></a> <!-- http://nco.sf.net/nco.html#cnt -->
<a name="ppl"></a> <!-- http://nco.sf.net/nco.html#ppl -->
@end html
@menu
* Contributors::
* Proposals for Institutional Funding::
@end menu
@node Contributors, Proposals for Institutional Funding, Contributing, Contributing
@section Contributors
@cindex contributors
The primary contributors to @acronym{NCO} development have been:
@table @asis
@cindex Charlie Zender
@item Charlie Zender
Concept, design and implementation of operators from 1995-2000.
Since then autotools, bug-squashing, documentation, packing,
@acronym{NCO} library redesign, @command{ncap2} features, @command{ncbo},
@command{ncpdq}, SMP threading and MPI parallelization,
netCDF4 integration, project coordination, and releases.
@cindex Henry Butowsky
@item Henry Butowsky
Non-linear operations and @code{min()}, @code{max()}, @code{total()}
support in @command{ncra} and @command{ncwa}.
Type conversion for arithmetic.
Migration to netCDF3 API.
@command{ncap} parser, lexer, @w{and I/O}.
Multislabbing algorithm.
Variable wildcarding.
Various hacks.
@command{ncap2} language.
@cindex Rorik Peterson
@item Rorik Peterson
Autotool build support.
Long command-line options.
UDUnits support.
Debianization.
Numerous bug-fixes.
@cindex Daniel Wang
@item Daniel Wang
Script Workflow Analysis for MultiProcessing (SWAMP).
RPM support.
@cindex Harry Mangalam
@item Harry Mangalam
Benchmarking.
OPeNDAP configuration.
@cindex Brian Mays
@item Brian Mays
Packaging for Debian GNU/Linux, @command{nroff} man pages.
@cindex George Shapovalov
@item George Shapovalov
Packaging for Gentoo GNU/Linux.
@cindex Bill Kocik
@item Bill Kocik
Memory management.
@cindex Len Makin
@item Len Makin
NEC SX architecture support.
@cindex Jim Edwards
@item Jim Edwards
AIX architecture support.
@cindex Juliana Rew
@item Juliana Rew
Compatibility with large PIDs.
@cindex Karen Schuchardt
@item Karen Schuchardt
Auxiliary coordinate support.
@cindex Gayathri Venkitachalam
@item Gayathri Venkitachalam
@acronym{MPI} implementation.
@cindex Scott Capps
@item Scott Capps
Large work-load testing
@cindex Mark Flanner
@cindex Keith Lindsay
@cindex Martin Dix
@cindex Mike Page
@cindex Martin Schmidt
@cindex Michael Schulz
@cindex Remik Ziemlinski
@item Martin Dix, Mark Flanner, Keith Lindsay, Mike Page, Martin Schmidt, Michael Schulz, Remik Ziemlinski
Excellent bug reports and feature requests.
@end table
@html
<a name="prp"></a> <!-- http://nco.sf.net/nco.html#prp -->
<a name="prp_sei"></a> <!-- http://nco.sf.net/nco.html#prp_sei -->
<a name="fnd"></a> <!-- http://nco.sf.net/nco.html#fnd -->
@end html
@menu
* Proposals for Institutional Funding::
@end menu
@node Proposals for Institutional Funding, , Contributors, Contributing
@section Proposals for Institutional Funding
@cindex funding
@cindex proposals
@cindex @acronym{NSF}
@cindex server-side processing
@cindex Distributed Data Reduction & Analysis
@cindex Scientific Data Operators
@cindex @acronym{DDRA}
@cindex Server-Side Distributed Data Reduction & Analysis
@cindex @acronym{SSDDRA}
@cindex @acronym{CCSM}
@cindex @acronym{IPCC}
@cindex @acronym{NSF}
@cindex @acronym{SDO}
@cindex @acronym{SEIII}
@cindex OptIPuter
@acronym{NSF} has funded a
@uref{http://nco.sf.net#prp_sei, project}
to improve Distributed Data Reduction & Analysis (@acronym{DDRA}) by
evolving @acronym{NCO} into a suite of Scientific Data Operators called
@acronym{SDO}.
@cindex parallelism
The two main components of this project are @acronym{NCO} parallelism
(OpenMP, @acronym{MPI}) and Server-Side @acronym{DDRA}
(@acronym{SSDDRA}) implemented through extensions to @acronym{OPeNDAP}
and netCDF4.
This project will dramatically reduce bandwidth usage for @acronym{NCO}
@acronym{DDRA}.
@cindex @acronym{NASA}
@cindex @acronym{NRA}
@cindex @acronym{HDF}
With this first @acronym{NCO} proposal funded, the content of the
next @acronym{NCO} proposal is clear.
We are interested in obtaining @acronym{NASA} support for
@acronym{HDF}-specific enhancements to @acronym{NCO}.
We plan to submit a proposal to the next suitable @acronym{NASA}
@acronym{NRA} or @acronym{NSF} opportunity.
We are considering a lot of interesting ideas for still more proposals.
Please contact us if you wish to be involved with any future
@acronym{NCO}-related proposals.
Comments on the proposals and letters of support are also very welcome.
@html
<a name="ccsm"></a> <!-- http://nco.sf.net/nco.html#ccsm -->
@end html
@node CCSM Example, General Index, Contributing, Top
@chapter CCSM Example
@cindex CCSM
This chapter presents an in depth example of using @acronym{NCO} to
process and analyze the results of a @acronym{CCSM} climate simulation.
@example
************************************************************************
Task 0: Finding input files
************************************************************************
The CCSM model outputs files to a local directory like:
/ptmp/zender/archive/T42x1_40
Each component model has its own subdirectory, e.g.,
/ptmp/zender/archive/T42x1_40/atm
/ptmp/zender/archive/T42x1_40/cpl
/ptmp/zender/archive/T42x1_40/ice
/ptmp/zender/archive/T42x1_40/lnd
/ptmp/zender/archive/T42x1_40/ocn
within which model output is tagged with the particular model name
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-03.nc
...
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0001-12.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-01.nc
/ptmp/zender/archive/T42x1_40/atm/T42x1_40.cam2.h0.0002-02.nc
...
or
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-01.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-02.nc
/ptmp/zender/archive/T42x1_40/lnd/T42x1_40.clm2.h0.0001-03.nc
...
************************************************************************
Task 1: Regional processing
************************************************************************
The first task in data processing is often creating seasonal cycles.
Imagine a 100-year simulation with its 1200 monthly mean files.
Our goal is to create a single file containing 12 months of data.
Each month in the output file is the mean of 100 input files.
Normally, we store the "reduced" data in a smaller, local directory.
caseid='T42x1_40'
#drc_in="$@{DATA@}/archive/$@{caseid@}/atm"
drc_in="$@{DATA@}/$@{caseid@}"
drc_out="$@{DATA@}/$@{caseid@}"
mkdir -p $@{drc_out@}
cd $@{drc_out@}
Method 1: Assume all data in directory applies
for mth in @{1..12@}; do
mm=`printf "%02d" $mth`
ncra -O -D 1 -o $@{drc_out@}/$@{caseid@}_clm$@{mm@}.nc \
$@{drc_in@}/$@{caseid@}.cam2.h0.*-$@{mm@}.nc
done # end loop over mth
Method 2: Use shell 'globbing' to construct input filenames
for mth in @{1..12@}; do
mm=`printf "%02d" $mth`
ncra -O -D 1 -o $@{drc_out@}/$@{caseid@}_clm$@{mm@}.nc \
$@{drc_in@}/$@{caseid@}.cam2.h0.00??-$@{mm@}.nc \
$@{drc_in@}/$@{caseid@}.cam2.h0.0100-$@{mm@}.nc
done # end loop over mth
Method 3: Construct input filename list explicitly
for mth in @{1..12@}; do
mm=`printf "%02d" $mth`
fl_lst_in=''
for yr in @{1..100@}; do
yyyy=`printf "%04d" $yr`
fl_in=$@{caseid@}.cam2.h0.$@{yyyy@}-$@{mm@}.nc
fl_lst_in="$@{fl_lst_in@} $@{caseid@}.cam2.h0.$@{yyyy@}-$@{mm@}.nc"
done # end loop over yr
ncra -O -D 1 -o $@{drc_out@}/$@{caseid@}_clm$@{mm@}.nc -p $@{drc_in@} \
$@{fl_lst_in@}
done # end loop over mth
Make sure the output file averages correct input files!
ncks -M prints global metadata:
ncks -M $@{drc_out@}/$@{caseid@}_clm01.nc
The input files ncra used to create the climatological monthly mean
will appear in the global attribute named 'history'.
Use ncrcat to aggregate the climatological monthly means
ncrcat -O -D 1 \
$@{drc_out@}/$@{caseid@}_clm??.nc $@{drc_out@}/$@{caseid@}_clm_0112.nc
Finally, create climatological means for reference.
The climatological time-mean:
ncra -O -D 1 \
$@{drc_out@}/$@{caseid@}_clm_0112.nc $@{drc_out@}/$@{caseid@}_clm.nc
The climatological zonal-mean:
ncwa -O -D 1 -a lon \
$@{drc_out@}/$@{caseid@}_clm.nc $@{drc_out@}/$@{caseid@}_clm_x.nc
The climatological time- and spatial-mean:
ncwa -O -D 1 -a lon,lat,time -w gw \
$@{drc_out@}/$@{caseid@}_clm.nc $@{drc_out@}/$@{caseid@}_clm_xyt.nc
This file contains only scalars, e.g., "global mean temperature",
used for summarizing global results of a climate experiment.
Climatological monthly anomalies = Annual Cycle:
Subtract climatological mean from climatological monthly means.
Result is annual cycle, i.e., climate-mean has been removed.
ncbo -O -D 1 -o $@{drc_out@}/$@{caseid@}_clm_0112_anm.nc \
$@{drc_out@}/$@{caseid@}_clm_0112.nc $@{drc_out@}/$@{caseid@}_clm_xyt.nc
************************************************************************
Task 2: Correcting monthly averages
************************************************************************
The previous step appoximates all months as being equal, so, e.g.,
February weighs slightly too much in the climatological mean.
This approximation can be removed by weighting months appropriately.
We must add the number of days per month to the monthly mean files.
First, create a shell variable dpm:
unset dpm # Days per month
declare -a dpm
dpm=(0 31 28.25 31 30 31 30 31 31 30 31 30 31) # Allows 1-based indexing
Method 1: Create dpm directly in climatological monthly means
for mth in @{1..12@}; do
mm=`printf "%02d" $@{mth@}`
ncap2 -O -s "dpm=0.0*date+$@{dpm[$@{mth@}]@}" \
$@{drc_out@}/$@{caseid@}_clm$@{mm@}.nc $@{drc_out@}/$@{caseid@}_clm$@{mm@}.nc
done # end loop over mth
Method 2: Create dpm by aggregating small files
for mth in @{1..12@}; do
mm=`printf "%02d" $@{mth@}`
ncap2 -O -v -s "dpm=$@{dpm[$@{mth@}]@}" ~/nco/data/in.nc \
$@{drc_out@}/foo_$@{mm@}.nc
done # end loop over mth
ncecat -O -D 1 -p $@{drc_out@} -n 12,2,2 foo_$@{mm@}.nc foo.nc
ncrename -O -D 1 -d record,time $@{drc_out@}/foo.nc
ncatted -O -h \
-a long_name,dpm,o,c,"Days per month" \
-a units,dpm,o,c,"days" \
$@{drc_out@}/$@{caseid@}_clm_0112.nc
ncks -A -v dpm $@{drc_out@}/foo.nc $@{drc_out@}/$@{caseid@}_clm_0112.nc
Method 3: Create small netCDF file using ncgen
cat > foo.cdl << EOF
netcdf foo @{
dimensions:
time=unlimited;
variables:
float dpm(time);
dpm:long_name="Days per month";
dpm:units="days";
data:
dpm=31,28.25,31,30,31,30,31,31,30,31,30,31;
@}
EOF
ncgen -b -o foo.nc foo.cdl
ncks -A -v dpm $@{drc_out@}/foo.nc $@{drc_out@}/$@{caseid@}_clm_0112.nc
Another way to get correct monthly weighting is to average daily
output files, if available.
************************************************************************
Task 3: Regional processing
************************************************************************
Let's say you are interested in examining the California region.
Hyperslab your dataset to isolate the appropriate latitude/longitudes.
ncks -O -D 1 -d lat,30.0,37.0 -d lon,240.0,270.0 \
$@{drc_out@}/$@{caseid@}_clm_0112.nc $@{drc_out@}/$@{caseid@}_clm_0112_Cal.nc
The dataset is now much smaller!
To examine particular metrics.
************************************************************************
Task 4: Accessing data stored remotely
************************************************************************
OPeNDAP server examples:
UCI DAP servers:
ncks -M -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata in.nc
ncrcat -O -C -D 3 -p http://soot.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
-l /tmp in.nc in.nc ~/foo.nc
ncrcat -O -C -D 3 -p http://dust.ess.uci.edu/cgi-bin/dods/nph-dods/dodsdata \
-l /tmp in.nc in.nc ~/foo.nc
NOAA DAP servers:
ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l /tmp -p \
http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc ~/foo.nc
LLNL PCMDI IPCC OPeNDAP Data Portal:
ncks -M -p http://username:password@@esgcet.llnl.gov/cgi-bin/dap-cgi.py/ipcc4/sresa1b/ncar_ccsm3_0 pcmdi.ipcc4.ncar_ccsm3_0.sresa1b.run1.atm.mo.xml
Earth System Grid (ESG): http://www.earthsystemgrid.org
caseid='b30.025.ES01'
CCSM3.0 1% increasing CO2 run, T42_gx1v3, 200 years starting in year 400
Atmospheric post-processed data, monthly averages, e.g.,
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0449-12.nc
/data/zender/tmp/b30.025.ES01.cam2.h0.TREFHT.0400-01_cat_0599-12.nc
ESG supports password-protected FTP access by registered users
NCO uses the .netrc file, if present, for password-protected FTP access
Syntax for accessing single file is, e.g.,
ncks -O -D 3 \
-p ftp://climate.llnl.gov/sresa1b/atm/mo/tas/ncar_ccsm3_0/run1 \
-l /tmp tas_A1.SRESA1B_1.CCSM.atmm.2000-01_cat_2099-12.nc ~/foo.nc
# Average surface air temperature tas for SRESA1B scenario
# This loop is illustrative and will not work until NCO correctly
# translates '*' to FTP 'mget' all remote files
for var in 'tas'; do
for scn in 'sresa1b'; do
for mdl in 'cccma_cgcm3_1 cccma_cgcm3_1_t63 cnrm_cm3 csiro_mk3_0 \
gfdl_cm2_0 gfdl_cm2_1 giss_aom giss_model_e_h giss_model_e_r \
iap_fgoals1_0_g inmcm3_0 ipsl_cm4 miroc3_2_hires miroc3_2_medres \
miub_echo_g mpi_echam5 mri_cgcm2_3_2a ncar_ccsm3_0 ncar_pcm1 \
ukmo_hadcm3 ukmo_hadgem1'; do
for run in '1'; do
ncks -R -O -D 3 -p ftp://climate.llnl.gov/$@{scn@}/atm/mo/$@{var@}/$@{mdl@}/run$@{run@} -l $@{DATA@}/$@{scn@}/atm/mo/$@{var@}/$@{mdl@}/run$@{run@} '*' $@{scn@}_$@{mdl@}_$@{run@}_$@{var@}_$@{yyyymm@}_$@{yyyymm@}.nc
done # end loop over run
done # end loop over mdl
done # end loop over scn
done # end loop over var
cd sresa1b/atm/mo/tas/ukmo_hadcm3/run1/
ncks -H -m -v lat,lon,lat_bnds,lon_bnds -M tas_A1.nc | m
bds -x 096 -y 073 -m 33 -o $@{DATA@}/data/dst_3.75x2.5.nc # ukmo_hadcm3
ncview $@{DATA@}/data/dst_3.75x2.5.nc
# msk_rgn is California mask on ukmo_hadcm3 grid
# area is correct area weight on ukmo_hadcm3 grid
ncks -A -v area,msk_rgn $@{DATA@}/data/dst_3.75x2.5.nc \
$@{DATA@}/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/area_msk_ukmo_hadcm3.nc
Template for standardized data:
$@{scn@}_$@{mdl@}_$@{run@}_$@{var@}_$@{yyyymm@}_$@{yyyymm@}.nc
e.g., raw data
$@{DATA@}/sresa1b/atm/mo/tas/ukmo_hadcm3/run1/tas_A1.nc
becomes standardized data
Level 0: raw from IPCC site--no changes except for name
Make symbolic link name match raw data
Template: $@{scn@}_$@{mdl@}_$@{run@}_$@{var@}_$@{yyyymm@}_$@{yyyymm@}.nc
ln -s -f tas_A1.nc sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc
area_msk_ukmo_hadcm3.nc
Level I: Add all variables (but not standardized in time)
to file containing msk_rgn and area
Template: $@{scn@}_$@{mdl@}_$@{run@}_$@{yyyymm@}_$@{yyyymm@}.nc
/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas sresa1b_ukmo_hadcm3_run1_tas_200101_209911.nc \
sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v pr sresa1b_ukmo_hadcm3_run1_pr_200101_209911.nc \
sresa1b_ukmo_hadcm3_run1_200101_209911.nc
If already have file then:
mv sresa1b_ukmo_hadcm3_run1_200101_209911.nc foo.nc
/bin/cp area_msk_ukmo_hadcm3.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
ncks -A -v tas,pr foo.nc sresa1b_ukmo_hadcm3_run1_200101_209911.nc
Level II: Correct # years, months
Template: $@{scn@}_$@{mdl@}_$@{run@}_$@{var@}_$@{yyyymm@}_$@{yyyymm@}.nc
ncks -d time,....... file1.nc file2.nc
ncrcat file2.nc file3.nc sresa1b_ukmo_hadcm3_run1_200001_209912.nc
Level III: Many derived products from level II, e.g.,
A. Global mean timeseries
ncwa -w area -a lat,lon \
sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
sresa1b_ukmo_hadcm3_run1_200001_209912_xy.nc
B. Califoria average timeseries
ncwa -m msk_rgn -w area -a lat,lon \
sresa1b_ukmo_hadcm3_run1_200001_209912.nc \
sresa1b_ukmo_hadcm3_run1_200001_209912_xy_Cal.nc
@end example
@c @node Name Index, General Index, Operators, Top
@c @unnumbered Function and Variable Index
@c @printindex fn
@html
<a name="index"></a> <!-- http://nco.sf.net/nco.html#index -->
<a name="idx"></a> <!-- http://nco.sf.net/nco.html#idx -->
@end html
@node General Index, , CCSM Example, Top
@unnumbered General Index
@syncodeindex fn cp
@printindex cp
@c Print table of contents
@contents
@c TTFN (Ta ta for now)
@bye
distrib
>
Mandriva
>
2010.0
>
i586
>
media
>
contrib-release
>
by-pkgid
>
c19e19dccb2f9a49f95629cbbd11307c
>
files
>
72
