<html>
<head>
<title>Chapter 13: Scalability and ZEO</title>
</head>
<body bgcolor="#FFFFFF">
<h1>Chapter 13: Scalability and ZEO</h1>
<p> When a web site gets more requests than it can handle it can become slow
and unresponsive. In the worst case too many requests to a web site can
cause the server to completely overload, stop handling requests and
possibly even crash. This can be a problem for <em>any</em> kind of server
application, not just Zope. The obvious solution to this problem is to
use more than one computer, so in case one computer fails, another
computer can continue to serve up your web site.</p><p> Using multiple computers has obvious benefits, but it also has some
drawbacks. For example, if you had five computers running Zope then you
must ensure that all five Zope installations have the same information on
them. This is not a very hard task if you're the only user and you have
only a few static objects, but for large organizations with thousands of
rapidly changing objects, keeping five separate Zope installations
synchronized manually would be a nightmare. To solve this problem,
Zope Corporation created <a href="http://www.zope.org/Products/ZEO">Zope Enterprise
Objects</a>, or ZEO. This chapter gives you
a brief overview on installing ZEO, but there are many other options we
don't cover. For more in-depth information, see the documentation that
comes with the ZEO package, and also take a look at the <a href="http://www.zope.org/Wikis/ZODB/FrontPage">ZEO discussion
area</a>.</p><h2> What is ZEO?</h2>
<p> ZEO is a system that allows you to run your site on more than one
computer. This is often called <em>clustering</em> and <em>load balancing</em>.
By running Zope on multiple computers, you can spread the requests
evenly around and add more computers as the number of requests
grows. Further, if one computer fails or crashes, other computers
can still service requests while you fix the broken one.</p><p> ZEO runs Zope on multiple computers and takes care of making sure
all the Zope installations share the exact same database at all
times. ZEO uses a client/server architecture. The Zope
installations on multiple computers are the <em>ZEO Clients</em>. All of
the clients connect to one, central <em>ZEO Storage Server</em>, as shown
in <a href="#11-1">Figure 11-1</a>.</p><p> <a name="11-1"></a>
<img src="Figures/11-1.png" alt="Simple ZEO illustration">
<p><b>Figure 11-1</b> Simple ZEO illustration</p>
</p><p> The terminology can be a bit confusing, because normally you think
of Zope as a server, not a client. When using ZEO, your Zope
processes act as both servers (for web requests) and clients (for
data from the ZEO server).</p><p> ZEO clients and servers communicate using standard Internet protocols,
so they can be in the same room or in different countries. ZEO, in
fact, can distribute a Zope site all over the world. In this chapter
we'll explore some interesting ways you can distribute your ZEO
clients.</p><h2> When you should use ZEO</h2>
<p> ZEO serves many hits in a fail-safe way. If your site does not get
millions of hits, then you probably don't need ZEO. There is no
hard-and-fast rule about when you should and should not use ZEO, but
for the most part you should not need to run ZEO unless:<ul>
<li>Your site is getting too many hits for your computer to handle them
quickly. Zope is a high-performance system, and one Zope can
handle millions of hits per day (depending on your hardware, of
course). If you need to serve more hits than that, then you
should use ZEO.</li>
<li>Your site is very critical and requires constant, 24/7 uptime. In
this case, ZEO will allow you to have multiple fail-over servers.</li>
<li>You want to distribute your site globally to many different mirror
ZEO clients.</li>
<li>You want to debug one ZEO client while others are still serving
requests. This is a very advanced technique for Python developers
and is not covered in this book.</li>
</ul>
</p><p> All of these cases are fairly advanced, high-end uses of Zope.
Installing, configuring, and maintaining systems like these requires
advanced system administration knowledge and resources. Most Zope
users will not need ZEO, or may not have the expertise necessary to
maintain a distributed server system like ZEO. ZEO is fun, and can be
very useful, but before jumping head-first and installing ZEO in your
system you should weigh the extra administrative burden ZEO creates
against the simplicity of running just a simple, stand-alone Zope.</p><h2> Installing and Running ZEO</h2>
<p> The most common ZEO setup is one ZEO server and multiple ZEO clients.
Before installing and configuring ZEO though, consider the following
issues:<ul>
<li>All of the ZEO clients and servers must run the same version of
Zope. Make sure all of your computers use the latest version.
This is necessary, or Zope may behave abnormally or not work at
all.</li>
<li>All of your ZEO clients must have the same third party Products
installed and they must be the same version. This is necessary, or
your third-party objects may behave abnormally or not work at all.</li>
<li>If your Zope system requires access to external resources, like
mail servers or relational databases, ensure that all of your ZEO
clients have access to those resources.</li>
<li>Slow or intermittent network connections between clients and server
degrade the performance of your ZEO clients. Your ZEO clients
should have a good connection to their server.</li>
</ul>
</p><p> ZEO is not distributed with Zope, you must download it from the
<a href="http://www.zope.org/Products/ZEO">Products Section</a> of Zope.org.</p><p> Installing ZEO requires a little bit of manual preparation. To
install ZEO, download the <em>ZEO-1.0.tgz</em> from the
<a href="http://www.zope.org/Products/ZEO">Zope.org</a> web site and place it
in your Zope installation directory. Now, unpack the tarball. On
Unix, this can be done with the following command:<pre> $ tar -zxf ZEO-1.0.tgz</pre>
</p><p> On Windows, you can unpack the archive with WinZip. Before
installing ZEO, make sure you back up your Zope system first.</p><p> Now you should have a <em>ZEO-1.0</em> directory. Next, you have to copy
some files into your Zope top level <em>lib/python</em> directory. This
can be done on UNIX with:<pre> $ cp -R ZEO-1.0/ZEO lib/python</pre>
</p><p> If you're running windows, you can use the following DOS commands
to copy your ZEO files:<pre> C:\...Zope\>xcopy ZEO-1.0\* lib\python /S</pre>
</p><p> Now, you have to create a special file in your Zope root directory
called <em>custom_zodb.py</em>. In that file, put the following python code:<pre> import ZEO.ClientStorage
Storage=ZEO.ClientStorage.ClientStorage(('localhost',7700))</pre>
</p><p> This will configure your Zope to run as a ZEO client. If you pass
ClientStorage a tuple, as this code does, the tuple must have two
elements, a string which contains the address to the server, and the
port that the server is listening on. In this example, we're going to
show you how to run both the clients and the servers on the same
machine, so the machine name is set to <code>localhost</code>.</p><p> Now, you have ZEO properly configured to run on one computer. Try it
out by first starting the server. Go to your Zope top level directory
in a terminal window or DOS box and type:<pre> python lib/python/ZEO/start.py -p 7700</pre>
</p><p> This will start the ZEO server listening on port 7700 on your
computer. Now, in another window, start up Zope like you normally
would, with the <em>z2.py</em> script:<pre> $ python z2.py -D
------
2000-10-04T20:43:11 INFO(0) client Trying to connect to server
------
2000-10-04T20:43:11 INFO(0) ClientStorage Connected to storage
------
2000-10-04T20:43:12 PROBLEM(100) ZServer Computing default pinky
------
2000-10-04T20:43:12 INFO(0) ZServer Medusa (V1.19) started at Wed Oct 4 15:43:12 2000
Hostname: pinky.zopezoo.org
Port:8080</pre>
</p><p> Notice how in the above example, Zope tells you <em>client Trying to
connect to server</em> and then <em>ClientStorage Connected to storage</em>. This
means your ZEO client has successfully connected to your ZEO server.
Now, you can visit <em>http://localhost:8080/manage</em> (or whatever URL your
ZEO client is listening on) and log into Zope as usual.</p><p> As you can see, everything looks the same. Go to the <em>Control
Panel</em> and click on <em>Database Managment</em>. Here, you see that Zope
is connected to a <em>ZEO Storage</em> and that its state is <em>connected</em>.</p><p> Running ZEO on one computer is a great way to familiarize yourself
with ZEO and how it works. Running ZEO on one computer does not,
however, improve the speed of your site, and in fact, it may slow
it down just a little. To really get the speed benefits that ZEO
provides, you need to run ZEO on several computers, which is
explained in the next section.</p><h2> How to Run ZEO on Many Computers</h2>
<p> Setting up ZEO to run on multiple computers is very similar to running
ZEO on one computer. There are generally two steps, the first step is
to start the ZEO server, and the second step is to start one or more
ZEO clients.</p><p> For example, let's say you have four computers. One computer named
<em>zooserver</em> will be your ZEO server, and the other three computers, named
<em>zeoclient1</em>, <em>zeoclient2</em> and <em>zeoclient3</em>, will be your ZEO clients. </p><p> The first step is to run the server on <em>zooserver</em>. To tell your
ZEO server to listen on the tcp socket at port 9999 on the
<em>zooserver</em> interface, run the server with the <em>start.py</em> script
like this:<pre> $ python lib/python/ZEO/start.py -p 9999 -h zooserver.zopezoo.org</pre>
</p><p> This will start the ZEO server. Now, you can start up your clients by
going to each client and configuring each of them with the following
<em>custom_zodb.py</em>:<pre> import ZEO.ClientStorage
Storage=ZEO.ClientStorage.ClientStorage(('zooserver.zopezoo.org',9999))</pre>
</p><p> Now, you can start each client's z2.py script as shown in the
previous section, <em>Installing and Running ZEO</em>. Notice how the
host and port for each client is the same, this is so they all
connect to the same server. By following this procedure for each
of your three clients you will have three different Zope's all
serving the same Zope site. You can verify this by going visiting
port 8080 on all three of your ZEO client machines.</p><p> You probably want to run ZEO on more than one computer so that you
can take advantage of the speed increase this gives you. Running
more computers means that you can serve more hits per second than
with just one computer. Distributing the load of your web site's
visitors however does require a bit more elaboration in your
system. The next section describes why, and how, you distribute
the load of your visitors among many computers.</p><h2> How to Distribute Load</h2>
<p> In the previous example you have a ZEO server named <em>zooServer</em>
and three ZEO clients named <em>zeoclient1</em>, <em>zeoclient2</em>, and
<em>zeoclient3</em>. The three ZEO clients are connected to the ZEO
server and each client is verified to work properly.</p><p> Now you have three computers that serve content to your users.
The next problem is how to actually spread the incoming web
requests evenly among the three ZEO clients. Your users only know
about <em>www.zopezoo.org</em>, not <em>zeoclient1</em>, <em>zeoclient2</em> or
<em>zeoclient3</em>. It would be a hassle to tell only some users to use
<em>zeoclient1</em>, and others to use <em>zeoclient3</em>, and it wouldn't be
very good use of your computing resources. You want to automate,
or at least make very easy, the process of evenly distributing
requests to your various ZEO clients.</p><p> There are a number of solutions to this problem, some easy, some
advanced, and some expensive. The next section goes over the more
common ways of spreading web requests around various computers
using different kinds of technology, some of them based on
freely-available or commercial software, and some of them based on
special hardware.</p><h3> User Chooses a Mirror</h3>
<p> The easiest way to distribute requests across many web servers
is to pick from a list of <em>mirrored sites</em>, each of which is a
ZEO client. Using this method requires no extra software or
hardware, it just requires the maintenance of a list of mirror
servers. By presenting your users with a menu of mirrors, they
can use to choose which server to use.</p><p> Note that this method of distributing requests is passive (you have
no active control over which clients are used) and voluntary (your
users need to make a voluntary choice to use another ZEO client).
If your users do not use a mirror, then the requests will go to your
ZEO client that serves <em>www.zopezoo.org</em>.</p><p> If you do not have any administrative control over your mirrors,
then this can be a pretty easy solution. If your mirrors go
off-line, your users can always choose to come back to the
master site which you <em>do</em> have administrative control over and
choose a different mirror.</p><p> On a global level, this method improves performance. Your users can
choose to use a server that is geographically closer to them, which
probably results in faster access. For example, if your main server
was in Portland, Oregon on the west coast of the USA and you had
users in London, England, they could choose your London mirror and
their request would not have to go half-way across the world and
back.</p><p> To use this method, create a property in your root folder of type
<em>lines</em> named "mirror_servers". On each line of this property, put
the URL to your various ZEO clients, as shown in <a href="#11-2">Figure 11-2</a>.</p><p> <a name="11-2"></a>
<img src="Figures/11-2.png" alt="Figure of property with URLs to mirrors">
<p><b>Figure 11-2</b> Figure of property with URLs to mirrors</p>
</p><p> Now, add some simple DTML to your site to display a list of your
mirrors:<pre> <h2>Please choose from the following mirrors:
<ul>
<dtml-in mirror_servers>
<li><a href="&dtml-sequence-item;"><dtml-var
sequence-item></a></li>
</dtml-in>
</ul></pre>
</p><p> This DTML displays a list of all mirrors your users can choose from.
When using this model, it is good to name your computers in ways that
assist your users in their choice of mirror. For example, if you
spread the load geographically, then choose names of countries for
your computer names.</p><p> Alternatively, if you do not want users voluntarily choosing a
mirror, you can have the <em>index_html</em> method of your www.zopezoo.org
site issue HTTP redirects. For example, use the following code in
your <em>www.zopezoo.org</em> site's <em>index_html</em> method:<pre> <dtml-call expr="RESPONSE.redirect(_.whrandom.choice(mirror_servers))"></pre>
</p><p> This code will redirect any visitors to <em>www.zopezoo.org</em> to a random
mirror server. </p><h3> Using Round-robin DNS to Distribute Load</h3>
<p> The <em>Domain Name System</em>, or DNS, is the Internet mechanism that
translates computer names (like "www.zope.org") into numeric
addresses. This mechanism can map one name to many addresses.</p><p> The simplest method for load-balancing is to use round-robin
DNS, as illustrated in <a href="#11-3">Figure 11-3</a>.</p><p> <a name="11-3"></a>
<img src="Figures/11-3.png" alt="Load balancing with round-robin DNS.">
<p><b>Figure 11-3</b> Load balancing with round-robin DNS.</p>
</p><p> When <em>www.zopezoo.org</em> gets resolved, BIND answers with the
address of either <em>zeoclient1</em>, <em>zeoclient2</em>, or <em>zeoclient3</em> -
but in a rotated order every time. For example, one user may
resolve <em>www.zopezoo.org</em> and get the address for <em>zeoclient1</em>,
and another user may resolve <em>www.zopezoo.org</em> and get the
address for <em>zeoclient2</em>. This way your users are spread over
the various ZEO clients.</p><p> This not a perfect load balancing scheme, because DNS resolve
information gets cached by the other nameservers on the
net. Once a user has resolved <em>www.zopezoo.org</em> to a particular
ZEO client, all subsequent requests for that user also go to the
same ZEO client. The final result is generally alright, because
the total sum of the requests are really spread over your
various ZEO clients.</p><p> One down-side to this solution is that it can take from hours to days
for name servers to refresh their cached copy of what they think the
address of <em>www.zopezoo.org</em> is. If you are not responsible for the
maintenance of your ZEO clients and one fails, then 1/Nth of your
users (where N is the number of ZEO clients) will not be able to
reach your site until their name server cache refreshes.</p><p> Configuring your DNS server to do round-robin name resolution is a
pretty advanced technique that is not covered in this book. A good
reference on how to do this can be found in the <a href="http://www.engelschall.com/pw/apache/rewriteguide/#ToC29">Apache
Documentation</a>. </p><p> Distributing the load with round-robin DNS is useful, and cheap,
but not 100% effective. DNS servers can have strange caching
policies, and you are relying on a particular quirk in the way
DNS works to distribute the load. The next section describes a
more complex, but much more powerful way of distributing load
called <em>Layer 4 Switching</em>.</p><h3> Using Layer 4 Switching to Distribute Load</h3>
<p> Layer 4 switching lets one computer transparently hand requests to a
farm of computers. This is a pretty advanced technique that is
beyond the scope of this book, but it is worth pointing out several
products that do Layer 4 switching for you.</p><p> Layer 4 switching involves a <em>switch</em> that, according to your
preferences, chooses from a group of ZEO clients whenever a request
comes in, as shown in <a href="#11-4">Figure 11-4</a>.</p><p> <a name="11-4"></a>
<img src="Figures/11-4.png" alt="Illustration of Layer 4 switching">
<p><b>Figure 11-4</b> Illustration of Layer 4 switching</p>
</p><p> There are hardware and software Layer 4 switches. There are a number
of software solutions, but one in general that stands out is the
<em>Linux Virtual Server</em> (LVS). This is an extension to the free
Linux operating system that lets you turn a Linux computer into a
Layer 4 switch. More information on the LVS can be found on <a href="http://www.linuxvirtualserver.org">its
web site</a>.</p><p> There are also a number of hardware solutions that claim higher
performance than software based solutions like LVS. Cisco
Systems has a hardware router called LocalDirector that works as
a Layer 4 switch, and Alteon also makes a popular Layer 4
switch.</p><h3> Dealing with a Single Point of Failure </h3>
<p> Without ZEO, your entire Zope system is a single point of
failure. ZEO allows you to spread that point of failure around
to many different computers. If one of your ZEO clients fails,
other clients can answer requests on the failed clients behalf.</p><p> Note that as of this writing, the single point of failure can't
be <em>entirely</em> eliminated, because there is still one central
storage server. The methods described in this section, however,
do minimize the risks of failure by spreading most of Zope
across many computers.</p><p> What this means is that, while this does remove a lot of risk
away from your web servers as a single point of failure, it does
not eliminate all risk because now the ZEO server is a single
point of failure. There are several ways of dealing with this
issue.</p><p> One popular method is to accept the single point of failure risk
and mitigate that risk as much as possible by using very
high-end, reliable equipment for your ZEO server, frequently
backing up your data, and using inexpensive, off-the-shelf
hardware for your ZEO clients. By investing the bulk of your
infrastructure budget on making your ZEO server rock solid
(redundant power supplies, RAID, and other fail-safe methods)
you can be pretty well assured that your ZEO server will remain
up, even if a handful of your inexpensive ZEO clients fail.</p><p> Some applications, however, require absolute 100% up-time.
There is still a chance, with the solution described above, that
your ZEO server will fail. If this happens, you want a backup
ZEO server to jump in and take over for the failed server right
away.</p><p> Like Layer 4 switching, there are a number of products, software
and hardware, that help you mitigate this kind of risk. One
popular software solution for linux is called
<a href="http://vergenet.net/linux/fake/">fake</a>. Fake is a Linux based
utility that can make a backup computer take over for a failed
primary computer by "faking out" network addresses. When used
in conjunction with monitoring utilities like
<a href="http://www.kernel.org/software/mon/">mon</a> or
<a href="http://www.linux-ha.org/">heartbeat</a>, fake can guarantee almost
100% up-time of your ZEO server and Layer 4 switches. Using
<code>fake</code> in this way is beyond the scope of this book.</p><p> So far, we've explained these techniques for mitigating a single
point of failure:<ul>
<li>Various tools (mirrors, round-robin DNS, Layer 4 switching) can
be used to multiplex requests across multiple computers.</li>
<li>ZEO can be used to distribute your database (ZEO server) to
multiple ZEO clients.</li>
<li><em>fake</em>, and other tools can be used to provide redundant servers
and Layer 4 switches.</li>
</ul>
</p><p> The final piece of the puzzle is the ZEO server itself, and where it
stores its information. If your primary ZEO server fails, how can
your backup ZEO server ensure it has the most recent information that
was contained in the primary server? As usual, there are several
ways to solve this problem, and they are covered in the next section.</p><h3> ZEO Server Details</h3>
<p> Before explaining the details of how the ZEO server works, it is
worth understanding some details about how Zope <em>storages</em> work in
general.</p><p> Zope does not save any of its object or information directly to
disk. Instead, Zope uses a <em>storage</em> component that takes care of
all the details of where objects should be saved.</p><p> This is a very flexible model, because Zope no longer needs to be
concerned about opening files, or reading and writing from databases,
or sending data across a network (in the case of ZEO). Each
particular storage takes care of that task on Zope's behalf.</p><p> For example, a plain, stand-alone Zope system can be illustrated in
<a href="#11-5">Figure 11-5</a>.</p><p> <a name="11-5"></a>
<img src="Figures/11-5.png" alt="Zope connected to a filestorage">
<p><b>Figure 11-5</b> Zope connected to a filestorage</p>
</p><p> You can see there is one Zope application which plugs into a
<em>FileStorage</em>. This storage, as its name implies, saves all of its
information to a file on the computer's filesystem.</p><p> When using ZEO, you simple replace the FileStorage with a
<em>ClientStorage</em>, as illustrated in <a href="#11-6">Figure 11-6</a>.</p><p> <a name="11-6"></a>
<img src="Figures/11-6.png" alt="Zope with a Client Storage and Storage server">
<p><b>Figure 11-6</b> Zope with a Client Storage and Storage server</p>
</p><p> Instead of saving objects to a file, a ClientStorage sends
objects over a network connection to a <em>Storage Server</em>. As you
can see in the illustration, the Storage Server uses a
FileStorage to save that information to a file on the ZEO
server's filesystem.</p><p> Storages are interchangeable and easy to implement. Because of their
interchangeable nature, ZEO Storage Servers can use ZEO
ClientStorages to pass on object data to yet another ZEO Storage
Server. This is illustrated in <a href="#11-7">Figure 11-7</a>.</p><p> <a name="11-7"></a>
<img src="Figures/11-7.png" alt="Multi-tiered ZEO system">
<p><b>Figure 11-7</b> Multi-tiered ZEO system</p>
</p><p> Here, you can see a number of ZEO clients funnel down through three
ZEO servers, which in turn act as ZEO clients themselves and funnel
down into the final, central ZEO server than saves its information in
a FileStorage. Now, that central ZEO server is the single point
of failure in the system. If any of your other clients, or
intermediate servers fail, the system will still continue to
work, but if the central server fails, then you need an
alternative.</p><p> Using <code>fake</code> you can have a back-up storage server strategy, but
this method is not very well proven and hasn't been explored by
the authors. In the future, ZEO will have a "multiple-server"
feature, that allows a group of storage servers to act as a
quorum, so if one or more storage servers fail, the remaining
servers in the quorum can continue to serve objects. </p><p> There are a number of advantages to an approaches like these,
especially if you are interested in creating a massively
distributed network object database. Of course, with any system
of advantages, there are some drawbacks as well, which are
discussed in the next section.</p><h2> ZEO Caveats</h2>
<p> For the most part, running ZEO is exactly like running Zope by itself,
but there are a few issues to keep in mind.</p><p> First, it takes longer for information to be written to the Zope object
database. This does not slow down your ability to use Zope (because
Zope does not block you during this write operation) but it does
increase your chances of getting a <em>ConflictError</em>. Conflict errors
happen when two ZEO clients try to write to the same object at the same
time. One of the ZEO clients wins the conflict and continues on
normally. The other ZEO client looses the conflict and has to try
again.</p><p> Conflict errors should be as infrequent as possible because they could
slow down your system. While it's normal to have a <em>few</em> conflict
errors (due to the concurrent nature of Zope) it is abnormal to have <em>a
lot</em> of conflict errors. The pathological case is when more than one
ZEO client tries to write to the same object over and over again very
quickly. In this case, there will be lots of conflict errors, and
therefore lots of retries. If a ZEO client tries to write to the
database three times and gets three conflict errors in a row, then the
request is aborted and the data is not written.</p><p> Because ZEO takes longer to write this information, the chances of
getting a ConflictError are higher than if you are not running
ZEO. Because of this, ZEO is more <em>write sensitive</em> than running
Zope without ZEO. You may have to keep this in mind when you are
designing your network or application. As a rule of thumb, more
and more frequent writes to the database increase your chances of
getting a ConflictError. On the flip side, faster and more
reliable network connections and computers lower your chances of
getting a ConflictError. By taking these two factors into
account, conflict errors can be mostly avoided.</p><p> Finally, as of this writing, there is no built in encryption or
authentication between ZEO servers and clients. This means that you
must be very careful about who you expose your ZEO servers to. If you
leave your ZEO servers open to the whole Internet, then anyone can
connect to your ZEO server and write data into your database, and that
can be bad news.</p><p> This is not an unsolveable problem however, because you can use
other tools, like firewalls, to protect your ZEO servers. If you
are running a ZEO client/server connection over an unsecure
network and you want guarantee that your information is kept
private, you can use tools like <a href="http://www.openssh.org">OpenSSH</a>
and <a href="http://www.stunnel.org/">stunnel</a> to set up secure, encrypted
communication channels between your ZEO clients and servers. How
these tools work and how to set them up is beyond the scope of
this book, but both packages are adequately documented on their
web sites. For more information on firewalls, with Linux in
particular, we recommend the book "Linux Firewalls" by Robert
Ziegler, which is published by New Riders.</p><h2> Conclusion</h2>
<p> In this chapter we looked at ZEO, and how ZEO can substantially
increases the capacity of your website. In addition to running
ZEO on one computer to get familiarized, we looked at running ZEO
on many computers, and various techniques for spreading the load of
your visitors among those many computers.</p><p> ZEO is not a magic bullet solution, and like other system designed
to work with many computers, it adds another level of complexity
to your web site. This complexity pays off however when you need
to serve up lots of dynamic content to your audience.</p></body>
</html>
distrib
>
Mandriva
>
2010.0
>
i586
>
media
>
contrib-release
>
by-pkgid
>
8b32d38f0094104a5fd847636983daa8
>
files
>
132
