MEAP VERSION 6

About this MEAP

This book file has been tested with iBooks 1.5 on iOS 5, and the Kindle App for iPad 2.9. If you are using an earlier version of iBooks and are experiencing problems, please try upgrading iBooks.

To get a full-screen view of images:

iBooks: Double-tap an image.

Kindle app: Use two fingers to stretch the image.

You can download the most up-to-date version of your electronic books from your Manning Account at account.manning.com. For customer support write to support@manning.com.

© Manning Publications Co. We welcome reader comments about anything in the manuscript - other than typos and other simple mistakes. These will be cleaned up during production of the book by copyeditors and proofreaders.

https://forums.manning.com/forums/gnuplot-in-action-second-edition

Welcome

Thank you for purchasing the MEAP for Gnuplot in Action, Second Edition. Since its first publication in 2008, Gnuplot in Action has been popular and successful as the "missing manual" to gnuplot, serving as a tutorial introduction for those readers new to gnuplot, but also providing the depth and detail necessary to become a power- user! I want to retain both of these aspects, while including new gnuplot features and responding to some changes in the environment.

Gnuplot is and has been the most popular open-source tool for plotting and visualizing data.

Early this year, the gnuplot development team released gnuplot 5.0—the first major gnuplot release in almost a decade.

I took this opportunity to bring my book Gnuplot in Action up to date. Since its first publication in 2008, Gnuplot in Action has been popular and successful as the “missing manual” to gnuplot, serving both as a tutorial introduction for those readers new to gnuplot, but also providing the depth and detail necessary to become a poweruser! I want to retain both of these aspects, while including new gnuplot features and responding to some changes in the environment.

WL.1  What's really new in gnuplot 5?

Gnuplot 5 is not a break with the past; in fact, if you are already familiar with gnuplot, you will feel right at home. Nevertheless, the sum of many small and incremental changes generates a different feeling when working with gnuplot 5 over some time. Here are some highlights:

Color—With gnuplot 5, color is the way to go. It is no longer an afterthought and, moreover, it is no longer optional: gnuplot 5 assumes that you are creating color graphs. Monochrome is now a special case.

Color scheme—Gnuplot 5 uses a new default color sequence; you can get the old one back using the command: set colorsequence classic.

Consistency across terminals—Using gnuplot 5, you can generate output in all common formats (that is, PDF, PNG, and for the screen) using a consistent set of routines (or "terminals"), all built on the Cairo graphics libraries. The differences among terminals for various output formats are now largely a thing of the past.

Better terminals—Terminals are not only more consistent, they are also much better and generally more powerful. Text handling in particular is much improved, with “enhanced text mode” now the default mode for contemporary terminals.

PDF—The “standard” print-quality output format is now PDF, not Postscript anymore.

Unicode and UTF-8—Gnuplot 5 understands UTF-8 encoded Unicode. This may sound like a nuisance, but it is, in fact, a tremendous improvement. You can now use almost any character in a string or text label, and do so in a straightforward and consistent manner. (I explain all this in chapter 5.)

Loops—Gnuplot 5 includes support for some labor-saving features that can speed up repetitive tasks. It is now possible to automate a series of plot or set commands through “inline loops.”

More consistent styles—The handling of line and point styles has become more consistent and powerful.

If you are familiar with previous versions of gnuplot, you will realize that gnuplot 5 has done away with many former pain points and old gripes! (Just the consistency across terminals is worth upgrading for, as is the change to Unicode text.) At the same time, gnuplot has become more complex, with more features, and more ways to combine them. Gnuplot 5 is the best gnuplot there ever was, but it is no longer as simple as it used to be.

WL.2  Conventions

As you look over the new chapters, please keep in mind that you are looking at a “rough cut,” not the final product!

You should be aware of the following conventions:

Based on your local configuration, the graphs that you see when you type in commands at your own computer may look somewhat different from the figures in the book. Do not be disturbed if colors or point types look different—these are merely configuration details! The overall content of the figure should always be the same as in the book.

Gnuplot's plot command requires the name of a file with data to be present. Example graphs of many actual data files are presented in this book (and the files will, eventually, also be available for download). Whenever I show the plot command for an actual data file, I will use the proper file name in the plot command, like so:

plot "prices" using 1:2 with lines

But at other times, I only want to demonstrate the syntax of the plot command. In these cases, there is no data file, and I use the generic file name "data" as a placeholder. Hence, whenever you see the file name "data", you know that this command is only there to illustrate the surrounding discussion. This will always be clear from the context; in any case, I never use the name "data" for an actual data file.

Your feedback is essential to creating the best book possible and I invite you to post any feedback you have about the book in the Author Online forum..

— Philipp K. Janert

Brief Table of Contents

  1 Prelude: Understanding data with gnuplot

  2 Tutorial: Essential gnuplot

  3 The heart of the matter: the plot command

  4 Managing data sets and files

  5 Practical matters: strings, loops, history

  6 A catalog of styles

  7 Decorations

  8 All about axes

  9 Color, style, and appearance

10 Terminals in depth

11 Automation, scripting, and animation

12 Beyond the defaults: workflow and styles

13 Basic techniques of graphical analysis

14 Topics in graphical analysis

15 Coda: Understanding data with graphs

Appendixes:

  A Obtaining, building, and installing gnuplot

  B Resources

The following appendixes will be available in the eBook:

  C Surface and contour plots

  D Palettes and false-color plots

  E Special plots

  F Higher math

1

Prelude: Understanding data with gnuplot

This chapter covers:

Warmup examples

What is graphical analysis?

What is gnuplot?

Gnuplot is probably the most widely used open source program for plotting and visualizing data. In this book, I want to show you how to use gnuplot to make plots and graphs of your data: both quick and easy graphs for your own use and highly polished graphs for presentations and publications.

But I also want to show you something else: how to solve data analysis problems using graphical methods. The art of discovering relationships in data and extracting information from it by visual means is called graphical analysis and I believe gnuplot to be an excellent tool for it.

As a teaser, let’s take a look at some problems and how we might be able to approach them using graphical methods. The graphs here and in the rest of the book (with very few exceptions) have been, of course, generated with gnuplot.

1.1  A busy weekend

To get a feeling for the kinds of problems that we may be dealing with, and for the kinds of solutions that gnuplot can help us find, let’s look at two examples. Both take place during a long and busy weekend.

1.1.1  Planning a marathon

Imagine you’re in charge of organizing the local city marathon. There will be more than 2,000 starters, traffic closed around the city, plenty of spectators—and a major Finish Line Festival to celebrate the victors. The big question is: when should the Finish Line crew be ready to deal with the majority of runners? At what point do we expect the big influx of the masses?

You still have the results from last year’s event. Assuming that the starters haven’t improved dramatically over the last year (probably a safe assumption), you do a quick average on the completion times and find that last year’s average was 172 minutes. To be on the safe side, you calculate the standard deviation as well, which comes out to about 15 minutes. So you tell your crew to be ready for the big rush starting two and a half hours (150 minutes) after the start, and feel reasonably well prepared for the event.

So it comes as a surprise when on the big day, plenty of runners start showing up on the finish line after only 130 minutes—a good 20 minutes earlier than the expected onset of the rush. In terms of event management, being off by 20 or 30 minutes isn’t catastrophic, yet it is a bit strange. The next day you wonder: what went wrong?

Let’s look at the data to see what we can learn about it. So far, all we know of it is the mean and the standard deviation.

The mean is convenient: it is easy to calculate and it summarizes the entire data set in a single number. But in forming the mean, we lost a lot of information. To understand the whole data set, we have to look at it. And since we can’t understand data by looking at more than 2,000 individual finish times, this means we’ll have to plot it.

It will be convenient to group the runners by completion time and to count the number of participants that completed during each full minute. The resulting file might start like this:

# Minutes Runners
133       1
134       7
135       1
136       4
137       3
138       3
141       7
142       24
...

Now we plot the number of runners against the completion (see figure 1.1).

Figure 1.1. Number of finishers versus time to complete (in minutes)

It is immediately obvious where we went wrong: the data is bimodal, meaning it has two peaks. There is an early peak at around 150 minutes, and a later main peak at 180 minutes.

Actually, this makes sense: a major sporting event such as a city marathon attracts two very different groups of people: athletes, who train and compete throughout the year and are in it to win, and a much larger group of amateurs, who come out once a year for a big event and are mostly there to participate.

The problem is that for such data, the mean and standard deviation are obviously bad representations—so much so that at the time when we expected the big rush (170 minutes), there’s actually a bit of a lull at the finish line!

The take-home message here is that it is usually not a good idea to rely on summary statistics (such as the mean) for unknown data sets. We always should investigate what the data looks like. Once we’ve confirmed the basic shape, we can choose how to summarize our findings best.

And of course, there is always more to learn. In this example, for instance, we see that after about 200 minutes, almost everybody has made it, and we can start winding down the operation. The actual “tail” of the distribution is quite small—actually, a bit surprisingly so (I would’ve expected to see a greater number of stragglers, but possibly many runners who are really slow drop out of the race when they realize they’ll place badly).

Using gnuplot

Let’s look at the gnuplot command that was used to generate figure 1.1. Gnuplot is command-line–oriented: after you start gnuplot, it drops you into an interactive command session, and all commands are typed at the interactive gnuplot prompt.

Gnuplot reads data from simple text files, with the data arranged in columns as shown previously. To plot a data file takes only a single command, plot, like this:[1]

plot "marathon" using 1:2 with boxes

The plot command requires the name of the data file as argument in quotes. The rest of the command line specifies which columns to use for the plot, and in which way to represent the data. The using 1:2 declaration tells gnuplot to use the first and second column in the file called marathon. The final part of the command, with boxes, selects a box style, which is often suitable to display counts of events.

Gnuplot handles most everything else by itself: it sizes the graph and selects the most interesting plot range, it draws the border, and it draws the tic marks and their labels. All these details can be customized, but gnuplot typically does a good job at anticipating what the user wants.

1.1.2  Determining the future

The same weekend when 2,000 runners are running through the city, a diligent graduate student is working on his research topic. He studies diffusion limited aggregation (DLA), a process wherein a particle performs a random walk until it comes in contact with a growing cluster of particles. At the moment of contact, the particle sticks to the cluster at the location where the contact occurred and becomes part of the cluster. Now, a new random walker is released to perform a random walk, until it sticks to the cluster. And so on.

Clusters grown through this process have a remarkably open, tenuous structure (as in figure 1.2). DLA clusters are fractals, but rather little is known about them with certainty.[2]

Figure 1.2. A DLA cluster of N=30,000 particles, drawn with gnuplot

The DLA process is very simple, so it seems straightforward to write a computer program to grow such clusters in a computer, and this is what our busy graduate student has done. Initially, all seems well, but as the simulation progresses, the cluster seems to grow more and more slowly. Excruciatingly slowly, in fact. The goal was to grow a DLA cluster of N=100,000 particles. Will the program ever finish?

Luckily, the simulation program periodically writes information about its progress to a log file: for each new particle added to the cluster, the time (in seconds) since the start of the simulation is recorded. We should be able to predict the completion time from this data, but an initial plot (figure 1.3) is just not very helpful; there are too many ways that this curve can be extrapolated to larger cluster sizes.

Figure 1.3. Time required to grow a DLA cluster

The time consumed by many computer algorithms grows as a simple power of the size of the problem. In our case, this would be the number N of particles in the cluster T ~ Nk , for some value of k . Our research student therefore plots the running time of his simulation program on a double logarithmic plot versus the cluster size (see figure 1.4). The data points fall on a straight line, indicating a power law. (I’ll explain later how and why this works.) Through a little trial and error, he also finds an equation that approximates the data quite well. The equation can be extended to any cluster size desired and will give the time required. For N=100,000 (which was the original goal), we can read off T=300,000 seconds (or more), corresponding to 83 hours or four days, so we can tell our friend that there is no point in spending the weekend in the lab—he should go out (maybe run a marathon), and come back on Monday. Or perhaps work on a better algorithm. (For simulations of DLA cluster growth, dramatic speedups over the naive implementation are possible. Try it if you like.)

Figure 1.4. Time required to grow a DLA cluster in a double-logarithmic plot, together with an approximate mathematical model

Using gnuplot

Again, let’s see how the graphs in this section were created. The easiest to understand is figure 1.3. Given a file containing two columns, one listing the cluster size and the other listing the completion time, the command is simply

plot "runtime" using 1:2 with lines

The only difference compared to figure 1.1 is the style: rather than boxes, I use line segments to connect consecutive data points: with lines.

Did you notice that figure 1.3 and figure 1.4 contain more than just data? Both axes are now labelled! Details such as labels and other helpful decorations often make the difference between a mediocre and a high-quality graph, because they provide the observer with the necessary context to fully understand the graph.

In gnuplot, all details of a graph’s appearance are handled by setting the appropriate options. To place the labels on the x and y axes in figure 1.3, I used

set xlabel "Cluster Size"
set ylabel "Run Time [sec]"

Figure 1.4 is drawn using double-logarithmic axes. This is another option, which is set as follows:

set logscale

Figure 1.4 shows two curves: the data together with a best “fit.” Plotting several data sets or mathematical functions together in one plot is easy: we just list them one after another on the command line for the plot command:

plot "runtime" using 1:2 title "data" with lines,
»    1.2*(x/1000)**2.7 title "model"

This command introduces a further gnuplot feature: the title directive. It takes a string as argument, which will be displayed together with a line sample in the plot’s key or legend (visible in the upper left of figure 1.4).

Finally, we come to figure 1.2. That’s a somewhat different beast. You’ll notice that the border and the tic marks are missing. The aspect ratio (the ratio of the graph’s width to its height) has been constrained to 1, and a single dot has been placed at the position of each particle in the cluster. Here are the most important commands that I used:

unset border
unset xtics
unset ytics

set size square

plot "cluster" using 1:2 with dots

You see that gnuplot is really simple to use. In the next section, I’d like to talk more about using graphical methods to understand a data set, before coming back to gnuplot and discussing why it is my favorite tool for this kind of activity.

[1] Depending on your gnuplot set-up and initialization, your graphs may look slightly different than the figures shown in this chapter. We’ll discuss user-defined appearance options starting in part 2.

[2] The original paper on DLA was “Diffusion Limited Aggregation, A Kinetic Critical Phenomenon” by T. A. Witten and L. M. Sander, and appeared in Physical Review Letters Vol. 41, p. 1400 in 1981. It is one of the most quoted papers from that journal of all time. If you want to learn more about DLA and similar processes, check out Fractals, Scaling, and Growth Far From Equilibrium by Paul Meakin (1998).

1.2  What is graphical analysis?

These two examples should have given you an idea of what graphical analysis is and how it works. The basic steps are always the same:

Plot the data.

Inspect it, trying to find some recognizable behavior.

Compare the actual data to data that represents the hypothesis from the previous step (as we did in the second example earlier, when we plotted running time of the simulation program together with a power-law function).

Repeat.

We may try more sophisticated things, but this is the basic idea. If the hypothesis in the second step seems reasonably justified, we’ll often try and remove its effect, for instance by subtracting a formula from the data, to see whether there is any recognizable pattern in the residual. And so on.

Iteration is a crucial aspect of graphical analysis: plotting the data this way and that way; comparing it to mathematical functions or to other data sets; zooming in on interesting regions or zooming out to detect the overall trend; applying logarithms or other data transformations to change its shape; using a smoothing algorithm to tame a noisy data set; and so on. During an intense analysis session using a new but promising data set, it’s not uncommon to produce literally dozens of graphs.

None of these graphs will be around for long. They’re transient, persisting just long enough for us to form a new hypothesis, which we’ll try to justify in the next graph we draw. This also means that these graphs won’t be “polished” in any way, since they’re the graphical equivalent to scratch paper: notes of work in progress, not intended for anyone but ourselves.

This isn’t to say that polishing doesn’t have its place. But it comes later in the process: once we know what the results of our analysis are, we need to communicate them to others. At this point, we’ll create “permanent” graphs, which will be around for a long time—maybe until the next departmental presentation, or (if the graph will be part of a scientific publication, for instance) possibly forever!

Such permanent graphs have different requirements: other people must be able to understand them, possibly years later, and most likely without us there to explain them. Therefore, graph elements such as labels, captions, and other contextual information become very important. Presentation graphs must be able to stand by themselves.

Presentation graphs also should make their point clearly. Now that we know the results of our analysis, we should find the clearest and most easily understood way of presenting our findings. A presentation graph should make one point and make it well.

Finally, some would argue that a presentation graph should “look good.” Maybe. If it makes its point well, there is no reason why it shouldn’t be visually pleasing as well. But that is an afterthought. Even a presentation graph is about the content, not the packaging.

1.2.1  Why graphical analysis?

Graphical analysis is a discovery tool. We can use it to reveal as-yet unknown information in data. In comparison to statistical methods, it helps us discover new and possibly quite unexpected behavior.

Moreover, it helps us develop an intuitive understanding of the data and the information it contains. Since it doesn’t require particular math skills, it is accessible to anyone with an interest and a certain amount of intuition.

Even if rigorous model building is our ultimate goal, graphical methods still need to be the first step, so that we can develop a sense for the data, its behavior, and quality. Knowing this, we can then select the most appropriate formal methods.

1.2.2  Limitations of graphical analysis

Of course, graphical analysis has limitations and its own share of problems.

Graphical analysis doesn’t scale.

Graphical analysis is a manual process that can’t easily be automated. Each data set is treated as a separate special case, which isn’t feasible if there are thousands of data sets.

But this problem is sometimes more apparent than real. It can be remarkably effective to generate a large number of graphs and browse them without studying each one in great depth. It’s totally possible to scan a few hundred graphs visually, and doing so may already lead to a high-level hypothesis regarding the classification of the graphs into a few subgroups, which can then be investigated in detail. (Thank goodness gnuplot is scriptable, so that preparing a few hundred graphs poses no problem.)

Graphical analysis yields qualitative, not quantitative results.

Whether you regard this as a strength or a weakness depends on your situation. If you’re looking for new behavior, graphical analysis is your friend. If you’re trying to determine the percentage by which a new fertilizer treatment increases crop production, statistical analysis is the way to go.

It takes skill and experience.

Graphical analysis is a creative process, using inductive logic to move from observations to hypothesis. There is no prescribed set of steps to move from a data set to conclusions about the underlying phenomena, and not much that can be taught in a conventional, classroom format.

But by the same token, it does not require formal training, either. Ingenuity, intuition, and curiosity are the most important character traits. Everyone can play this game, if they’re interested in finding out what the data tries to tell them.

1.3  What is gnuplot?

Gnuplot is a program for exploring data graphically. Its purpose is to generate plots and graphs from data or functions. It can produce highly polished graphs, suitable for publication, and simple throw-away graphs, when we’re merely playing with an idea.

Gnuplot is command-line–driven: you issue commands at a prompt, and gnuplot will redraw the current plot in response. Gnuplot is also interactive: the output is generated and displayed immediately in an output window. Although gnuplot can be used as a background process in batch-mode, typical use is highly interactive. On the other hand, its primary user interaction is through a command language, not through a point-and-click GUI interface.

Don’t let the notion of a command language throw you: gnuplot is easy to use—really easy to use! It takes only one line to read and plot a data file, and most of the command syntax is straightforward and quite intuitive. Gnuplot does not require programming or any deeper understanding of its command syntax to get started.

So this is the fundamental workflow of all work with gnuplot: plot, examine, repeat—until you have found out whatever you wanted to learn from the data. Gnuplot supports the iterative process model required for exploratory work perfectly.

1.3.1  Gnuplot isn’t GNU

To dispel one common confusion right away: gnuplot isn’t GNU software, has nothing to do with the GNU project, and isn’t released under the GNU Public License (GPL). Gnuplot is released under a permissive open source license.

Gnuplot has been around a long time—a very long time! It was started by Thomas Williams and Colin Kelley in 1986. On the gnuplot FAQ, Thomas has this to say about how gnuplot was started and why it is named the way it is:

I was taking a differential equation class and Colin was taking
Electromagnetics, we both thought it'd be helpful to visualize the
mathematics behind them. We were both working as sys admin for an EE
VLSI lab, so we had the graphics terminals and the time to do some
coding. The posting was better received than we expected, and prompted
us to add some, albeit lame, support for file data.

Any reference to GNUplot is incorrect. The real name of the program is
"gnuplot". You see people use "Gnuplot" quite a bit because many of us
have an aversion to starting a sentence with a lower case letter, even
in the case of proper nouns and titles. gnuplot is not related to the
GNU project or the FSF in any but the most peripheral sense. Our
software was designed completely independently and the name "gnuplot"
was actually a compromise. I wanted to call it "llamaplot" and Colin
wanted to call it "nplot." We agreed that "newplot" was acceptable
but, we then discovered that there was an absolutely ghastly pascal
program of that name that the Computer Science Dept. occasionally
used. I decided that "gnuplot" would make a nice pun and after a
fashion Colin agreed.

1.3.2  Why gnuplot?

I have already mentioned the most important reasons why I like gnuplot: easy to learn, easy to use, excellent support for iterative, exploratory use, yet nevertheless scriptable for bulk or offline processing, able to produce publication-quality graphs.

Here are some other reasons why I believe gnuplot is a good tool for many situations. Gnuplot is

Stable, mature, and actively maintained.

Free and open source.

Available on all three commonly used platforms: Linux/Unix, Windows, Mac OS X.

Able to generate polished, publication-quality graphs, and offering detailed control over the final appearance of plots.

Supporting all common graphics formats (and quite a few less common ones).

Able to read regular text files as input, and is tolerant regarding the specifics of the input file format. (No need for the data to be in some special archive file format!)

Capable of handling large data sets (easily many millions of data points) and fast.

Scriptable and easy to integrate with other tools and programs.

Modest in its resource consumption.

1.3.3  Limitations

It is important to remember that gnuplot is a data-plotting tool, nothing more, nothing less. In particular, it is neither a numeric nor symbolic workbench, nor a statistics package. It can therefore only perform rather simple calculations on the data. On the other hand, it has a flat learning curve, requiring no programming knowledge and only the most basic math skills.

Gnuplot is also no drawing tool. All its graphs are depictions of some data set. It has only very limited support for arbitrary box-and-line diagrams, and none at all for free-hand graphics.

Finally, gnuplot makes no attempt at what I earlier called “reality representation.” It is a tool for quantitative analysis, and therefore its bread and butter are dot and line plots. It has no support for three-dimensional solid body imaging, ray-tracing, fisheye functionality, and similar techniques.

Overall, though, I regard these limitations more as strengths in disguise: in the Unix tradition, gnuplot is a rather simple tool, doing (mostly) one thing, and doing it very, very well.

1.3.4  Gnuplot 5: the best gnuplot there ever was!

TBD

1.4  Summary

In this chapter, I showed you a couple of examples that demonstrate the power of graphical methods for understanding data. I have also tried to suggest a suitable method for dealing with data analysis problems. Start with a plot of the data and use it to identify the essential features of the data set. Then iterate the process to bring out the behavior you’re most interested in. And finally (not always, but often) develop a mathematical description for the data, which can then be used to make predictions (which, by their nature, go beyond the information contained in the actual data set).

Our tool in doing this kind of analysis will be gnuplot. And it is gnuplot, and how to use it, that we’ll turn to next. Once we have developed the skills to use gnuplot well, we’ll return to graphical analysis and discuss useful techniques for extracting the most information possible from a data set, using graphs.

2

Tutorial: essential gnuplot

This chapter covers:

Invoking gnuplot

Plotting functions and data

Saving and exporting

Managing options

Getting help

In this chapter, we introduce gnuplot’s most important features: generating plots, saving them to a file, and exporting graphs to common graphics file formats. We’ll also explain briefly how to set options and how to access gnuplot’s built-in documentation. This chapter covers those commands that you’ll find yourself using almost every time you start up gnuplot. In the next couple of chapters, you’ll learn about further features for graphical analysis and how to manage data sets and files. By the end of chapter 4, you’ll know most of the commands you’ll use on a day-to-day basis.

Are you surprised that just a few chapters are sufficient to get us this far? Congratulations, you just discovered why gnuplot is cool: it makes easy things easy, and hard things possible. This chapter and the next two cover the easy parts; as to the hard parts... well, that’s what the rest of this book is all about.

2.1  Simple plots

Since gnuplot is a plotting program, it should come as no surprise that the most important gnuplot command is plot. It can be used to plot both functions (such as sin(x) ) and data (typically from a file). The plot command has a variety of options and subcommands, through which we can control the appearance of the graph as well as the interpretation of the data in the file. The plot command can even perform arbitrary transformations on the data as we plot it.

2.1.1  Invoking gnuplot and first plots

Gnuplot is a text-based plotting program: we interact with it through command-line-like syntax, as opposed to manipulating graphs using the mouse in a WYSIWYG fashion. Gnuplot is also interactive: it provides a prompt at which we type our commands. When we enter a complete command, the resulting graph immediately pops up in a separate window. This is in contrast to a graphics programming language (such as PIC), where writing the command, generating the graph, and viewing the result are separate activities, requiring separate tools. Gnuplot has a history feature, making it easy to recall, modify, and reissue previous commands. The entire setup encourages you to play with the data: making a simple plot, changing some parameters to hone in on the interesting sections, eventually adding decorations and labels for final presentation, and in the end exporting the finished graph in a standard graphics format.

If gnuplot is installed on your system, it can usually be invoked by issuing the command:

gnuplot

at the shell prompt. (Check appendix A for instructions on obtaining and installing gnuplot, if your system doesn’t have it already.) Once launched, gnuplot displays a welcome message and then replaces the shell prompt with a gnuplot> prompt. Anything entered at this prompt will be interpreted as gnuplot commands until you issue an exit or quit command, or type an end-of-file (EOF) character, usually by hitting Control-D.

Probably the simplest plotting command we can issue is

plot sin(x)

(Here and in the following, the gnuplot> prompt is suppressed to save space. Any code shown should be understood as having been entered at the gnuplot prompt, unless otherwise stated.)

On Unix running a graphical user interface (an arbitrary window manager running on top of X11), this command opens a new window showing the resulting graph, looking something like figure 2.1.

Figure 2.1. Our first plot: plot sin(x)

Please note how gnuplot has selected a “reasonable” range for the x values automatically (by default from -10 to +10) and adjusted the y range according to the values of the function.

Let’s say we want to add some more functions to plot together with the sine. We recall the last command (using the up-arrow key or Control-P for “previous”) and edit it to give

plot sin(x), x, x-(x**3)/6

This will plot the sine together with the linear function x and the third-order polynomial x - ⅙ x³ , which are the first few terms in the Taylor expansion of the sine.[3] (Gnuplot’s syntax for mathematical expressions is straightforward and similar to the one found in almost any other programming language. Note the ** exponentiation operator, familiar from Fortran or Perl. Appendix B has a table of all available operators and their precedences.) The resulting plot (see figure 2.2) is probably not what we expected.

Figure 2.2. An unsuitable default plot range: plot sin(x), x, x-(x**3)/6

The range of y values is far too large, compared to the previous graph. We can’t even see the wiggles of the original function (the sine wave) at all anymore. Gnuplot adjusts the y range to fit in all function values, but for our plot, we’re only interested in points with small y values. So, we recall the last command again (using the up-arrow key) and define the plot range that we are interested in:

plot [][-2:2] sin(x), x, x-(x**3)/6

The range is given in square brackets immediately after the plot command. The first pair of brackets defines the range of x values (we leave it empty, since we’re happy with the defaults in this case); the second restricts the range of y values shown. This results in the graph shown in figure 2.3.

Figure 2.3. Using explicit plot ranges: plot [][-2:2] sin(x), x, x-(x**3)/6

We can play much longer with function plots, zoning in on different regions of interest and trying out different functions (check the reference section in appendix B for a full list of available functions and operators), but instead let’s move on and discuss what gnuplot is most useful for: plotting data from a file.

2.1.2  Plotting data from a file

Gnuplot reads data from text files. The data is expected to be numerical and to be stored in the file in whitespace-separated columns. Lines beginning with a hashmark (#) are considered to be comment lines and are ignored. Listing 2.1 shows a typical data file containing the share prices of two fictitious companies, with the equally fictitious ticker symbols PQR and XYZ, over a number of years.

Listing 2.1. A typical data file: stock prices over time (prices)

# Average PQR and XYZ stock price (in dollars per share) per calendar year
1975     49    162
1976     52    144
1977     67    140
1978     53    122
1979     67    125
1980     46    117
1981     60    116
1982     50    113
1983     66     96
1984     70    101
1985     91     93
1986    133     92
1987    127     95
1988    136     79
1989    154     78
1990    127     85
1991    147     71
1992    146     54
1993    133     51
1994    144     49
1995    158     43

The canonical way to think about this is that the x value is in column 1 and the y value is in column 2. If there are additional y values corresponding to each x value, they are listed in subsequent columns. (We’ll see later that there’s nothing special about the first column. In fact, any column can be plotted along either the x or the y axis.)

This format, simple as it is, has proven to be extremely useful—so much so that long-time gnuplot users usually generate data in this way to begin with. In particular, the ability to keep related data sets in the same file is a big help (so that we don’t need to keep PQR’s stock price in a separate file from XYZ’s—although we could if we wanted to).

While whitespace-separated numerical data is what gnuplot expects natively, gnuplot can parse and interpret significant deviations from this norm, including text columns (with embedded whitespace if enclosed in double quotes), missing data, and a variety of textual representations for calendar dates, as well as binary data (see chapter 4 for a more detailed discussion of input file formats, chapter 5 for strings, and chapter 8 for the special case when one of the columns represents date/time information).

Plotting data from a file is simple. Assuming that the file shown in listing 2.1 is called prices and is in the current working directory (typically, the directory from which gnuplot was started), we can simply type

plot "prices"

Since data files typically contain many different data sets, we’ll usually want to select the columns to be used as x and y values. This is done through the using directive to the plot command:

plot "prices" using 1:2

This will plot the price of PQR shares as a function of time: the first argument to the using directive specifies the column in the input file to be plotted along the horizontal (x) axis, while the second argument specifies the column for the vertical (y) axis. If we want to plot the price of XYZ shares in the same plot, we can do so easily (as in figure 2.4):

plot "prices" using 1:2, "prices" using 1:3

Figure 2.4. Plotting from a file: plot "prices" using 1:2, "prices" using 1:3

[Tip]
Use plot ... using ... to select columns

The using directive tells the plot command which columns to use: plot "data" using 1:2 selects the first column for the x axis and the second column for the y axis.

By default, data points from a file are plotted using unconnected symbols. Often this isn’t what we want, so we need to tell gnuplot what style to use for the data. This is done using the with directive. Many different styles are available. Among the most useful ones are with linespoints, which plots each data point as a symbol and also connects subsequent points, and with lines, which just plots the connecting lines, omitting the individual symbols.

[Tip]
Use plot ... with ... to select the style

The with directive to the plot commands selects the plotting style. The most frequently used styles include with points, with lines, and with linespoints.

plot "prices" using 1:2 with lines,
» "prices" using 1:3 with linespoints

This looks good, but it’s not clear from the graph which line is which. Gnuplot automatically provides a key, which shows a sample of the line or symbol type used for each data set together with a text description, but the default description isn’t very meaningful in our case. We can do much better by including a title for each data set as part of the plot command:

plot "prices" using 1:2 title "PQR" with lines,
» "prices" using 1:3 title "XYZ" with linespoints

This changes the text in the key to the string given as the title (figure 2.5). The title has to come after the using directive in the plot command. A good way to memorize this order is to remember that we must specify the data set to plot first and provide the description second: define it first, then describe what you defined.

[Tip]
Use plot ... title "..." to attach a description to a curve

The title directive can be used to place a descriptive string in the graph’s legend.

Figure 2.5. Introducing styles and the title keyword: plot "prices" using 1:2 title "PQR" with lines, "prices" using 1:3 title "XYZ" with linespoints

Want to see how PQR’s price correlates with XYZ’s? No problem; just plot one against the other, using PQR’s share price for x values and XYZ’s for y values, like so:

plot "prices" using 2:3 with points

We see here that there’s nothing special about the first column. Any column can be plotted against either the x or the y axis; we just pick whichever combination we need through the using directive. Since it makes no sense to connect the data points in the last plot, we’ve chosen the style with points, which just plots a symbol for each data point, but no connecting lines (figure 2.6).

Figure 2.6. Any column can be used for either x or y axis: plot "prices" using 2:3 with points

A graph like figure 2.6 is known as a scatter plot and can show correlations between two data sets. In this graph, we can see a clear negative correlation: the better PQR is doing, the worse XYZ’s stock price develops. We’ll revisit scatter plots and their uses later in chapter 18.

Now that we’ve seen the most important, basic commands, let’s step back for a moment and quickly introduce some creature comforts that gnuplot provides to the more experienced user.

2.1.3  Abbreviations and defaults

Gnuplot is very good at encouraging iterative, exploratory data analysis. Whenever we complete a command, the resulting graph is shown immediately and all changes take effect at once. Writing commands isn’t a different activity from generating graphs, and there’s no need for a separate viewer program. (Graphs are also created almost instantaneously; only for data sets including millions of points is there any noticeable delay.) Previous commands can be recalled, modified, and reissued, making it easy to keep playing with the data.

There are two more features which gnuplot offers to the more proficient user: abbreviations and sensible defaults.

Any command and subcommand or option can be abbreviated to the shortest, nonambiguous form. So the command

plot "prices" using 1:2 with lines,
» "prices" using 1:3 with linespoints

would much more likely have been issued as

plot "prices" u 1:2 w l, "prices" u 1:3 w linesp

In fact, we can shorten the linespoints style description even further to lp, so that the command becomes:

plot "prices" u 1:2 w l, "prices" u 1:3 w lp

This compact style is very useful when doing interactive work and should be mastered. From here on, I’ll increasingly start using it. (A list of the most frequently used abbreviations can be found in table 0.2.)

But this is still not the most compact form possible. Whenever a part of the command isn’t given explicitly, gnuplot first tries to interpolate the missing values with values the user has provided, and, failing that, falls back to sensible defaults. We’ve already seen how gnuplot defaults the range of x values to [10:10], but adjusts the y range to include all data points.

[Tip]
Abbreviations

Abbreviations for common keywords facilitate quick, interactive, and iterative work. You can find a list of common abbreviations in table 0.2.

Whenever a filename is missing, the most recent filename is interpolated. We can use this to abbreviate the last command even further:

plot "prices" u 1:2 w l, "" u 1:3 w lp

Note that the second set of quotation marks must be there.

In general, any user input (or part of user input) will remain unaffected until explicitly overridden by subsequent input. The way the filename is interpolated in the preceding example is a good example of this behavior. In later chapters, we’ll see how options can be built up step by step, by subsequently providing values for different suboptions. Gnuplot helps to keep commands short by remembering previous commands as much as possible.

One last example concerns the using directive. If it’s missing entirely and the data file contains multiple columns, gnuplot plots the second column versus the first (this is equivalent to using 1:2). If a using directive is given, but lists only a single column, gnuplot will use this column for y values and provide x values as integers starting at zero. This is also what happens when no using is given and the data file contains only a single column.

[3] A Taylor expansion is a local approximation of an arbitrary, possibly quite complicated, function in terms of powers of x . We won’t need this concept in the rest of this book. Check your favorite calculus book if you want to know more.

2.2  Saving commands and exporting graphics

There are two ways to save our work in gnuplot: we can save the gnuplot commands used to generate a plot, so that we can regenerate the plot at a later time. Or we can export the actual graph to a file in a standard graphics file formats, so that we can print it or include it in web pages, documents, or presentations.

2.2.1  Saving and loading commands

If we save the commands that we used to generate a plot to file, we can later load them again and in this way regenerate the plot where we left off. Gnuplot commands can be saved to a file simply using the save command:

save "graph.gp"

This will save the current values of all options, as well as the most recent plot command, to the specified file. This file can later be loaded again using the load command:

load "graph.gp"

The effect of loading a file is the same as issuing all the contained commands (including the actual plot command) at the gnuplot prompt.

[Tip]
Always save your work

Never forget to save your commands using save, so that you can recreate your graph later on using load.

An alternative to load is the call command, which is similar to load, but also takes up to 9 additional parameters after the filename to load. The parameters are available inside the loaded file in the variables ARG1 through ARG9. The special variable ARG0 is set to the filename, and the variable ARGC holds the number of parameters supplied to call.[4] We can use call to write some simple scripts for gnuplot.

Command files are plain text files, usually containing exactly one command per line. Several commands can be combined on a single line by separating them with a semicolon (;)—this also works at the interactive command-line. The hashmark (#) is interpreted as a comment character: the rest of the line following a hashmark is ignored. The hashmark isn’t interpreted as a comment character when it appears inside quoted strings.

The recommended file extension for gnuplot command files is .gp, but you may also find people using .plt instead.

Since command files are plain text files, they can be edited using a regular text editor. It’s sometimes useful to author them manually and load them into gnuplot (see section 16.1 for further discussion of that point), and they are also used for batch operations (chapter 15) and configurations (chapter 16).

2.2.2  Exporting graphs

The save command that we have just introduced saves the commands required to generate (or regenerate) the graph into a text file, but it doesn’t save the actual graph itself. How then do we get the graph that we see in the plot window on the screen into a file? Turns out that there are three ways to do it: easy, roundabout, and complicated.

Using the GUI button

In gnuplot 5, both the wxt and the Qt terminals include a GUI button that allows you to export a graph to file. The button will open a standard file selection dialog where you can select the desired name of the target file and the file format (PNG, PDF, or SVG). Easy.

Using the clipboard

A roundabout way to save the graph to file is to take a screenshot of the graph, using your favorite screenshot utility, and to save the result as a GIF or PNG file. After all, the graphics window on the screen is nothing more than a bitmap image, and a screenshot saves it to file.

Gnuplot facilitates this process by offering a way to copy the current content of the plot window to the window manager’s “clipboard”, from which the graph can be pasted into a paint program (such as the Gimp) or saved directly to a file. Above the actual graph in the plot window, you’ll find a row of buttons. One of them (at the time of this wring, the one all the way on the left), will copy the canvas with the graph to the clipboard, without the tool and title bars, and without any of the decorations put on the window by the window manager. Don’t be disturbed if nothing appears to be happening: there is no visual feedback, but the window contents will have been copied to the clipboard, from which it can be pasted into other applications. (The details of how to do that depend on your choice of window manager and the way the clipboard is configured on your local computer.)

This method may appear a bit unsophisticated, but it has two significant advantages: the workflow is simple and the resulting graph corresponds exactly to the graph that was on the screen.

Understanding terminals

Traditionally, the process for exporting a graph from gnuplot to a file has been rather cumbersome and error-prone. The reasons are historical: gnuplot predates many of today’s common graphics file formats, and some of the steps involved date back to the times when creating graphical output from a computer required hooking up special hardware (such as a pen-and-ink plotter). We will have much more to say about this topic in chapter 14 on terminals; for now I’ll only give you a short and hands-on introduction that will be sufficient to get you going.

All graphical output in gnuplot occurs via a terminal. In gnuplot parlance, a terminal describes a graphics-capable device, and also the set features it supports (color or black-and-white, available linewidths, and so on). Originally, a terminal tended to be a dedicated piece of hardware, but nowadays, a gnuplot “terminal” is merely a reference to the underlying graphics library.

[Tip]
Terminals

A terminal is gnuplot’s abstraction for a graphics-capable end point. Terminals can be interactive (screen oriented) or file-based. All graphics output must be directed to a terminal.

Contemporary terminals come in two flavors: interactive and file-based. Interactive terminals create the graphs that we see on the screen. On Linux, you have a choice among three interactive terminals, each built using a different widget set: the qt terminal uses Qt, the wxt uses wxt, and the old x11 terminal is a pure Xlib application. (On Windows, gnuplot exists as a GUI-like application, using the native Windows graphics elements.)

To save a graph to a file, however, we need to employ a file-based terminal. In this case, we need to specify the format (such as PNG, PDF, SVG) and the desired name of the file. These parameters need to be specified separately using the set command, and after the file has been created, the interactive terminal needs to be restored explicitly in the same way. It’s easy to forget one of these steps—that’s what makes the process awkward.

Next, I’ll walk you through the entire sequence, step-by-step. Then I’ll discuss the ways things can go wrong. Finally, I’ll show you a simple script that takes most of the pain out of the experience.

Saving to file: a step-by-step guide

As we just learned, in order to save a graph to a file, we need to specify two things: the file format (PNG, PDF, SVG, and so on) and the filename. In gnuplot, we do this using the set command:

set terminal pngcairo      # choose the file format
set output "mygraph.png"   # choose the filename

This is the first time we encounter gnuplot’s set command—we’ll talk about it in more detail in the next section (and subsequently in almost every chapter of this book!). For now, it’s enough to understand that it sets a parameter (such as terminal) to a value. However, and this is often forgotten, it does not generate a plot! The only commands to do so are plot, splot (which is used for three-dimensional graphs, which we’ll discuss in chapter 10), and replot and refresh (which both simply repeats the most recent plot or splot command).

So, with this in mind, the complete sequence to export a graph from gnuplot and to resume working is shown in listing 2.2.

Listing 2.2. The complete workflow to generate a PNG file from gnuplot

plot exp(-x**2)              # some plot command
set terminal pngcairo        # select the file format
set output "graph.png"       # specify the output filename
replot                       # repeat the most recent plot command,
                             #   with the output now going to the
                             #   specified file.
set terminal wxt             # restore the terminal settings
set output                   # send output to the screen again,
                             #   by using an empty filename.

This example demonstrates an important point: after exporting to a file, gnuplot does not automatically revert back to interactive mode—instead, all further output will be directed to the specified file. Therefore, we need to explicitly restore the interactive terminal (wxt in this example) and the output device. (The command set output without an argument sends all output to the interactive device, usually the screen.) This should come as no surprise. As we’ve seen before, gnuplot remembers any previous settings, and so neither the terminal nor the output setting change until we explicitly assign them a different value.

[Tip]
Don’t forget to restore terminal and output device

It’s a common mistake to forget to restore the terminal and output device to their previous settings after saving a graph to a file. Using a script can help to avoid errors.

Nevertheless, this behavior is rather different than what we’ve come to expect from user interfaces in most programs: we usually do not have to restore the interactive session explicitly after exporting to a file. It’s also unexpected that three separate commands are required to generate a file (set terminal, set output, and replot), making it easy to forget one.

What we therefore really want most of the time is a simple export routine, which takes the name of a file to export to, as well as the desired file format, and does all the required steps in one fell swoop. In the next section, I show you how to build one yourself.

2.2.3  One-step export script

The multistep process we just described to generate printable graphics from gnuplot is clearly a nuisance. Luckily, we can use the call command introduced earlier to bundle all required steps into one handy one-liner.

The call command executes all commands in a single file. Therefore, we can put all commands required to generate (for example) a PNG file and to restore the gnuplot session back to its original state into a command file, which we can then invoke through a single call command. And because call can take arguments, we can even pass the name of the desired output file as part of the same command.

If the commands shown in listing 2.3 are placed into a file, this file can be executed using call and will write the most recent plot to a PNG file and restore the initial session to its original state.

Listing 2.3. A useful script to export the current plot to file

set terminal push       # save the current terminal settings
set terminal pngcairo   # change terminal to PNG
set output ARG1         # set the output filename to the first option
replot                  # repeat the most recent plot command
set output              # restore output to interactive mode
set terminal pop        # restore the terminal

Here we’ve used the two pseudoterminals push and pop to help with the back-and-forth between the interactive and file terminals. The former (push) saves the current terminal settings onto a stack; the latter (pop) restores the terminal to the state saved on the stack. Neither makes any assumptions about the choice of interactive terminal, and therefore both can safely be used in scripts that must be portable across architectures.

Assuming the file shown in listing 2.3 was named export.gp in the current directory, we would call it like this, to write the current plot to a file called graph.png:

call "export.gp" "graph.png"

Before leaving this section, one last word of advice: always save the commands used to generate a plot to a command file before exporting to a printable format. Always. It’s almost guaranteed that you’ll want to regenerate the plot to make a minor modification (such as fixing the typo in a label, or adding one more data set, or adjusting the plot range slightly) at a later time. This can only be done from the commands saved to file using save, not from plots exported to a graphics file. In chapter 14, I’ll give an improved version of the export script which does both at the same time—that’s how I generate all of my graphs.

[Tip]
Save the commands, not just the graph

Don’t just store the finished graph. Always save the commands required to generate it as well, so that you can come back and modify it later on.

[4] Earlier versions of gnuplot used the special tokens #0 through #9. That syntax is now deprecated.

2.3  Managing options with set and show

Gnuplot has relatively few commands (such as the plot, save, and load commands we’ve already encountered), but a large number of options. These options are used to control everything, from the format of the decimal point to the name of the output file. There are more than 100 such options available, and countless suboptions for each. Check appendix B for a complete categorized reference of available options.

[Tip]
Gnuplot options

Many aspects of gnuplot’s behavior can be controlled through options. Use the set command to change their value; see appendix B for a full list of options.

The three commands used to manipulate individual options are

show to display the current value of an option

set to change the value of an option

unset to disable a specific option, or to return it to its default value

There’s also a fourth command, reset, which returns all options to their default values. The only options not affected by reset are the ones directly influencing output generation: terminal and output.

The syntax of all three commands is straightforward. To assign a new value to an option, use the set keyword, followed by the name of the option and the new value. The show command takes only a single argument: the name of the option you want to inspect.

The show command is also used more generally to display all kinds of information about gnuplot’s internal state, not just options which can be changed using set. For example, show variables displays all variables that are defined in the current session together with their values, and show functions lists all user-defined functions.

Another useful command is

show version long

This prints the current version of gnuplot, together with a copyright notice and some pointers to the online documentation. More importantly, it also shows the compile-time flags that gnuplot was compiled with. This is particularly relevant, since some features have only recently been added to gnuplot, and may not be enabled on all platforms. You can use show version long to see which flags your version of gnuplot was built with.

Finally, we can use show all to see a listing of all possible options and their values. Be prepared for a long listing!

[Tip]
The show command

You can use the show command to learn about your gnuplot installation and to inspect the current state of a gnuplot session.

2.4  Getting help

Gnuplot has an extensive, built-in online help system (“online” in the sense that it’s accessible from within the gnuplot session; it has nothing to do with network connectivity to the Internet). To get started, enter help at the gnuplot prompt. Alternatively, you can go directly to the reference page for a specific command or option by entering the name of the command or option as argument to the help command. For example, to learn about the plot command, you’d use

help plot

The online help is very detailed and comprehensive, so you should become familiar with it. But keep in mind that it’s a reference, not a tutorial. If you know the name of the command or option you are looking for, it’s great. But if you want to find all relevant options for a specific task, navigating the online help can be very frustrating.

This is what appendix B of this book is for: there, you’ll find all commands and options, grouped by topic. The appendix can therefore be used as a roadmap to the online help system.

[Tip]
Gnuplot’s online help system

Type help, followed by the command or option name, to access gnuplot’s built-in reference documentation.

2.5  Summary

In this chapter, we learned how to do the most important things with gnuplot: plotting, saving, and exporting. We also learned how to set and inspect options, and how to access the built-in reference documentation. In detail, we discussed

How to plot functions or data with the plot command: plot sin(x).

How to restrict the plot range using bracket notation: plot [0:5] sin(x).

How to select which columns from a data file to plot through using: plot "data" using 1:2.

How to change the plotting style through with: plot "data" with points or with lines or with linespoints.

How to save our work to file with the save command and how to load it again using load.

How to export a graph to a printable file format using set output, set terminal, and replot.

How to write simple scripts and use them through the call command

How to set options with set and how to inspect them with show.

How to access the online help using help.

This means that we can do the three most important things for day-to-day work already: generate a plot, save it to file, and export it. In the next chapter, we’ll begin to learn how to use gnuplot to analyze data and understand what it is telling us.

3

The heart of the matter: the plot command

This chapter covers:

Plotting functions and data

Math with gnuplot

Inline transformations

Logarithmic plots

Smoothing data

In gnuplot, you use the plot command to create plots of functions and data. It is, hands down, the most frequently used and most important of all gnuplot commands. In this chapter, we will take a systematic look at the plot command and the ways it can be used.

We begin by making simple plots of functions and data. Then we introduce inline transformations: gnuplot is capable of applying arbitrary transformations to the data as it is being plotted. You will learn how to use this useful feature, after we have taken a quick detour describing how to do math with gnuplot. Logarithmic plots are another indispensable technique in graphical analysis, and I’ll show you how and when to use them. (If you have never seen them before, you’ll find a sidebar giving you more background on them.) Finally, we’ll be ready to discuss the gnuplot’s built-in features to simplify, smooth, and summarize data.

3.1  Plotting functions and data

We use the plot command to plot both functions and data. Let’s discuss plotting functions first, and then move on to plotting data.

3.1.1  Plotting functions

To plot a function, you use the plot command, followed by the function to plot, and (optionally) any style directives you wish to use (see chapters 9 and 6 to learn about available styles and how to specify them).

plot sin(x)                   # plot in the default style
plot cos(x) with linespoints  # plotted with lines and points

Including two or more functions in the same graph is easy: just list them all after the initial plot command, separated by commas:

plot sin(x), cos(x) w lp

If you don’t specify a style, gnuplot will chose a different line color or dash pattern for each new function in the plot. If you include more functions than there are currently defined styles, gnuplot will cycle through them again from the beginning. (This is explained in more detail in chapters 6 and 9.)

Gnuplot comes with a large number of built-in functions; and it is also possible to define your own. We will talk about this in the section on math later in this chapter.

The sampling rate for functions

The set samples option controls the number of points at which a function is evaluated to generate a plot. It defaults to 100 points, but this will not be sufficient for curves including cusps or singularities. A value of 500–1,000 works well—it rarely makes sense to set a sampling rate much higher than the horizontal number of pixels of the output device (which is usually around 600). This option has no effect when plotting data, unless one of the smoothing algorithms is used (see section 3.5 later in this chapter); the smooth curve will be sampled according to the value of the set samples option.

Changing the number of sampling points is easy: simply give an integer argument to set samples:

set samples 1000    # Now 1000 points are used

You may want to include that line in your personal gnuplot initialization file, overriding gnuplot’s (ridiculously) small default value (see chapter 16 to learn how to set up a gnuplot initialization file).

[Tip]
Use an appropriate value for set samples

Choose an appropriate value (such as 1,000 or even 3,000) for the number of points gnuplot uses to plot a function. The default value of 100 is unreasonably low given the capabilities of today’s computers and displays.

3.1.2  Plotting data

To plot data, you again use the plot command, only that in place of the function to plot, you now specify the name of the file containing the data set. There is one additional wrinkle: you also need to specify which columns from the file to select. And finally, you can again chose which style gnuplot should use for your plot.

We will talk about data files and their permissible formats in more detail in the next chapter(Chapter 4 Managing data sets and files). For now, suffice it to say that gnuplot expects data files to be whitespace-separated, column-oriented text files, with one line per record. Listing 3.1 shows an example—in this case, the world records (as of late 2014), for both men and women running events in a variety of distances. (Distances in meters and times in seconds—the last two entries with the “odd” distances are for half-marathon and full-marathon races.)

Listing 3.1. World record running times with date (records)

# Distance    Men      Women    Men           Women
100          9.58      10.49    2009-08-16    1988-07-16
200         19.19      21.34    2009-08-20    1988-09-29
400         43.18      47.60    1999-08-26    1985-10-06
800        100.91     113.28    2012-08-09    1983-07-26
1000       131.96     148.98    1999-09-05    1996-08-23
1500       206.00     230.46    1998-07-14    1993-09-11
3000       440.67     486.11    1996-09-01    1993-09-13
5000       757.35     851.15    2005-05-31    2008-06-06
10000     1577.53    1771.78    2005-08-26    1993-09-08
21098     3503.00    3912.00    2010-03-21    2014-02-16    # Half marathon
42195     7377.00    8262.00    2014-09-28    2003-04-13    # Full marathon

Figure 3.1 displays a plot of the finishing times, for events up to 10,000 meters as a function of the distance, together with two function plots for comparison (the using, title, and w p directives have already been introduced in section 2.1.2):

plot [:12000] "records" u 1:2 title "Men" w p, "" u 1:3 title "Women" w p,
»             0.14*x, 0.16*x

Not surprisingly, the time required increases roughly in proportion with the distance, but not precisely so. A comparison with the straight lines included in the graph shows that for longer distances the required time is greater than what one would expect given the short-distance results. Again, that’s no surprise: as the distance gets longer, the runners slow down. (We’ll come back to this point later.)

Figure 3.1. Current athletic world record running times, as a function of the competition distance, for both men and women. The data is shown in listing 3.1.

In this example, we plot two (simple) functions together with the data, so that we can compare the data against a known behavior (as expressed by the function). This kind of comparison constitutes an important part of graphical analysis, and gnuplot makes it particularly easy to combine multiple data sets and functions together in a single graph.

[Tip]
Comparing data against known behavior

Comparing data against a known function can tell us a lot about the data set. Including one or several suitable functions in the graph brings out the behavior of the data.

3.2  Math with gnuplot

Not surprisingly, gnuplot can evaluate mathematical expressions and includes support for many mathematical functions. Gnuplot’s syntax for mathematical expressions is straightforward and similar to the conventions found in other programming languages.

Gnuplot ships with its own math library, which these days is only used when the system math library is found insufficient. (The most notable extension that gnuplot’s math library provides is support for complex numbers: see chapter 13).

3.2.1  Mathematical expressions

Gnuplot uses standard infix syntax for mathematical expressions, including the normal operators for the four basic arithmetical operations, as in most C-like languages. Parentheses can be used to change the order of evaluation. Gnuplot has the exponentiation operator (**), found in Fortran and Perl. All these operate either on floating-point values or on integers.

It is important to understand that integer division truncates, so that 3/2 yields 1. Integers are promoted to floating-point values in mixed expressions, but in some situations truncation occurs in expressions involving variables. If nontruncating division is desired, it’s always safer to force the use of floating-point arithmetic by writing x/2. or x/2.0. You can also force a mixed expression by adding a floating-point zero: x + 0.0.

[Tip]
Integer division truncates

Keep in mind that integer division truncates, hence an expression like 3/2 evaluates to 1, not to 1.5. To force floating-point division, include a decimal point: 3/2. will evaluate to 1.5.

Gnuplot also offers a range of operators for integer arithmetic, such as the familiar modulo operator %, and a unary factorial operator: n! = 1 * 2 * ... * (n-1) * n. You can also find bit-shift and bitwise logical operators.

All the usual relational and logical operators are available as well, including the ternary conditional operator (?:). A full list (sorted by precedence) can be found in appendix B.

3.2.2  Built-in functions

Gnuplot provides all the mathematical functions you’ve come to expect on any scientific calculator: square root, exponential function and logarithms, and trigonometric and hyperbolic functions. It also supports some less-standard functions that are relevant in physics, applied math, and statistics (such as Bessel functions, the error integral, and the gamma function). You can execute help expressions functions from within gnuplot or check appendix B for the full list.

[Tip]
Predefined probability functions

Check the file stat.inc in gnuplot’s demo/ folder for many useful functions related to probability distributions.

Gnuplot also includes a built-in random number generator, which can be quite useful at times. It can be accessed through the function rand(x). If called with an argument that is equal to zero, the function returns a pseudo random number between zero and one.

The algorithm used to generate the pseudo random numbers requires two seeds to be set; given the same values for the seeds, exactly the same sequence of numbers will be returned. Gnuplot doesn’t have a separate function to set the seeds; instead this is accomplished by providing special values as arguments to rand(x). Check table 3.1 for details. Keep in mind that rand(x) is a function, and therefore must be called as an expression, even if only setting the seeds. The command print rand(-1), for example, can be used to reset the seeds to their default values.

Table 3.1. The rand(x) function is used both to access the random number generator and to set its seeds, depending on the value of the argument.

Invocation

Description

rand(0)

Returns the next pseudo random number, using the current values of both seeds

rand(-1)

Resets both seeds to a standard value

rand(x)

For x>0, sets both seeds to a value based on the value of x

rand({x,y})

For x,y>0, sets the first seed to x and the other seed to y

The standard gnuplot distribution contains a file called stat.inc in the demo/ folder, which defines many additional functions, most of them related to various probability distributions. You can load this file into your current gnuplot session using load, and then use these functions the same way you would use built-ins.

3.2.3  User-defined variables and functions

It’s easy to define new variables simply by assigning an expression to a name. For instance, we might want to define some useful mathematical constants. (We don’t need to define pi, since it’s already defined by gnuplot.) Listing 3.2 shows some more useful constants we may want to define.

Listing 3.2. Examples of user-defined variables (constants)

e = 2.71828182845905       # The base of the natural logarithm
sqrt2 = sqrt(2)            # The square-root of 2
euler = 0.57721566490153   # The Euler-Mascheroni constant

Here, euler is the Euler-Mascheroni constant (usually represented using the Greek letter gamma), which is “the most important number that you’ve never heard of before” (J. Stopple).[5]

Functions can be defined in a similar fashion, as shown in listing 3.3.

Listing 3.3. Examples of user-defined functions

f(x) = -x * log(x)

gauss( x, m, s ) = exp( -0.5*( (x-m)/s )**2 )/sqrt(2*pi*s**2)

# Using the factorial operator to generate binomial coefficients
binom( n, k ) = n!/(k!*(n-k)!)

min( a, b ) = ( a < b ) ? a : b  # Using the ternary "?" operator
step( x ) = ( x < 0 ) ? 0 : 1    # A piece-wise defined function

Functions can have up to 12 variables and can contain other functions and operators. We use them as we would use any other function. Assuming we had issued the definitions in listing 3.3, we could then simply write plot sin(x), f(x).

By default, gnuplot assumes that the independent dummy variable, which is automatically replaced by a range of x values when plotting a function, is labeled x (as in: plot sin(x)), but this can be changed using the set dummy command. For example, set dummy t will make t the independent variable so that now we would say: plot sin(t).

All other variables occurring in a function definition (parameters) must have been assigned values before we can plot (that is, evaluate) the function. For convenience, it’s possible to assign values to parameters as part of the plot command. For example, we could do the following (notice that no comma is required between the variable assignment and the function):

g(x) = exp(-0.5*(x/s)**2)/s
plot s=1 g(x), s=2 g(x), s=3 g(x)

But in general, I think it is better to make all parameters explicit arguments in the function definition. The previous example could (and probably should) be written as follows (except for the entries in the graphs legend or “key”, the resulting figure are identical—see figure 3.2):

g(x, s) = exp(-0.5*(x/s)**2)/s
plot g(x,1), g(x,2), g(x,3)

Figure 3.2. Plotting a function for different values of a parameter: plot g(x,1), g(x,2), g(x,3) where g(x, s) = exp(-0.5*(x/s)**2)/s.

All functions and variables have global scope. There’s no such thing as a private variable or local scope!

Lists of all user-defined variables and functions can be generated using the following two commands (remember to use help expressions functions to list built-in functions):

show variables
show functions

3.2.4  Mathematically undefined values and NaN (Not-a-Number)

Gnuplot tends to be pretty tolerant when encountering undefined values: rather than failing, it just doesn’t produce any graphical output for data points with undefined values. For instance, you may safely plot the square root (which is only defined for positive values) over any plot range: if the argument is negative, gnuplot will not fail—it just won’t show anything on the graph for x < 0. Similarly, you never have to worry whether the denominator of a function might equal zero for any value in the plot range: if it does, gnuplot will silently skip those points.

This behavior can be used to suppress data points or define piecewise functions, usually in conjunction with the ternary conditional operator. For example, consider the following function:

f(x) = abs(x) < 1 ? 1 : NaN

It’s only defined on the interval [-1:1]. For values with an absolute value equal to one or greater, it returns NaN, which stands for for “Not a Number” and indicates a mathematically undefined value.[6] A plot of this function will show data points only on the interval [-1:1] and be empty otherwise.

This NaN technique is a good trick that’s frequently useful, in particular in conjunction with the ternary operator, as in these examples.

[Tip]
Gnuplot ignores undefined values in a plot

Gnuplot does not fail when it encounters a mathematically undefined value during a plot: it just does not generate graphical output for such points. You can leverage this effect to exclude intervals or data points from plots.

[5] Read his wonderful A Primer of Analytic Number Theory (2003) to find out why you should care. The quote can be found on page 49.

[6] Using the NaN special value from the IEEE 754 standard for floating-point computations is new with gnuplot 5; in previous releases it was customary to achieve the same effect using the expression 1/0.

3.3  Data transformations

As stated before, gnuplot is first and foremost a plotting tool: a program that allows us to generate straightforward plots of raw data in a simple and efficient manner. Specifically, it’s not a statistics package or a workbench for numerical analysis. Large-scale data transformations are not what gnuplot is designed for. Properly understood, this is one of gnuplot’s main strengths: it does a simple task and does it well, and does not require learning an entire toolset or programming language to use.

Nevertheless, gnuplot has the ability to perform arbitrary data transformations “on the fly” as part of the plot command. This allows us to apply filters to the data from within gnuplot, without having to take recourse to external tools or programming languages.

3.3.1  Simple data transformations

An arbitrary function can be applied to each data point as part of the using directive in the plot command. If an argument to using is enclosed in parentheses, it’s not treated as a column number, but as an expression to be evaluated. Inside the parentheses, you can access the values of the column values for the current record by preceding the column number with a dollar sign ($) (as in shell or awk programming). Some examples will help to clarify.

To plot the square root of the values found in the second column versus the values in the first column, use

plot "data" using 1:( sqrt($2) ) with lines

To plot the average of the second and third columns, use

plot "data" using 1:( ($2+$3)/2 ) with lines

To generate a log/log plot, we can use the following command (although the logscale style, discussed in section 3.4, is the preferred way to achieve the same effect):

plot "data" using ( log($1) ):( log($2) ) with lines

[Tip]
Inline transformations

Inline transformations allow you to apply an arbitrary transformation to each point as it is being plotted.

A fundamental limitation to all these transforms is that they can only be applied to a single record at a time. If you need aggregate functions over several records (sums or averages, for example), or across different data sets, you’ll usually have to perform them externally to gnuplot. Nevertheless, the ability to apply an arbitrary filter to each data point, and to combine different data points for the same x value, is often tremendously useful.

A worked example

Remember the running world records from listing 3.1? Based on the basic plot of time versus distance, we were led to suspect that athletes run more slowly on longer distances. The best way to confirm this would be a plot of the average speed for the different events. The data set does not contain the speed itself, but we can calculate it easily as distance (column 1) divided by speed (in column 2 or 3, respectively). This is precisely the kind of simple combination of column values that inline transformations are intended for (remember than an empty file name is a placeholder for the most recently accessed file):

plot [9:][5:11] "records" u 2:($1/$2) t "Men" w lp,
»               "" u 3:($1/$3) t "Women" w lp

Figure 3.3 shows the speed as a function of the time (not distance), in a a semi-logarithmic plot (in the horizontal axis: set logsc x). The figure confirms our expectation that the average speed decreases for longer distances, but it shows us more: the rate of decrease is not constant. We can distinguish three regimes: a sprint regime (up to 20 seconds), a long-duration regime (longer than about 2 minutes), and an transition regime in between where the decrease in speed is particularly drastic.

Figure 3.3. The average speed of the current world record holder, as a function of the distance. The speed has been calculated from the distance and the finishing time as part of the plot command itself, using an inline transformation: plot "records" u 2:($1/$2) w lp. (See listing 3.1 for the data.)

More ideas for inline transformations

Here are some more creative uses for inline transformations. To plot two data sets of different magnitude on a similar scale, use this (assuming that the data in column three is typically greater by a factor of 100 than the data in column two):

plot "data" using 1:2 with lines, "" using 1:( $3/100.0 ) with lines

If the data file contains the x value in the first column, the mean in the second, and the variance in the third, we can plot the band in which we expect 68 percent of all data to fall as

plot "data" using 1:( $2+sqrt($3) ) with lines,
»    "" using 1:( $2-sqrt($3) ) with lines

All expressions involving operators or functions can be part of using expressions, including the conditional operator:

plot "data" using 1:( $2 > 0 ? log($2) : 0 ) with lines

A similar method can be used to exclude certain data points when plotting data from a file. For example, the following command will only plot data points for which the y value is less than 10:

plot "data" using 1:( $2 < 10 ? $2 : NaN ) with linespoints

Finally, it should be kept in mind that the expression supplied in parentheses can be a constant. We will see some examples for this usage in section 3.5 later in this chapter.

3.4  Logarithmic plots

Lastly, let’s see how we can generate logarithmic plots. Logarithmic plots are a crucial technique in graphical analysis. In gnuplot, it’s easy to switch to and from logarithmic plots:

set logscale        # turn on double logarithmic plotting
set logscale x      # turn on logarithmic plotting for x-axis only
set logscale y      #                              for y-axis only

unset logscale      # turn off logarithmic plotting for all axes
unset logscale x    #                               for x-axis only
unset logscale y    #                               for y-axis only

We can provide a base as a second argument: set logscale y 2 turns on binary logarithms for the y axis. (The default is to use base 10.)

We’ll talk some more about uses for set logscale in chapter 18.

3.4.1  Digression: How do logarithmic plots work?

Logarithmic plots are a truly indispensable tool in graphical analysis. Fortunately, it’s possible to understand what they do even without detailed understanding of the mathematics behind them. However, the math isn’t actually all that hard, so in this section, I’ll try to explain how logarithmic plots work and how they’re used.

Let’s just quickly recapitulate the most relevant properties of logarithms. The defining relation for the log function is

n = logb bn

There isn’t a single log function, but infinitely many; we always need to specify the base b , with respect to which the logarithm needs to be taken.

Logarithms with respect to different bases are related to one another in a simple way:

logax = logab logbx

Logarithms to two specific bases are used most often: the logarithm to base 10 (typically abbreviated log ) and the so-called natural logarithm to base e=2.7818... (abbreviated ln ). Be aware that the notation just introduced ( log(x) and ln(x) ) is common in mathematical contexts, but when using gnuplot’s built-in functions, log(x) refers to the natural logarithm; you must use log10(x) for the logarithm to base 10.

Logarithms have many interesting properties, two of which are of fundamental importance:

log( x y ) = log( x ) + log( y )

and

log( xk ) = k log( x )

We can see that the logarithm of a product equals the sum of the logarithms, so these two relations can be summarized by saying that logarithms turn products into sums and powers into products. We’ll see how these relationships are at the core of logarithmic plots.

In a logarithmic plot, we don’t plot the actual values; instead we plot their logarithms. But tic marks on the axes are usually labeled with the actual values (not their logarithms). Here’s an example: we want to plot the values of 1, 10, 100, and 1000. Their logarithms (to basis 10) are 0, 1, 2, and 3, respectively (since 10³ = 1000 and so on). In a logarithmic plot, we’ll place tic marks at equal distance from each other (at the locations 0, 1, 2, and 3), but label them 1, 10, 100, and 1000.

In figure 3.4 we demonstrate these aspects. The y axis on the left is scaled logarithmically. Comparing the values on the left-hand y axis to the units of the right-hand y axis shows how a logarithmic axis is linear in the exponent to which the base must be raised to obtain the actual value (that is, it’s linear in the logarithm). Also note how the same relative change results in the same visual length when using logarithmic scales: both scale arrows have the same length and represent a 100 percent change, although the absolute change is different for both arrows (1 to 2 on the left, 100 to 200 on the right).

Figure 3.4. A semi-log plot. See the text for details. (Labels for some of the minor tic marks on the left axis have been provided.)

We can switch on logarithmic plotting either on only a single axis (usually the y axis) or on both, yielding a single or double logarithmic plot, respectively. Single logarithmic plots are sometimes referred to as semi-log plots, and double logarithmic plots are also known as log-log plots. Both plots are used for different purposes.

There are two uses for single logarithmic plots. First of all, if we have data with exponential behavior, showing it on a single logarithmic plot will let the data fall onto a straight line:

y(x) = ea x ⇒ log( y(x) ) = a x

In other words, if y(x) is an exponential function of x , then log( y(x) ) is a linear function (straight line) of x . We can use single logarithmic plots in this way to determine, for instance, the half-life of a radioactive substance from measured decays.

The other use for single logarithmic plots is common in finance: on a single logarithmic plot, all relative changes have the same size, no matter how large the underlying absolute changes.

Here’s an example: If a stock we own falls by $10 in a single day, our reaction will be different depending on whether we paid $20 or $100 initially. In the first case, we lost 50 percent of our money, but in the second, we lost only 10 percent. What we really care about are relative changes, not the absolute ones. What we therefore want is a plot in which all changes of the same relative magnitude result in the same distance on the graph. That’s exactly what semi-log plots do for us. If the value before the change is y₁ and the value after the change is y₂ = g y₁ , where g equals 1.1 (corresponding to a 10 percent gain), then the change in absolute terms is

y₁ - y₂ = y₁ - g y₁ = (1 -g) y₁

which depends on the overall size of the value y₁ . But if we consider logarithms, we find

log( y₁ ) - log( y₂ ) = log( y₁ ) - log( g y₁ ) = log( y₁ ) - log( g ) - log( y₁ ) = - log( g )

which is independent of the overall value and depends only on the relative size of the change, g . This is why long-term charts of, for example, the Dow-Jones index are usually shown as semi-log plots: a 10 percent drop today or in 1929 results in the same distance covered on the graph, despite the fact that the Dow-Jones is over 17,000 today, but stood around 300 then.

Double logarithmic plots serve a different purpose: they help us identify power law behavior—that is, data that follows an equation such as the following ( C is a constant):

y( x ) = C xk

The analysis goes through as before, but we end up with logarithms now on both sides of the equation:

log( y( x ) ) = k log( x ) + log( C )

The resulting graph is a straight line, with a slope that depends on the exponent k . We’ve seen an example of this in chapter 1, when estimating the completion time of a long-running computer program.

Double logarithmic plots are very important. Power laws occur in many different contexts in the real world, but aren’t always easy to spot. Go back to figure 1.3 in chapter 1: many different curves will seem to fit the data about equally well. But once plotted on a double-log plot (see figure 1.4), the linear shape of the data stands out and provides a strong and easily recognizable indicator of the underlying power law behavior.

[Tip]
Become familiar with log plots

Log and log-log plots are part of the standard toolset. They reveal exponential and power-law behavior, and they are generally useful when dealing with data that changes over many orders of magnitude. Learn how to use them!

3.5  Smooth interpolation and approximation

Gnuplot can construct a variety of smooth curves as representations of a data set. We can invoke this feature using the smooth directive to the plot command, followed by an additional keyword that selects the specific interpolation method. Table 3.2 lists the available smoothing methods, together with a brief description.

Although all these smoothing methods are invoked using the smooth keyword, they fall into three broad groups, which do rather different things and and apply to different situations. Methods 1–5 construct a smooth curve through a set of (x,y) coordinate pairs. Methods 6–8 treat the data set as a one-dimensional (or univariate) point distribution and generate visual representations of this distribution. Methods 9 and 10 remove multiple entries from a data sets. As it turns out, method 10 (frequency) can also be used to count data points in order to construct a histogram.

In the remainder of this chapter, we will describe each group of options in a section by itself. In chapter 18, we will see how these methods can be applied to solve common problems in the graphical analysis of data sets.

Table 3.2. Smoothing methods available for plot ... smooth ....

Keyword

Description

1

csplines

Cubic spline interpolation

2

msplines

Monotone cubic spline interpolation

3

acsplines

Approximate cubic spline

4

bezier

Bézier curve

5

sbezier

Sorted Bézier curve (constructed after sorting the data points on their x values).

6

kdensity

Kernel density estimate

7

cumul

Cumulative distribution function

8

cnormal

TBD

9

unique

Sort data points on their x values, dedupe repeated entries, replace the y value for repeated x values by the average of the y values.

10

frequency

Sort data points on their x values, dedupe repeated entries, replace the y value for repeated x values by the sum of the y values.

3.5.1  Interpolation curves

Gnuplot provides two different sets of methods to draw a smooth curve through a set of (x,y) points: those based on cubic splines, and those based on Bézier curves.

Spline interpolations

Splines[7] are smooth curves, which are constructed from piecewise-polynomial functions that are smoothly joined together. If splines are called cubic, then the polynomials used are third order. Splines can be used to give a smooth approximation to a set of points; interpolating splines pass through each of the points exactly, approximating splines only pass the vicinity of each point.

Both csplines and mcsplines construct interpolating cubic splines, meaning that the resulting curve passes through every point in the data set. The second form (mcsplines)[8] also guarantees that the curve has the same monotonicity and convexity properties as the original data: look at figure 3.5. The curve drawn using csplines occasionally “overshoots” near locations of greatest curvature, whereas the line drawn using mcsplines doesn’t. The second curve therefore may give a better, more faithful representation of the data.

Figure 3.5. Interpolating splines: both csplines and mcsplines interpolate the data points smoothly; but mcsplines don’t “overshoot” the data. By contrast to splines, which are locally defined, the Bézier curve constitutes a global approximation. plot "splines" u 1:2 smooth csplines, "" u 1:2 smooth mcsplines,"" u 1:2 smooth bezier.

The third keyword (acsplines) creates an approximating spline, meaning that the curve does not pass exactly through the data points. To control how close the curve passes to each point, we need a new parameter (in addition to the x and y coordinates of each point), which is supplied to gnuplot using a third “column” in the using directive, like so:

plot "data" u 1:2:(0.01) smooth acsplines

This third parameter is a measure for the weight with which each point contributes to the spline: the greater it is, the more the point will influence the curve, and the closer the curve will pass to each point; the smaller it is, the less will each point influence the curve. The two extreme cases are instructive: as the weight parameter approaches infinity, the curve will approach the the interpolating spline (which passes through every point exactly), but as the weight parameter approaches zero, the resulting spline will approach the linear least-squares fit to the data. (See figure 3.6.)

I’d like to draw attention to the way we have handled the third “column” specifier in the using directive: we supplied an expression in parentheses, which means that it was interpreted (because of the parentheses) as a “transformation” to be applied to the data. The expression itself, however, was a constant, so that the value of this “third column” is simply the value of that constant. Since the third column gives the weight with which the point should contribute to the spline approximation, every point received the same weight. Instead of supplying a numerical constant, we could have specified an actual column from which to take the weights, like so: plot "data" u 1:2:3 smooth acsplines (note the missing parentheses, which means that the 3 is interpreted as a column number, not as a numerical constant). If the third column in the data file contains some measure of the reliability or accuracy with which each point is known, we can have each point contribute with a different weight, according to the certainty with which it is known.

[Tip]
Constant expressions in inline transformations

It is frequently useful to supply a constant expression in inline transformations. For example, to have every point contribute equally to a spline approximation, you can use: plot "data" u 1:2:(0.1) smooth acsplines.

Figure 3.6. Influence of the weight parameter on approximate splines. As the weight increases, the curve approximates an interpolating spline; as the weight goes to zero, the curve approaches a straight line: plot "splines" u 1:2 w p, "" u 1:2:(10e6) s acs, "" u 1:2:(10) s acs, "" u 1:2:(1./10) s acs, "" u 1:2:(10e-6) s acs.

Approximating splines are an important tool for graphical analysis. In chapter 18 I will show you how to use it to visualize trends and features in noisy data sets.

Bézier approximations

Bézier curves[9] are constructed as the path that would be traced out by a particle which, starting at one point, steers towards each of the subsequent points in turn. By construction, Bézier curves pass through the first and last point exactly, but in general won’t pass through any of the internal points. Whereas splines are defined locally (between neighboring points), Bézier curves are defined for the entire data set. As a result, Bézier curves usually appear particularly “smooth”—which is also why they are used so frequently in computer graphics and design.

Gnuplot creates a Bézier curve of order n (where n is the number of data points) using smooth bezier. See figure 3.5.

Because Bézier curves are traced out by a particle heading towards a set of “waypoints” in turn, not only the positions of the waypoints matter, but also the order in which they are traversed. When using smooth sbezier, the data points are first sorted by their x coordinates, and a Bézier curve is constructed for the resulting data set.

3.5.2  Point distributions

Gnuplot includes a few different ways to visualize univariate point distributions, some involving the smooth facility, some not. Here we are primarily interested in the technicalities of using the smooth feature—consult chapter 18 for additional information about the math involved and to learn about the kinds of questions that these methods help you answer.

Kernel density estimates

Assume that you have a number of measurements of a single quantity, like the ones in example 3.4. These numbers may be weights of different shipments, or the time it took a bunch of queries to finish, or the the temperatures in various parts of the house: it doesn’t matter, in every case, we want to know how much the points scatter about.

Listing 3.4. Scattered measurements for figure 3.7

2.14
1.25
1.10
1.00
1.60
1.31
0.95
0.90
4.01
1.95
2.53
0.65

We can just plot all the points along a horizontal line (see figure 3.7) and take a look, but if the number of points becomes large, it can be difficult to recognize the details of the distribution. In a kernel density estimate, we replace each point with a strongly peaked, but smooth curve (the “kernel”) centered at the location of each point. We then sum the contributions from all these curves and plot the result.

Gnuplot’s smooth kdens feature uses Gaussian curves as kernels and takes two columns in the using specifier: the first column gives the position of each point, the second the weight of each point. If we want to weight all points equally, we may again supply a constant. To obtain a normalized density estimate, the weight of each point should be 1/n, where n is the number of points in the data set.

Figure 3.7. Visualizing point distributions using kernel density estimates and the cumulative distribution function. plot "measurements" u 1:(-1) w p, "" u 1:(1/12.) s kdens bandwidth .1, "" u 1:(1/12.) s kdens bandwidth 0.2, "" u 1:(1/12.) s cumul The calculated default bandwidth is 0.58. Compare listing 3.4 for the data.

With a third parameter, we control the width of each kernel. If you choose a large value for this parameter, each point will appear very “smeared out” and the resulting curve will show few details; if you choose a small bandwidth, more features will show, but the resulting curve may be noisy. (This is explained in greater detail in section 18.2.2.) The bandwidth parameter is not passed as a pseudo-column; instead, it follows after the bandwidth keyword:

plot "data" u 1:(1) s kdens, "" u 1:(1) s kdens 1

If you don’t supply a bandwidth explicitly, gnuplot will calculate one based on the standard deviation of the data set and store it in a variable called GPVAL_KDENSITY_BANDWIDTH. This default bandwidth will often appear too broad in practice. You can inspect its value using print GPVAL_KDENSITY_BANDWIDTH. (We will formally introduce gnuplot’s internal variables in section 5.5 in chapter 5.)

Cumulative distribution functions

Another way to visualize a point distribution is to count the fraction of points that lie to the left of the current position: this is the cumulative distribution function of the data set. By construction, the cumulative distribution function is monotonically increasing: it is zero to the left of the point with the smallest value, and is equal to one to the right of the point with the greatest value. (See figure 3.7.)

In gnuplot you use smooth cumul to create a cumulative distribution function. It takes two columns in the using specifier: the first holds the position of each measurement, the second the associated weight. (It is not necessary for the data set to be sorted.) If you supply a weight of 1, then the value of the resulting curve at any position will be equal to the number of data points to the left of the current position; if you supply a weight of 1/n to cumul where n is the number of points in the data set, then the curve will be equal to the fraction of data points to the left.

3.5.3  Deduping repeated entries

The last two smoothing methods available are less sophisticated. When using smooth unique on a data set consisting of (x,y) coordinate pairs, gnuplot first sorts the points in ascending order on their x coordinates. It then replaces all points that have the same x coordinate with a single point that has the same x coordinate, but whose y coordinate is the average over all the y coordinates of such points.

The smooth frequency method works in the same way, except that it replaces the y coordinate with the sum of all the affected points. This is not very useful by itself, but can be used to construct histograms of point distributions—I will show how to do that when we revisit this topic in chapter 18.

[7] We’ll have more to say about splines in section 18.1.1.4.

[8] Monotone Piecewise Cubic Interpolation” by F. N. Fritsch and R. E. Carlson, SIAM Journal on Numerical Analysis, Vol. 17 (1980), pages 238–246.

[9] The Wikipedia entry on Bézier curves is a good place to start.

3.6  Summary

The plot command is the most important gnuplot command. Not only does it generate plots of functions and data, but it can also apply arbitrary transformations to the data as it is being plotted, and it can perform a variety of operations to interpolate or approximate the data.

We also took a look at the way gnuplot handles mathematical expressions and at its built-in functions.

Next, we will take a closer look at data files and data file formats. We will see how gnuplot reads data from files, and learn more about accessing column values in conjunction with inline transformations.

4

Managing data sets and files

This chapter covers:

Data file formats and options

Managing structured data sets

Accessing columns and pseudocolumns

Pseudofiles and input/output redirection

Metadata in data files

Working with data is what gnuplot is all about. Data is usually presented to gnuplot as files, and in this chapter you will learn a range of useful tricks that can make you more productive when working with gnuplot.

We begin with a review of gnuplot’s standard file format and discuss options to modify its defaults. We then look at some special gnuplot syntax to pick out only parts from larger files. This is required when dealing with data sets that are larger or more complicated than the straightforward ones we’ve encountered so far, and often useful.

Next, we will systematically consider all ways to access columns, in particular in conjunction with inline transformations. In the process, we will also encounter “pseudocolumns”, which are synthetic columns that gnuplot automatically provides when reading a file. Pseudocolumns hold useful information (such as the line number) and often come in handy when constructing inline transformations.

Then we’ll turn to files that aren’t: “pseudofiles” are data sources that don’t live on the disk, but that we can treat almost as if they were regular files.

But data files often contain information beyond the actual data that goes onto a plot. Such metadata typically consists of textual information that we may want to place onto a plot in the form of text labels. Here, where we talk about data files, we will merely summarize the kind of metadata gnuplot can take from a file and use in a plot. (The details will follow in their proper context when we discuss how to handle text labels and similar auxiliary information in a plot in chapters 6 through 8.)

4.1  Quickstart: The standard data file format

Data is usually fed to gnuplot as plain text in a simple, whitespace-separated format: gnuplot does neither use nor require special proprietary binary formats or weird markup; nor do we need to employ special tools or libraries to generate data that constitutes suitable input for gnuplot.

Input files may also contain comments, text strings, and other information, in addition to the actual data. Furthermore, many aspects of this format can be changed through options. We will consider each of these topics in turn.

By default, gnuplot data files are text files that contain data in whitespace-separated columns, with one line per record. This is a natural tabular format—basically the way you would lay out information in a spreadsheet, or the way it would result from a database query. Listing 4.1 shows the beginning of a data set with information on customer orders as received by an online retailer. We might want to plot data from this file directly in a scatter plot, for example to learn about the relationship between the number of items in the order (the ItemCount) and the time it took the warehouse to complete the order (the CycleTime): plot "orders" u 4:7 w p. (See chapter 18 to learn more about using scatter plots to understand relationships.) Alternatively, we might want to understand the relative number of orders containing only a single item, or up to two items, or up to three, and so on. This is what cumulative distribution functions are for: plot "orders" u 4:(1) smooth cumul (see section 3.5 for smooth cumul and chapter 18 for cumulative distribution functions more generally.)

Listing 4.1. The beginning of a generic data file

# ID     Date         CustomerID   ItemCount   Weight   Value   CycleTime
oid001   2014-12-01   cid0017      1           1.5      34.33     65
oid032   2014-12-01   cid0523      1           2.1       4.78     83
oid171   2014-12-01   cid0738      3           3.2      48.01    122
oid284   2014-12-01   cid0023      1           0.4      12.35     72
oid314   2014-12-02   cid1003      5           3.9      78.62    159
...

4.1.1  Comments and header lines

Lines beginning with a # are considered comment lines and are ignored. You can also instruct gnuplot to ignore the first n lines of a file by using the skip directive in the plot command:

plot "data" skip 2 using 1:2

Now gnuplot will discard the first two lines of the file, regardless of whether they are commented out or not. This is a convenient way to get rid of header-lines without having to comment them out explicitly.

[Tip]
Use plot ... skip ... to get rid of unwanted header information

You can use the new skip keyword to discard the first n lines of a file as it is being plotted.

4.1.2  Selecting columns

We can pick out any column and plot its values against any other column by means of the using directive. If no using directive is found, gnuplot will plot the first column against the line number; if only a single column is specified, gnuplot will plot that column against the line number.

For the most part, input file parsing with gnuplot is very robust and works without much tinkering. One good piece of advice is to always specify all required columns explicitly via the using directive. If this is done, gnuplot silently skips any garbage (fields it can’t parse) in the file, treating them as missing values. If one relies on the default columns (without the using directive), gnuplot will instead silently bail when it encounters an unparseable field. This is most likely to happen when doing casual work with small files containing only two columns, sometimes leading to mysterious failures. My advice: make it a habit always to specify all columns with using.

[Tip]
Always use an explicit using directive

Specifying all columns to plot explicitly in the using directive will allow gnuplot to parse correctly even files that are partially garbled or that contain garbage.

Time and date information in a data file constitutes a special case. Gnuplot provides special facilities to parse timestamps, which we will discuss in detail in section 8.4.3. Do notemphasis> attempt to parse time and date information yourself—use the appropriate options!

Plain text files as data format

Using text files as primary data store has a number of advantages: we don’t need special tools to generate or read them, and it’s easy to write scripts to pre- or postprocess them. If they are large, it’s easy to compress them. And if necessary, we can even load them into any text editor (or a spreadsheet!) and manipulate them by hand.

Furthermore, text files are largely portable across most current computer architectures (so that files generated on Unix, say, can be opened and read on Windows)—except for the choice of the linebreak indicator (newline character). Gnuplot expects lines in input files to be terminated by whatever the local operating system (or rather, the local C library) considers the “native” newline character. If you encounter problems reading files generated on a different platform, try converting newlines to the local format. (The same argument applies to the files gnuplot writes: lines are terminated with the native newline character on the given system.)

Gnuplot string handling is based around Unicode and UTF-8. We will have more to say about both topics in section 5.1.5 in the next chapter.

4.2  Managing structured data sets

Sometimes data sets have more internal structure than the simple one-record-per-line model we have seen so far. Two cases in particular are reasonably common and must be dealt with: files containing data blocks—that is, different data sets one after another in a single file—and files containing records that span several lines each. We’ll show how to handle each case in turn.

4.2.1  Multiple data sets per file: index

Here is a common scenario: web traffic monitoring across multiple hosts. Some script logs into each host, pulls the log files for the last few days, summarizes the hits per day, and writes the results to an output file before moving on to the next host. We want to use gnuplot to look at the contents of the resulting file.

The important issue here is that the resulting file does not contain individual data points, but entire data sets: one for each host. Each data set spans several rows: one for each day. The file might look something like the one shown in listing 4.2.[10]

Listing 4.2. A data file containing several sets (traffic1)

# Day   Hits
# host=main
0       101
1       100
2       102
3       103

# host=staging
0        25
1        26
2        27
3        28

# host=test
0         5
1         4
2         3
3         2

This data file looks as if several distinct data files (for different hosts) had been appended to one another. Because this situation is sufficiently common, gnuplot provides a way to handle it. But first, we need to look at the meaning of blank lines in a data file.

For gnuplot, blank lines in a data file are significant. A single blank line indicates a discontinuity in the data. The data above and below the blank line will be treated as belonging to the same data set (and be shown using the same line color and line style), but no connecting line will be drawn between the record before and after the blank.

In contrast, double blank lines are used to distinguish data sets in the file. Each set can be addressed in the plot command as if it were in a separate file using the index directive to plot. Table 4.1 summarizes these differences.

The index directive follows after the filename in the plot syntax and takes at least one argument specifying which data set from the file to select. The argument can be either numeric or a string.

Table 4.1. Blank Lines

Blank lines

Meaning

single

Indicates a discontinuity in the data. No connecting line will be drawn between points separated by a single blank line.

double

Indicates different data set (or data blocks) in the same file. Different data blocks can be selected using the index keyword with the plot command.

Selecting data sets by position

A numeric argument is treated as an index into an array, following the C language convention of counting from 0 (zero). Therefore, to plot only the traffic for the staging site, we could use

plot "traffic1" index 1 using 1:2 w linespoints

This picks out only the data set with index 1 and shows it with linespoints.

The index directive can be abbreviated as i. It can take up to three arguments, separated by colons (similar to the syntax familiar from using):

index {int:start}[:{int:end}][:{int:step}]

If only a single argument is given, only the corresponding data set is plotted. If two arguments are present, they’re treated as the index of the first and last data set (inclusive) to be shown: plot "data" index 2:5 will plot four data sets total. A third argument is interpreted as a step size. Accordingly, plot "data" index 2:5:2 will plot only the data in sets 2 and 4. Only the first argument is mandatory.

Selecting data sets by name

Gnuplot 5 offers a new way to pick out a single data set from a file: if it is preceded by a comment line that includes the ID or name of the data set, we can use this identifier in the index directive. The comment character and any leading whitespace are removed; the following characters are treated as the “name” of the data set that follows the comment line in the file.

Referring again to the file shown in listing 4.2, the following two commands are therefore equivalent:

plot "traffic1" index 1 using 1:2 w linespoints
plot "traffic1" index "host=staging" using 1:2 w linespoints

Only a single data set can be selected in this way. If the “name” given is ambiguous, then gnuplot plots the first data set in the file that matches. (Matching is case-sensitive.) Referring again to listing 4.2, the following three commands are both unambiguous and will result in an identical plot:

plot "traffic1" index 2 using 1:2 w linespoints
plot "traffic1" index "host=t" using 1:2 w linespoints
plot "traffic1" index "host=test" using 1:2 w linespoints

In contrast, plot "traffic1" index "host" u 1:2 w lp is ambiguous, and gnuplot would plot the first matching data set in the file (which happens to be the one for the main host); that is, the very first data set in the file).

4.2.2  Records spanning multiple lines: every

Whereas the index directive lets us select consecutive sets of data from a file, the every option, which we discuss now, solves a different problem.

Consider again the same arrangement of hosts as in the previous section, together with an auxiliary script that gathers traffic data from all hosts and dumps it into a single file. But in contrast to the previous scenario, the script now only retrieves a single days worth of traffic and appends it to the output file before moving on to the next host. After a few days, the output file might look like the one in listing 4.3.

Listing 4.3. A data file containing interleaved data sets (traffic2)

# Day     Hits
0         101       # host=main
0          25       # host=staging
0           5       # host=test
1         100
1          26
1           4
2         102
2          27
2           3
3         103
3          28
3           2

Here, each record for a single day really spans three lines: one for each host. (Compare this file against the one in listing 4.2: the data in both is exactly the same!). If we want to plot the traffic for each host separately, we can use the every directive to pick up only the relevant subset of all lines. The following command, for instance, will plot only the traffic for the staging host:

plot "traffic2" every 3::1 using 1:2 with lp

Using the every directive, we can control how we step through individual lines. The syntax looks similar to the syntax used for index, except that individual arguments are separated by two colons. Unfortunately, this similarity is somewhat deceiving, because the order of the arguments isn’t the same for every as it is for index:

every {int:step}[::{int:start}[::{int:end}]]

The first argument is the increment, followed (optionally) by the first and last line number. Line numbers are counted from zero. Don’t forget to use double colons with the every directive: single colons won’t generate an error message, but will lead to strange and hard-to-predict behavior.[11]

We can play other tricks with the every feature. For instance, the following command will skip the first two lines of every data set in the file:

plot "data" every ::2 using 1:2

This is similar to, but not quite the same as the skip facility: skip only discards lines at the beginning of the file, not at the beginning of very data set in the file. Moreover, every only applies to data line, whereas skip also skips comments. Feel free to devise other creative uses for the every keyword!

[Tip]
Prefer index to every

I recommend that you use a file format suitable for the index facility when you need to combine several data sets in a single file. The every keyword is a bit of a legacy feature, and data handling based on index is more robust and transparent.

[10] The data in this data set is dorky. This is intentional. I want to make sure we can distinguish data from the each of the three hosts by looking data (the number of hits) itself. This will be important when we compare this data set with the one in listing 4.3.

[11] I am simplifying here. Gnuplot recognizes an additional concept known as a data block in a file, as a set of consecutive lines delimited from each other using single blank lines. Data blocks are functionally redundant with data sets (delimited by double blank lines). Data blocks can be selected through additional arguments to the every directive, which are placed between the double colons. This is why it’s not illegal to use single colons in this context. If you want to know more about data blocks, check the standard gnuplot reference documentation.

4.3  File format options in detail

Several aspects of the file format can be controlled through options. For instance, we can choose additional characters to indicate comment lines in data files. In this section, we look at the fine points of input file formats: numbers and missing values, comments, and strings.

4.3.1  Number formats

Gnuplot can read both integers and floating point numbers, as well as numbers in scientific notation (strictly speaking scientific E notation): a floating-point number, followed by an upper- or lowercase character 'e', followed by an integer, which is interpreted as a power of 10. The numeric value of such a field is obtained after multiplying the floating-point part by 10 raised to the appropriate power. An example will make this clear: in scientific notation, the value 35,100 would be encoded 3.51e4; the value -0.0001 would be written -1e-4.

We can also allow the letters 'd' or 'q' (both upper- and lowercase) instead of 'e' or 'E' for Fortran D or Q constants, by setting

set datafile fortran

This option is off by default, since it requires additional parsing, and should only be enabled if actually needed.

4.3.2  Comments

Comments can be included in a data file on lines starting with the comment character (#). The line must start with the comment character and is ignored entirely. If gnuplot encounters a # in any location other than the first one in the line, it isn’t interpreted as a comment character, and any text following it is interpreted as additional data. This isn’t a problem as long as only columns preceding it are specified in the using declaration of the plot command.

We can make gnuplot interpret additional characters as comment characters by using the set datafile commentschars command:

set datafile commentschar ["{str:chars}"]

For example, to tell gnuplot that the exclamation point indicates a comment line in a data file, we’d say

set datafile commentschar "!"

The string can contain any number of characters, all of which will be interpreted as a comment character if found at the beginning of a line, which will result in the line being ignored. Resetting this option to a new value overrides all previous settings.

4.3.3  Field separator

By default, fields (columns) are separated from one another by whitespace, which means any number of space or tab characters. We can change the field separator using the set datafile separator command:

set datafile separator [ "{str:char}" | whitespace | tab | comma ]

For example, to make both the colon and the pipe character individually field separators, we’d use

set datafile separator ":|"

To let gnuplot parse comma-separated (CSV) files, either of the following two commands would work:

set datafile separator ","
set datafile separator comma

Separator characters aren’t interpreted as separators when inside quoted strings: quoted strings are always interpreted as the entry of a single column.

It is possible to specify several character, each of which by itself will be interpreted as a field separator. It is not possible to specify a multicharacter pattern as field separator. (This is not true when using whitespace.)

To reset, we can issue set datafile separator whitespace or simply set datafile separator, so that columns will be split on whitespace again.

4.3.4  Missing values

We can use the set datafile missing command to specify the string that will be used in a data file to denote missing data:

set datafile missing ["{str:str}"]

An example would be set datafile missing "NaN", which interprets the IEEE floating-point indicator NaN (“Not-a-Number”) as missing value. There is no default value for this parameter.

Having an indicator for missing values is important when using a whitespace-separated file format: if the missing value were just left blank, gnuplot wouldn’t recognize it as a column value at all, and use the value from the next column instead.

[Tip]
Missing-value indicator

In a whitespace-separated format, you must employ an explicit missing-value indicator. It is not permissible to leave a missing value blank.

The interpretation of missing values in a data set depends on the precise syntax of the using directive. Let’s look at two examples. Listing 4.4 shows a file containing a missing value.

Listing 4.4. A data file containing a missing value (missing)

1    10
2    10
3    11
4    12
5    NaN
6    11
7    11
8    10
9    10

First, let’s assume that datafile missing has been set to "NaN". If we now plot this file using

plot "missing" using 1:2 with linespoints

then the fifth record (the one containing the missing value) will be ignored and the data will be plotted with one continuous, unbroken line. In contrast, if we use the command

plot "missing" using 1:($2) with linespoints

or if the datafile missing option does not equal "NaN" (either because this option is undefined or because it has been set to a different value) then gnuplot will also ignore the fifth record, but will treat it as a blank line and therefore not draw a connecting line across the gap (see figure 4.1).

Figure 4.1. Gnuplot treats missing values differently, depending on the value of the datafile missing option and the plot syntax. The data file is shown in listing 4.4.

4.3.5  Strings in data files

Gnuplot can read and process text fields found in input files. A valid text field can be any string of printable characters not including blank spaces. If the string contains blanks, it must be enclosed in double quotes to prevent gnuplot from interpreting the blanks as column separator. (Single quotes don’t work!) The enclosing double quotes are stripped off and aren’t part of the field’s value. If a field contains whitespace and is protected by enclosing double quotes, it must not contain double quotes as part of the string value. If you need to use quotation marks and blanks in the same string, you must use single quotes inside the string and double quotes to enclose the entire field. If you have designated a non-whitespace character as column separator using set datafile separator (see the previous section), the same considerations apply: strings containing the separator must be protected with double quotes. Listing 4.5 shows some ways that strings can be used in a data file.

[Tip]
Strings in data files

You can include strings in data files. If the strings contain whitespace, they must be enclosed in double quotes. Single quotes won’t work.

Listing 4.5. Strings in data files need only be quoted if they contain whitespace

# Year  Title
1965    Yesterday                  # Bare string without blanks
1966    "Yellow Submarine"         # Blanks require double quotes
1969    "Maxwell's Silver Hammer"  # Sgl quote inside dbl quotes

For more information on string handling, check section 5.1 on string handling in the next chapter.

4.4  Accessing columns and pseudocolumns

In chapters 2 and 3 we have already seen how to access columns in using directives. Here, we will review what we know already and add some new features. Table  4.2 , summarizes all methods of accessing columns in gnuplot.

Table 4.2. Column access methods, pseudocolumns, and column access functions. (An entry in the first column in italics is a placeholder for any matching value. Non-italic entries must be used verbatim.)

Specifier

Example

Description

number

using 1:2

Access a column by its horizontal position in the file.

name

using "Height":"Weight"

Access a column by its name as given in the first non-comment line in the file.

0

using 0:2

Pseudocolumn, containing the record number (starting from zero) in the current data set.

-1

using -1:2

Pseudocolumn, containing the line number (starting from zero). Reset by single blank line.

-2

using -2:2

Pseudocolumn, containing the index (starting from zero) of the current data set. Reset by double blank line.

(expression)

using 1:($2+$3)

Inline transformation. The expression in parentheses is evaluated and the value of the expression is plotted. The column values for the current record are available through the short-hands $1, $2, and so on.

(constant)

using 1:(1)

Same as before, only that the expression is a constant, independent of the values in the data file.

(column(expression))

using 1:(column($3+1))

The argument to the column() function should be an expression that evaluates to an integer. The value of this expression is interpreted as column number, and the column() function returns the value of the current record for the desired column as a numeric value. (For example, column(3) returns the current value of the third column as a numeric value.)

(column(string))

using 1:(column("Weight"))

If the argument to the column() function is a string, it is interpreted as the name of a column (as given in the first non-comment line in the data file) The column() function returns the value of the current record for the desired column as a numeric value.

(stringcolumn(expression)) or (strcol(expression))

using 1:(stringcolumn($3+1))

The argument to the stringcolumn() function should be an expression that evaluates to an integer. The stringcolumn() function returns the value of the current record for the desired column as a string value.

(stringcolumn(string)) or (strcol(string))

using 1:(stringcolumn("ID"))

If the argument to the stringcolumn() function is a string, it is interpreted as the name of a column (as given in the first non-comment line in the data file). The stringcolumn() function returns the value of the current record for the desired column as a string value.

(timecolumn(expression, formatstring))

using 1:(timecolumn(3, "%Y-%m-%d"))

This function parses a column entry as a timestamp and returns the corresponding Unix epoch second as numeric value. The first argument must evaluate to an integer, identifying a column. The second argument specifies a time format, according to tables 8.4 or 8.5.

(valid(expression))

using 1:(valid(2)?$2:$3)

The argument to the valid() function must be an expression that evaluates to an integer (not a string). The valid() function returns 1 if the current record for the identified column is a valid number and zero otherwise.

4.4.1  Accessing columns by position or name

Columns are usually accessed by their position in the file (counting left to right, starting at 1).

plot "data" using 1:2

This is very convenient for files with only a few columns in it, but it can also work well for truly large files, because it is easy to iterate over numeric column specifiers with gnuplot’s new “inline” looping feature. The following command will plot the columns 2 through 24 against the first one in a single plot (more on this kind of looping constructs in section 5.4 in the next chapter; we will treat loops in greater detail in chapter 15):

plot for [j=2:24] "data" u 1:j

We can also specify a column by name if the data files contains a set of column labels in the first row of the file (or the data set, if the file contains multiple data set addressed using index). For example, for the file shown in listing 4.6, the following commands are equivalent:

plot "grains" skip 1 using 1:2
plot "grains" using 1:"Wheat"
plot "grains" using "Year":"Wheat"

Listing 4.6. Column names in a data file (grains)

Year    Wheat   Barley  Rye    # Not a comment line!
1990    8       6       14
1991    10      5       12
1992    10      7       15
1993    11      5       13
1994    9       6       12

Observe that the line containing the labels must not be a comment line. Furthermore, if any of the entries in the using phrase is a string, the entire first line is interpreted as labels and is not included in the plot.

4.4.2  Pseudocolumns

Gnuplot supplies three pseudocolumns for each file it reads. They are numbered 0, -1, and -2.

The pseudocolumn 0 contains the line number in the current file (or data set), starting at zero, and without counting any comment, label, or skipped lines. You can also access this column in inline transformations using $0. (This counter is reset to zero when encountering the double blank line that separates data sets.)

The pseudocolumn -1 contains the line number, starting at zero, and is reset by a single blank line. (This is relevant if the data file is in grid format—see chapter 10 for more details.)

The pseudocolumn -2 contains the index of the current data set within the data file. When a double blank line is encountered in the file, the line number (corresponding to the value of pseudocolumn 0) resets to zero and the index is incremented.

We could use these pseudocolumns, for instance, like this:

plot "data" using 0:1  # Plot first column against line number
plot "data" using 1:-2 # Plot data set index against first column

4.4.3  Column access functions

You can use the column() function whenever an expression has become to complicated for the using syntax, or in contexts where the $ shorthand is not available. The function evaluates its argument into a column identifier and returns the value of the identified column as a numeric value. (There is also a stringcolumn() or strcol() function that returns the column value as a string. See the next chapter for more information on strings. )

The argument to column() (or stringcolumn()) can be either numeric or string. A numeric argument is evaluated into a column number. A string argument is only suitable if the file contains a non-comment header line with column identifiers (as in listing 4.6): if that’s the case, the argument should evaluate into one of the column names; an error results if no column with the supplied identifier can be found.

The column() function and pseudocolumns work together quite well. For instance, in this example, we plot all values from a file, but add a constant vertical offset of 1.5 to values from different data sets (to separate curves from different data sets from each other, so that we can distinguish them more easily):

plot "data" using 1:($2 + 1.5*column(-2)) with lines

The following example (found in the standard reference documentation) plots each data set in a different color (see chapter 9 to learn more about linecolor variable):

plot "data" u 1:2:(column(-2)) linecolor variable

Further column access functions

If a column contains date and time values, then the timecolumn() function may be useful. It takes two arguments: the first must be an integer identifying a column (it can’t be a string!), the second must be a string specifying the format of the timestamp in the data file. The function returns the Unix epoch seconds corresponding to the parsed value as a number. This value may, for example, be formatted into a text label using the strftime() function.

The format specifiers are based on those used by the strftime() and strptime() functions from the C standard library. We will discuss the handling of date and time information in section 8.4.3 in the context of time series plotting; here only a brief preview. Imagine a file containing dates and values like this:

2014-12-01   3
2014-12-02   5
2014-12-05   1

Now the following command will convert the dates in the first column to their corresponding epoch seconds and use them as horizontal positions:

plot "data" u (timecolumn(1, "%Y-%m-%d")):2 w lp

There’s one other function that can be useful for inspecting column values: valid(). It returns true only if the value of the column with column number x is a valid number. You can use this function to test values from messy files, and only plot those that are valid.

4.5  Pseudofiles

Typically, the data rendered by the plot command is read from a file, but there are situations when it makes sense to accept data from a different source. Gnuplot includes a number of “special” file names that you can read from as if they were proper files, although they don’t actually exist on any disk.

As of gnuplot 5, data can be embedded in command files, and accessed through a variable-like identifier called a heredoc. Data can also be read from the command line interactively and stored in a heredoc—this can be useful when we want to add a few data points to an existing graph.

On platforms supporting pipes, gnuplot can read data from standard input—the latter is particularly useful when scripting gnuplot to run in batch-mode. When they are available, it’s also possible to send output to another process via pipes.

Finally, there are two special file names that enable us to apply the full flexibility of the using facility to function plots.

Table 4.3 lists all these special file names and describes their meaning.

Table 4.3. Pseudofiles

Pseudofile

Description

"" (empty filename)

Reuse the most recently encountered file name.

"-"

Read from standard input. In an interactive session, read from the command window. (Terminate input with EOF character or e.)

"+"

Generate samples equally spaced x values, covering the entire current plot range.

"++"

Generate isosamples equally spaced x and y values, covering the entire current plot range. (Two-dimensional equivalent to "+".)

"< ..."

Read via pipe from a subprocess. [a]

"| ..."

Write to subprocess via pipe. [b]

$d

Read from a heredoc. (No quotes around the name.)

[a] Only available on Unix-like platforms.

[b] Only available on Unix-like platforms.

4.5.1  Reading data from standard input

When given the special filename - (as in: plot "-" u 1:2), gnuplot attempts to read data from standard input, which in an interactive session is the command window. Gnuplot will show a prompt at which data can be typed. Finish each line by hitting the return key. Gnuplot will keep prompting for data until either an end-of-file (EOF) character (typically Ctrl-d) is encountered or the character e is entered on a line by itself.

It’s even possible to read data from standard input multiple times within the same plot command: plot '-', '-' will read data until an end-of-file character is encountered and then expect to read more data (for the second “file”) until finding a second EOF character. Of course, the data entered at a prompt this way can have multiple columns, from which we can select some with using, and all the other features of the plot command can be used as well.

Although this feature can be used interactively, it’s mostly intended for situations where gnuplot is used in batch-mode as part of larger scripts (we’ll talk more about that in chapter 15). When used interactively, this feature quickly becomes inconvenient, because (as explained earlier), gnuplot doesn’t maintain data sets in memory and therefore all data has to be manually reentered every single time one wants to plot or replot the graph.

4.5.2  Heredocs

Gnuplot 5 contains a new feature that makes it possible embed data in a command file (as opposed to a data file) or to enter data in the command window and store the data in a variable for the duration of the gnuplot session. This is somewhat of a radical departure, because traditionally gnuplot never maintained data in memory—data was read from file every time the plot command was executed. This new feature is referred to as “heredocs”, because its syntax is modeled after the eponymous feature in Perl and Unix shells.

Defining and using heredocs

It goes like this: you define a heredoc by preceding a variable identifier with a $ (dollar sign), followed by the << redirection operator, and an arbitrary sequence of characters that will mark the end of the data section:

$d << EOD

Now follows the actual data section, just as if it was a data file. The data is terminated by the end-of-data indicator (on a line by itself—the indicator must match exactly). The whole affair may look like listing 4.7.

Listing 4.7. Defining a heredoc.

$d << EOD
1     0.5
2     0.75
3     0.99
EOD

You can now use the heredoc as you would use a file; in particular, you can plot its contents (note that there are no quotes around the heredoc identifier):

plot $d using 1:2

Listing 4.7 could be part of a command file; in this case, it could be loaded (with load). Alternatively, it could be entered interactively during a gnuplot session.

A heredoc, once defined, occupies memory. To release these resources, use the undefine command: undefine $d. Using undefine $* drops all currently existing heredocs.

Discussion

Heredocs are a new and somewhat controversial feature, and some of their aspects may have changed by the time you read this. Heredocs break with several of gnuplot’s long-standing and fundamental design principles. It is important to understand how they came about.

Heredocs arose from a desire to embed data in command files, in order to enable command files that would be “self-contained” and would not require a separate data file. Because traditionally gnuplot command files contain commands that could be issued, without change, at the gnuplot command prompt, they can be issued interactively as well. This gives the impression as if heredocs are akin to session variables, but that was not how they were meant to be used.

Heredocs live in the gnuplot session, but their contents is not included in the information persisted by the save command. In this regard, heredocs behave differently than regular variables (which are persisted, together with their values by the save). To persist the contents of a heredoc, you must use either the set print option together with the print command, or the set table option together with plot (see section 5.3).

Traditionally, gnuplot did not maintain any “memory”: data had to be read from file every time it was used: in that sense, gnuplot was stateless (in regards to data) and thus avoided the problem of “stale” data (if data changed on disk, the next invocation of the plot command would read the new data).

Although heredocs occupy resources (and so maintain a form of state), it is not possible to manipulate them. In particular, you can’t combine to heredocs or perform mathematical operations on them. The only way they can be used is as data source in a plot (or splot) command.

Heredocs introduce the notion of a gnuplot-specific “data set” and thereby break with the principle that data files are plain, unstructured, and generic text files.

Although I do understand the motivation for heredocs, I personally think they are a bad idea, improperly implemented: I am not convinced that mixing commands and data in the same file is a good idea to begin with. But if one chooses to do so, then the “data” portion of the command file should be treated as a file itself—rather than loading its contents into the global session state, and creating the appearance of a structured data structure that is not really there.

4.5.3  Reading data from a subprocess

On platforms supporting input/output redirection, gnuplot can read from a subprocess via a pipe. Let’s say that a.out is a program that prints data suitable for gnuplot to standard output. The following command runs a.out, captures and plots its output, without accessing the file system:

plot "< a.out" using 1:2

The < character must always be the first character of the file name, even when reading from an entire pipeline. For instance, if the output of the first program a.out is piped through a second program b.out, the gnuplot command would be: plot "< a.out | b.out".

You need to watch out for several things when reading from a subprocess like this. First of all, keep in mind that gnuplot does not plot anything until the process has finished, at which point gnuplot plots all of the data received at once. If you want to monitor a live stream with data trickling in over time, you need to use one of the methods from section 17.7. Furthermore, if you attempt to read from the same subprocess twice, for instance like so: plot "< a.out" u 1:2, "" u 1:3, gnuplot actually runs the program twice (and you will have to wait until both instances complete). Repeating a plot program (or using the replot command) will start the subprocesses anew.

For these reasons, it is usually a better idea to write the process output to a file, which can then be plotted at leisure and as often as necessary. Alternatively, you might consider running gnuplot itself as a subprocess and piping data to it—we will come back to this idea in chapter 15.

4.5.4  Writing to a pipe

If pipes are available, gnuplot can also write to them. As gnuplot’s output consists primarily of graphs, this feature is mostly of interest for post-processing image files. A single example shall suffice.

Among gnuplot’s terminals based on the cairo library (see section 14.2) there isn’t one that can generate files in GIF format. The following sequence of commands uses gnuplot’s pngcairo terminal to create a graph in the PNG format, which is then piped through the convert utility from the ImageMagick[12] to create the desired GIF:

set terminal pngcairo
set output "| convert - graph.gif"
plot sin(x)
set output

The pipe character sends gnuplot’s output to a pipe; the dash instructs convert to read from standard input. The final set output is necessary to flush gnuplot’s output channel. (Compare chapter 14 for additional information.)

4.5.5  Generating data

When gnuplot plots a function (as in: plot sin(x)), gnuplot evaluates the function on a set of equally spaced points, which are evenly distributed over the entire plot range. The number of points is controlled by the set samples option (see 3.1), and the plot range is either given (in square brackets) as part of the plot command (see chapter 2) or specified using set xrange (see chapter 8).

Reading from the special file name "+" is like reading from a file that contains exactly those x positions at which the function would be evaluated, in a single column. In other words, the following two commands are equivalent:

plot sin(x)
plot "+" using 1:(sin($1)) w l

In the second command, you must specify the with lines style explicitly, because—as far as gnuplot is concerned—you are plotting data (not a function), and hence the default data style applies (see chapter 9).

The special file "++" is the two-dimensional equivalent to the "+". It generates regularly spaced positions on a two-dimensional grid as used by the splot command. (See chapter 10 for more information about splot.)

You probably won’t need the "+" and "++" pseudofiles very often—they are special-purpose tools for handling certain edge cases. Sections 6.3.4, 8.2.2, and 9.1.5 discuss potential applications.

[12] www.imagemagick.org

4.6  Metadata in data files

Gnuplot can read a variety of (mostly textual) information from a data file—in addition to the actual data points being plotted. We have neither discussed gnuplot’s string handling facilities in detail yet (we will do so in the next chapter), nor have we introduced the context in which the need for metadata arises (we will do so in chapter 7 on decorations that can be placed on a plot and 8 on axes), so this section is mostly a teaser.

Strings can be read from a data file and placed directly onto the plot itself using the with labels style (see section 5.1.4 in the next chapter). With the ticslabels() family of functions (see chapter 8) we can use strings read from file as labels on the axes of a plot (see section 8.3.5 and the examples in chapter 18.) Special rules apply if the data in the file represents calendar dates and times (see the section on time series in 8.4.3 Finally, it’s possible to use the values of the first line in an input file (the “column labels”) as entries into the legend or key of a plot (see section 7.4.5.2 in chapter 7).

Table 4.4 summarizes all the metadata that can be found in a data file.

Table 4.4. Metadata in data files

Section

Description

#

4.3.2

If found at beginning of line: treat the entire line as comment.

plot skip n

4.1.1

Skip the first n lines.

index name

4.2.1

Identify a data set in a file by a unique identifier in the comment line immediately preceding the data block.

Named column

4.4.1

Identify a column by a name, which is found in the first non-comment line of the file.

Key entry explanations

7.4.5.2

Read explanations to be used in graph’s key from the first non-comment line of the file.

with labels

6.3.5.1

Read text labels from data file and place on graph.

xtics()

8.3.5

Read tic labels from data file.

xdata time

8.4.3

In time series mode, read and parse appropriately formatted strings as time stamps.

4.7  Other file formats

In passing, let’s mention some other file formats used by gnuplot that we have not used so far.

It is possible to parse more complicated record formats than the ones we’ve discussed so far by passing a format string, which describes the format of each record, to using. The format string must be compatible with the scanf() family of functions, familiar from the standard C library. Check the standard gnuplot reference documentation if you believe this is of relevance to you, but given the well-known fussiness of scanf(), this is rarely the best path forward. If a file has a format that can’t be parsed normally by gnuplot, it’s usually a better idea to convert it to a gnuplot-compatible format using a small conversion program in Perl, awk, or a similar tool.

Date and time strings are handled in a special way: don’t attempt to parse them using a scanf()-like format string. Use the timecolumn() function (see table 4.2) or the special commands described in section 8.4 in chapter 8 instead to parse and process such data.

[Tip]
Parsing date and time columns

Time and date information constitutes a special case. Use gnuplot’s dedicated facilities for handling such data (see section 8.4.3).

Besides the regular, column-oriented, tabular file format we have seen so far, gnuplot can also handle data on a two-dimensional grid. There are two formats to choose from: one that contains the actual data values together with the coordinates of the grid points, and a packed “matrix” format. We’ll discuss them in chapter 10 on multidimensional plots.

Finally, gnuplot can read certain binary packed file formats. Again, if this is of relevance to you, I suggest the standard gnuplot reference documentation. Unless you have very special needs, I recommend you stick with text files.

4.8  Summary

Gnuplot is all about graphing data, and in this chapter we covered how to get data into gnuplot. You learned about data files and formats. Gnuplot reads data from text files, which are portable, easy to generate, and easy to modify.

You learned how to keep multiple data sets in a single data file and select from those with index, and how to select a subset of records with every. You also learned different ways to identify and access individual columns in a data file.

We discussed pseudofiles, which are data sources that don’t live on the disk, and we also mentioned that gnuplot can read data from another process (if it is installed on a platform that permits it).

Finally, we gave a brief outlook to other information that can be found in a data file and mentioned alternative file formats in addition to the whitespace-separated columns we have concerned ourselves with exclusively until now.

This concludes what we need to know to get started doing data analysis using gnuplot. In the next chapter, we’ll turn away from gnuplot’s core functionality of dealing with data and generating plots, and instead talk about a host of useful features which make our work with gnuplot easier.

5

Practical matters: strings, loops, history

This chapter covers:

Handling strings and textual data

Understanding string operations and macros

Simplifying repetitive tasks with inline loops

Using gnuplot’s internal variables

Accessing gnuplot’s command history

In the previous chapters, we looked at ways to manipulate and visualize data using gnuplot, which is the core task that gnuplot was designed for. Before moving on to more sophisticated application areas, let’s look at some features of gnuplot that don’t directly have much to do with data and visualization, but that are useful to have around.

First we will learn about strings and textual information. Although gnuplot is primarily a tool for plotting data, handling textual labels is nevertheless very important: it’s only through text (included on a graph) that we can tell users what it is they are looking at!

Then we will examine some facilities that that make it easier to work with gnuplot: internal variables allow us to inspect the gnuplot’s internal state and make use this information to adjust options accordingly; and inline loops can help reduce the number of keystrokes required significantly when our commands contain repeated operations (for instance when dealing with a large number of files).

This chapter is a bit of a grab-bag of useful features. What they all have in common is that they help us be more productive with gnuplot, without directly being related to the manipulation and plotting of data. This chapter deals with the “other stuff.”

5.1  Strings

In this section, we’ll first look at the way strings are defined, and then introduce operations that gnuplot can perform on strings, before moving on too some example applications that demonstrate what we just learned. This section concludes with a lengthy digression on Unicode and UTF-8: this may sound boring, but is, in fact, tremendously good news.

5.1.1  Quotes

You can assign a string constant to a gnuplot variable. String constants must be enclosed in quotes, either single or double quotes. The difference is that escaped control characters (such as \n) are interpreted as control characters within double quoted strings, but are treated literally (that is, not interpreted) inside of single quoted strings. (Reminiscent of the behavior of Perl strings, for instance.)

A double-quoted string can contain single quotes; to obtain a double quote inside of a double quoted string, it must be escaped with a preceding backslash. Within single-quoted strings (which don’t recognize the backslash as escape operator), we can get a single quote by doubling it:

a = 'This is a string.'
b = "First Line\nSecond Line."

c = "Double quote\" escaped."
d = 'Single quote'' escaped.'

5.1.2  String operations

Strings can be assigned to variables just as numbers can. Strings are converted to numbers silently, if possible; bur only integers (not floating-point numbers) are promoted to strings:

x = '3.14'     # String
y = 2 + x      # Promotes to number: now y = 5.14

a = 4          # Number (integer!)
b = 'foo' . a  # Promotes to string: now b = foo4

z = 0.0 + x    # Force evaluation as number
c = '' . z     # Force evaluation as string

[Tip]
Automatic conversion to string or number

Depending on context, gnuplot automatically interprets the value of a parameter as number or as string. If necessary, you can coerce evaluation as number by adding 0.0 or as string by concatenating an empty string.

There are three operators acting on strings and a handful of functions. The first of the operators is the concatenation or “dot” operator:

a = 'baz'
b = "foo" . 'bar'
c = b . a          # c is now "foobarbaz"

The other two operators are comparison operators, for use in conditionals: eq (equals) returns true if both of its arguments are equal, and ne (not equals) returns true if they differ.

a = 'foo'
b = 'bar'

c = a eq 'foo' ? 'equal' : 'different'

Strings can be indexed like arrays, using a syntax similar to the syntax used to indicate plot ranges:

a = "Gnuplot"
b = a[2:4]         # b is now "nup"
c = a[4:]          # c is now "plot"

The first character in the string has index 1 (not zero!). Also notice that both limits are inclusive. If we leave either the beginning or the end of the substring empty, it will default to the beginning or the end of the entire string, as shown.

There is a small number of functions to analyze and parse strings (see table 5.1) and we can define our own string functions, in the same way that we define functions operating on numbers:

head( s ) = s[1:3]
a = head( "January" )     # a is now "Jan"

Table 5.1. String functions

Function

Description

strlen( "str" )

Takes a string and returns the number of characters in the string.

substr( "str", i, j )

Takes a string and two integers. Returns the substring beginning at i and ending at j, inclusively. Equivalent to str[i:j].

strstrt( "str", "key" )

Takes two strings. Returns the index of the first character of the string key in the string str, or zero if not found.

words( "str" )

Takes a string. Strips leading and trailing whitespace, then breaks the string into tokens on whitespace. Returns the number of tokens found. Newlines and tabs are interpreted as whitespace only if the string was double-quoted.

word( "str", n )

Takes a string and an integer. Strips leading and trailing whitespace, then breaks the string into tokens on whitespace. Returns the n th token found. (Tokens are counted starting at 1, not at zero.) Newlines and tabs are interpreted as whitespace only if the string was double-quoted.

sprintf( "format", ... )

Takes a format string and an arbitrary number of arguments. Returns a formatted string. Equivalent to the sprintf() function in the C standard library.

gprintf( "format", ... )

Takes a format string and a single argument to format. Returns a formatted string. Similar to sprintf(), but uses gnuplot’s format specifiers (compare table 8.2).

system( "string" )

Takes a shell command as string and executes it in a subshell. Returns the output of the shell command. (More detail in chapter 15.)

5.1.3  String applications

Let’s conclude this section on strings and string handling with a brief look at practical uses for strings within gnuplot.

There are two different ways to place strings onto a graph: either as individually (or manually) placed decorations using the set labels command, or read from a data file and placed automatically by gnuplot. We’ll discuss manual placement of labels and tic marks in chapters 7 and 8 on decorations and axes.

In regards to automatic placement of strings, three different uses stand out. First of all, we can use labels as data and place strings, rather than symbols, onto a plot using the with labels style (see listing 5.1). When using this style, we must supply a third column to the using directive (see section 6.3.5.1 for details). Values in this column will be interpreted as strings and placed onto the plot at the position specified by the values from the first and second column.

Listing 5.1. Reading string labels from file: the with labels style

plot "data" using 1:2:3 with labels

We can apply inline data transformations to the with labels style. Of course, the only possible operations are string operations. The stringcolumn(n) function (see table 4.2) is often useful in this context: it returns the value of column n as a string (whereas the column(n) function returns the value of column n as a number). We’ll show an example for all of this in the next section.

We can also read the labels to be used for tic marks from a file, through the ticlabels() family of functions, which we’ll describe in section 8.3.5, when we discuss axes labeling in general. You can find an example of its use in listing 5.2.

Listing 5.2. Reading the tic labels from file: the xticlabels() function

plot "data" using 1:2:xticlabels(3) with lines

Finally, we can use the first noncomment entry in the data file as label for the data set in the plot’s legend (the key), by giving the column number as argument to the title option of the plot command (see listing 5.3—more details in section 7.4.5.2).

Listing 5.3. Reading key entries from file: title keyword with column number

plot "data" using 1:2 title 2 with lines

5.1.4  Crazy example: plotting the Unix password file

As a crazy example of what is possible, let’s plot a typical Unix password file with gnuplot!

Here is the file (see listing 5.4). (For non-Unix users: each line in the file describes a user. Each line consists of several fields, separated by colons. The first field is the username, the third field is a numeric user ID, and the fifth field is a textual description of the user. The other fields are of no relevance to us here.)

Listing 5.4. A text file that can be plotted by gnuplot

at:x:25:25:Batch jobs daemon:/var/spool/atjobs:/bin/bash
daemon:x:2:2:Daemon:/sbin:/bin/bash
ftp:x:40:49:FTP account:/srv/ftp:/bin/bash
games:x:12:100:Games account:/var/games:/bin/bash
ldap:x:76:70:User for OpenLDAP:/var/lib/ldap:/bin/bash
lp:x:4:7:Printing daemon:/var/spool/lpd:/bin/bash
mail:x:8:12:Mailer daemon:/var/spool/clientmqueue:/bin/false
man:x:13:62:Manual pages viewer:/var/cache/man:/bin/bash
mysql:x:60:108:MySQL database admin:/var/lib/mysql:/bin/false
news:x:9:13:News system:/etc/news:/bin/bash
ntp:x:74:103:NTP daemon:/var/lib/ntp:/bin/false
postfix:x:51:51:Postfix Daemon:/var/spool/postfix:/bin/false
sshd:x:71:65:SSH daemon:/var/lib/sshd:/bin/false
uucp:x:10:14:Unix-to-Unix CoPy system:/etc/uucp:/bin/bash
wwwrun:x:30:8:WWW daemon apache:/var/lib/wwwrun:/bin/false

To plot this file, we need to set the field separator to be the colon (:) and are then able to plot it using the with labels style (see chapter 7).

Just for fun, we also make the letter 'm' the comment character. Verify how the records starting with an 'm' don’t show up in the graph!

In the plot (see listing 5.5; the resulting graph is shown in figure 5.1), we use the numeric user ID as the x coordinate and the line number in the file as the y coordinate. The label, printed at the resulting position, consists of each user’s login name, stacked (by virtue of a newline character) on top of the textual description of the user.

Listing 5.5. Commands for figure 5.1 using data from listing 5.4.

set datafile separator ':'
set datafile commentschar "m"
plot [-20:150][:27] "/etc/passwd"
» u 3:($0+2):( stringcolumn(1) . "\n" . stringcolumn(5) ) w labels

Figure 5.1. Demonstrating string functions and the with labels plot style: plotting the Unix password file with gnuplot.

5.1.5  Digression: Unicode and UTF-8

Gnuplot 5 can handle Unicode strings that are encoded using UTF-8. Although this may sound like the typical configuration and localization nightmare, it is actually really good news—once you understand why.

Unicode

Unicode is based on the concept of a character. A character is a logical abstraction, such as “the first capital letter in the Latin alphabet” (that is: the letter “A”), or the “question mark” character “?”. In Unicode, every character is identified by a unique integer value, known as that character’s code point. Unicode defines over 1 million code points, which are usually specified as U+xxxx, where each x stands for a hexadecimal digit. The capital letter “A”, for instance, can be found at code point U+0041.[13]

Separate from the notion of the character is the concept of the glyph. A glyph is the geometric shape or symbol used to represent a character visually. The same (logical) character can be represented by a different glyph, depending on the typeface or font.[14]

Here comes the first possible stumble: not every font provides a glyph for every possible code point—in fact, most fonts will map only a small subset of characters. That should not be surprising: a calligraphic font, designed to print wedding invitations (in English!) need not include glyphs for mathematical symbols, or for characters from Asian languages. Hence it is essential to use a font that provides glyphs for all the characters you want to display. Fortunately, several open-source fonts exist that provide a wide variety of glyphs that may be useful in scientific or technical contexts. I recommend Nimbus Sans L, which is designed to be similar to Helvetica. A good font manager is very helpful for browsing available fonts and glyphs—if you don’t have a preference yet, you might want to check out Fontmatrix.[15] (We will have more to say about font name resolution in gnuplot in chapter 14.)

It is important to remember that a Unicode code point uniquely identifies a character, and that the only remaining task is to find a font that provides a glyph for that character. All this is very different from the old way of doing things, where the number of characters was limited to 255, and different encodings were used to map these 255 characters to different glyphs, depending on the language or symbols one wanted to use. In the first edition of this book, I explained how to switch from the main text font (such as Helvetica) to the Symbol font to get access to Greek letters and a (limited) choice of mathematical symbols. Although changing fonts is still possible, there is much less need for it today, because it is now possible for a single font to include not only Latin letters, but also a wide range of mathematical and geometric symbols, as well as letters from non-Latin alphabets.

UTF-8

We have not yet explained how a Unicode string is actually represented internally. Fortunately, we (for the most part), don’t need to know.

Clearly, one byte per character is not enough to represent all the 1.1 million possible Unicode characters, and several encoding schemes have been developed to map Unicode characters to “bits and bytes”. The good news is that one encoding came out as the clear winner: UTF-8.[16] The (very clever) details of UTF-8 are of no interest, as long as you stick with UTF-8, always and exclusively. If you present your strings to gnuplot in UTF-8 (and use a font that provides glyphs for all characters in your strings), things will “just work”.

This brings us to the second possible stumble: how to generate arbitrary strings encoded in UTF-8? The most common case (in my experience) is that I know the code point for a character that I would like to use (like the “White Smiling Face” character found at code point U+263A)—but gnuplot is not prepared to handle code points directly. I must translate the code point to a UTF-8 encoded string first.

There are several ways to do that. Some particular combination of keys on your keyboard may generate the desired character in the interactive terminal already. Your favorite editor may allow input of characters by code point or Unicode name:[17] you can then save the string to file and let gnuplot read it, or cut-and-paste the string from the editor window to the gnuplot session (yes, this works). Finally, current programming languages will be able to handle both Unicode and UTF-8. In Python, for example, the following works:

python>>> print u'\u263a'.encode( 'utf-8' )            # Unicode literal
python>>> print unichr( 0x263a ).encode( 'utf-8' )     # Function call

[Tip]
Become familiar with Unicode and UTF-8

All strings in gnuplot (and the libraries it uses) can handle Unicode that is encoded using UTF-8. Together with an appropriate font, this should take much fear out of handling “special characters” in gnuplot.

[13] Tables of Unicode characters and their code points can be found on the Unicode website: http://www.unicode.org/charts/.

[14] Technically, the typeface is the abstract design, whereas the font is an implementation of that design. In the times of hot-lead typesetting that distinction was a lot more important than today: you actually needed to be in physical possession of the metal pieces that made up a “font” to be able to print something.

[15] Although its development seems to have been stagnant for several years, the currently available version works well and is available for a large number of Linux distributions.

[16] It is amusing to note that UTF-8 was designed, quite literally, on the back of a napkin: http://www.cl.cam.ac.uk/~mgk25/ucs/utf-8-history.txt.

[17] In Emacs, for instance, typing C-x 8 RET, followed by the code point in hex or the Unicode name will work—notice that tab-completion of the name is available. You may also want to check out the TeX input method: you enable it using C-x RET C-\, followed by TeX. You can now input Greek letters and other symbols as in LaTeX, for instance typing \lambda will produce the corresponding Greek letter.

5.2  String macros

The string-handling capabilities just introduced make it possible to construct command lines programmatically in a number of ways.

First of all, anywhere a command expects a string, a string expression can be substituted as well. For instance, this will work just fine:

file = "data.txt"        # Assign string "data.txt" to variable
desc = "My Data"         # Assign description to a variable

plot file title desc     # Equivalent to:
                         #     plot "data.txt" title "My Data"

Functions that return a string can also be used. Here, we use a function to provide the file extension for us:

f( x ) = x . '.txt'
plot f( 'data' )      # Plots the contents of "data.txt"

The example is whimsical, of course, but it’s easy to come up with more useful applications. The string functions (see table 5.1) will often be helpful in this context.

When we want to replace a part of a command line where gnuplot doesn’t expect a string, we can’t use a string variable (or string expression) directly. Instead, we must resort to a string macro.[18] String macros let us insert the contents of string variables at arbitrary positions within a gnuplot command. Here’s an example:

cols  = "using 1:3"           # Assign a string variable
style = "with lines 3"        # Assign another one

plot "data" @cols @style      # Equivalent to:
                              #     plot "data" using 1:3 w lines 3

The symbol @ in front of a variable name indicates that the value of the variable should be interpolated into the command line at the current location. The command line is evaluated after all string macros have been expanded.

We can play all kinds of games with string macros. In the previous example, we used it simply to avoid having to repeat the column specification. Obviously, we could do the same thing to define mnemonic names for custom styles. We can even define entirely new commands:

redblue =
» "set pm3d; set palette defined( 0 'blue', 1 'white', 2 'red' )"

Now it suffices to say

@redblue
splot cos(x)*sin(y)

Macros aren’t evaluated within quotes (either single or double), but we can usually achieve the desired effect through simple string manipulations. This example shows both situations:

tool = "(made with gnuplot)"     # String variable

set title "My plot @tool"        # Macro expansion does not work...
set title "My plot " . tool      # but string concatenation is fine!

Only string variables are evaluated when prefixed with an @, but more general string expressions aren’t. The return value from a string function must therefore always be assigned to a variable before it can be evaluated. Using such a two-step process, we can write string functions that are able to process arguments. For example, here we define a function that takes a filename and then prepares a command line consisting of all commands necessary to export a plot to the indicated file as PNG:

export( file ) =
» sprintf("set t push;set t pngcairo;set o '%s';replot;set o;set t pop",
» file );

We’d use it like this:

cmd = export( 'out.png' )    # setup
@cmd                         # evaluate

Macro expansion can provide convenience by allowing us to define and then reuse certain repetitive expressions. We may even want to predefine certain macros so that they’re available whenever we start a gnuplot session. That’s a topic we’ll pick up again in chapter 16.

[18] At the time of this writing, the future of string macros within gnuplot is a bit uncertain. You can run show version long and look for an entry of the form +MACROS to see whether macros are enabled in your gnuplot installation.

5.3  Generating textual output

Gnuplot creates graphs—after all, that’s the whole point! Nevertheless, sometimes it can be useful to have gnuplot create textual output. For example, we may want to export the results from gnuplot’s spline interpolation algorithm to a file, so that we can use them in another application. Or we may have applied some inline data transformation and want to get our hands on the resulting data for some reason.

Gnuplot has two different facilities for generating text: the print command and the set table option.

5.3.1  The print command

The print command evaluates one or more expressions (separated by commas) and prints them to the currently active printing channel—usually the screen:

print sin(1.5*pi)
print "The value of pi is: ", pi

The device to which print will send its output can be changed through the set print option:

set print                              # print to STDERR
set print "-"                          # print to STDOUT
set print "{str:filename}" [ append ]  # print to file
set print $heredoc [append]            # print to heredoc

By default, print sends its output to standard error (usually the current terminal if gnuplot is run interactively). But output can be redirected either to standard output (using the special filename -), or to a regular file (simply by giving the filename as argument to set print). Each invocation of set print creates a new file (clobbering any existing file of the same name), unless the additional keyword append has been specified. The destination file may be a pseudofile (see section 4.5.1): on systems that support input/output redirection, it is possible to specify a shell command as recipient of the output from the print command, by starting the filename with the pipe symbol (|). It is also possible to send the output from print to a heredoc.

Some gnuplot commands that generate textual output use the set print option internally to determine where to send their output (an example is the show palette palette command; check section 11.1.2.4).

5.3.2  The set table option

The set table facility gives us access to the data that makes up a graph as text. In other words, using set table, we can obtain the values of all points shown in a plot as numeric expressions.

The command to generate output as numbers, as opposed to graphics, is the set table command:

set table [ "{str:filename}" | $heredoc ]

As long as set table is active, all output will be directed to the specified file (or to the current value of set output if no file was named). To return gnuplot to creating graphics, use unset table. (Don’t forget!) It is also possible to use a heredoc as destination.

Similar to what we found for the set terminal command, it’s important to remember that set table doesn’t generate any output: it merely selects a channel to which the output will be sent. Only the plot, splot, replot and refresh commands generate actual output. One of them must be issued after set table; otherwise the resulting file will be empty. (In fact, in previous versions of gnuplot, what is now the set table facility was exposed as one of the terminals: set terminal table.) You can use the with table style to suppress any graphics-related processing when doing so:

set table "output.txt"
plot "data" u 1:2 with table
unset table

Both x and y values will be printed to the file as two separate columns. If the plot contains several curves, they’ll be written to the file one after another, as separate data blocks. The data format for each column is equal to the format used for the tic labels of the corresponding axis (see section 8.3.3). The third column contains a marker indicating whether this data point was within the plot range or outside of it: an i for “inside,” an o for “outside”, or u if the point is undefined.

For data from a file, only the data points read from the input file will be written to the output file. For functions, including Bézier and Spline approximations, the number of data points depends on the value of the set samples option (see section 3.1).

5.4  Simplifying work with inline loops

Gnuplot 5 has acquired features commonly associated with programming languages: loops and conditionals. We will investigate them in detail in chapter 15. But the looping facility also has a “short-hand” version that is so simple and useful that we will introduce it here.

This “short-hand” is a form of for-loop that is used to simplify plot or set commands that contain a repeated element. The loop itself is embedded in the plot or set command itself, which is why I call them inline loops. (This terminology is not used in the standard gnuplot reference documentation.) A few examples will make that more clear.

5.4.1  Loops over numbers

Let’s assume that we wish to plot the second, third, and fourth column of a file against the first. We have learned how to do this:

plot "data" u 1:2 w l, "" u 1:3 w l, "" u 1:4 w l

Now, for three columns it is not a big deal to type this out explicitly, but you can see how quickly this will become tedious when we include more columns! This is where the looping construct comes in handy. We can obtain the same plot as in the previous command in the following way:

plot for [j=2:4] "data" u 1:j

The variable j is set to the values 2, 3, and 4 successively, and the data from all three columns is included in the graph. Notice how there are no parentheses around the second column specifier!

The “body” of the loop extends to the first comma, and multiple loops can be combined in the same plot command:

plot for [j=2:4] "data1" u 1:j, "file2" u 1:2
plot for [j=2:4] "data1" u 1:j, for [j=2:4] "data2" u 1:j

The first of these commands plots three columns from data1 and a single column from data2. The second command plots three columns from data1 together with three columns from data2.

The loop condition is specified in square brackets:

for [ variable = start : end : increment ]

where the increment defaults to 1 if it is not included explicitly. A negative increment is possible and will lead to a descending loop. An increment value that is not an integer will be truncated to the next integer. (Truncation occurs in the way C truncates: (int)1.5 = 1, but (int)-1.5=-1.)

Both start and end point of the loop are inclusive. In other words, gnuplot’s for-loop is equivalent to the following construct in a C-like language:

for( var = start; var <= end; var += increment ) {...}

All our examples so far used the plot command, but the same operations are also available for set. For instance, the following command will place a text label at three positions on the graph (we have not yet encountered the set label option, we will remedy this in chapter 7):

set for [j=1:3] label "Here!" at j,0

5.4.2  Loops over strings

In the previous examples, we loop extended over a range of numbers. This is convenient if we want to select columns from a file. But what if we want to loop over a bunch of files? To do that, we need to loop over strings. As it turns out, we can do this, too.

To make this concrete, let’s say that we want to plot the daily temperatures as measured in three cities. The observations are contained in files named according to the locations. Then the following two commands are equivalent:

plot "newyork" u 1:2, "chicago" u 1:2, "sanfranciso" u 1:2
plot for [name in "newyork chicago sanfranciso"] name u 1:2

The loop condition in this case is:

for [ variable in tokens ]

where tokens is a string of whitespace-separated tokens.

String operations are available when using loops. For instance, we can use the string concatenation with the . (dot) operator to build up more complicated filenames:

plot for [name in "newyork chicago sanfranciso"] name . ".txt" u 1:2

The word() and words() functions (see table 5.1) may come in handy. Remember: words(s) returns the number of tokens found in the string s; word(s,k) returns the kth token found. For example, the following command iterates over all files. For each file, it plots the second against the first column, but it also adds a vertical offset:

files = "newyork chicago sanfranciso"
plot for [j=1:words(files)] word(files,j) using 1:($2+5*j)

5.4.3  Summary of inline loops

Inline loops are a new addition and have the potential to change the way we interact we gnuplot. For this reason, I want to summarize their most relevant features.

[Tip]
Inline loops

Become familiar with inline loops. They can help a lot, in particular if you work with many files or with files containing a large number of columns.

Inline loops are available for the plot and the set command. In both cases, they follow directly upon the command keyword. The loop’s body extends to the next comma (for plot) or to the end of the command line. Loops can extend over numeric ranges or over the tokens found in a string. (See listing 5.6.)

Listing 5.6. Inline loop syntax for plot and set commands.

plot for [ variable = start : end : increment ] ...
plot for [ variable in tokenstring ] ...

set for [ variable = start : end : increment ] ...
set for [ variable in tokenstring ] ...

For numeric ranges, the loop includes both start and end point. The increment should be an integer; it is truncated to an integer otherwise. For iteration over strings, the loop condition consists of a string of whitespace-separated tokens.

Finally, I’d like to point out that inline loops nest. The following is legal and works as expected:

set for [j=1:3] for [k=1:3] label sprintf( "%d-%d", j, k) at j,k

5.5  Gnuplot’s internal variables

The frequent use of “internal variables” is a relatively recent trend in gnuplot development. I use the term “internal” variable to refer to variables that are set by gnuplot itself, and that can be read out by the user to investigate various aspects of gnuplot’s internal state. The purpose of these internal variables is twofold:

to make information about gnuplot’s internal state available to user-defined functions and expressions.

to report the result of an internal calculation back to the user.

If you know the name of a specific variable, then you can inspect its value using print and you can use it in expressions just like a user-defined variable (as introduced in section 3.2.3). If you don’t know the variable name, you can list all currently defined variables using

show variables all

To avoid conflicts with user-defined variables the names of internal variables are prefixed by strings in capital letters (see table 5.2). You can restrict the show command to variables beginning with a a certain prefix:

show variables GPVAL

will only show internal variables with names beginning with GPVAL. (Note that there are no quotes around the prefix in the argument to the show command.)

Table 5.2. Prefixes used for internal variables

Prefix

Description

GPVAL_

For general purpose variables (most of which refer to the most recent plot and graph (see section 9.5.4).

GPFUN_

For every user-defined function (see section 3.2.3), the function definition—as a string—is stored in an internal variable prefixed with GPFUN_, followed by the name of the user-defined function.

MOUSE_

For variables containing information about mouse events (see chapter 16).

STATS_

For variables set by the stats command (see chapter 12).

FIT_

For variables used by the fit command (see section 13.4).

An example will help to clarify this. In section 3.5.2.1, we learned that the smooth kdens algorithm can calculate a “default” bandwidth. Upon completion of the plot ... smooth kdens command, the internal variable GPVAL_KDENSITY_BANDWIDTH contains numerical value of the default bandwidth. I also mentioned that this default bandwidth will often appear to broad to reveal all details in the data. We may therefore want to redraw the graph with a bandwidth that is (for instance) half the default bandwidth. We can achieve this effect through the sequence of the following two commands:

plot "data" u 1:(1) s kdens
plot "data" u 1:(1) s kdens bandw 0.5*GPVAL_KDENSITY_BANDWIDTH

It is necessary first to run the plot command without specifying a bandwidth, since it is this operation that creates and populates the internal variable. This leads us to two important properties about internal variables:

Internal variables are only created when needed. Before the first plot command (in a newly started gnuplot session), most internal variables are not just empty—they are not even defined (as show variables all will tell).

Internal variables are read-only. Internal variables inform the user of gnuplot’s internal state—but not the other way around: assigning a new value to such a variable does not change gnuplot’s behavior (even if gnuplot allows the assignment, the new value is silently ignored). Assigning to internal variables is not a way to modify options: you must use the set command to do that.

Not all option values can be inspected through internal variables at the moment, but it is relatively easy to add a new internal variable to the gnuplot implementation (if you are a gnuplot developer, that is), and so you should expect that the information available through internal variables will only grow over time! Use show variables all to see what’s been included by the time you read this.

5.6  Inspecting compile-time options

Whereas internal variables tell us something about the dynamic state of the current gnuplot session, we can inspect compile-time options to learn about the static details of the current gnuplot executable.

The pertinent commands are show version and show version long. The first version will tell you little more than the version number, but the second version will also print the compile-time options to the screen. A - prefix indicates an option that is disabled, a + prefix one that is enabled.

Most of this information is primarily relevant if you are building gnuplot from source yourself (see appendix A) or if you want to report bugs to the gnuplot developers’ mailing list.[19] But even if some feature does not work as expected, it might be worth checking whether your gnuplot executable was built with support for that feature to begin with.

[19] gnuplot-beta@lists.sourceforge.net

5.7  Command history

Gnuplot has a history feature, making it possible to recall the most recent commands. The history feature is bound to the up- and down-arrow keys, as well as to the Ctrl-p and Ctrl-n key combinations: the easiest way to modify an existing plot is to recall the command (using the up-arrow) and to make whatever change is desired to the command line.

The most common application is to regenerate the current graph. In the next section, we discuss some special commands to do that. In section 5.7.2, we discuss gnuplot’s history command, which gives you greater control over the command history.

5.7.1  Redrawing a graph

We have already (in section 2.2.2.4) encountered the replot command. It simply runs the most recent plot command again. We usually use it after making changes to some options using set.

We can add new data or functions to replot:

plot sin(x)     # plots a sine-wave
replot cos(x)   # plots both a sine- and a cosine-wave

The refresh command is like the replot command, except that it does not read the data again. This is convenient if data was read from standard input (using the special filename "-"), or when the file may have changed on disk. It is not possible to add further data or functions using refresh.

[Tip]
The refresh command

The refresh command reruns the most recent plot command (just like replot does), but without having to reread the data from disk.

It is fairly common to load a script using load, and then wanting to make some edits to the plot command. Unfortunately, the load command does not enter the contents of the file into the command history, hence the plot from the file command will not be available using the up-arrow key. Here a new option to the show command can help:

show plot add2history

This will not only print the command line containing the plot command to screen, but it will also enter it as the most recent entry to the command history.

[Tip]
The show plot add2history command

You can enter plot commands read from a file into the command history using the (optional) add2history argument to show plot.

5.7.2  The general history feature

If you want more control over the command history, you can use the history command. The history command can be used in three ways: to print all or parts of the command history, to search it, or to reexecute a command:

history [quiet] [{int:max}] ["{str:filename}" [append]]  # show
history ?"{str:cmd}"                                     # search
history ![ {int:pos} | "{str:cmd}" ]                     # execute

The first line shows how to print the command history to the screen, or (if a filename is given) to a file. Optionally, history clobbers any existing file of the same name, unless the append keyword is used. By default, the command history is shown together with line numbers, but they can be suppressed through the quiet option. Finally, we can restrict the output to the last max entries by providing a numerical argument. For example, the command history quiet 5 will plot the last five commands, as well as the current one, to screen, suppressing line numbers. The current output on my system is shown in listing 5.7.

Listing 5.7. Typical output from the command: history quiet 5

plot cos(x)
plot sin(x)
plot [-2:2] sin(x)
plot [-2:2] sin(x), x
plot [-2:2][-1:1] sin(x), x
history quiet 5

Alternatively, we can give a command, or just its beginning, as argument to history, prefixed by either a question mark (?) or an exclamation point (!). If the question mark is used, gnuplot will search the command history and show only those entries that start with the string provided. If the exclamation point is used, the most recent matching entry in the command history will be executed again. For example, history ?"set term" will show all commands that changed the terminal setting. Finally, the exclamation mark, together with a number will execute the command at the specified history position.

There is also a history option that can modify the behavior of the history command:

set history [ size {int:max} ] [quiet|numbers] [full|trim] [default]

The size keyword takes an integer argument that specified how many entries in the history should be saved to file when a gnuplot session is ended. A value of -1 allows an unlimited number of entries to be written to file.

Specifying set history quiet suppresses numbers generally when using the history command. The trim option removes duplicate entries in the command history, retaining only the most recent one.

The default keyword restores the default settings, which are set history size 500 numbers trim.

5.7.3  Restoring session defaults

At times it may be desirable remove all user-defined variables and options and to restore your gnuplot session to a defined state. This can be accomplished using the reset command. The reset command does not affect options having to do with terminal settings (set terminal and set output), with locale settings (set locale, set encoding, and set decimalsign), and with search paths (set loadpath, set fontpath, and set psdir).

The reset command restores gnuplot to its default settings, period. If you also want to reload your personal initialization file (see chapter 16), then you must use reset session.

[Tip]
Restoring your personal defaults

Use reset session to restore gnuplot’s default state and to load your personal initialization file.

Finally, there are two special forms of this command: reset errors clears only the two internal variables GPVAL_ERRNO and GPVAL_ERRMSG. The command reset bind restores all hotkey bindings (see 16)to their default values.

5.8  Summary

This chapter was somewhat of a grab-bag of useful features that make life easier. Such material often appears dry, so let’s highlight what was important and why:

Strings and string operations are important whenever you want to include textual information on a graph. That’s usually not a concern for initial and exploratory work, but is crucial to provide semantics to finished graphs intended for public presentation. Feel free to come back to this topic whenever the need arises!

String macros and inline loops can simplify repetitive tasks. Inline loops (which are a new feature in gnuplot 5) can be a major convenience. You should at least be aware of this feature.

Internal variables, compile-time options, and the command history all are methods to inspect gnuplot’s internal state. The need for these features usually arises only when you are trying to tackle more complex tasks—for now, it should suffice to know that these possibilities exist.

This concludes our introduction to gnuplot fundamentals. You now have all features and commands at your disposal to work efficiently with gnuplot. In the next part, we’ll take the basics for granted, and instead talk in detail about the different ways that we can affect the appearance of a graph: the “looks.” Stay tuned.

6

A catalog of styles

This chapter covers:

Directives for choosing styles

Gallery of plotting styles

Pointers to additional styles

The next few chapters describe the different ways to control the appearance of a plot: how to make it look just right. In the present chapter, we’ll discuss the various ways to display data, and in the next chapter we’ll talk about all the other stuff that goes onto a plot, such as labels, borders, arrows, and similar decorations. Since axes and their labels can provide so much relevant information about a plot, they’ve been given their own chapter (chapter 8). And finally (in chapter 9) we’ll discuss how you can customize the appearance of all plot elements, in case you don’t like the defaults.

Most of this chapter is taken up by a comprehensive gallery of available plotting styles. Feel free to treat it as a catalog: just look at the pictures to see whether there is anything you like and might want to use right away. As your needs change, you can always come back and check out what other styles are available.

In any case, make sure to read the first section, because here is where I explain the mechanics of how to choose plot styles in gnuplot. The remainder of the chapter is optional (at this point).

6.1  Why use different plot styles?

Different types of data call for different display styles. For instance, it makes sense to plot a smooth function with one continuous line, but to use separate symbols for a sparse data set in which each individual point counts. Experimental data often requires error bars together with the data, whereas counts of discrete events call for histograms. Choosing an appropriate style for the data leads to graphs that are both informative and aesthetically pleasing.

But the choice of plot style is not just an aesthetic issue: different graphical representations give a data set context and may even imply specific semantics:

Continuous lines indicate a function or a dense data set with little noise.

Explicit point symbols emphasize the discrete nature of a sparse data set and make individual data points stand out.

Boxes are mostly used for histograms and so imply counts of discrete events.

Error bars make the uncertainty in a data set explicit.

[Tip]
Use the right style for the job

Remember: the style you choose not only determines what people are seeing, but also how they will interpret it!

6.2  Styles and aspects

Gnuplot can create graphs using different overall styles: we have already encountered the with lines, with points, and with linespoints. In this chapter you will learn about additional styles for drawing boxes, error bars, circles and arrows, and so on. In any case, the style determines the overall appearance of the plot. Styles are usually specified using the familiar with keyword.

All graphs are constructed from lines and point symbols. In addition to the overall choice of style, you can also customize many aspects of lines and points: things like the width, color, and dash pattern of lines, or the shape, size, and color of points. These aspects can be set globally or locally for each data set individually.

Unless you specify aspects explicitly, gnuplot will choose a different set of aspects for each new data set that you include in a graph. For example, if you create a graph from four data sets, by default each data set will be drawn using a different color (or dash pattern) and point shape. You can override this default sequence locally (as part of the plot command) or redefine the overall sequence globally.

In this section, we will first review how to use inline customizations to modify the appearance of graph elements as part of the plot command, and then give an overview over the aspects that can be changed in this way.

6.2.1 Choosing styles inline through with

Generally, plot styles are chosen inline, as part of the plot command. (There are also ways to set global style preferences—we’ll come back to this in chapter 9.) We have already seen inline styles in chapter 2: by giving the with keyword as part of the plot command, you can specify which style to use:

plot "data" u 1:2 with lines, "" u 1:3 with linespoints,
»    "" u 1:4 with points

As usual, keywords can be abbreviated to the shortest unambiguous form, so we’d probably write w l, w linesp (or even: w lp), and so on.

6.2.2  The default sequence

In graphical analysis, you often want to plot several similar data sets, possibly together with functions, all on the same plot—usually so that they can be compared directly to one another:

plot [0:4][-0.25:1.5] "sequence" u 1:2 w lp, "" u 1:3 w lp, "" u 1:4 w lp,
»                     -log(x)+(x-1) w l

All data sets are plotted using the same overall style (with linespoints), but gnuplot makes sure to change the color (or dash pattern) of the lines used. Gnuplot also uses a different point symbol for each data set (see figure 6.1).

Figure 6.1. Gnuplot chooses a different line style for each curve automatically.

Overriding the default sequence

By default, lines and points are chosen automatically from a sequence of available patterns. If there are more data sets in the plot than there are patterns in the sequence, the selection starts again at the beginning.

You can override the automatic selection by specifying the index of one particular element from the default sequence explicitly. For example, if in the previous example you would like to draw the second data set in the same style as the first, you could say:

plot [0:4][-0.25:1.5] "sequence" u 1:2 w lp, "" u 1:3 w lp lt 1,
»                     "" u 1:4 w lp, -log(x)+(x-1) w l

Here, lt is short for linetype (which refers to elements of the default sequence) and lt 1 asks for the first element from the sequence. Fixing a specific line like this doesn’t affect the way the internal counter is incremented, so the way the the fourth column of the data and the function are displayed don’t change.

So far, I only showed you how to select a specific entry from the default sequence. But instead of making such a local change, you can also customize the default sequence globally—or even use different default sequences for different purposes. We’ll discuss how to do that in chapter 9, but that’s pretty advanced stuff. Most of the time, you will find that gnuplot cycles through the default sequence in a way that “just works”.

6.2.3  Customizing graph elements

I already mentioned that you can also exercise detailed control over various aspects of graph elements. Table 6.1 lists all commonly used properties. Many are reasonably self-explanatory, but some may not be. In particular color specifications and customized dash patterns require more detail—we’ll come back to them in chapter 9.

There are several contexts in which the properties from table 6.1 can be specified:

Some properties can be set globally. For example, there is a set pointsize option that can be used to adjust the size of point symbols globally. Others can be configured globally using set style (see chapter 9.

Several terminals allow you to modify the line widths and point size to be used for the selected terminal (see chapter 14).

But by far the most common way to specify appearance aspects is inline as part of the plot command. It is straightforward to do this, and I will explain how to do it next.

Table 6.1. Modifiers for the appearance of lines, points, and other graph elements.

Keyword

Abbreviation

Description

linetype

lt

A line type is a combination of all aspects of both lines and points: line width, color, and dashtype, as well as point type, point size, and point interval. If not instructed otherwise, gnuplot will cycle through a predefined sequence of line types for each new curve in a plot.

linestyle

ls

User-defined combination of appearance aspects. Similar to linetype, but not eligible for automatic assignment to curves. (See section 9.4.)

linewidth

lw

A numerical multiplier, multiplying the default line width.

linecolor

lc

Color used to draw graphical elements (lines and points). (See section 9.1 for the syntax of color specifications.)

dashtype

dt

Pattern of dots and dashes used to draw the line. (See section 9.2.2 for the syntax of dash patterns customizations.)

pointtype

pt

The shape or symbol used to display a data point. Chosen from a selection of available symbols. (See section 9.2.1.)

pointsize

ps

A numerical multiplier, multiplying the default point size.

pointinterval

pi

Controls whether only a subset of records is displayed with an explicitly symbol when using the linespoints style. (See section 6.3.1.)

textcolor

tc

Color used for textual labels. (See section 9.1 for the syntax of color specifications.)

fillstyle

fs

Closed shapes (boxes, circles, ellipses, candlesticks) can be filled with a color or a pattern. The keyword solid, followed by a density between 0.0 and 1.0, fills the shape with the current color at the specified density; pattern uses the current pattern from the range of available patterns; empty leaves the shape unfilled. (See chapter 9.4.4 for more information on fill styles.)

fillcolor

fc

Color used to fill closed shapes. (See section 9.1 for the syntax of color specifications.)

border

bo

Describes the border used for filled areas. Expects any combination of line options as argument. (See section 6.2.3.2 for the lineoptions shorthand.)

arrowstyle

as

Selects a separately defined arrow style when using with vectors. (See section 7.3.2 for more information on arrows and section 9.4.3 for information on arrow styles.)

Inline customizations

To change any appearance property, you simply tack on the desired specifiers to the function or data set in the plot command:

plot sin(x) w l lw 3          # use line width 3
plot "data" u 1:2 w lp lt 2   # inherit all properties from line type 2

If there are multiple curves in a plot, attach the specifiers to desired curve:

plot "data" u 1:2 w lp lt 3, sin(x) w l lt 1

This will plot the data set using the properties of line type 3 and the sine function with the properties of line type 1.

You can include more than one modifier (but gnuplot will prevent you from including the same modifier multiple times):

plot "data" u 1:2 w lp pt 6 ps 2  # point type 6, at point size 2
plot cos(x) w l lw 3 lc rgb "red" # triple line width, in red

Line types and line styles are special: they are containers for all available properties. Each line type (or line style) indicates a line width, color, and dash pattern, as well as a point shape and size. By specifying either a line type or a line style, you choose values for all properties at once. Gnuplot allows you to override individual aspects, and it is quite common to do so:

plot sin(x) w l lt 3 lw 2         # use line type 3, but double the width
plot "data" u 1:2 w p lt 4 pt 3   # use color and point size of lt 4,
                                  # but override the shape of the points

As you can see, if a line type (or line style) has been given, then the curve “inherits” the applicable properties, unless they are explicitly overridden. In the previous example, the data set was plotted with points (that is, without connecting lines), and so the graph inherited point shape, size, and color from the indicated lt 4. But because the point type was selected explicitly, the resulting graph will use color and point size from lt 4 but the point shape corresponding to pt 3.

Shorthand: “line options”

Most styles and decorations that we will introduce in this and the next few chapters are constructed from lines, and the appearance of these lines is usually subject to the appropriate modifiers from table 6.1. In particular, linetype, linestyle, linewidth, linecolor, and dashtype are usually recognized.

To condense the presentation, I will not list these modifiers individually going forward, but instead refer to them as line options. In command summaries, the string lineoptions will henceforth be used as a placeholder for any combination of the aforementioned explicit appearance modifiers. Of course, lineoptions is not a legal gnuplot keyword—it is just a notational shorthand![1]

[Tip]
The lineoptions shorthand

In command summaries, the placeholder lineoptions stands for any combination of the explicit line appearance modifiers listed earlier. This is just a convenient shorthand; lineoptions is not a legal gnuplot keyword!

[1] That being said, as a symbolic placeholder for all available line modifiers, the lineoptions shorthand is more robust and often more accurate than an explicit list that becomes incomplete whenever gnuplot’s selection of line modifiers grows. The standard gnuplot reference documentation contains many instances where explicit lists of line modifiers have not kept up with gnuplot’s increased capabilities and are now incomplete.

6.3  A catalog of plotting styles

There are well over two dozen styles available in gnuplot. Here we look at those most useful for ordinary, two-dimensional data. We’ll provide pointers to additional styles (for more complicated data sets) in section 6.5 towards the end of this chapter.

6.3.1  Core styles: lines and points

There are four styles I would call “core” styles, because they are so generally useful: with points, with lines, with linespoints, and with dots. These styles represent data with simple symbols or lines on the plot (see 6.2).

Figure 6.2. The four core styles: with points, with lines, with linespoints, and with dots

Figure 6.3. Variants of the with linespoints style. Top to bottom: default behavior, using a (regular) character as symbol, omitting every second symbol, and using a character from an extended range of glyphs while omitting two out of every three points. Note how in the last case, the lines have been erased to make room for the characters, by giving a negative argument to the pi keyword.

Points

The points style plots a small symbol for each data point. The symbols aren’t connected to each other. This is the default style for data (see figure 6.2).

The size of the symbol can be changed globally using the set pointsize command. The parameter is a multiplier, defaulting to 1.0:

set pointsize {flt:mult}

It is also possible to change the pointsize (or abbreviated ps) inline:

plot "data" u 1:2 w points pointsize 3

The point type (or point shape) can be selected using pointtype (or pt). Remember that ps stands for pointsize, not for point style (there is no such thing)! Figure 9.6 shows all commonly available point shapes.

Besides the selection of available pointtypes, you can also specify a single letter as argument to pointtype. The selection of symbols available in this way depends, of course, on the fonts installed on your system. Modern fonts that provide glyphs for a large number of Unicode code points may offer some interesting possibilities (see figure 6.3)! Note that when using a character as symbol, its color must be changed using textcolor instead of using linecolor. (Compare section 5.1.5 for information on Unicode and how to access characters from extended sets.)

Lines

The lines style does not plot individual data points, only straight lines connecting adjacent points. This is the default style for functions, and the preferred style for dense data sets without too much noise.

Many aspects of lines, including their width, color, and dash pattern, can be customized. For instance, to double the width of a single line in a plot, we can use the following inline directive:

plot "data" u 1:2 w l linewidth 2

Linespoints

The linespoints (abbreviated lp) style is a combination of the previous two: each data point is marked with a symbol, and adjacent points are connected with straight lines. This style is mostly useful for sparse data sets.

What has been said about with points and with lines styles generally applies to with linespoints: the default size of the symbol is controlled by the set pointsize option, but can be overruled inline, as can the line type and the point type.

A property that is only available for linespoints is the pointinterval (short: pi).[2] If this property is set to a positive integer n, then gnuplot plots a symbol only for every nth data point; all other points are drawn with lines only (see figure 6.3). This can be extremely useful for dense data sets (see chapter 19 for an example).

If pointinterval is negative, then it has the same effect as just explained, but in addition, a circular region centered at the location of the point is cleared, using the special bgnd color before the symbol is drawn (see figure 6.3). (The background color bgnd is a property of the terminal; use set terminal ... background ... to set it. See chapter 14.) The size of the cleared region is controlled using the set pointintervalbox option:

set pointintervalbox {flt:mult}

The value of pointintervalbox multiplies the point size to find the radius of the region to be cleared. All this is demonstrated in figure 6.3.

Dots

The dots style prints a “minimal” dot (a single pixel for bitmap terminals) for each data point. This style is occasionally useful for very large, unsorted data sets (such as large scatter plots). Figure 1.2 in chapter 1 was drawn using dots.

6.3.2  Indicating uncertainty: styles with errorbars or ranges

Sometimes we don’t just want to show a single data point, but also want to indicate some range with it. This may be the experimental uncertainty (the errorbar), or it may be the range over which some quantity has changed during the observation interval (this is typical of financial charts). Gnuplot offers several styles that place an indicator for such a range onto the plot. First we’ll look at styles that draw regular errorbars (both in vertical and in horizontal directions). Then we’ll go on to discuss styles that allow us to indicate several ranges at once (but only in the vertical direction).

Styles with errorbars

There are two basic styles to show data with errorbars in gnuplot: errorbars and errorlines (see figure 6.4). The errorlines style is similar to the linespoints style (a symbol for each data point, adjacent points connected by straight lines), while the errorbars style is similar to the points style (unconnected symbols).

These styles draw errorbars in addition to the actual data. Errorbars can be drawn in either x or y direction, or in both. To select a direction, prefix the style with x, y, or xy respectively, as in plot "data" with xyerrorlines. Table 6.2 summarizes all available combinations.

Table 6.2. All possible combinations of errorbars and errorlines styles

Errorbars in x direction

Errorbars in y direction

Errorbars in both directions

Unconnected symbols

xerrorbars

yerrorbars

xyerrorbars

Connected symbols

xerrorlines

yerrorlines

xyerrorlines

The appearance of both the errorlines and errorbars styles is determined by the current line style. Both styles will draw a symbol at the location of the actual data point—this is unnecessary, because the intersection of the errorbars already indicates this position! You can suppress the symbol by specifying pointtype 0, like so: plot "data" u 1:2:3 with yerrorlines pt 0.

The errorbars themselves are drawn in the current line style. A tic mark is placed at the ends of each errorbar (see figure 6.4). The size of the tic mark can be controlled using the set bars option:

set bars [ small | large | fullwidth | {flt:mult} ]

The parameter is a multiplier, defaulting to 1.0. The symbolic names small and large stand for the values 0.0 and 1.0, respectively. The value fullwidth is only relevant to histogram styles (more on that in a minute). Finally, the tic marks can be turned off entirely using unset bars.

Errorbars require additional information: just the x and y coordinates aren’t enough; you must also provide data about the size of the uncertainties. Usually, this data comes from the data file, in the form of one or two additional columns. If one additional column is given, it’s interpreted as a range dy to be added and subtracted from the corresponding data value, so that the errorbar would be drawn from (x, y-dy) to (x, y+dy) . If two additional columns are supplied, they are interpreted as the absolute coordinates of the lower and upper end of the errorbar (not the ranges), so that errorbars are drawn from (x, ylow) to (x, yhigh) . Corresponding logic applies to errorbars drawn in x direction.

As usual, the columns to use are indicated through the using directive to plot:

plot "data" using 1:2:3 w yerrorbars          # ( x, y, dy )
plot "data" using 1:2:3:4:5:6 w xyerrorbars   # ( x, y, xlow, xhigh,
                                              #         ylow, yhigh )

Data transformations (see section 3.3 in chapter 3) are often useful in this context. Here are some examples:

If the input file contains only the variance (instead of the standard deviation, which is usually plotted as error) together with the data, we can apply the necessary square root inline: plot "data" u 1:2:(sqrt($3)) w yerrorb.

If we know that the uncertainty in the data is a fixed number (such as 0.1 ), we can supply it directly: plot "data" u 1:2:(0.1) w yerrorl.

If the data supplied in the file is of the unsupported form (x, y, ylow, yhigh, dx) , we can build up the required plot command manually: plot "data" u 1:2:($2-$5):($2+$5):3:4 w xyerrorl.

Figure 6.4. Different plot styles showing uncertainty in the data. From top to bottom: connected symbols using errorlines, unconnected symbols using errorbars, ranges indicated as boxes using boxxyerrorbars, and finally errors on top of a histogram using boxerrorbars.

As a final style to visualize data with uncertainty in both directions, there’s the boxxyerrorbars style. It’s similar to the xyerrorbars style, except that the range of uncertainty is shown as a rectangular box centered at the data point, rather than as a cross of errorbars. (If you would prefer to use circles or ellipses instead of boxes to indicate a confidence range you may want to peek ahead to section 6.3.5.2.)

The last style that uses errorbars is boxerrorbars (not to be confused with boxxyerrorbars), which is a combination of the boxes and yerrorbars styles. It is displayed as a box, with a vertical errorbar centered at its top. It might be used, for instance, for histograms that have some uncertainty in their counting statistics. The additional values required for the errorbar are supplied as the third (or third and fourth) arguments to the using directive. In any case, the box width is provided as the last argument to using.

The styles that we’ve discussed in this section are mostly used to plot data stemming from scientific experiments or calculations, where we want to show the uncertainty in the data clearly. But there are other situations, where we want to indicate a range (or even several ranges) together with the data. Those will be the topic of the next section.

Time series styles

Gnuplot offers two styles that are mostly useful for time series data, although they can be used for other purposes as well: the candlesticks style (also known as bar-and-whiskers plot) and the financebars style (see figure 6.5). Both have the ability to show two ranges in a single (the vertical) direction, for instance the typical band of variation and the highest and lowest values ever. Both are frequently used for financial data (such as stock prices) and I’ll discuss them in those terms.

Figure 6.5. Styles for time series: financebars and whiskerbars. Filled bars indicate that the closing value is lower than the opening value for the current record.

Both require five columns of data: the x value, followed (in order) by the opening, low, high, and closing prices. As usual, the appropriate columns are selected through the using directive to the plot command. Additional columns may optionally be specified to indicate the box width or the color.

Both styles represent the maximum range (low to high) by a vertical line. They differ in the way the secondary (opening to closing) range is displayed: in the candlesticks style, a box of finite width is overlaid the vertical line; in the financebars style, tic marks indicate the opening and closing values. The size of the tic marks is controlled by the set bars option familiar from errorbars styles (see figure 6.5).

Details of the candlesticks can be controlled through some additional options. If the current fill style is empty, and the closing value is smaller than the opening one, the box is filled with the current color; otherwise, the box is left empty. (If the a particular fill style has been chosen using set style fill, then this style is used to fill the box, independent of the opening and closing values. See chapter 9 for more information on set style fill.)

The width of the box can either be changed through the set boxwidth option or through the value supplied in a sixth column. If the width is read from a column, it is assumed to be given in the same units as the x coordinate. (If boxwidth is unset, the value of set bars will be used instead, but this usage is deprecated for candlesticks. For financebars, the set bars option is the only way to change the size of the opening and closing markers.)

Tic marks can be placed at the ends of the vertical line by appending the keyword whiskerbars (or whisker) to the plot command. The size of these tic marks can be controlled independently from the box width by appending a numerical value to the whiskerbars keyword. This value is interpreted as a multiplier giving the length of the tic mark relative to the box width.

Finally, you can choose a color for each individual symbol using the linecolor variable style. If this style is used, the last column specified in the using directive is treated as argument to lc var. (See section 9.1.5 for information about linecolor variable.)

A few examples will make this more clear:

plot "data" u 1:2:3:4:5 w candlesticks      # Plain
plot "data" u 1:2:3:4:5 w cand whiskerbars  # Tic marks same length
                                            #    as boxwidth
plot "data" u 1:2:3:4:5 w cand whisker 0.1  # Tic marks one tenth
                                            #    of boxwidth

Neither the financebar nor the candlesticks style connect consecutive entries. If that’s what we want, we’ll have to do so explicitly. Keep in mind that it’s not even clear what should be connected in these styles—they don’t have a concept of a “middle” value. You will have to supply this information separately.

The candlesticks style in particular is quite versatile and can be used to good effect in a variety of situations.

6.3.3  Styles with steps and boxes

Box styles, which draw a box of finite width, are sometimes useful for counting statistics, or for other data sets where the x values cannot take on a continuous spectrum of values.

Steps

Gnuplot offers three styles to generate steplike graphs, consisting only of vertical and horizontal lines (see figure 6.6). The only difference between the three styles is the location of the vertical step:

histeps style centers each bin around the supplied x value.

steps style places the vertical step at the end of the bin.

fsteps style places the vertical step at the front of the bin.

If in doubt, the histeps style is probably the most useful one.

Figure 6.6. The three steps styles. The same data set is shown three times (vertically shifted). Individual data points are represented by symbols; the three steps styles are shown with continuous lines. Note how different the same data set can appear, depending on the exact location of the vertical steps.

Boxes and impulses

In contrast to the step styles from the previous section, the boxes style plots a box centered at the given x coordinate from the x axis (not from the graph border) to the y coordinate (see figure 6.7). The width of the box can be set in one of three ways:

Supplied as third parameter to using.

Set globally through the set boxwidth option.

Otherwise, boxes are sized automatically to touch adjacent boxes.

If a third column is supplied in the using directive, it is interpreted as the total width of the box in the same coordinates that are used for the x axis. The set boxwidth option has the following syntax:

set boxwidth [ {flt:size} ] [ absolute | relative ]

The size parameter can either be a measure of the absolute size of the box in x axis coordinates, or it can denote a fraction of the default box size, which is the width of the box if it touches adjacent boxes. If absolute mode isn’t stated explicitly, relative sizing is assumed. A boxwidth of -2 can be used to force automatic sizing of boxes (with adjacent boxes touching each other). The impulses style is similar to the boxes style with a boxwidth set to zero. The examples in figure 6.7 make this more clear.

Boxes can be filled or shaded, according to the global set style fill option (see section 9.4.4) or the inline fillstyle specification. The fillsteps style a filled equivalent to the steps style.

Figure 6.7. Box and impulse styles. The widths of boxes can be set globally or for each box individually. The second data set uses a fixed width (enclosed in parentheses in the using directive); the third one reads values for variable box widths from file.

6.3.4   Filled styles

Gnuplot has the ability to fill the area between two curves in two-dimensional plots with color or patterns. This is accomplished through the filledcurves style. The appearance of the filled regions is determined by the settings of the fill style, which is controlled by the set style fill option (see section 9.4.4) or given inline.

To fill the area between two curves requires a data set with at least three columns: for the x coordinate, and the y coordinates of the two curves: the area between them will be filled. If the two lines cross each other, we can distinguish the enclosed areas depending on whether the first or the second line is greater than (that is, above) the other. By default, all enclosed areas are shaded, but we can restrict shading to only one of the two kinds of areas by appending either the keyword above or below.

This style is only available when plotting data from a file, but we can “fake” it using the "+" pseudofile (see 4.5). That’s what’s done in the following code snippet (see figure 6.8):

plot [0:3*pi][-1.1:2.1] "+" u 1:(sin($1)):(0.5*sin($1)) w filledc,
                         "" u 1:(1+sin($1)):(1+0.5*sin($1)) w filledc above,
                         "" u 1:(2+sin($1)):(2+0.5*sin($1)) w filledc below

All three versions are shown: the default (with all enclosed areas filled), and—vertically shifted—the same curves, but filling only the areas above and below.

Figure 6.8. Shading the area between two curves in a single data set: plot "data" u 1:2:3 w filledcurves. It is possible to select only those areas where one of the curves is above or below the other curve. (See text for details.)

In addition to filling the areas between two explicitly given curves, gnuplot can also handle other boundaries:

Fill the area between one curve and one straight line (which may be one of the coordinate axes or a plot boundary.

Treat a single curve as a closed polygon and fill its interior.

Specify an additional point that will be included when constructing the polygon.

I suggest you consult the standard gnuplot reference documentation for the pertinent details.[3]

6.3.5  Beyond lines and points: multivariate visualization

Several styles allow us to encode information by other means than the position on the graph: with labels lets us read a test string from the input file and place it on the plot, and with pointsize variable we can change the symbol size according to the values in the data set. Finally, we can place arrows or entire geometric shapes on to the plot using the with arrows, with circles, and with ellipses styles.

All these styles have in common that they consist of graphical elements that are not simple, but that instead have internal degrees of freedom—that’s what enables them to encode additional information, beyond their position on the graph. (We will come back to multivariate visualization in section 18.4.) At the same time, by being more than “lines and points”, the distinction between these styles and some of the decorations that will be introduced in the next chapter is becoming a bit fuzzy—and, in fact, these styles share many implementation details with the corresponding decorations. Frequently, what applies to one, applies to the other, even if it does not say so explicitly in the text!

[Tip]
Multivariate styles and decorations

The with labels, with arrows, with circles, and with ellipses styles share many properties with the set label, set arrow, and set object decorations.

Labels and point size

Let’s look at an example that demonstrates both the with labels style and the pointsize variable facility. Listing 6.1 shows a short data file containing the additional information we’ll use for labels and symbol sizes as additional columns. Given this file, we can generate the plot in figure 6.9 using the following commands:

plot [0:6][0:3.5] "labels" u 1:2:3 w p pt 6 ps var,
»                 "" u ($1+0.25):($2-.25):4 w labels

Both styles require a third column as part of the using declaration, the contents of which are interpreted as labels or desired symbol sizes, respectively. Variable symbol sizes are most easily recognized if the symbols are circles, which by default are chosen by pointtype 6 (or pt 6 for short). Then follows the pointsize variable (abbreviated ps var) specification. Labels are chosen using with labels, and I make sure that all labels are offset a little down and to the right, so that they don’t overlap with any of the circles.

Listing 6.1. Data for figures 6.9 and 6.10 (labels)

# x     y       size    label
1       2.6     3       ABC
2       2.1     6       EFG
3       1.0     2       PQR
4       1.2     1       UVW
5       1.6     4       XYZ

Figure 6.9. Encoding additional information through symbol size or the use of textual labels: pointsize variable and with labels. The corresponding data file is shown in listing 6.1.

You should exercise some caution when using pointsize variable: it is often difficult for the observer to judge the size of symbols accurately. Moreover, it’s not necessarily clear to the observer whether the radius or the area of the symbol is proportional to the encoded quantity. I’ll have more to say about visual perception in chapter 19.

By contrast, the with labels style is quite generally useful and very versatile. As mentioned earlier, many of the of the options from set label carry over to with labels. See section 7.3.3 for details, here only a few remarks:

The label can be rotated using rotate by.

The text font and its size can be changed using font "...".

The text color can be selected with textcolor (or tc). Data-dependent text coloring is available through tc var (see section 9.1.5 for data-dependent color selection).

A point symbol can be drawn using point, and the text label shifted relative to it using offset.

If the hypertext directive is included, the text label is usually hidden and is only displayed when the hovering with the mouse near the location of the point. (Obviously, this is only relevant for interactive terminals.)

Figure 6.10 shows some of these options in action—the data is again the one shown in listing 6.1. The plot command is:

plot [0:6][0:3.5] "labels"
»                 u 1:2:(sprintf("{/=%d %s}",4*column(3), strcol(4))):1
»                 w labels rotate by 30 point pt 5 offset 1,-1 tc var

Mind you, I am not saying that this is an example for a successful visualization! But it does demonstrate many of the concepts just introduced. (The variable text size makes use of enhanced text mode—we will formally introduce this concept in section 14.1.)

Figure 6.10. Further capabilities of the with labels style: plotting a point symbol together with the text label, and changing color, size, and orientation of the label itself.

Finally, a last word of warning: in contrast to what’s customary elsewhere, the with labels style directive cannot be abbreviated: neither with lab, nor with label will work!

Shapes and Arrows

Gnuplot defines three shapes that can be used as plotting symbols: arrows, circles, and ellipses. What’s different about these plot elements is that they have a non-trivial internal structure that can be used to encode information: the arrow does not only have a position (as does any other point symbol), but it also has a direction and a length. An ellipse has a variable shape and orientation, and circles are interesting, because gnuplot allows us to also draw arc segments, not only full circles. (As I pointed out earlier, these styles share many properties with set arrow and set object—see sections 7.3.2 and 7.3.4 for additional details).

As example, consider the short data file shown in listing 6.2. We will use the first two columns as x and y coordinates, and will display the data in the third and fourth columns as size and angle, respectively. (Compare figure 6.11.)

Listing 6.2. Data for figure 6.11 and listing 6.3 (shapes)

# x     y       size    angle
1       1       0.1     0
2       1.2     0.2     15
3       1.8     0.3     45
4       1.7     0.2     90
5       1.6     0.15    160
6       1.5     0.2     250
7       1.4     0.3     300

Figure 6.11. The data from listing 6.2, plotted using with vectors, with circles, and with ellipses. See listing 6.3 for details of the plot command.

Arrows are drawn using the with vectors style. They require four columns in the using directive: x and y position, followed by the offset of the arrow’s tip from it’s starting position. The way the arrow is drawn can be controlled, either by giving explicit style details inline (see section 7.3.2 to learn how to control an arrow’s appearance), or by referencing a separately defined arrow style (see section 9.4.3 for globally defined styles). With arrowstyle variable, it is also possible to read the index of a predefined arrow style from an additional column.

plot ... using 1:2:3:4 with vectors nohead                 # inline
plot ... using 1:2:3:4 with vectors arrowstyle 3           # global style
plot ... using 1:2:3:4:5 with vectors arrowstyle variable  # from file

Circles only requires two columns, for the x and y positions. If a third column is specified, it’s used for the radius, otherwise, the radius is taken from the global set style circle setting. (The radius is always measured in x coordinates.) Fourth and fifth columns, if present, measure the beginning and ending angle of the drawn arc segment (the “pie slice”). Both are measured in degrees against the positive x axis. The circle (or circle segment) may be filled, using either an inline fillstyle specification or the global set style fill setting. Finally, the color of the entire shape may be read from an additional column, using linecolor variable. One thing to keep in mind is that the resulting shape is always a circle, independent of the selected y range. If you want to control the ratio of the horizontal to the vertical axis, you must use with ellipses.

Ellipses require four columns, for the x and y coordinates of the center, and for the length of the major and minor diameters. A fifth column may be used to give the angle of the ellipse with the positive x axis. Fill and color properties are the same as for circles. (There are some fineries about the units in which the diameters of an ellipse are measured. Check the standard reference manual for details.)

Let’s now discuss the commands used to create figure 6.11. The commands are shown in listing 6.3. This listing is worth studying in detail, because it demonstrates how judiciously employed inline transformations can be used to adapt plot styles to a given purpose.

Because the various shape-styles expect angles in degrees (not radians), we first make sure that trigonometric functions will interpret their arguments accordingly, using the set angle option (for more information, see chapter 13). We then plot the data set several times, shifted vertically. A few notes should suffice: we use basic trigonometric relations to calculate the offsets of the arrow’s tip from the angle and the radius. In the first row of complete circles, we ignore the angle argument in the fourth column; but for both the incomplete circles and the ellipses, we instead ignore the y coordinate and plot all shapes at the same vertical elevation.

Listing 6.3. Commands for figure 6.11 using data from listing 6.2.

set angle degrees

plot "shapes" u 1:($2+2):(2*$3*cos($4)):(2*$3*sin($4)) w vectors,
     "" u 1:($2+1):($3) w circles,
     "" u 1:(1):($3):(0):4 w circles,
     "" u 1:(0):(0.5):($3):4 w ellipses

[2] This feature is new in gnuplot 5.

[3] At the time of this writing, some of these additional features did not seem to work—check the release notes for up-to-date details.

6.4  Putting it together

In this section, we will put together everything we have learned in this chapter to create figure 6.12. Imagine that you have been given a set of daily weather observations (the beginning of the file is shown in listing 6.4). We can see the day of the month, followed by the temperature (in Celsius) and the dew point,[4] cloud cover (in percent) and visibility (in miles—10 miles meaning unrestricted visibility). The last columns give the wind speed (in knots) and direction, the air pressure (in millibar) and its short-term trend (up or down). That’s a lot of data, and we’d like to present it both compactly and intuitively.

Listing 6.4. Beginning of meteorological data for figure 6.12. See listing 6.5. (weather)

# Day    Temp   Dew     Cloud   Visib   Wind    Direct  Press    Trend
1        14.3   14      20      7       0       0       1007.6   dn
2        14.3   14      80      2.5     7       0       1006.4   up
3        13.8   14      70      10      6       150     1006.4   dn
4        14.3   13.9    50      10      8       190     1009.6   --
5        14.3   13.9    10      10      6       170     1012     dn
6        12.4   11.8    80      1.75    6       210     1008.2   up
7        12.4   11.5    90      2.5     3       190     1005.8   dn
...

Figure 6.12. All-in-one visualization of the data set in listing 6.4. Temperature and dew point in red, atmospheric pressure and its trend with symbols, visibility with grey bars. Wind speed and direction across the top, cloud cover along the bottom. See listing 6.5 for the commands.

In figure 6.12, temperature and dew point are shown using continuous lines, pressure is shown using unconnected symbols (in millibars above 1000), and the symbols itself indicate the most recently observed tendency: light triangles pointing upwards for rising tendency, and dark, downwards triangles for falling. Visibility, or rather: restrictions to visibility are shown as grey bars in the background—the shorter the remaining white area at the bottom of the graph, the shorter the visible range. Finally, wind is shown with vectors along the top, and cloud cover using circles along the bottom. The complete commands are given in listing 6.5.

Listing 6.5. Commands for figure 6.12 using data from listing 6.4.

set angle degrees
unset key

set bar 0          # disable whiskers on the errorbars
set boxwidth .7    # fix width of candlesticks

sym(s) = s eq 'up' ? "△" : (s eq 'dn' ? "▼" : "◎")

plot [0:32][0:18]
"weather" u 1:($5+2):($5+2):(10+2):(10+2) w cand lc rgb "grey" fs solid,
"" u 1:2:3:2 w yerrorlines pt 0 lt 1,
"" u 1:(1):(0.4):(90-0.01*$4*359):(90) w circ lc rgb "blue" fs solid,
"" u 1:(1):(0.4) w circ lc rgb "black",
2 lc rgb 'dark-grey', 10+2 lc rgb 'dark-grey',
"" u 1:(16) w p lt 2 pt 7 ps .75,
"" u 1:(16):(.05*$6*cos(90-$7)):(.05*$6*sin(90-$7)) w vectors lw 2 lt 2 nohead,
"" u 1:($8-1000):(sym(stringcolumn(9))) w labels

This set of commands uses many of the concepts we have introduced previously. The plot command in this example is uncommonly complicated (including nine rather different quantities!). Here are a few pointers as you step through it:

I use the candlestick style to draw the grey bars that indicate the visible range. This style is convenient, because it draws a box, all dimensions of which I can control explicitly. The whiskers have zero length.

Gnuplot draws items onto the graph in the order in which they appear in the plot command. For this reason, it is important that the visibility comes first: the grey bars should be in the background and not obscure any of the other plot elements.

The yerrorlines style comes in handy to draw the temperature and the dew point.

Next come the circle segments for the cloud cover. There is a little math to account for the fact that gnuplot measures angles counter-clockwise from the horizontal axis, but we expect angles to increase clockwise from the “twelve o’clock” position. The filled pie slices are drawn first, the empty black circles that provide a “rim” for each symbol are drawn afterwards, so that they are visually in front.

Next follow two horizontal lines that visually divide the graph. This part of the plot command could also have come earlier or later.

Next come the wind speed and direction indicators. First, a circular symbol (pointtype 7) is drawn to indicate the origin of each vector—this is important if the wind speed is so low that the vector itself has zero length. Then follows the vector, again adjusting the angles to be measured clockwise from the top, rather than counter-clockwise from the right. I suppress the arrow head using nohead: because the origin of each arrow is already clearly marked, there is no need to clutter the graph by drawing explicit heads.

Finally, the symbols are drawn for the pressure. Here, I make use of the ability to use a single character as plotting symbol. Moreover, I use some symbols from the “geometric shapes” section of the Unicode standard: the light upwards triangle can be found at code point U+25b3, the filled downwards triangle at U+25bc, and the “bullseye” character has code point U+25cd. The latter has been chosen to be as neutral as possible, compared to the other two symbols (no angles, and of intermediate visual weight, due to the inner circle on an unfilled background).

The sym(s) function consists of two nested conditional operators, and converts the strings used in the data file to the characters used in the plot. It is necessary to use the stringcolumn(i) function here—the $9 short-cut is not available in this evaluation context. Remember, also, that stringcolumn(i) takes a numeric argument: the position of the column from which a value should be taken. (See section 4.4 for more information about stringcolumn(i) and section 5.1.5 for more information about Unicode.)

Figure 6.12 is certainly impressive, and so are the details of listing 6.5—but remember that most graphs (and commands) are much, much simpler!

More important is the question whether figure 6.12 actually succeeds in its visualization task. Here are some things to think about:

The graph packs a lot of information into a relatively compact space. The middle section is shared by three totally different quantities, but because they are represented using different visual cues (lines, symbols, and filled regions), their interference is reduced. The amount of information encoded in the different symbols used to represent pressure readings seems too high: I find it difficult to get an intuitive sense of the behavior of the air pressure from this graph.

The dew point is represented particularly well, I think. The absolute value of the dew point is less relevant than its distance to the actual temperature, and it is precisely this distance that is emphasized by the vertical error lines. The continuous lines used for the ambient temperature highlights the actual values.

For the visibility data, I wanted to stress the reduction in visibility, represented by the grey bars intruding from above, possibly leaving only a small, unobscured “visible region” at the bottom of the graph. Although good in concept, I don’t think this idea works well in practice: simply drawing the visual range with lines would have been clearer.

Both the wind and the cloud cover visualizations represent their underlying data “intuitively”: we can, literally, see “which way the wind blows”. It would have been weird to use with lines to indicate the wind direction!

Nevertheless, the visualization of the wind speed and direction has a couple of problems: although we have a relative sense for the wind speed, there is no indication of its numerical range—if you are trying to land a plane, that matters a lot! And if the wind speed is high and the direction variable, the arrows will have a tendency to obscure each other. (That this is not an issue here is solely a consequence of this particular data set.)

One last problem: although we have good numerical tic marks for the temperature and dew point values, things are less clear for the visibility and pressure readings. A single axis is not enough to represent quantities which have such totally different numerical ranges.

[4] The dew point is the temperature at which water vapor in the air begins to condense. The dew point is always lower than or equal to the actual temperature, and the closeness of those two temperatures is a measure for the humidity in the air.

6.5  Other styles

The styles we discussed in this chapter are all very general: they can be used to represent all sorts of data, and in a variety of situations. Besides these “general purpose” styles, gnuplot offers further possibilities, for special applications:

In chapter 12 we will see boxplots (to represent point distributions), histograms (for counting statistics), and parallel coordinate plots (a particular kind of multivariate technique).

In chapters 10 and 11, we will introduce surface and contour plots, as well as false-color plots and heatmaps. All of these methods apply to cases where we would like to represent how one dependent quantity depends on two independent variables.

In chapter 13 we’ll also encounter some special plot types for non-cartesian coordinates.

6.6  Summary

This chapter was a bit of an illustrated catalog: a place where all of gnuplot’s major plot styles are demonstrated by way of an example. As such, I’m not disappointed if you didn’t read the chapter from start to finish: it’s perfectly fine to just look at the figures until you find one that does what you need, and then read the accompanying section.

The only section of this chapter that you should have read, even on a first pass, is section 6.2, because it explains how styles are chosen and how you can customize them to meet your needs. This is fundamental material that you will need basically every time you use gnuplot.

Everything we did in this chapter is there to visualize data. By contrast, in the next chapter (chapter 7), we look at things we may want to put on a plot in addition to the data: labels, arrows, and other decorations.

7

Decorations

This chapter covers:

Understanding layers and locations of a graph

Adding arrows, labels, and other decorations

Providing explanations using a key

Changing the overall appearance

Data alone doesn’t tell a story. To be useful, the data needs to be placed into context: at least, we must tell the observer what the data is (such as position versus time, particle count versus scattering angle, stock price versus date, or whatever) and what units the data is plotted in (centimeters or inches, seconds or minutes, dollars or euros). No plot is complete without this information.

But we can do much more to make a graph useful and informative: we can add arrows and annotations on the graph to point out and explain interesting features. We may also want to provide special tic marks and labels to make quantitative information stand out more. Or we may need to change the overall size and shape of the entire graph to accommodate a specific data set.

In this chapter, we discuss all the means that gnuplot offers to put additional information on a plot (in addition to the actual data). Because much of this material is quite dry, I’ve gathered the most important commands and options together in the next section. Unless you have special needs, this may be all you need to know right now—gnuplot is quite good at automatically “doing the right thing” in most situations.

7.1  Quick start: minimal context for data

For the sake of concreteness, let’s go back to the DLA (diffusion limited aggregation) example of section 1.1.2 and in particular figure 1.4. We’ll use this plot as our example throughout this section.

The absolute quickest way to add the most important contextual information to the plot is to give it a title, such as

set title "Run Time (in seconds)
» vs. Cluster Size (in thousands of particles)"

This tells the observer both what is plotted (computation time versus system size) as well as the relevant units (time is given in seconds, while particles are counted by the thousands), conveniently packaged into a single command. This is the bare minimum.

The title is centered at the top of the graph (as in figure 7.1—you may want to go back to section 1.1.2 to learn about the context in which this example arose).

With a little additional effort we can put labels on the axes, using the set xlabel and set ylabel commands. This frees up the title for contextual information about the data in the plot, like so:

set xlabel "Cluster Size [thousands]"
set ylabel "Running Time [sec]"
set title "DLA Growth Program: Run Time
» vs Cluster Size (Double Logarithmic)"

Finally, we have the key (or legend), which relates line types to data sets. By default, it’s placed in the top-right corner of the graph, but this may not be suitable if it would interfere with data. We can change the position of the key using the keywords

left, right, top, bottom, center

in the set key command, for example, set key right center, to place the key vertically centered on the right side of the graph. The order of the keywords doesn’t matter, and if only one is given, all other options retain their values. Finally, the key can be suppressed entirely using unset key.

Figure 7.1. Providing minimal context to a plot using a title and axes labels. See listing 7.1.

We have already seen (in section 2.1.2) how we can change the string describing the data set in the key using the title option to the plot command. To suppress an entry in the key, we can either use the notitle option or provide an empty title string. In the following command, only the actual data curve will have an entry in the key, but the fitted curve won’t:

plot [0.5:40][.5:30000] "dla-log-short" u 2:6 title "data" w p,
» 1.2*x**2.7 notitle

Now we can put all of this together. Listing 7.1 summarizes all commands that were used to build figure 7.1.

Listing 7.1. Commands for figure 7.1

set xlabel "Cluster Size [thousands]"
set ylabel "Running Time [sec]"
set title "DLA Growth Program: Run Time
» vs Cluster Size (Double Logarithmic)"

set key top left
set logscale

plot [0.5:40][.5:30000] "dla-log-short" u 2:6 title "data" w p,
» 1.2*x**2.7 notitle

This concludes our quick start, and may be all you need for now. In the rest of this chapter, we’ll discuss all the miscellaneous options that are available in case we aren’t happy with the defaults. But first I need to describe the different ways you can refer to a location on the plot.

[Tip]
Always provide context for your plots

You can use set xlabel and set ylabel to tell the viewer what quantities are plotted along the horizontal and vertical axes. The set title command is an easy way to add general context to a plot. And don’t forget the plot ... title "..." facility to add a meaningful string to the plot’s legend.

7.2  Digression: layers and locations

Before we can add decorations to a plot, we first must know how to tell gnuplot where on the graph the new element should be placed. There are two different concepts involved here: in the next section, we’ll see how to specify positions laterally within the plane of the graph. Thereafter, we’ll learn about the stacking order that enables us to place one elements visually into the foreground or background, relative to each other.

7.2.1  Locations

First, let’s establish some terminology. The entire area of the plot is referred to as the screen or the canvas. On the canvas there is the actual graph, which is surrounded by a border (unless we explicitly turn the border off). The region outside the border is called the margin. All these are indicated in the figure 7.2.

Figure 7.2. The parts of a gnuplot graph: canvas, border, margin

We can provide up to two different sets of axes to a plot. This is occasionally useful when comparing different data sets side by side: each data set can be presented with its own coordinate system in a single graph. The primary coordinate system (named first) is plotted along the bottom and left borders. The secondary coordinate system (second) is plotted along the top and right borders. By default, the secondary system not shown, instead the primary system is displayed on all four borders. (You’ll find more information on axes and coordinate systems in chapter 8.)

Now that we know what all parts of a graph are called, we can talk about the different ways to specify locations. Gnuplot uses five different coordinate systems:

first, second, graph, screen, character

The first two refer to the coordinates of the plot itself. The third and fourth (graph and screen) refer to the actual graph area or the entire canvas, respectively, placing the origin (0,0) at the bottom-left corner and the point (1,1) in the top-right corner (see figure 7.2). Finally, the character system gives positions in character widths and heights from the origin (0,0) of the screen area. Obviously, positions in this last coordinate system depend on the font size of the default font for the current terminal.

Coordinates are given as pairs of numbers separated by a comma. As necessary, each number in this pair can be preceded by one of the five coordinate specifiers. The default is first, and if no coordinate system is given explicitly for y, the one for x is used for both values.[1]

We’ll see many examples of coordinate specifications in the rest of this chapter.

7.2.2  Layers

Gnuplot draws graph elements onto the canvas in a well-defined sequence. We can leverage this stacking order to place some elements visually “in front of” or “behind” other elements.

Plot elements are organized into layers (see figure 7.3). Graph elements in a layer that is closer to the front obscure elements in layers further back.

Figure 7.3. Stacking order of layers, front to back. Within each layer, objects are drawn first, followed by labels and arrows. Curves are drawn in the order in which they appear in the plot command.

Labels, arrows, and geometric shapes (“objects”) are organized in two layers, one of which is (visually) behind the plotted data, the other one is on the very top of the visual stack. When creating a decoration, we can assign it to either of these two layers using the keywords front and back; the default is back. Within each layer, arrows are visually in front of labels, and geometric shapes are even further behind. As you will learn in the next section, each arrow (or label or object) has a numeric tag and this tag is used to sort elements of the same type: arrows with lower tag numbers are drawn first, and all arrows are drawn before any text label is drawn.

There is a special layer, identified using the keyword behind, which resides at the very bottom of the visual stack. Only objects (but no arrows or labels) can be added to this layer. This layer can be used to provide a visual background (such as a solid color or possibly a color gradient) to the entire graph.

Data and functions finally are plotted in the order in which they appear in the plot command, left to right. This means that elements specified later in the plot command will appear in front of those that were given earlier.

Whereas the locations within the plane of the graph are essential when working with decorations, the stacking order only becomes relevant for complicated graphs, consisting of many plot elements. For most day-to-day operations, you probably won’t need to concern yourself with it. But you have control over it, for those special situations when you need it!

[1] Although it may seem weird to use different coordinate systems in the same coordinate pair, this feature is occasionally very useful. For example, we can use the following command to place a vertical line at a fixed x location and let it extend over the full range of the graph, independent of the y range: set arrow from 1.25, graph 0 to 1.25, graph 1.

7.3  Additional graph elements: decorations

I use the term decorations for all additional graphical elements that can be placed onto the graph, but do not (primarily) represent data. The most important decorations are text labels, closely followed by arrows, but gnuplot can also draw rectangles, circles, ellipses, and polygons.

7.3.1  Common conventions

Labels, arrows, and objects share some conventions that apply to all of them equally. I’ll summarize them here, so that we don’t need to repeat this information for each of the individual styles again and again.

Creating decorations

All decorations are created using the set ... command (see chapter 5). It’s very important to remember that this command does not generate a replot event: the decorations won’t appear on the plot until the next plot, splot, replot, or refresh command has been issued!

Keep in mind that decorations aren’t taken into account by the autoscale feature, which automatically attempts to adjust the plot ranges in such a way as to display the relevant parts of the data. In other words, if a decoration (or a part of it) falls outside the plot range (either the explicitly given or the automatically selected one), it will not appear on the plot. (More on autoscaling in section 8.2.)

[Tip]
Creating a decoration does not generate a plot event

Decorations created with set ... won’t appear on the graph until the entire plot is redrawn through a plot, replot or equivalent command.

Identifying decorations with tags

So that we can later refer to a specific object (such as an arrow or a text label), we can give each object a numeric tag, for instance set arrow 3 .... Now we can make changes to this arrow only by providing this tag in the next call to set arrow or even eliminate it using unset arrow 3. If we omit the label, gnuplot will assign the next unused integer automatically. Arrows, labels, and objects have separate counters.

Because decorations are drawn onto the plot in the order of their tag number, we can control the visual stacking through appropriately assigned tag numbers.

You can see all defined arrows, together with their location and appearance options, using show arrow, including a tag (for example show arrow 3) limits the command to the corresponding element. Labels and objects can be listed using equivalent commands.

[Tip]
Use explicit tags

When using gnuplot interactively, it’s usually a good idea to assign numeric tags explicitly and not to rely on the automatically generated tags. Almost invariably, we will want to modify a graph element created earlier, and always using an explicit tag number ensures that we really only edit the one element we intend to.

7.3.2  Arrows

Arrows are generated using the set arrow command, which has the following set of options:

set arrow [{idx:tag}] from {pos:from} to {pos:to}
set arrow [{idx:tag}] from {pos:from} rto {pos:offset}
set arrow [{idx:tag}] from {pos:from} length {pos:len} angle {flt:degrees}

set arrow [{idx:tag}] [ nohead | head | heads | backhead ]
                      [ size {flt:length} [,{flt:angle}] [,{flt:backangle}]
                        [fixed] ]
                      [ nofilled | empty | filled | noborder ]
                      [ front | back ]
                      [ lineoptions ]

set arrow [{idx:tag}] [ arrowstyle | as ] {idx:style}

Each arrow has a beginning and an end point. The starting point is always given explicitly, as a coordinate pair in any of gnuplot’s coordinate systems, but the end point can be given in three different ways (see the first three lines in the listing above):

As an explicit end point coordinate using the to keyword. (Example: set arrow from 0,0 to 2,1.)

As a relative offset from the starting point, using rto. The horizontal and vertical length of the offset are measured in units of the x and v axis, respectively.[2] (Example: set arrow from 0,0 rto 2,0 draws a horizontal angle that is two x axis units long.)

As a length and an angle. The length is measured in units of the horizontal axis, but can be preceded by any of the permissible coordinate system specifiers. The angle is given in degrees, and measured counterclockwise from the positive x axis. (Example: set arrow from 0,0 length 2 angle 45.)

Many aspects of the arrow’s appearance can be changed using the set arrow command. These can either be given explicitly or by referencing a previously defined arrow style (using the arrowstyle keyword). In the next section, I’ll explain the the different appearance options; we will come back to predefined styles in chapter 9.

Appearance options can either be specified separately, or combined with the coordinates in a single call. The following two code snippets shows to equivalent ways to define the same arrow:

set arrow 1 from 0,0 to 2,1
set arrow 1 nohead front

set arrow 1 from 0,0 to 2,1 nohead front

Customizing arrow appearance

The appearance of an arrow, in particular the size and shape of its head (or heads) and the look of its “backbone” can be customized:

Heads—By default, a head is drawn at the destination endpoint. Heads can be suppressed entirely using nohead. The keyword heads draws heads at both ends; backheads draws only a single head at the start point.

Head size and shape—The size and shape of the head can be changed using the keyword size, followed by the length and the angle at the arrow tip, and—optionally—the angle at the back of the head (see figure 7.4). Unless otherwise specified, the length is measured in units of the primary horizontal axis, this can be changed using any of the coordinate system specifiers (see figure 7.2). All angles are measured in degrees; the back angle is ignored if the fill style is nofilled (which is the default).[3]

By default, the size of the arrow head is reduced for very short arrows; this can be disabled using fixed.

Head color and fill style—By default, arrow heads are not filled and the back angle is ignored. The keyword empty draws only the outline, filled draws the outline and fills it, and noborder draws only the filled area, but not the outline. This matters, because the outline extends, by one line width, beyond the end position of the arrow, hence it may be necessary to suppress the outline.

It is not possible to choose the fill color separately from the outline color—if you want a red-filled arrow head with a black outline, you need to construct it from two arrows, one on top of the other.

Layer assignment—Arrows can be assigned to the front or back layers.

Line options—All available line options can be used and apply to the arrow’s backbone and its head or outline.

Figure 7.4. The parameters that control the shape of an arrowhead

Some examples of custom arrow heads are shown in figure 7.5.

Figure 7.5. Different arrow forms and the commands used to generate them. See listing 7.2 for the last one.

More tricks with arrows

The arrow facility is quite flexible and can be used for purposes other than placing explicit arrows on a plot. Here are some ideas.

Since an arrow without a head is just a straight line, we can use set arrow ... to draw arbitrary straight lines on a plot. In particular, arrows can be used to draw vertical lines. (We don’t need arrows to draw horizontal lines of course; just say plot 1, -1 to draw horizontal lines at y=-1 and y=+1.)

The ability to mix different coordinate systems in the same coordinate specification can come in handy in this context. For example, if we want to draw a vertical line at x=0.5 from the lower to the upper boundary, we can simply say set arrow from 0.5, graph 0 to 0.5, graph 1, without having to worry about the exact values of the vertical plot range.

A set of short arrows without heads can be used to draw custom tic marks, if gnuplot’s built-in tic marks are insufficient (see chapter 8 for more information on tic marks).

Arrows with customized heads can be used to indicate a scale or range on a graph—figure 7.8 contains such an arrow used for this purpose.

Finally, we can generate an arrow with two different heads by generating two single-headed arrows back to back. We can also overlay arrows to generate more sophisticated effects. In listing 7.2, we show the commands used to generate the bottom arrow in figure 7.5.

Listing 7.2. Commands used to generate the double-feathered arrow in figure 7.5

set arrow 1 from 0,-9 to 6,    -9 size 0.5,30
set arrow 2 from 0,-9 to 5.75, -9 size 0.5,30
set arrow 3 from 0,-9 to 1,    -9 backhead size 0.3,90

As we’ve seen, arrows are a versatile graph element. But more often than not, we’ll want to combine them with some textual explanation. That’s what labels are for.

7.3.3  Text labels

Text labels are a natural companion to arrows: the arrow shows the observer where to look, and the label explains what is happening.

set label [{idx:tag}] [ "{str:text}" ] [ at {pos:location} ]
                      [ left | center | right ]
                      [ rotate [ by {int:degrees} ] | norotate ]
                      [ boxed ]
                      [ front | back ]

                      [ point lt|pt {idx:pointtype}
                              | ps {flt:pointsize} | nopoint ]
                      [ offset {pos:off} ]
                      [ hypertext ]

                      [ noenhanced ]
                      [ font "{str:name}[,{int:size}]" ]
                      [ textcolor | tc [ {clr:colorspec}
                                       | lt {idx:type} | ls {idx:style} ] ]

The label text is typically a constant, but it can also be a string variable or any string-valued expression. To obtain a formatted string, the sprintf() or gprintf() functions are often useful. For example, to create a label (at the origin) showing the value of π, truncated to two digits, you could use: set label at 0,0 sprintf( "%.2f", pi ). (See section 5.1 in chapter 5 for more information about string operations, in particular table 5.1.)

By default, the text is placed flush left at the position specified by the at ... location, but this can be controlled using the left, center, and right keywords.

Gnuplot allows text to be rotated, but not all terminals support arbitrary angles. Use the test command to see what’s possible (see section 14.1). Unrotated text can be placed inside a surrounding frame using boxed. The stacking order can be controlled with front and back, as for arrows.

Using point, a symbol can be placed at the position specified with at and the text label is shifted relative to it. The point style and size of the symbol can be fixed using lt (line type), pt (point type), or ps (point size), and the offset can be customized using offset. The nopoint option suppresses the point.

The hypertext option creates a label that is visible only when the mouse hovers near the location of the label—specifically: near the symbol that is placed at the location of the label. (The hypertext option can only be used together with the point option.) The hypertext option is not available for all terminals. Currently, only the (interactive) wxt, qt, and windows terminals support it, as well as the (browser-based) svg and canvas terminals.

The color of the text can be selected using textcolor (abbreviated tc), followed either by a color specification (see chapter 9), or by the keyword linetype (or lt) and the index of a predefined line type. In this case, the color of the line type is applied to the text.

Finally, a specific font can be chosen using the font option. The keyword noenhanced suppresses the interpretation of enhanced-mode control characters in the text string, even if enhanced mode is active for the current terminal. (Read chapter 14 for an in-depth discussion of font selection and enhanced text mode).

Besides the fully general set label facility that I just described, gnuplot offers two specialized labelling commands: to add a label to any axis of the plot, or to add a title to the entire graph. Both provide convenient shortcuts for common situations.

Title

The title option is a label with some special defaults. For instance, it’s automatically placed centered at the top of the graph.

The title option takes the following arguments, which should be familiar from the label discussed previously.

set title [ "{str:text}" ] [ offset {pos:off} ]
          [ font "{str:name}[,{int:size}]" ][ noenhanced ]
          [ textcolor | tc [ {clr:colorspec}
                             | lt {idx:type} | ls {idx:style} ] ]

Only the behavior of the offset directive requires an explanation. It can be used to shift the title from its default position by a specified amount. What’s unusual is that by default it interprets its argument as given in the character coordinate system. With this in mind, the following commands are entirely equivalent—both shift the title down by the height of a single character:

set title "..." offset 0,-1
set title "..." offset character 0,-1

Axis labels

The commands that place labels on the axes of a plot are also variants of the standard set label command. Since labels can be placed onto any axis, there’s an entire family of related commands, all differentiated by prefixes (such as x and y), that indicate which axis the label belongs to. (We haven’t discussed these prefixes yet, but will do so in chapter 8). In the following synopsis, the underscore (_) stands for any of the permissible prefixes.

  set _label ["{str:text}"] [ offset {pos:offset} ]
             [ rotate by {int:degrees} | parallel | norotate ]
             [ font "{str:name}[,{int:size}]" ] [ noenhanced ]
             [ textcolor | tc [ {clr:color}
                               | lt {idx:type}
                               | lt {idx:style} ] ]

The options are an almost strict subset of the ones available for the plain set label command. Check section 7.3.3 for more details. The only new feature is the rotate parallel directive. It only applies to three-dimensional plots (see chapter 10 and rotates each label such that it is parallel to its axis (by default, the labels are printed horizontally in three-dimensional plots).

Finally, always keep in mind that both set title and the axis label commands are merely standard text labels with some convenient defaults. If they don’t give you sufficient control to achieve the effect you’re looking for, you can always use explicit set label commands instead.

Besides arrows and labels, which primarily are intended to add explanations to a graph, gnuplot also offers the user the ability to place arbitrary graph objects on a plot. This is done using the set object command.

7.3.4  Shapes or “Objects”

The set object ... facility can place a variety of geometrical objects on a graph, such as rectangles, circles, ellipses, and even arbitrary polygons. Shapes are different from other graphical elements we have encountered so far: they are less suited to graph data, because their coordinates cannot be read from a file and instead have to be set manually. Neither do shapes provide semantic support for a graph in the way that arrows and labels do. They add functionality to gnuplot that you would expect in a general-purpose drawing program, rather than in a data visualization tool—but here they are, in case that (special) need arises.

Because gnuplot’s shapes and objects are a bit outside our primary goal and interest (which is to understand data with graphs), I’ll not discuss them much further, except for some cautionary notes. (See the standard reference documentation for the details.)

All objects share a single tag counter, which is separate from the tag counters for arrows and labels, but keep in mind that there is no separate tag counter for (say) circles as opposed to rectangles. All shapes are of type object; the particular shape (whether circle, rectangle, ellipse, or polygon) is specified as property of each individual object. For rectangles, you can define a global style using set style rectangle, the other shapes inherit properties from the global fill style, which is defined through set style fill. (See chapter 9 for information on global styles.) Finally, let me remind you that elements of type object are the only items that can be placed in the layer that is at the bottom of the visual stack (using the behind keyword). This can be useful to give a colored background to the entire graph.

Arrows, labels, and objects are graphical elements that we can use to make graphs more interesting and more informative. But they aren’t the only elements that can go on a plot besides data, and aren’t even the most useful ones. The single most useful element is probably the graph’s legend or key, which explains what all the lines and symbols on the plot stand for. Gnuplot’s set key facility is very powerful, and therefore deserves a section by itself.

[2] For logarithmic coordinates, the relative offset is multiplicative, not additive.

[3] The default values for the head size are: size screen 0.0225,15,90. Using the screen coordinate system guarantees that the head size does not change if the plot range does..

7.4  The graph’s legend or key

We all know the little boxes on hiking maps explaining what all the symbols mean: thick red lines indicate highways, white is for country roads, and thin dashed lines mean unpaved gravel. And the odd-looking symbol with a roof: right, that’s an outhouse. In gnuplot, this box is called the key.

The key (or legend) explains the meaning of each type of line or symbol placed on the plot. Because gnuplot generates a key automatically, it’s the most convenient way to provide this sort of explanation. On the other hand, relying on the key separates the information from the actual data. I therefore often find it preferable to place arrows and labels directly on the graph instead, to explain the meaning of each curve or data set. A separate key excels again when there are so many curves that individual arrows and labels would clutter the graph.

As usual, almost anything about the key can be configured. The full command synopsis is as follows:

set key [ on|off ] [ default ]

        [ [ at {pos:position} ]
          | [ inside | outside ] [ lmargin | rmargin | tmargin | bmargin ] ]
        [ left | right | center ] [ top | bottom | center ]

        [ vertical | horizontal ] [ Left | Right ]
        [ [no]reverse ] [ [no]invert ]
        [ maxrows [ auto | {int:rows} ] ] [ maxcols [ auto | {int:cols} ] ]

        [ [no]opaque ]
        [ samplen {flt:len} ] [ spacing {flt:factor} ]
        [ width {int:chars} ] [ height {int:chars} ]

        [ [no]title "{str:text}" ]
        [ [no]box [ lineoptions ] ]

        [ [no]enhanced ]
        [ font "str:font,int:size" ] [ textcolor | tc {clr:color} ]

        [ [no]autotitle [columnheader] ]

]

As you can see, there are a lot of suboptions! To make information easier to find, I’ve broken the following explanation into separate sections under their own headings.

7.4.1  Turning the key on and off

The entire key can be suppressed using either of the following two commands:

set key off
unset key

The command set key on enables it again.

7.4.2  Placement

The key can be placed at any position on the entire canvas. The most straightforward (but not necessarily the most convenient) method is to fix a specific location using the at option. The location can be prefixed by any of the standard coordinate system prefixes. The keywords left, right, top, bottom, and center can be used to align the key relative to the specified position. For instance,

set key at 0,0 top left

places the top-left corner of the key at the origin of the coordinate system.

As usual, with great power comes great responsibility. When using the explicit at option, gnuplot does not rearrange the plot in any way to make room for the key: this must be done explicitly by the user. Instead, we can use several predefined keywords to place the key relative to the graph. When using these keywords, the borders of the graph are automatically adjusted to make room for the key.

We can use the keywords left, right, top, bottom, and center to push the key into the desired position. We can place the key inside the plot using inside (this is the default), or on any one of the margins, using lmargin, rmargin, tmargin, bmargin (meaning the left, right, top, and bottom margin, respectively). We can use combinations of position specifiers, such as set key top left . The effect of these options is cumulative, so that the following two examples will have the same effect:

set key bottom left

or

set key bottom
set key left

One surprising exception is that the keyword center by itself is interpreted as the center of the graph (both horizontally and vertically).

7.4.3  Layout

The samples in the key can either be stacked vertically or aligned horizontally using the vertical and horizontal keywords. The alignment of the labels within the key is controlled using the Left and Right options (note the capitals). The default is vertical Left.

Usually, the textual description of the line sample is to the left, with the line sample on the right. This arrangement can be interchanged using reverse.

Entries in the key are made in the order in which the corresponding data sets occur in the plot command. If you want to sort entries in some specific way, you need to list the data sets in the appropriate order in the plot command. But you can invert the stacking order of the samples within the key through invert. This is mainly useful to force the ordering of labels in the key to match the order of box types in a stacked histogram (see section 12.2).

In general, the key is built up in its primary direction as far as possible. Assuming that key entries are vertically stacked (which is the default), then gnuplot will continue to add lines to the key, and only begin a new column when there is no more vertical room on the graph. This behavior can be changed using the maxrows keyword together with a numeric argument: gnuplot will create a new column in the key whenever the number of rows exceeds the specified value. The maxcols keyword has the equivalent effect with a key that built up horizontally.

7.4.4  Appearance

By default, each entry in the key is created at the same time that the corresponding line is drawn on the graph. Later plot elements may therefore obscure parts of the key. When the opaque keyword is given, the key will be drawn after the graph is complete; moreover, the key area will be cleared before the key is created.

The length of the line sample in the key can be controlled using samplen. The sample length is the sum of the tic length (see chapter 8), and the argument given to samplen times the character width. A negative argument to samplen suppresses the line sample.

The vertical spacing between lines in the key is controlled using spacing. The spacing is the product of the point size, the vertical tic size, and the argument to spacing. This parameter does not influence the horizontal spacing between entries in horizontal mode.

The entire key can be given a title (which will sit at the top of the key), using the title option, and be surrounded by a frame using box. In this case, it may be necessary to adjust the size of the box: this can be done using the width and height parameters. Both take the number of characters to be added or subtracted from the calculated size of the key, before the box is drawn. In particular, if any text label in the key contains control characters for enhanced text mode, the size of the autogenerated box may be incorrect. Finally, the parameters of the lines used to draw the box can be adjusted using the usual options.

Enhanced text mode (see chapter 14 can be switched on or off for the key, independent from the terminal’s behavior. The default font for the key can be set by using the font keyword, but keep in mind that these settings can be overruled in the actual explanations themselves (see next section).

7.4.5  Explanations

The explanation is a bit of text that assigns a meaning to each line sample or symbol type included in the key. The string for the explanation can come from one of two places: either from plot command itself, or from the data file. Let’s take a look at both methods.

Taking explanations from the plot command

Usually, the explanatory text is taken from the plot command, using the title keyword:

plot "data" u 1:2 title "Experiment" w l, sin(x) title "Theory" w l

If no explicit title has been set in the plot command, by default gnuplot will generate a standard description, based on the filename and the selection of columns plotted. If you don’t like this behavior, you can turn it off using set key noautotitle: now only curves with an explicit title keyword in the plot command will have an entry in the key. (The command set key autotitle restores the default behavior.)

There two further ways to suppress parts or all of a key entry:

If an empty string is given as argument to the title keyword, gnuplot will generate no key entry for the corresponding data set at all: neither line sample, nor explanation will be shown.

To generate a key entry consisting only of the line sample, but without a visible explanation, use a string consisting only of whitespace as argument to the title keyword in the plot command.

Taking explanations from the data file

Instead of taking them from the plot command, gnuplot can take the explanations for the key from the first noncomment line in the data file. This is the same information that we used in section 4.4.1 to identify a column in the using specification—no we use it for the graph’s key, as well.

Although the strings used in the key always come from the first noncomment line in the data file, there are three different ways to instruct gnuplot what to do:

Globally, using set key autotitle columnhead.

As part of the plot command, via title columnhead.

To have maximum control, using the function columnheader() as part of an inline transformation in the using specification.

Let’s look at an example. Listing 7.3 shows a data file that we have seen before (in listing 4.6). Note how the first line, containing the column headings (which will be used in the key of the plot) isn’t a comment line!

Listing 7.3. Data for figures 7.6 and 7.7. (grains)

Year    Wheat   Barley  Rye    # Not a comment line!
1990    8       6       14
1991    10      5       12
1992    10      7       15
1993    11      5       13
1994    9       6       12

We already know how to plot a file like this and generate proper key entries using explicit explanations in the plot command (technically, we need to use skip 1, because the first line is not commented out):

plot "grains" skip 1 u 1:2 t "Wheat" w lp, "" skip 1 u 1:3 t "Barley" w lp,
»    "" skip 1 u 1:4 t "Rye" w lp

If we use global style settings instead, then the plot command itself can be particularly simple (see figure 7.6):

set style data linesp                   # global style setting...
set key autotitle columnhead            # and global key configuration...
plot "grains" u 1:2, "" u 1:3, "" u 1:4 # ... result in simple plot command

Rather than using a global directive, we can tell gnuplot to take the explanation from the data file within the plot command. Doing so lets us modify the behavior on a case-by-case basis as desired:

plot "grains" u 1:2 t columnhead w lp,        # take explanation from file
»    "" skip 1 u 1:3 t "Brown Barley" w lp,   # explicitly provided
»    "" skip 1 u 1:4 t "" w lp                # suppress key entry entirely

Using title columnhead plot command instructs gnuplot to take the explanation for that column from the header entry for the same column. If you want more control (for instance, because you want to specify a different column, or you are applying an inline transformation that combines multiple columns), you can use the columnheader() function.[4] The argument to columnheader should evaluate to an integer, which is interpreted interpreted as a column number, and the function returns the first noncomment line in this column as a string. This string can either be supplied to the title keyword in the plot directly, or be part of a string expression. The next code snippet shows all these variants:

plot "grains" u 1:2 t columnhead w lp,          # same column
»    "" u 1:3 t columnhead(3) w lp,             # arbitrary column
»    "" u 1:4 t "Black " . columnhead(4) w lp   # string expression

Note that the title can be taken from a different column than the data, which can be useful in combination with data transformations: commands such as plot "data" using 1:($2/$3) title stringcolumn(2) are legal.

Figure 7.6. Using set key autotitle columnhead, key entries are taken from the first non-comment line in the data file. See listing 7.3 for the data file.

The ability to take explanations directly from the data file is an interesting new feature, adding significant convenience in particular when plotting many data sets in the same plot. Nevertheless, I am a bit uncomfortable with the way it mixes data and comments in the same file without an easy way to distinguish the two, making files using this format harder to use with data processing filters or as input to other programs. (I’d probably have preferred to make the line containing the column heads a comment line, but indicate it through a special marker, such as a doubled hashmark or other comment character at the beginning of the line.) Judge for yourself.

Placing explanations on the graph itself

In passing, I quickly want to mention another new feature: traditionally, explanations are visually grouped together to form the key. It is now also possible to put some (or all) explanations onto the graph itself, either at the beginning or the end of the line that they are describing. To do so, include at beginning or at end after the title keyword in the plot command:

unset key

plot [1989:1995] "grains" u 1:2 t "Wheat" at beginning w lp,
»                "" u 1:3 t columnhead at end w lp,
»                "" u 1:4 t at beginning w lp

Notice that the explanation string may be missing, but the title keyword (or its abbreviated form t) must be present. Also keep in mind that you have no fine control over the placement of the explanation. Furthermore, gnuplot will not adjust plot ranges automatically to make room for these labels: you have to do so yourself. If you want more control, use the set label facility (see section 7.3.3).

Figure 7.7. Explanations can be placed at the beginning or end of the curves themselves, rather than grouped together in a global key.

7.4.6  Default settings

Just to provide a reference, the default settings for the key are

set key on right top vertical Right noreverse noinvert autotitle
» samplen 4 spacing 1.25 title '' nobox

The previous command is equivalent to the much shorter

set key default

which can occasionally be helpful to restore sanity when experimenting with key placement and layout.

Now that we’ve seen all of the different graph elements that can be used in a gnuplot figure, it’s time to put everything together and look at a worked example. That’s what we’ll do next.

[4] The columnheader() function is new; in previous versions of gnuplot this effect was achieved by supplying the column number directly as argument to the title keyword.

7.5  Worked example: features of a spectrum

To put all the things we’ve learned together, let’s study a more extensive example (see figure 7.8). The plot shows some experimental data, together with two theoretical curves that might explain the data from the experiment.

Figure 7.8. A complicated plot. See listing 7.4 for the commands used to generate it.

Listing 7.4 gives the commands used to generate this graph, as they would have been entered at the gnuplot prompt. Let’s step through them in detail—there is much to learn.

Listing 7.4. The commands for figure 7.8

set title "Scattering: Data and Theory"

set xlabel "Frequency Offset"
set ylabel "Relative Intensity"

set key top left box title "Legend" height .5

set arrow 1 from -4,.35 to -3.4,0.2
set label 1 "Cusp at x=-π?" at -6,0.4

set arrow 2 from 5,-.2 to 5,.95 nohead dashtype '-' lc black
set label 2 "Singularity at x=5" at 5,1 center

set arrow 3 from -1,0.2 to 1,0.2 heads size 0.1,90
set label 3 "Full Width at\nHalf Maximum" at 0,0.125 center

plot [-7:7][-.2:1.1] "spectrum.dat" u 1:2 t "Data",
»                    1/(1+x**2) t "Lorentzian", exp(-x**2) t "Gaussian"

First, we set an overall title for the graph, and put labels on the horizontal and vertical axes.

We move the key to the top-left corner, so that it doesn’t interfere with the data and surround the key with a box. We also give the key its own title and increase the overall height of the key a bit to give it a less crowded appearance.

We place an arrow and a label onto the graph, pointing out an oddity in the data. The label contains the Unicode character π (see section 5.1.5 for more information on Unicode and how to enter Unicode characters in gnuplot).

We use a dashed arrow without heads to place a vertical line on the plot. The line shows the vertical asymptote as the data approaches the singularity at x=5. (Dash types will be introduced in section 9.2.2.)

Yet another creative use for an arrow, this time with two customized heads. This arrow isn’t used to point out a feature in the graph, but to give an indication of a scale. (The “Full Width at Half Maximum” is commonly used in spectroscopy to measure the width of a spectral line.)

The label for the last arrow contains an explicit line break, so that it spreads over two lines. Remember from chapter 5 that escaped characters (such as the \n here) in double-quoted strings are interpreted as control characters.

Finally, finally: plot the data!

I hope this example gives you an idea of what can be achieved using decorations. Arrows and labels can be used to explain the data and point out specific features, but nothing prevents us from getting more creative.

7.6  Summary

This chapter explained techniques for putting additional, decorative graphics elements onto a plot: things like arrows and text labels. Because it is such an important tool to let a viewer understand a complicated graph, we spent a lot of time explaining how to customize and tweak the “key” or legend of a graph.

We also learned how to specify the locations of a graph, and introduced the different coordinate systems gnuplot uses for that purpose. We also discussed the layers that make up the graphical “stacking order” of a graph.

Decorative graphical elements (such as those in this chapter) are important, even though they do not represent data themselves: it is through these additional decorations and text labels that a graph receives its semantics: only if there is text can an onlooker understand what the data means that is shown in the graph.

In the following chapter, we’ll take a look at the last important thing that gives a plot context: the axes, and their subdivisions and labels. Stay tuned.

8

All about axes

This chapter covers:

Using multiple axes

Selecting plot ranges

Customizing tic marks and tic labels

Plotting time series with gnuplot

In this chapter, we finally come to coordinate axes and their labeling. Treating them last, after discussing plot styles and decorations, may seem surprising, given how critical well-labeled coordinate axes are to achieving an informative graph. On the other hand, gnuplot’s default behavior for axes-related options is perfectly adequate in almost all situations, so that explicit customization is rarely required.

One topic deserves special consideration, namely, the use of multiple axes on the same plot, and this is what we’ll discuss first. Then we move on and describe all the ways that axes and their labels can be customized. Lastly, we treat the special case when one axis (usually the x axis) represents time, in other words, when the plot shows a time series. Time series plots pose special challenges, since now the labels aren’t simply numeric. Instead, we need to worry about things such as the names of months and weekdays, potentially in different languages, too! This has long been a problem to gnuplot users, and so I’ll devote significant space to this application.

But first, let’s talk about multiple axes on the same plot.

8.1  Multiple axes

Gnuplot gives us the ability to plot graphs using two different coordinate systems within the same plot. Typically, these coordinate systems will share one axis (otherwise there’s no good reason to have them on the same plot), but they may also be entirely independent.

Plots involving two different y axes usually make the most sense when we want to compare two data sets side by side which have very different units. As a typical example, let’s study figure 8.1, which compares the average ice cream consumption (in some community) over consecutive four-week periods with the mean temperature during the same period. [5]

Figure 8.1 is a good example of why we might want to use multiple axes on a plot: the two quantities (ice cream consumption and temperatures) are of different nature, and are also numerically quite different. Yet, once we put them next to each other, the correlation becomes clear (not too surprisingly, in this example).

Figure 8.1. Using multiple axes on a plot to compare two different quantities side by side. (See listing 8.2 find out how this plot was made.)

8.1.1  Terminology

As we’ve just seen, gnuplot can handle two sets of axes on a single plot. The consequence is that all commands and options to manipulate axes-related properties come in two versions—one for each set of axes. In this section, we summarize the naming conventions associated with these commands.

The primary coordinate system is usually plotted along the bottom and left borders of the graph. (This corresponds to the first coordinate system we introduced in, section 7.2.) If the secondary system (second) is used, it’s plotted along the top and right borders.

Be default, the secondary system isn’t used. Instead, the tic marks (but not the labels) of the primary system are mirrored on the opposite sides of the plot.

All options that modify aspects of the coordinate systems can be applied to any of the axes: either the x or the y axis, in either the primary or the secondary coordinate system. The actual commands and options are prefixed to indicate which specific axis a command should be applied to (see table 8.1). Omitting the prefix applies the option to all axes.

Table 8.1. Prefixes used to indicate the selected coordinate system

Primary

Secondary

x axis

x

x2

y axis

y

y2

In the rest of this chapter, I’ll frequently discuss only one variant of any option—typically the one for the x axis of the primary coordinate system. It should be understood that everything applies to all other axes as well, just by selecting the appropriate prefix per table 8.1.

8.1.2  Plotting with two coordinate systems

The best way to understand how multiple coordinate systems are used in the same plot is through an example. Listing 8.1 shows the beginning of the data file from figure 8.1, and the complete set of commands used to generate the plot from the data file is in listing 8.2.

Listing 8.1. The beginning of the data for figure 8.1 (icecream)

# Date         Consumption[g]   Temperature[Celsius]
1951-04-01     179.8             6.01
1951-04-29     180.8            13.34
1951-05-27     186.5            17.78
1951-06-24     202.1            19.38
1951-07-22     190.1            19.48
...

The first three lines (from set timefmt to set xdata) tell gnuplot how to parse and format the calendar date used along the x axis. We’ll discuss them in a later section (8.4).

Next, we switch off the mirroring of the primary axis’s tic marks on the opposite (right) side of the plot (set ytics nomirror) and instead explicitly switch on tic marks for the secondary y axis (set y2tics).

We make sure that labels are placed on the graph—this step is crucial when using multiple axes, since otherwise the viewer has no chance of figuring out which data set goes with which axis. We also modify the key from its default location and appearance (see section 7.4 if you need a refresher on any of the options).

Finally, the actual plot command. The only thing new here are the axes keywords and their arguments. These directives tell the plot command which combination of axes to use for each data set. For example, axes x1y2 means that the data should be plotted according to the primary x axis, but the secondary y axis.

There are four possible combinations of axes that can be used, and they can be selected using x1y1, x1y2, x2y1, and x2y2.

Listing 8.2. The commands for figure 8.1 using data from listing 8.1

set timefmt "%Y-%m-%d"
set format x "%d%b%y"
set xdata time

set ytics nomirror  # Switch mirroring of principal system OFF
set y2tics          # Switch secondary system ON

set ylabel  "Mean Icecream Consumption per Head [Grams]"
set y2label "Mean Temperature [Celsius]"

set key top left reverse Left

plot ["1951-03-25":]
» "icecream" u 1:2 t "Icecream" axes x1y1 w linesp,
» "" u 1:3 axes x1y2 t "Temperature" w linesp

8.1.3  Sidebar: Should you do it?

Multiple axes on a single plot are occasionally frowned upon, because they can easily be abused to manipulate the message of a graph. Look at figure 8.2. The middle panel shows both data sets drawn in a single coordinate system. We can see that both curves grow, but also that one grows more strongly than the other.

In the panels to both sides, we show exactly the same data, but how different is the appearance! In the panel on the left, it seems as if both curves are almost identical, while in the panel on the right, one seems to be growing much more strongly than the other one. (Look closely—the seemingly strongly growing curve is the one that changed least in the middle panel.)

Figure 8.2. The malicious effect of not-to-scale graphs. The data in all three panels is the same, but the scales have been changed for both curves independently. The scale for the solid curve is always on the left, the scale for the dashed curve is on the right. Only in the middle panel are both curves drawn to the same scale.

These dramatically different appearances have been achieved solely by manipulating the plot ranges for each curve individually. Being able to select different plot ranges for the same data on a single plot is what makes dual axes plots open to the kind of abuse you see in figure 8.2.

Also note how in the figure no indication is given which curve is plotted on which axis, making it impossible to determine the actual meaning of the graph!

I think dual axes plots have their use. Yet, in all such graphs, care must be taken that the selected plot range is chosen suitably for the data at hand. I’ll have more to say about the effect of scales and plot ranges in chapter 19.

8.1.4  Linking axes

In the previous section, we saw how to plot two curves, representing different and independent quantities (ice cream consumption and temperature), on the same plot. In this section, we’ll consider a related problem: how to plot two curves, representing different but related quantities.

Let’s consider an example. Figure 8.3 shows two data files, side-by-side. Both contain temperature readings for the same date range, but for two different locations (New York City, USA and Frankfurt a.M., Germany) and in two different units (Fahrenheit and Celsius).[6]

Figure 8.3. Two data sets, showing daily temperatures. Note that the data sets use different units.

Because both data sets show the same kind of data, it makes sense to plot them in the same graph. Nevertheless, because the units are different, the two data sets can’t just share a single y axis. Of course, we could use inline transformations (see 3.3) to transform (say) the Fahrenheit values into Celsius and only plot that—but in order to easily go back and forth between both temperature systems, it would be nice if we could retain both units in the same plot.

This is were gnuplot’s new set link facility comes in. Using set link, we can make the values on one axis be dependent on the values on the opposite axis. All we need to do is to provide a link function, which maps values from one system to the other, and its inverse.

In our example, we want to map Celsius to Fahrenheit, and vice versa. This is accomplished using the following formulas:

Fahrenheit = (9/5)*Celsius + 32
Celsius    = (5/9)*(Fahrenheit - 32)

Using these formulas, we can set up the link between the two vertical axes:

set link y2 via (9/5.)*y+32 inverse (5/9.)*(y-32)

This establishes a link between the primary vertical axis (the y axis, on the left), which shows temperatures in Celsius, and the secondary vertical axis (the y2 axis, on the right), which shows values in Fahrenheit. We can now plot both files using the commands from listing 8.3, the resulting graph is shown in figure 8.4. (See listing 8.2 for an explanation of the commands at the beginning of this example.)

Listing 8.3. Commands for figure 8.4 using data from figure 8.3.

set timefmt "%Y-%m-%d"
set format x "%d%b%y"
set xdata time

set ytics nomirror
set y2tics

set ylabel "Celsius"
set y2label "Fahrenheit"

set link y2 via (9/5.)*y+32 inverse (5/9.)*(y-32)

plot "frankfurt" u 1:2 axis x1y1 w lp, "newyork" u 1:2 axis x1y2 w lp

Figure 8.4. Linking two axes together using link. See the text for details, and listing 8.3 for the commands.

This example should have given you a sense how the set link feature can be used. A few additional details:

Links can be established between the two horizontal and between the two vertical axis, using set link x2 or set link y2, respectively.

When linking the horizontal axes using set link x2, the dummy variable in the link functions must be called x; when linking the vertical axes using set link y2, the dummy variable in the link functions must be called y.

The link between either set of axes can be removed using unset link x2 or unset link y2.

Keep in mind that the link function must be invertible! This is not a problem when the link function is linear (as in our example), but non-linear functions are, in general, not invertible (or are invertible only over a limited domain—the function x² for instance is invertible for all non-negative values of x).

The set link feature is relatively new and strikes me (at the time of this writing) as still somewhat immature. Feel free to experiment further!

Do-it-yourself log plots

The link feature can be used to create log plots. Doing so may shed more light on what link can do.

Let’s consider the data set shown in figure 3.4. There, we used set logscale y to generate a semi-logarithmic plot. Now, we will use link instead.

In a semi-log plot, you plot the logarithm of the y value. We can achieve this using a simple inline transformation:

plot "data" u 1:(log10($2)) w lp

This will give the appearance of a semi-log plot, but the tic marks along the vertical axis will not show the actual y values from the data sets, but their logarithms (to base 10)—so, a value of 10 will correspond to a tic mark labelled “1”, a value of 100 to a tic mark labelled “2”, and so on. (See the right hand scale of figure 3.4.)

The link feature enables us to have an axis showing the original y values:

set link y2 via 10**y inverse log10(y)

One problem is that gnuplot is not very clever at figuring out where to place tic marks for the dependent (“linked”) axis, which means that you have to specify them explicitly, using a command like this (the set _tics family of options will be explained later in this chapter):

set y2tics ( 1, 2, 5, 10, 20, 50, 100, 200, 500, 1000, 2000, 5000 )

Although we had to specify the tic marks explicitly, link enables gnuplot to figure out where along the vertical axis these tic marks should go. The resulting plot is equivalent to figure 3.4 (with the tic marks on the left and right interchanged).

[5] This example was inspired by the “Ice Cream Consumption” story, found on the StatLib’s Data and Story Library (DASL) at http://lib.stat.cmu.edu/DASL/Datafiles/IceCream.html.

[6] Weather data can be obtained from the respective government services: http://www.nws.noaa.gov/climate/index.php?wfo=okx and http://www.dwd.de/bvbw/appmanager/bvbw/dwdwwwDesktop?_nfpb=true.

8.2  Selecting plot ranges

We’ve already encountered plot ranges in chapter 2, but only in a limited form as an inline specification to the plot command, looking something like this:

plot [-10:10][-2:2] sin(x)

The first pair of numbers in brackets sets the desired x range, while the second pair of numbers in brackets fixes the y range. Both terms are optional—if you leave them out, gnuplot will select ranges automatically.

This is enough—most of the time. However, using this syntax, only the plot ranges of the primary coordinate system can be fixed, which is insufficient if we want to use multiple axes on the same plot. Also, the inline syntax doesn’t work well when attempting to change the plot range with the mouse. Let’s look at gnuplot’s facilities for controlling plot ranges in more detail.

[Tip]
Plot ranges are essential

An distinctive property of good graphs is that they show that what is relevant. Plot ranges are the primary means of achieving this goal. Well selected plot ranges are essential!

In chapter 19 I will have more to say about ways to create meaningful, high-quality graphs. But for now, let’s take a look at gnuplot’s facilities to control them.

8.2.1  What you need to know for interactive work

The inline syntax is a shorthand for the family of _range commands. (Here and in the following, the underscore is intended as a placeholder for any one of the prefixes from table 8.1.) To adjust plot ranges for the primary and the secondary systems independently, we need to issue separate set _range commands using different prefixes.

The explicit set _range commands expect a pair of numbers enclosed in square brackets, similar to the syntax for inline range specifications. Listing 8.4 shows some examples of the _range commands in action.

Listing 8.4. Permissible syntax for setting plot ranges

set xrange [-1:5]      # Explicit min and max
set xrange [:10]       # Leave min unaffected, set max explicitly
set yrange [3:*]       # Set min explicitly, use autoscaling for max
set yrange [:sqrt(2)]  # Numeric expressions are legal
set yrange [1:0]       # Inverted axes are possible

Besides providing explicit lower and upper boundaries, we can leave one or both of the numbers blank, in which case the corresponding value won’t be changed. Alternatively, we can supply a star (*) to turn on autoscaling for that particular value. Autoscaling means that gnuplot selects a plot range that is large enough to include all data points, but gnuplot may extend the range in order to include the next full tic mark position. This behavior can be suppressed using the noextend specifier. Using the following setting, the vertical range will be bounded by the minimal and maximal value in the plotted data set precisely.

set yrange [*:*] noextend

The inline shorthand (like plot [0:1] ...) only affects the command line that it is part of—it does not reset the overall plot ranges, the way set _range does. This is the reason why you don’t need to use the * indicator to turn on vertical autoscaling when using only the shorthand: by default, the plot ranges are [-10:10] horizontally and [*:*] vertically. Issuing a command like plot [][0:1] ... does not change the global autoscale setting for the y axis, and hence a subsequent command like plot [][0:] ... will autoscale the positive y axis. But if the vertical range has been fixed globally using set yrange [0:5] for example, then plot [][0:] ... will honor the explicit plot range. To get autoscaling behavior, you now need to turn it on explicitly, either inline (plot [][0:*] ...) or using set yrange [0:*].

8.2.2  What you might want to know for batch processing

Special considerations apply when creating large numbers of graphs automatically. Imagine, for instance, that you want to monitor traffic across many websites—in hosted or enterprise environments, this may mean following hundreds of separate traffic streams. Furthermore, the traffic to these sites is likely to vary by several orders of magnitude: a busy site may see tens of thousands of clicks a day (or more), other sites may see few if any clicks.

Obviously, you don’t want to select plot ranges manually (for all the sites—every day!). But relying entirely on gnuplot to adapt the plot range to the available data will often not be satisfactory either. For some of the very slow sites (those that show at most a single click a day), gnuplot will select a vertical plot range of [0:1], which is visually not a pleasing solution. It would be better if you could make sure the plot range is always at least [0:10], but larger if need be. Fortunately, gnuplot let’s you do just that, because it is possible to define bounds for autoscaled ranges. Listing 8.5 shows you the syntax.

Listing 8.5. Restricting plot ranges to bounded intervals

set yrange [*:*]         # Unbounded autoscaling
set yrange [0:5<*]       # Lower bound fixed, upper bound at least 5 or higher
set yrange [0:5<*<10]    # Lower bound fixed, upper bound between 5 and 10
set yrange [-10<*:*<10]  # Double-sided bound not exceeding [-10:10]

At other times, it might make sense to let autoscaling apply to only some of the lines in a plot. For example, imagine that you want to plot two data sets together, one of which is corrupted by occasional crazy outliers, whereas the other is not. In that case, it may be desirable to let gnuplot use only the smoother curve to determine the plot range. There are two ways to do this. Using the noautoscale (or noauto for short) keyword, you can flag those data sets in a plot command that should be disregarded when determining the plot range:[7]

plot "smooth" u 1:2, "noisy" u 1:2 noautoscale

Another approach is to evaluate the data sets first, and then use information obtained to calculate the plot range. The set _range family of options provide a way of doing this (using writeback and restore), but the stats command (see chapter 12 and the internal variables it defines are probably more straightforward to use.[8]

[7] The noautoscale directive is only available when plotting data files, but not when plotting functions. If you want to use it with functions, you need to “fake” it via the "+" pseudofile facility (see section 4.5).

[8] Another related option is set autoscale, possibly in conjunction with set offset, but it seems overly complicated and largely redundant. For more information about this, as well as about writeback, restore see the standard reference documentation.

8.3  Tic marks

Tic marks are the subdivision markers placed onto the axes of a plot to indicate the scale of the graph. Only if tic marks are present can a viewer infer quantitative information from a graph. Suitably chosen tic marks are therefore of critical importance to any well-constructed graph. Fortunately, gnuplot handles tic marks really well on its own and we rarely need to customize tic mark generation and labeling. But for the few cases when we do have special requests, here’s how to do it.

8.3.1  Overview and common conventions

Gnuplot distinguishes between major and minor tic marks. The difference is that major tic marks also carry a textual label (normally a number), while minor tic marks don’t. By default, only major tic marks are used, except for logarithmic axes, where both major and minor tic marks are drawn by default.

Major tic marks are by far the more important of the two. The appearance of major tic marks and their labels is controlled through the family of set _tics options. The available suboptions are shown in the next code sample.

set _tics [ axis | border ]
          [ in | out ] [ front | back ]
          [ scale [ default | {flt:major} [,{flt:minor}] ] ]
          [ [no]mirror ]

          [ rotate [by {flt:ang}] | norotate ]
          [ left | right | center | autojustify ]
          [ offset {pos:offset} | nooffset ]
          [ font "{str:name} [,{int:size}]" ]
          [ textcolor | tc {clr:color} ]
          [ format {"str:format"} ]

As before, the underscore _ is a placeholder for any one of the available axes identifiers (see table 8.1), indicating which axis the settings should be applied to. It is also possible to omit in the identifier (in this case) to signal that the chosen style should take effect for all axes.

The location and frequency of tic marks is controlled through another set of suboptions to set _tics. These suboptions only makes sense when applied to a specific axis (meaning that you can’t leave the _ placeholder blank in this case).

set _tics ...
          [ add ]
          [ autofreq
            | {flt:incr}
            | {flt:start}, {flt:incr} [,{flt:end}]
            | ( ["{str:label}"] {flt:pos} [ 0 | 1 ]
                [, ["{str:label}"] ... ] ) ]

We will next step through these options: first we’ll discuss how to modify the appearance and placement of tic marks, then we’ll talk about labels, before discussing ways to control location and frequency of tic marks. We’ll round this section up with some related features that are handled using commands other than set _tics.

8.3.2  Tic mark appearance and placement

By default, gnuplot draws tic marks on the border of the plot, extending inwards into the plot region, but they can be drawn toward the outside using the out option. This is useful in particular when the tic marks would interfere with the data. Tic marks can be drawn in either the front or the back layer of the visual stack (see section 7.2.)

Instead of being placed along the borders of the plot, tic marks can also be drawn along the zero axis (for instance, set xtics axis will draw tic marks along the line of the plot where y equals 0). If the zero axis isn’t within the plot range, the tic marks will always be drawn along the border.

The scale parameter controls the size of both major and minor tic marks. If no size for the minor tic marks is given explicitly, it’s set to half the size of the major marks. The size is given relative to the default size of 1.0 for major tic marks.

By default, the tic marks (but not their labels) of the primary coordinate system are mirrored on the opposite side. This behavior can be turned off using nomirror. You want to do this when using two coordinate systems in one plot (such as both the y and the y2 system), but there are other situations when mirroring is undesirable.

Why gnuplot’s defaults make sense

Graphs drawn using gnuplot can often be recognized by their overall appearance (unless the defaults have been modified significantly, of course): the plot is surrounded by a frame (or border), with tic marks and labels on the frame. This format may at times strike you as a bit “tired”, but I’d like to emphasize some advantages, that should make you hesitate at least a moment before rejecting it.

The frame on all four sides provides a clear visual boundary for the data. If only two axes were drawn along the bottom and left side, data points in the upper right-hand corner might get (visually) “lost”.

The other two sides of the frame (top and right side) can be used to mirror the primary tic marks or to carry a set of secondary tic marks, thus making qualitative evaluation of the data easier.

Because tic marks and their labels are drawn along the border of the graph (as opposed to along the zero axis) they are less likely to interfere with the data.

If you nevertheless want to give your graphs a different appearance, we’ll discuss some options in chapter 17.

A worked example

Figure 8.5 shows a graph that does not use the gnuplot default format: it has no border, but explicitly drawn coordinate axes (“zero axes”) instead. Because there is no border, the tic marks are drawn along the coordinate axes.

Figure 8.5. Non-standard placement of coordinate axes and tic marks. See listing 8.6.

You will find the commands for this figure in listing 8.6. First, the border is switched off, but the coordinate axis are drawn with solid black lines. (See section 8.3.6 later in this chapter for the set zeroaxis command. The line type -1 is guaranteed to draw a solid black line, see table 9.2.) Then gnuplot is instructed to place tic marks along the coordinate axis (instead of the border) with set xtics axis (and equivalently for the vertical axis). And finally, a function is plotted.[9]

Listing 8.6. Commands for figure 8.5.

unset border
set zeroaxis lt -1

set xtics axis
set ytics axis

plot [-8:4] airy(x)

The resulting graph is closer to what is common in mathematical contexts, but the disadvantages are clear: the curve interferes with the tic marks whenever it crosses the axes. (The partial obscuring of the zeros near the origin could be avoided through smarter placement of the labels and is not a fundamental problem.)

8.3.3  Tic labels

The text labels associated with major tic marks can be rotated and shifted using rotate and offset. If rotate is used without an explicit angle (in degrees), the labels will be turned by 90 degrees to the left. The position of the labels can be adjusted using offset. The relative shift can be specified in one of the five usual coordinate systems (see section 7.2); if none is specified, the character coordinate system is assumed.

By default, tic labels are justified depending on the axis and the rotation angle (autojustify). Alternatively, you can control the alignment explicitly with left, right, and center. Text font and color can be selected in the usual fashion.

Table 8.2. Conversion specifiers understood by the gprintf( ... ) function. See table 8.4 and table 8.5 for conversion specifiers for date and time values.

Conversion specifier

Description

%f

Floating point notation

%e or %E

Exponential notation, using ‘e’ or ‘E’ (respectively) to indicate exponent

%g or %G

Uses the shorter of %f and %e (or %E)

%h or %H

Like %g or %G, but use enhanced text mode to format the power of 10 as a proper superscript. Using %h, labels are formatted as a x 10^{b} (using the letter ‘x’); using %H, labels are formatted as a * 10^{b}.[a]

%x or %X

Hexadecimal representation

%o or %O

Octal representation

%P

Multiple of π

%c

Character replacement for scientific power, such as ‘k’ (kilo) for 1000, and so on

%%

Inserts a % character into the output (no conversion)

%t and %T

Coefficient and exponent to base 10

%l and %L

Coefficient and exponent to base of current logscale

%s and %S

Coefficient and exponent to base of current logscale; scientific power (restricts power to multiple of 3)

%b and %B

Coefficient and exponent using ISO/IEC 80000 notation

[a] This conversion specifier is new with gnuplot 5.

Table 8.3. Accuracy specifiers to be used with the gprintf() and sprintf() functions.

Accuracy Specifier

Description

"-"

Left-justify number.

"+"

Explicitly sign all numbers.

" "

Use a space instead of a positive sign.

"#"

Force a decimal point, even if all following digits are zero.

"a.b"

Field width a and number of decimal places b. (Both a and b must be positive integers.)

"0a.b"

Same as before, but pad the number with zeros instead of spaces on the left.

Formatting tic labels

We can change the formating used for the labels placed at the major tic marks, using set _tics format, followed by a string that describes the desired format.[10]

The format string is similar to the format string used by the printf() family of functions from the standard C library. In addition, gnuplot uses some extra format (or conversion) specifiers, which are listed in table 8.2. These conversion specifiers only apply to numeric arguments; for date/time values, check section 8.4.

[Tip]
Time series are different

Do not attempt to build up date and time strings manually; use dedicated conversion specifiers for time series (section 8.4).

If the % character is encountered in the format string, it’s interpreted as the beginning of a conversion specifier. It must be followed by one of the characters from table 8.2. Several modifiers can be placed between the % character and the conversion specifier, see table 8.3. In general, gnuplot’s functionality is built on top of the system’s C library—check its documentation for additional details.

The format string can also contain arbitrary characters, which are placed verbatim onto the plot. This makes it possible, for instance, to print the units (such as kg or cm) together with the numerical values.

Finally, providing an empty string as format specifier (as in: set xtics format "") is a way to suppress the generation of tic labels, although the tic marks will be drawn.

Scientific notation and enhanced text mode

Several of the conversion specifiers in table 8.2 give you individual access to the coefficient and exponent of a number in scientific notation. They’re intended for situations where you want to build up the combination of coefficient and exponent yourself.

In scientific notation, a number x is represented as

x = a × bc

where a is called the coefficient (also: mantissa or significand), b is the base and c is called the exponent or power. For example, the number 1024 can be written as 1.024 × 10³ (base 10) or 1 × 2¹⁰ (base 2).

When in enhanced text mode, gnuplot interprets certain regular character embedded in string labels as control or formatting characters. In this way, it is possible to change font face, size, and weight locally and “on the fly”.[11] A full discussion of enhanced text mode will have to wait till chapter 14; what is of interest in the context of tic marks and especially tic labels is the capability to include sub- and super-scripts in all text elements that are part of a graph. Subscripts are indicated through and underscore (_), superscripts through a caret (^). A single character following either of these (or a sequence of characters enclosed in curly braces) is formatted as sub- or superscript, respectively.

In particular for “powers of 10”, the ability to use real superscripts is very welcome. Imagine a number such as 5,000. In scientific notation, this becomes:

5 × 10³

Using the customary “E notation” (see 4.3), this would be written as 5.0e+03. But using enhanced text mode, we could format this string as:

"5 × 10^{3}"

(or simply as "5 × 10^3", because the exponent consists only of a single character) and gnuplot will display the exponent as a proper superscript.

The new conversion specifiers %h and %H use enhanced text mode to format numbers in this way. The lower-case version (%h) uses the letter ‘x’ to signify the multiplication operator following the coefficient, whereas the upper-case version (%H) uses the asterisk ‘*’.

But we can do better! Remember that gnuplot does handle Unicode (section 5.1.5). We can therefore replace the letter ‘x’ with a proper multiplication sign—either the saltire at code point U+00d7, or the multiplication dot at U+22c5. (If no glyph is available for the latter, you may substitute the “interpunct” at U+00b7.)

set ytics format "%.1t×10^{%T}"
set ytics format "%.1t·10^{%T}"

A worked example

Let’s study an example to demonstrate what we just learned. Figure 8.6 shows the weight of several mammals as a function of their heart rate in a double-logarithmic plot. Remarkably, the points seem to fall roughly onto a straight line: there is a distinct relationship between the size (or: weight) of an animal and the rate of its metabolism! This is known as allometric scaling and extends to other properties, such as the life time, as well: bigger animals live longer. Allometric scaling is a fascinating topic, but what interests us right now are the commands used to draw this graph. They are shown in listing 8.7.

Figure 8.6. Allometric scaling: larger animals have a slower heart rate. Notice the customized tic labels along both axes. See the text for details and listing 8.7 for the commands.

Choosing logarithmic scales and setting some axes labels should be familiar by now, but pay close attention to the way the tic labels are formatted. For the vertical axis, the format string is "%.0s%cg". According to table 8.2, %s is the coefficient (in scientific notation), when the power is restricted to being a multiple of 3, and %c is a character replacement for that power (‘k’ for kilo, ‘M’ for Mega, and so on). Hence a number like 2,000 will be split into the coefficient 2 (all decimals suppressed, because of %.0s), followed by ‘k’ for the factor of 1,000. I tack on an additional ‘g’, which carries through unchanged, to represent the weight unit “gram”. The result is perfect for grams and kilograms, but “megagrams” are more commonly known as (metric) tons. For the horizontal axes, we use the format with the multiplication dot discussed at the very end of the previous section. Finally, the plot command itself is an interesting example for the use of the with labels style (see section 6.3.5.1), with an explicitly selected point type and color.

Listing 8.7. Commands for figure 8.6

set logscale
unset key

set ylabel "Weight"
set xlabel "Heart rate [Beats per minute]"

set ytics format "%.0s%cg"
set ytics format "%.0t·10^{%T}"

plot [10:1000][:0.2e9]
» "mammals" u 3:2:1 w labels point pt 7 lc rgb 'red' offset 0,-1

8.3.4  Tic mark location and frequency

Finally, we can control where tic marks will be drawn. Using autofreq, gnuplot will automatically generate tic marks covering the plot range.[12] Alternatively, you can provide an increment alone, then tic marks will be drawn at integer multiples of the increment. Or you can specify a start point, an increment, and (optionally) an endpoint. (For logarithmic axes, the increment is interpreted multiplicatively instead of additively.)

Some examples will clarify (remember that the variable pi is predefined in gnuplot):

set xtics pi       # Increment alone: tic marks at pi, 1*pi, 2*pi, ...
set xtics 1, pi    # Start point and increment: tic marks at 1, 1+pi, 1+2pi, ...
set xtics 0,0.1,1  # Start point, increment, and end point:
                   # tic marks at 0, 0.1, 0.2, ... 0.9, 1 only

You can also provide a list of explicit labels and locations at which to draw tic marks. The list must be enclosed in regular parentheses, with list entries separated by commas. Each entry in the list consists of the text label for the tic mark, its location, and a third, optional parameter that indicates whether the tic mark should be drawn as a major or minor tic mark: 0 for major and 1 for minor.[13] The text label itself can be an empty string (so that no label is drawn for the tic mark), or it can be missing altogether (in this case, a textual representation of the location parameter is used as label).

In the following example, the first line creates five major tic labels with autogenerated (numeric) labels, and the second line creates three major tics using the explicitly provided textual labels. The third line creates two major tic marks (at 0 and 3, with labels A and B, respectively) and two minor tic marks between them.

set xtics ( 0, 1, 2, 3, 4 )                    # 5 major tic marks
set xtics ( "First" 1, "Second" 2, "Third" 3 ) # 3 major tic marks
set xtics ( "A" 0 0, 1 1, 2 1, "B" 3 0 )       # 2 major, 2 minor tic marks

Usually, any change to the location and frequency of tic marks wipes out any previous settings. To add new tic marks while retaining any existing ones, use the add keyword. This can be very useful for adding tic marks for special values to otherwise autogenerated tics, like so:

set xtics autofreq
set xtics add ( "π" pi )

These commands draw an additional tic mark at π, in addition to the automatically generated ones. Had we omitted the add keyword in the previous example, the second line would have clobbered the first, and the only tic mark would have been the one explicitly set at π.

Minor tic marks

Minor tic marks aren’t labeled, and are typically drawn smaller than the major tic marks. By default, minor tic marks are disabled for linear axes and enabled for logarithmic axes.

Minor tic marks can be switched on using the m_tics family of options, where the underscore is again used as a placeholder for any of the usual prefixes:

set m_tics [ {int:intervals} ]

The optional parameter counts the number of subintervals between major tics marks; the number of minor tic marks generated is one less than this number. (For example, let’s say that there are major tic marks at x=0 and x=5 and you would like to create minor tic marks for all the integers in between. Then set mxtics 5 would do the trick, because there are 5 integer-sized intervals between neighboring major tics, but the number of minor tic marks generated is only 4.)

Minor tic marks are only drawn when there are regularly spaced major tic marks. If all major tics are individually placed, m_tics will have no effect. Minor tic marks can still be created manually, using set _tics.

A worked example

You may have wondered how I generated the plot in figure 4.1 using two different y axis scales, each covering only part of the plot. Now we have all the information at hand to lift the secret.

The plot shows the same data twice, but vertically shifted. I achieve this—without the use of inline transformations!—by plotting the data once against the primary and once against the secondary vertical coordinate axes, with with properly chosen plot ranges for either axes. Note how the visible range (from min to max) is the same for both systems, but how the two ranges are offset from each other.

Now, only the tic marks are still missing. Here, I make sure to specify both a start and an end value for tic mark generation—this way, I achieve the partial labeling of each axis, only for the part of the plot that’s relevant to each curve.

Listing 8.8. Commands for figure 4.1 in chapter 4 using data from listing 4.4

set yrange [9:16]
set y2range [6:13]

set ytics 9,1,12 nomirror
set y2tics 10,1,13

set datafile missing "NaN"

plot [0:10] "missing" u 1:2 ax x1y2 w linesp lt 1 pt 7,
»           "" u 1:($2) ax x1y1 w linesp lt 1 pt 7

8.3.5  Reading tic labels from file

Finally, we can read the tic labels from the input file, using the _ticlabels() family of functions as part of the using directive to the plot command. (The underscore _ is again a placeholder for any of the axes identifiers from table 8.1.)

Let’s look at the data file in listing 8.9. We see that the x values are both present in numeric form (column 1) and as strings. Of course, it would be nice to use the strings for the tic labels. Here’s how to do that:

plot "months" u 1:2:xtic(3) w linesp

The xtic() function takes as argument the number of a column that will be used for the tic labels. Equivalent functions exist for the other coordinate axes; labels for the y axis are specified after labels for the x axis.

Listing 8.9. Time series data for figure 8.8 (months)

# Month  Data     Month Name
1        3        Jan
2        4        Feb
3        2        Mar
4        5        Apr
5        8        May
6        7        Jun
7        4        Jul
8        5        Aug
9        3        Sep
10       2        Oct
11       4        Nov
12       2        Dec

When employing any of the _tic() functions in a plot command, tic marks and labels will only be drawn at the locations explicitly read from the data file—in other words, autogeneration of tic marks is turned off.

The _tic() functions allow a variety of further tricks, beyond reading an explicit string from the data file. The following snippet calculates the quarter of the year for each month (such as “Q1”, “Q2”), and uses this as tic label:

plot "months" u 1:2:xtic( 'Q' . (1+int(($1-1)/3) )) w lp

In sections 8.4.2 and 8.4.3.4 we’ll see two further ways to produce figure 8.8 from the file in listing 8.9.

8.3.6  Grid and zero axis

In addition (or as alternative) to tic marks along the border of the graph, we can overlay a scale grid on the data. Grid lines are drawn at the position of major and, optionally, minor tic marks.

set grid [ [no]_tics ] [ [no]m_tics ]
         [ layerdefault | front | back ]
         [ polar [ {flt:angle} ] ]
         [ majorlineoptions [, minorlineoptions] ]

We can switch the grid on to be drawn at major or minor tic marks, for the primary or secondary coordinate system. (The underscore again must be replaced by any one of the prefixes from table 8.1.) Tic marks must be enabled—instructions to draw a grid at nonexistent tic locations will be ignored. The grid is either drawn in front (set grid front) or behind the data (set grid back).[14] The lines to use for the grid can be set separately for grid lines drawn at major and minor tic marks. If no style or type is given, the style ls 0, which draws the least visible lines possible (often using a dotted line), is assumed. The polar option is only relevant for plots using polar coordinates, which we’ll discuss in chapter 13.

Similar to grid lines, but less obtrusive, are zero axes. These are lines drawn across the graph for all the points where one of the coordinates is equal to zero:

set _zeroaxis [ lineoptions ]

For example, set xzeroaxis switches on a horizontal line at y=0 (representing the x axis). The default line type is ls 0, same as for the grid.

A worked example

A quick example (see figure 8.7) will help to clarify. The commands are in listing 8.10. Turning the grid on is simple; more interesting is are the tic marks and labels along the horizontal axis. With set xtic pi we tell gnuplot to place major tic marks at multiples of π. With the following format "%.0Pπ", we instruct gnuplot to use the integer multiple of π as label. Tacking on the character ‘π’ itself makes it clear to the viewer what the tic marks represent.

It is important to understand that the format specifier %P will interpret a value as multiple of π, but by itself does not ensure that tic marks will only be drawn at integer multiples of π. Instead, you must explicitly choose the locations where tic marks will be drawn using set xtics pi, then use format "%P" to format the labels at those positions accordingly.

Figure 8.7. Tic marks at multiples of π and the Greek letters used for the tic labels. See listing 8.10.

Listing 8.10. The commands for figure 8.7

set grid
set xtics pi format "%.0Pπ"

plot [-4*pi:4*pi] sin(x)/x

[9] The Airy function is the solution to the differential equation y"-xy=0 that is bounded everywhere. It is important in the approximation theory of differential equations.

[10] Traditionally, tic label formatting was accomplished using the set format option. This option is still available, but it is now redundant and should probably be avoided in favor of the more consistent method described in the text.

[11] Enhanced text mode is now the default in gnuplot 5; in general, it is no longer necessary to enable it explicitly. In fact, at times it may be necessary to disable it using the noenhanced keyword.

[12] To generate tic marks covering only the actual range of the data, rather than the plot range, you can use the rangelimited keyword. See the standard reference documentation for details.

[13] It is possible to add further levels of tic marks beyond major and minor. See the standard reference documentation for set tics and set xtics for details.

[14] The option layerdefault is only relevant for three-dimensional graphs—see the standard reference documentation for details.

8.4  Special case: Time series

Whenever we want to study how some quantity changes over time, we are dealing with a time series.[15] Time series are incredibly common—from stock charts and opinion polls to fever curves. Unfortunately, they pose special challenges, since the tic labels we would like to use for the x axis (such as the names of months or weekdays) aren’t strictly numeric.

Worse, they aren’t even universal, but locale-dependent. If we want to plot time series data, we therefore need to be able to parse arbitrary date/time formats from a file, and we must have the ability to format timestamps in a suitable, locale-dependent format.

Gnuplot offers three different ways to deal with date/time information as part of axes labels:

The “classic” way, using set _data, set timefmt, and set _tics format, which allows us to parse and reformat arbitrary date/time information in the input file, and which I’ll describe in detail in section 8.4.3. This is gnuplot’s most flexible and powerful method of handling timestamps.

The “new” style, which reads fully formatted tic labels directly from the input file using the _ticlabels() functions introduced in section 8.3.5. This method is very easy if the labels read from file are in the format you want, but it only works if the data set also contains numeric time information (to fix the x coordinate of the data points). It is also only suitable for relatively sparse data sets, because _ticlabels() create a tic mark and label for every data point. And it does not allow locale-dependent formatting.

For the special cases when you don’t require arbitrary date/time labels, but merely want to use the names of months or weekdays in a plot, gnuplot provides the simplified set _mtics and set _dtics facilities (see section 8.4.2).

In the next section, we first discuss the simpler case of using month or weekday names as tic labels. Afterwards, we’ll tackle the harder problem of dealing with arbitrary date/time information, both for input and for output. (Reading tic labels in files was discussed previously in section 8.3.5.)

8.4.1  Digression: Locales

The names of months and days of the week differ from country to country and language to language. The set of parameters defining such details of the user-interface is called the locale. Through examining the current locale, gnuplot determines the strings to be used in tic labels involving the names of months or weekdays.

The default is taken from the LANG environment variable, but can be changed using the following command:

set locale ["{str:locale}"]

The choice of available locales is system-dependent. On Unix systems, you can use the shell command locale -a for a list of available locales, or check the directory /usr/share/locale/. Note that some locales have country-specific variations (such as en_AU, en_CA, en_GB, and en_US). In this case it may not be sufficient to set the general locale (such as en), and a more specific locale must be chosen.

8.4.2  Turning numbers into names: months and weekdays

Gnuplot provides two simple commands to turn numbers into the names of months or days of the week. They offer much less flexibility than the general time series commands discussed in the next section, but are easy to use.

Let’s look back at the data file in listing 8.9. We want to label the x axis with the names of the month, but without using the explicit names in the third column. We can do this using the set xmtics command, which maps numbers to names of months (with 1=“January”, ..., 12=“December”). Don’t confuse this command with the set mxtics command introduced in section 8.3.4.1, which switches on minor tic marks!

The following sequence of commands was used to produce the plot in figure 8.8:

set xmtics                   # set primary tic mark formatting to Months
plot [][0:10] "months" u 1:2 w linesp

Figure 8.8. A time series plot, using set xmtics to use months for the labels along the horizontal axes. Also see listings 8.11 and 8.12. The data is shown in listing 8.9.

Besides the names of months, we also can use days of the week (such as “Mon”, “Tue”, and so on) as tic labels. We enable them using set xdtics (with 0=“Sunday”, ..., 6=“Saturday”), similar to what we’ve seen for set xmtics. Both set xmtics and set xdtics map overflows back into the legal range through a modulo operation (modulo 12 and modulo 7, respectively), as you would expect.

To restore normal (numerical) axes labeling through unset xmtics or unset xdtics.

Equivalent commands exist for all other axes, using the usual prefixes per table 8.1.

The set xmtics and set xdtics facilities are very limited. The strings follow the locale, but cannot be formatted any further. But as a quick-and-easy alternative to the fully general methods described next, the techniques introduced in this section can occasionally be very welcome.

8.4.3  General time series: the gory details

For more general time series, we face two problems: first we must be able to read arbitrary timestamps from the input file, and then format them again for output.

First, we must enable time series mode (for the primary x axis) using

set xdata time

Issuing set xdata (without an argument) restores normal operation again. Equivalent commands exist for all other axes, distinguished through the usual prefixes.

[Tip]
Time-series mode

To handle general date and time information in data files, you must first switch to time-series mode, using set xdata time. In time-series mode, timestamps are parsed (for input) according to the value of set timefmt and are formatted (for output) according to set _tics format.

In time series mode, input (parsing of timestamps from files) and output (formatting timestamps for inclusion in the plot) are controlled by the two commands set timefmt for input and set xtics format for output.[16] Both accept a format string using a syntax similar to the one found in the POSIX strftime() routine. (We already encountered set _tics format in section 8.3.3, but there we only talked about the formatting of plain numbers. To this, we now add the possibilities to format complex date/time values.)

Gnuplot assumes all data to be in universal time coordinates (UTC)—it has no facilities to perform time zone changes, adjust for daylight savings, or apply similar transformations. If they are required, they must be applied externally, before attempting to plot the data. [17]

Table 8.4. Alphabetically sorted conversion specifiers for date/time information for the set _tics format and set timefmt commands. See table 8.2 to format numeric values. See table 8.5 for a list sorted by topic.

Conversion specifier

Available for input: set timefmt

Available for output: set format

Values

Description

%a

x

Sun, Mon, ...

Abbreviated day of week

%A

x

Sunday, Monday, ...

Full day of week

%b

x

x (also %h)

Jan, Feb, ...

Abbreviated name of month (3 characters)

%B

x

x

January, February, ...

Full name of month

%d

x

x

01–31

Day of month (always two digits on output)

%D

x

e.g. “03/25/08”

Shorthand for “%m/%d/%y” (US date format)

%F

x

e.g. “2008-03-25”

Shorthand for “%Y-%m-%d” (extended ISO date format)

%H

x

x

00–24

Hour—24-hour clock (always two digits on output)

%I

x

00–12

Hour—12-hour clock (always two digits)

%j

x

x

001–366

Day of year (always three digits on output)

%k

x

0–24

Hour—24-hour clock (one or two digits on output)

%l

x

0–12

Hour—12-hour clock (one or two digits)

%m

x

x

01–12

Month (always two digits on output)

%M

x

x

00–60

Minute (always two digits on output)

%p

x

“am”, “pm”

a.m./p.m. indicator

%r

x

e.g. “10:55:48 pm”

Shorthand for “%I:%M:%S %p” (US time format)

%R

x

e.g. “22:12”

Shorthand for “%H:%M” (24-hour clock time format without seconds)

%s

x

0–...

Unix epoch seconds (input only!)

%S

x

x

00–60

Seconds (always two digits on output)

%T

x

e.g. “22:12:48”

Shorthand for “%H:%M:%S” (extended ISO time format: 24-hour clock with seconds)

%U

x

00–53

Week of the year (weeks starting on Sunday; always two digits)

%w

x

00–06

Day of the week (0=Sunday; always two digits)

%W

x

00–53

Week of the year (weeks starting on Monday; always two digits)

%y

x

x

00–99

Year (two-digit; always two digits on output)

%Y

x

x

0000–9999

Year (four-digit; always two digits on output)

Table 8.5. Conversion specifiers for date/time information for the set _tics format and set timefmt commands, sorted by topic. See table 8.2 to format numeric values. See table 8.4 for a list sorted alphabetically by conversion specifier.

Conversion specifier

Available for input: set timefmt

Available for output: set format

Values

Description

%s

x

0–...

Unix epoch seconds (input only)

%S

x

x

00–60

Seconds (always two digits on output)

%M

x

x

00–60

Minute (always two digits on output)

%k

x

0–24

Hour—24-hour clock (one or two digits on output)

%H

x

x

00–24

Hour—24-hour clock (always two digits on output)

%l

x

0–12

Hour—12-hour clock (one or two digits)

%I

x

00–12

Hour—12-hour clock (always two digits)

%p

x

“am”, “pm”

a.m./p.m. indicator

%j

x

x

001–366

Day of year (always three digits on output)

%d

x

x

01–31

Day of month (always two digits on output)

%m

x

x

01–12

Month (always two digits on output)

%b

x

x (also %h)

Jan, Feb, ...

Abbreviated name of month (3 characters)

%B

x

x

January, February, ...

Full name of month

%y

x

x

00–99

Year (two-digit; always two digits on output)

%Y

x

x

0000–9999

Year (four-digit; always four digits on output)

%w

x

00–06

Day of the week (0=Sunday; always two digits)

%a

x

Sun, Mon, ...

Abbreviated day of week

%A

x

Sunday, Monday, ...

Full day of week

%W

x

00–53

Week of the year (weeks starting on Monday; always two digits)

%U

x

00–53

Week of the year (weeks starting on Sunday; always two digits)

%R

x

e.g. “22:12”

Shorthand for “%H:%M” (24-hour clock time format w/o seconds)

%T

x

e.g. “22:12:48”

Shorthand for “%H:%M:%S” (extended ISO time format: 24-hour clock with seconds)

%r

x

e.g. “10:55:48 pm”

Shorthand for “%I:%M:%S %p” (US time format)

%F

x

e.g. “2008-03-25”

Shorthand for “%Y-%m-%d” (extended ISO date format)

%D

x

e.g. “03/25/08”

Shorthand for “%m/%d/%y” (US date format)

Input

Time/date information is parsed in a way reminiscent of the scanf() family of functions, and shares its familiar challenges.

The expected input format is indicated through a format string to set timefmt. The format string may contain several conversion specifiers, all of which begin with the % character, followed by a letter that indicates how an input value should be interpreted. Check tables 8.4 and 8.5 for a list of all possible conversion specifiers and their meanings.

The input format string may contain other characters besides format specifiers, but input strings must match the format exactly (with some exceptions regarding whitespace we’ll discuss shortly):

set timefmt "%Y-%m-%d"   # will match 2000-01-01, but also 2000-1-1
set timefmt "%d%b%y"     # will match 1JAN05, 01Jan05, 1jan05

If there are no characters separating different fields from one another, gnuplot consumes a fixed number of characters per field (left-to-right), and the fields must be left-zero-padded as necessary:

set timefmt "%Y%m%d" # will parse 20020101 as Jan 01st,
                     # but will parse 2002101 as Oct 01st,
                     # and will fail to parse 200211

Special rules apply when the date format to be parsed contains whitespace. First of all, gnuplot will interpret whitespace-separated data as occupying several columns. A timefmt format containing whitespace in turn will consume several columns. A blank space (not an escaped tab: \t) embedded in a formatting string matches one or more blanks (not tabs) in the input file. So, "%H %M" matches 12␣20 and 12␣␣␣20, but not 1220. (The ␣ symbol indicates a whitespace character.)

An example will help. The following input file

2005-01-01     8:41     3
2005-01-01     9:17     4
2005-01-01    22:46     2
2005-01-02    03:05     5

will be correctly parsed and plotted by the following commands:

set xdata time
set timefmt "%Y-%m-%d %H:%M"
set xtics format "%F\n%T" rotate
plot "data" u 1:3 w lp

Note that the column used for the y values is the third, since the time format consumes two columns. Also, the format string contains a single whitespace, but in the data file several blanks separate the date from the time. The file won’t parse correctly if the spaces between date and time are replaced by tabs.

Finally, gnuplot won’t parse strings enclosed in quotes (see section 4.3.3). Therefore, it’s not possible to parse a file that contains date/time information as strings with embedded whitespace:

"2005-01-01 8:41"     3  # will NOT parse
"2005-01-01 9:17"     4

Also keep in mind that gnuplot capabilities when it comes to parsing date/time information depend on the capabilities (and sensibilities) of the underlying system libraries. Do not expect edge cases to be absolutely portable!

Output

Compared to parsing time/date information, it’s much easier to format it into human-readable tic labels. Simply specify the desired output format using set _tics format "..." (where the underscore again is a placeholder for any of the possible prefixes from table 8.1) and you are done.[18]

The format string can contain arbitrary text besides the formatting characters. Here’s a useful snippet to stack the time on top of the date (note the embedded newline):

set xtics format "%T\n%D"     # Time stacked on top of date

But other text is also possible, such as formatting characters:

set xtics format "%Y-%m-%D %H:%M"     # Date, followed by time

and even plain text:

set xtics format "It happened on %A"     # Full day of week

In particular when used together with string functions, there is almost no limit to the appearance of tic labels for plots displaying time series.

Working in time series mode

Keep in mind that when working in time series mode (after issuing the set xdata time command), all x coordinates will be interpreted according to the current setting of the set timefmt format option (equivalently for all other axes).

In particular, this means that plotting ranges must be specified as quoted strings, in the format given by timefmt (the input time format): plot ["01Jan00":"15Jan00"] "data" u 1:2". Similar concerns hold for the coordinates supplied to set arrow or set label.

Also, don’t forget that the currently selected locale (as inherited from the environment when gnuplot was started, or set using set locale) will affect the tic labels (names of months and of days in week).

And finally, always remember that it is necessary to switch time series mode on using set xdata time. This necessary step is easy to forget, and doing so will lead to mysterious errors.

[Tip]
Don’t forget to turn on time-series mode

Before parsing date/time information from files and using it to plot data, you must enable time-series mode using set xdata time.

A worked example

Remember the file with monthly data from listing8.9? In section 8.3.5, we plotted it while reading the horizontal tic labels directly from the data file. In section 8.4.2, we plotted it using the simple set xmtics formatting. Now, we will plot it a third time, using the full machinery of gnuplot’s general time series mode.

Listing 8.11 shows one way to do it. (The resulting graph is exactly the same as figure 8.8.)

First of all: turn on time series mode using set xdata time.

Select the input format using set timefmt and the output format using set xtics format, using the conversion specifiers from tables 8.4 and 8.5.

Plot the data, using the first (numeric) column as input, while ignoring the strings in the third column entirely.

Listing 8.11. Alternative commands for figure 8.8

set xdata time                      # turn on time series mode!

set timefmt "%m"                    # input format: 1...12
set xtics format "%b"               # output format: Jan, Feb, ...

plot [][0:10] "months" u 1:2 w lp   # plot

Amusingly, there is another way to achieve the same effect. Remember that the data in listing 8.9 is redundant: the first and third column both contain information that can be interpreted as months. We can therefore ignore the first (numeric) column, and base the graph on the third column entirely. Listing 8.12 shows takes this approach.

Listing 8.12. Yet another set of commands for figure 8.8

set xdata time                      # turn on time series mode!

set timefmt "%b"                    # input format: Jan, Feb, ...
set xtics format "%b"               # output format: Jan, Feb, ...

plot [][0:10] "months" u 3:2 w lp   # plot, using the third column!

Notice how the strings in the third column are used to give the horizontal position for each point. (You may want to change the output format to full, rather than abbreviated, month names using set xtics format "%B" to convince yourself that the strings read from the file are actually parsed and interpreted and are not just passed through to the graph.)

8.4.4  Beyond tic labels: processing date/time information

The functionality we have just described for use with time series plots can be extended to more general applications. Most date/time manipulations involve the first three functions in table 8.6.

As an example, consider again the list of athletic records in listing 3.1. This data set contains not only the finishing times, but also the calendar date the record was set. We may want to include this information in a plot of the data (see figure 8.9).

Figure 8.9. Mens’ world records, together with the data the record was set. The dates are read from the data file, reformatted, and placed onto the graph. See text for details and listing 3.1 for the data.

This plot command to create this figure is shown here:

plot [0:1200] "records"
»             u 1:2:(strftime( "%d %b %y", timecolumn(4, "%Y-%m-%d")))
»             w labels point pt 7 offset 0,-1

The primary plot style is with labels (see section 6.3.5.1), using the ability to plot a point symbol together with the label. The labels themselves are first parsed (using timecolumn()), and then formatted (using strftime()). Notice that the date format in the graph is different than in the data file—the date strings are actually parsed and reformatted, not merely passed through to the plot.

Instead of using the timecolumn() function, you can use strptime() in conjunction with the basic stringcolumn() function:

plot [0:1200] "records"
»             u 1:2:(strftime("%d %b %y", strptime("%Y-%m-%d", strcol(4))))
»             w labels point pt 7 offset 0,-1

This example shall suffice to demonstrate the possibilities. Keep in mind that timecolumn() and strptime() return a numeric representation of a timestamp, thus making all kinds of further date calculations and manipulations possible.

Table 8.6. Functions to parse and format arbitrary time strings. See tables 8.4 and 8.5 for conversion specifiers.

Function

Description

strftime( format, seconds )

Formats its second argument, which is interpreted as Unix epoch seconds, according to the format string in the first argument. Returns a formatted string.

strptime( format, string )

Parses its second argument, which must be a string containing a timestamp, according to the format string in the first argument. Returns the corresponding number of Unix epoch seconds as a floating-point number.

timecolumn( column, format )

Takes an integer column identifier and a format string. Parses the current entry in the identified column according to the format string. Returns the corresponding number of Unix epoch seconds. Equivalent to strptime( format, stringcolumn( column ) ).

time(dummy)

Returns the current system time in Unix epoch seconds. A numeric argument must be supplied, but its value is ignored.

tm_year(seconds)

Takes a timestamp in Unix epoch seconds and returns the full year as floating-point number.

tm_yday(seconds)

Takes a timestamp in Unix epoch seconds and returns the day of the year as floating-point number (0.0–365.0).

tm_mon(seconds)

Takes a timestamp in Unix epoch seconds and returns the number of the month as floating-point number (0.0–11.0).

tm_mday(seconds)

Takes a timestamp in Unix epoch seconds and returns the day of the month as floating-point number (1.0–31.0).

tm_wday(seconds)

Takes a timestamp in Unix epoch seconds and returns the day of the week as floating-point number (Sunday=0.0).

tm_hour(seconds)

Takes a timestamp in Unix epoch seconds and returns the hour as floating-point number.

tm_min(seconds)

Takes a timestamp in Unix epoch seconds and returns the minute year as floating-point number.

tm_sec(seconds)

Takes a timestamp in Unix epoch seconds and returns the second year as floating-point number.

[15] This is not entirely correct: it’s only a time series if the time intervals between successive measurements are all equal. But for convenience, we will use the term here to mean any data set that captures the behavior of some quantity over time.

[16] The command set format x is an older alternative for controlling the output format. It is now redundant, but you will still find references to it in the standard reference documentation.

[17] Gnuplot 5 stores timestamps internally as seconds since the beginning of the Unix epoch. This is a change from previous versions of gnuplot.

[18] I also recommend that you avoid the former way of formatting tic labels based on the set format command. The new style, based on set xtics format is both clearer and more robust.

8.5  Summary

In this chapter, we talked about all the ways we can control the appearance of the axes on a plot. It’s been a long chapter, but axes—or rather, the tic marks and labels placed on them—are important: they enable the viewer to gain quantitative insight from the data displayed in the plot.

This chapter concludes our overview of what I would call “basic” gnuplot. In the following chapters, we’ll look at some exciting but distinctly more advanced topics, such as color in graphs, multidimensional plots, and other special-purpose features.

We’ll also take an in-depth look at ways to script and program gnuplot, and learn everything there is about exporting graphs to standard file formats.

9

Color, Style, and Appearance

This chapter covers:

Global appearance options

Color

Lines and points

Size and aspect ratio

Whereas the previous few chapters dealt with very local aspects of a graph (such as an individual curve, a single text label, or the details of tic mark formatting), the present chapter addresses global issues. We begin with a long and important section on color. Color is an exciting topic, and it opens up a whole range of options for data visualization, and we will explain gnuplot’s support for color handling in detail.

Next, we discuss a few details that matter if you want to modify the default appearance of point types, dash patterns, and the automatic selection of visual styles. The overall size and aspect ratio of the entire graph are the subjects of the closing section of this chapter.

9.1  Color

Color is always an exciting topic. At the base level, colors are often easier to distinguish than dash patterns or different symbol shapes. But colors open up many more options for visualization—in fact, colors by themselves are sufficient to represent semantic information (we will come back to this in chapter 11). And finally, color is just plain fun!

At the same time, color adds complexity: what colors do look good, and which are suitable for representing information? And then there is the problem of representing color in computer systems. In chapter 11, we’ll have more to say about choosing colors for visualization, here we are only interested in the technicalities of specifying color in gnuplot.

There are three fundamentally different ways in gnuplot to select a color:

Through explicit specification either by using a known color name (such as "red") or by giving its red-green-blue (RGB) components (like "#FF0000"). An explicit color specification is indicated by the keyword rgb.

By indexed lookup from a selection menu of discrete colors. (For example lc 1 picks the first color from an existing list of color styles). Indexed lookup is the default and does not use a special keyword.

By mapping a value into a continuous color gradient. Color gradients are called palettes in gnuplot terminology (with keyword palette), and are the principal topic of chapter 11. In this chapter, we will give only a brief preview.

Furthermore, each one of these methods can be used in two different ways:

Fixed—A specific color is selected for a given data set (to distinguish this data set from any other).

Variable or data-dependent—The color is determined by the value of the data plotted so that the curve changes its color depending on location. Data-dependent coloring is switched on by using the keyword variable in place of an explicit color name or index, and by palette z for gradient-mapped colors.

Keeping all these possibilities straight will be one of our primary goals in this section!

No matter which way color is used, a color specification is always indicated using the linecolor keyword for graph elements (not just lines, but all graph elements, like points or boxes), and by the keyword textcolor for all forms of text labels.

[Tip]
Color specifiers

Remember: all color specifications are either preceded by the keyword linecolor (or lc) for graph elements, or by textcolor (or tc) for text labels.

9.1.1  Explicit colors

The most fundamental way of indicating a color is by specifying it explicitly. This can be done by giving its name, or by spelling out its components explicitly.

Color names

Several colors can be referred to by their name. Gnuplot knows 111 predefined color names (corresponding to only 95 distinct colors; the duplications are mostly due to both “grey” and “gray” being valid spellings). You can see all color names and their corresponding RGB values using the show colornames command. Note that gnuplot’s list of color names is not identical to the list of of X11 colors.[1]

RGB components

The most common way in computer graphics to indicate an arbitrary color is through its RGB (red-green-blue) values: a triple of numbers giving the relative intensities of the red, green, and blue components that make up the desired color. By convention, each component is represented through a single byte, so that each RGB component lies between 0 and 255. Conventionally, the components are written as two-digit hexadecimal numbers (from 00 to FF), and an entire triple can be represented through a string of the following format:

rgb "#RRGGBB"

This is the “X11 format”. It is also possible to write the string explicitly as a hexadecimal constant in the format "0xRRGGBB". Here are three equivalent ways to specify the same color:

rgb "red"
rgb "#FF0000"
rgb "0xFF0000"

Instead of representing the RGB triple as a string, you can also use the integer value corresponding to the value of the three bytes making up the triple. Below is a function that will take the three components (as integers between 0 and 255) and “pack” them into an equivalent integer value.

pack( r, g, b ) = 2**16*r + 2**8*g + b

This gives us an additional way to obtain the color red:

rgb pack( 255, 0, 0 )

HSV color model

The RGB color model has the advantage that it corresponds directly to the way a computer display operates, but it’s not a particularly intuitive way for a human to use. (Quickly, what does #2e8b57 look like?)

The HSV (hue-saturation-value) scheme is an alternative to the RGB color description and the most popular intuitive color model. Hue describes the actual shade of the color (by convention, a hue value of zero corresponds to red, followed by yellow, green, cyan, blue, and magenta at equal intervals). The saturation measures the richness of the color, from pale pastel shades to full saturation. Finally, the third element of the HSV triple (value) describes the lightness of the color, from very dark to very bright. All HSV values vary from 0 to 1.[2]

Gnuplot’s built-in function hsv2rgb( h, s, v ) takes a triple of HSV values, performs the conversion to RGB, and finally returns an integer-packed representation of the corresponding RGB triple. This gives us yet another way to spell out “red” (remember that red corresponds to a hue value of 0):

rgb hsv2rgb( 0, 1, 1 )

In listing 9.1 we show all these forms together, this time for the color magenta, which is a mixture of full red and full blue components. (Its hue value in the HSV model is 5/6.)

Listing 1.1. Equivalent ways of explicit color specification

rgb "magenta"                # color name
rgb "#FF00FF"                # X11 RGB string
rgb "0xFF00FF"               # string with hex constant
rgb pack( 255, 0, 255 )      # integer packed
rgb hsv2rgb( 5/6., 1, 1 )    # conversion from HSV

Remember that the function hsv2rgb() is a built-in function, whereas the pack() function has only been defined earlier in this section.

[Tip]
The function hsv2rgb() is picky

The input values to the hsv2rgb() function must all lie in the interval [0:1]. The function silently replaces values outside this interval with the nearest legal value (so, 1.1 becomes 1.0 and -0.1 becomes 0). In particular the hue parameter does not wrap around for values greater than 1.0—you must map it into the unit interval yourself!

9.1.2  Alpha shading and transparency

So far, we always assumed that colors are fully opaque, so that an object drawn on top of another completely hides its background. But gnuplot can do more: several terminals allow for transparent colors.[3]

Making colors transparent

To indicate that a pixel should be transparent requires additional information. In practice, one adds a fourth byte to the RGB components; this is often called the alpha channel. In gnuplot’s string representation, it precedes the RGB value:

rgb "#ααRRGGBB"
rgb "0xααRRGGBB"

Transparency information is encoded in the highest order bit as follows:

α = 0   : fully opaque
α = 255 : fully transparent

This convention has the advantage that a fully opaque color rgb "#00RRGGBB" has the same numerical value as rgb "#RRGGBB", so that opaque colors do not require special treatment.

Alpha blending

So, what happens when two transparent colors are overlaid? Obviously, the color “underneath” will still be visible somehow—but how exactly? It turns out that there is no “obvious” answer to this question—in fact, the cairo graphics libraries defines 29 (!) different operators to combine two pixels containing alpha-channel information![4]

Even within gnuplot, the answer is not unique: lines and similar graph elements are combined using cairo’s OVER operator (the library’s default), filled objects use the SATURATE operator. (It is not possible for the user to specify a particular operator explicitly.)

To make the behavior of these two operators concrete, let’s assume that we want to add a new object of color α₁X₁ to an existing color α₀X₀ resulting in color α₂X₂. (Here X stands for any one of the three RGB components.) The new color is then calculated, separately for each RGB component, according to the following formulas:

OVER: α₂ = α₁ + α₀ ( 1 - α₁ ) X₂ = ( α₁ X₁ + α₀ (1 - α₁ ) X₀ ) / α₂ SATURATE: α₂ = min( 1, α₁ + α₀) X₂ = ( min( α₁, 1 - α₀) X₁ +α₀ X₀ )/α₂

These formulas are not so helpful to predict the outcome, but they can be used retrospectively to understand what happened. Two observations hold generally (see figure 9.1):

The two source colors do not commute: it matters which color is in the background and which is newly added. The color that is added on top of an existing background tends to prevail.

When all pixels are of the same hue, things are simpler: then the hue basically stays, only the intensity increases (in a roughly additive way).

Figure 1.1. Alpha blending in action. The color that is added to an existing background tends to prevail; adding it to a background of the same hue just increases the intensity.

Uses for transparent colors and alpha blending

Transparent colors can be useful when visualizing large data sets, consisting of many points. If such data is plotted in a scatter plot, points will frequently overlap. With fully opaque colors, overlapping points completely obscure each other, making the local density of points indistinguishable. But with partially transparent colors, overlapping points contribute additively, so that a higher color intensity reveals a higher point density.

Figure 1.2. Using alpha blending to visualize point density in a dense data set. All data points are drawn both with blue disks and red rims. Because contributions from overlapping points add visually, regions of high point density show up as areas of high color intensity. Regions where the density is high enough for the rims to contribute significantly appear red.

Figure 9.2 shows what such a scatter plot may look like. The plot command was:

plot [-0.4:1.2][-0.2:1.2] "alpha" u 2:1 w p pt 7 ps 1 lc rgb "#cc0000ff",
»                         "" u 2:1 w p pt 6 ps 1 lc rgb "#ddff0000"

All points are first drawn as filled circles (pt 7) in blue, and then the again as unfilled circles (pt 6) in red. Adding an outline to the circles significantly improves the appearance of the graph; giving the outline a different color sets off areas of truly high point density even more strikingly.

You may want to experiment further with this—for example, I have obtained good results when using the same color (say, blue) for both disk and outline, but making the outline slightly larger than the disk itself (for example ps 1.5 for the outline, but ps 1 for the disk).

9.1.3  Selecting a color through indexed lookup

The methods explained so far in this section allow the user to select individual colors, and when that’s what you want, they are very convenient: plot sin(x) lc rgb "red" does exactly that: it plots the sine in red.

It gets more tricky if you want to plot several curves all at once. To distinguish them, each should be plotted in a different color, and it would be convenient if gnuplot would do so automatically. In other words, now we want colors to be chosen from a discrete selection (rather than specified individually).

We have already seen that this is exactly what gnuplot does (see section 6.2): gnuplot maintains a set of distinct colors, and (by default) uses the next one for each new function or data set. (Later in this chapter we will learn how to customize the overall selection of colors available in this way.)

Colors (and line types and styles more generally) are identified by a positive integer. If you want to override gnuplot’s automatic choice of color, or if you simply want to pick a particular color from the predefined set, you do so by supplying the appropriate integer value:

plot "data" u 1:2 w l lc 2, "" u 1:3 w l lc 2

This plots both the second and third column against the first using the second color from the predefined set of colors for both data sets.

All of this is extremely straightforward. There is only one possible glitch: don’t confuse lc 1 with lc rgb 1. The former specifies the first color from the selection of predefined colors (whatever actual color this may be!). The latter selects a specific color, by giving its RGB components as integer-packed value. (The value 1 corresponds to "#000001", which is almost black.)

Special color indices

Gnuplot knows about two “special” color indices: black and bgnd. The first is a shorthand to obtain a black line through indexed lookup (that is, without having to identify a color explicitly). It is mostly convenient for drawing graph elements like arrows, borders, and labels: set arrow 1 from 0,0 to 1,1 lc black. (Note the absence of either quotes or the rgb keyword.)

Some terminals allow an explicit background color to be set using the background keyword: set terminal pdfcairo background rgb "grey" (see chapter 14). This color is then available through the bgnd index for all graphs sent to this terminal. This is mostly useful to “rub out” or erase parts of the graph:

set object 1 circle at 0,0 radius 1 fill solid fc bgnd front

This will clear out a circular area of radius 1, centered at the origin, by filling it with the background color.

9.1.4  Mapping a value into a continuous gradient

Continuous color gradients (called palettes in gnuplot) will be the topic of chapter 11. But a discussion of color in gnuplot would be incomplete without mentioning them, so I will give a brief preview here.

First, we need to define a gradient, for example like this:

set palette defined ( 0 'blue', 0.5 'white', 1 'red' )

This defines a color gradient from blue, over white, to red; white is centrally located in the mapped interval. We can now use any of the colors from this gradient in a plot like this:

plot sin(x) lc palette frac 0.25

This will plot the curve in the color which is half-way between “blue” and “white”. (The keyword palette can be abbreviated to pal.)

When selecting a color from a palette using the frac keyword, then the numeric argument must lie in the interval [0:1]. It is also possible to map the gradient to a different interval using set cbrange. (The identifier cb stands for “color box”.) For example, set cbrange [-5:5] maps the color gradient into that interval. We can now specify a color by providing a value from this interval together with the cb keyword. The following plot command will produce the same curve as in the previous listing (because -2.5 lies at the same relative position within the interval [-5:5] as 0.25 lies within the interval [0:1]):

plot sin(x) lc pal cb -2.5

Remapping a gradient to a specific numeric range makes sense when you want to indicate a specific numeric value with a color. That will be the topic of the next section, and—in much more detail—of chapter 11.

9.1.5  Using data-dependent colors

Gnuplot can change the color of a line or point, based on the data itself. As usual, there are three ways to do this: explicitly (using the rgb keyword), by indexed lookup (using an integer index), and using a gradient (with the palette keyword). In any case, an additional column must be supplied with the using specification of the plot command; the color information is always taken from the last (right-most) column in the using specification. (Value-dependent color is only available for data—if you want to use it with functions, you have to fake it using the "+" pseudofile—see section 4.4).

Explicit and indexed

Let’s consider an example. Here is a very short data file:

# x     y       color
1       2.0     1
2       2.1     2
3       1.9     3
4       2.0     4
5       2.0     5

Now, either one of the following two commands will work:

plot "data" u 1:2:3 w lp lc rgb variable     # explicit
plot "data" u 1:2:3 w lp lc variable         # indexed

The first thing to notice is that although the plot style is linespoints, we nevertheless supply three columns: the last (right-most) will be used for color information. The next thing to notice is the keyword variable in the color specification: that’s what turns on data-dependent coloring. If variable follows the rgb keyword, the value in the third (color) column is taken as an integer-packed RGB (or αRGB) tuple; if variable is used by itself, the entry in the color column is taken as the index of the color to use. (It is not possible to use either color names or RGB strings together with variable.)

A worked example using explicit data-dependent colors

In practice, you will almost never read either the color value or the color index from a file like this. Instead, think of the third column in the using specification as a place to perform some creative inline transformation (see section 3.3) to calculate a data-dependent color. Let’s consider a few examples.

Remember the DLA cluster in figure 1.2? For each of the particles, we not only know its coordinates, but also the sequence order in which it was added to the cluster. If we map the latter information onto a color spectrum, we can generate “tree rings” that indicate how the cluster grew (figure 9.3).

An easy way to turn sequence numbers into colors is to map them onto the hue value in the HSV model. Using the hsv2rgb() built-in function, this can be done on the fly:

plot "dla-cluster" u 2:3:(hsv2rgb($1/30000.,1,1)) w d lc rgb var

This graphs shows clearly that a DLA cluster tends to grow mostly at the rim: despite the tenuous structure, it is rare for a particle to penetrate deeply into an existing cluster; it is much more likely that it will get caught by one of the branches extending outward.

Figure 1.3. A DLA cluster (see section 1.1.2), using the HSV color model to indicate the order in which particles were added to the cluster.

Although it served us reasonably well in this example, in general you should use caution when mapping a quantity to the “rainbow” spectrum of the HSV color model. One problem with it is that it does not have any inherent sense of ordering: is yellow larger than blue or vice versa? Another problem is that the spectrum closes in on itself: red does represent both very small and very large values. In this example, it does not matter, because the geometric structure of the cluster provides the required sense of ordering, but in more general cases, a “rainbow” plot like this can be highly confusing. Carefully designed color gradients are almost always a better idea. Thankfully, gnuplot makes this easy—a topic we will come back to in chapter 11.

Worked examples using indexed data-dependent colors

Indexed data-dependent colors make sense when the data contains some attribute that assumes only a discrete set of values (or “levels”). In listing 9.3, some attributes (such as the physical dimensions, the year of presentation, and the available horsepowers) are all numeric and can have essentially any value. But others, such as the number of cylinders and the country of manufacture, assume only a finite number of discrete values. Those can be represented through distinct, indexed colors. (See listing 9.2 for the commands and figure 9.4 for the resulting plot.)

Listing 1.2. Command for figure 9.4 using data from listing 9.3

unset key
plot [2003:2014][500:1400] "supercars" u 1:2:($3/4-1) pt 7 ps 1.5 lc var,
»                          "" u 1:2:8 w labels offset 0,-1

Listing 1.3. Data for figure 9.4 (supercars)

# Year  PS      Cyl     Len     Height  Weight  Country Name
2005    1014    16      4462    1159    1888    France  "Bugatti Veyron"
2006    1061    8       4475    1092    1270    USA     "SSC Aero"
2012    1261    8       4655    1079.5  1244    UK      "Hennessey Venom GT"
2011    940     8       4293    1120    1435    Sweden  "Koenigsegg Agera"
2005    806     8       4293    1120    1456    Sweden  "Koenigsegg CCX"
2009    1104    8       4665    1198    1688    Denmark "Zenvo ST1"
2005    650     8       4460    1114    1100    Germany "Gumpert Apollo"
2009    750     12      4601    1222    1630    UK      "Aston Martin 0ne-77"
2011    700     12      4780    1136    1575    Italy   "Lamborghini Aventador"

Figure 1.4. Using indexed color to visualize discrete values or “levels” in a data set. For a set of current “supercars”, the date of presentation and the available horsepowers are visualized through the location on the graph, whereas the number of cylinders is represented via color. Compare listing 9.3 for the data and listing 9.2 for the commands.

But data-dependent indexed colors can also be used to represent metadata. Here is a clever trick:[5] consider a data file containing several data sets, separated by double blank lines. We learned already (section 4.4) that the index of each data set is available in the pseudocolumn -2. We can use this information to let gnuplot select a different color for each data set, by using the value of the pseudocolumn as index into the set of predefined colors.

plot "data" u 1:2:(1+column(-2)) w lp lc var

Because this example uses indexed lookup, the keyword rgb is absent. We need to use the column() function, because the dollar-sign shorthand does not work for a negative column number. And finally, we need to add 1 to the return value, because data sets within a file are counted starting at 0, whereas line colors are counted starting at 1.

Gradient

Indexed and explicit colors work well for discrete colors; if you want a curve that changes color continuously, then color gradients are the way to go. Using data-dependent color with a continuous gradient is similar to what we have seen before, except that the keyword now is z instead of variable. Assuming an appropriate color gradient has been defined, the following command will color the curve according to the gradient. Observe again that you need to specify a third column for the color information.

plot "data" u 1:2:3 w l lc palette z

Figure 9.5 shows a way that a color gradient can be used with a regular curve. The idea is that this curve monitors some important quantity which must not exceed some threshold. The graph uses a color gradient to change the appearance of the curve depending on the value of the monitored quantity. It is easy to set up the gradient and create the plot (more details will follow in chapter 11):

set palette defined ( 0 'green', 3 'green', 4 'red', 5 'red' )
plot [][0:5] "monitoring" u 1:2:2 w l palette z lw 2

Figure 1.5. Gradient-mapped color. The control curve changes its color if the monitored quantity raises above its safe value. See text for details.

9.1.6  The built-in color sequences

For many years, the default color sequence for the most common terminals consisted of red, green, blue, followed by magenta, cyan, and yellow: simply because these are the three basic colors of computer graphics, and their binary combinations. It’s a simple sequence, but not a very good one: the colors are bright and garish, and yellow is exceptionally difficult to see against a white background. Moreover, this sequence of colors (with its emphasis on red and green) is particularly challenging for people that are color-blind! There are also more subtle problems, having to do with color perception: the blue line, for example, always seems “heavier” than any of the other colors, and tends to dominate a graph.

In gnuplot 5, you can choose from three different color sequences, using the set colorsequence command.

set colorsequence [ default | classic | podo ]

The three color schemes are identified using keywords:

default

A terminal-independent sequence of 8 colors.

classic

The original behavior, which differs from terminal to terminal, but usually produces of the red/green/blue sequence mentioned earlier.

podo

A terminal-independent sequence of 8 colors, intended to be easily distinguishable for people that are color-blind.

Unfortunately, none of these three options amounts to a significant improvement, in my opinion: the new default scheme, for instance, may be much less garish than the classic scheme, but its colors are so muted that I at least now find it difficult to distinguish different lines to begin with! The set colorsequence facility is also strangely limited (for example, it is not possible to extend or modify the provided color choices), so you’ll probably want to construct your own sequence of colors (and other line properties) from scratch. That will be the topic of section 9.3.

9.1.7  Tips and tricks

Color specifications tend to be clumsy: both lengthy and obscure. It may therefore be a good idea to introduce some shorthands. Here are some ideas to get you started.

Instead of using a hex string, you can always assign its value to a semantically named variable:

gold = "#FFD700"                # hex string
gold = pack( 0xff, 0xd7, 0 )    # equivalent: integer value

plot "data" u 1:2 w l lc rgb gold

This is not ideal, because you still have to type out the keyword rgb every time. You can absorb it into the variable if you make use of gnuplot’s macro facility (see section 5.2):

blue = "rgb '#0000ff'"
plot "data" u 1:2 w l lc @blue

In fact, you can even include the lc into the macro itself:

green = "lc rgb 'green'"
plot "data" u 1:2 w l @green

Rather than using color names, you may also want to consider semantic names for colors:

alarm  = "red"
normal = "#00cd00"

The options are endless—adapt them to your needs.

[1] The Wikipedia page for X11 color names contains much useful information.

[2] The Wikipedia page for the HSL and HSV color models provides more detail.

[3] All cairo-based terminals support transparency, but several of the older terminals do not. In particular, neither the Postscript nor the GD-based bitmap terminals can handle transparency.

[4] http://cairographics.org/operators/

[5] The idea for this application was taken from the standard gnuplot reference documentation.

9.2  Lines and points

Gnuplot graphs are constructed from lines and points: these are the fundamental building blocks. (In keeping with the usage in the gnuplot documentation, the term “point” here and in the following refers to the symbol that is used to represent a single data point.) All aspects of lines and points can be customized: the width, color, and dash pattern of lines, and the shape, size, and color of points.

This is not news—we introduced all these concepts at the beginning of chapter 6 and have been using them ever since. (Table 6.1 provides a handy summary.) Most of these properties are pretty straightforward and won’t need explanation beyond what you can find in the table. But there are a few exceptions: the selection of available point types deserves a few words, and so do user-defined dash patterns.

9.2.1  Point types and shapes

Each terminal provides a set of shapes or points that can be used to represent a single data point. Because these shapes are relatively low-level graph elements, they are hard-coded into the implementation of each terminal driver. It is not possible for a gnuplot user to change the existing shapes or add new ones.[6] All commonly used terminals provide at least a dozen (or so) different shapes, but some terminals define many more (the old Postscript terminal defined 75 distinct point types!). In practice, less than about 10 point types are sufficiently different that they can be distinguished visually.

The sequence of shapes has been unified across all (contemporary) terminals, at least for the first few point types (see figure 9.6. Although the shape of the symbol cannot be changed, both its size and its color can be modified, using the pointsize (or ps) and linecolor (or lc) properties.

Figure 1.6. Available point symbols

Point shapes are usually indicated using their index: plot ... w p pt 7 will represent each data point by a filled circle. You may want to define symbolic names for point types:

frame = 4      # square, empty
box   = 5      # square, filled
circ  = 6      # circle, empty
disk  = 7      # circle, filled

plot "data" u 1:2 w p pt frame     # use empty squares

9.2.2  Dash pattern

The ability to customize dash patterns is a new feature in gnuplot 5; support in older terminals may be spotty. A dash pattern can either be specified inline, or defined separately using set dashtype. Once a dash pattern has been defined in this way, it can be referenced by its (integer) index.

There are two ways to specify a dash pattern: using a symbolic string, or using a tuple giving the relative lengths of solid and empty line segments.

Symbolic string

Dash patterns can be specified through a symbolic string, consisting of the symbols shown in table 9.1. The dash pattern can consist of up to four non-whitespace characters; leading whitespace and any further characters are ignored. Figure 9.7 shows some examples.

Symbolic strings can conveniently be used inline in a plot command. The following two commands are equivalent:

plot sin(x) dt '-'  # inline

set dashtype 1 '-'  # definition
plot sin(x) dt 1    # ... and indexed lookup

Table 1.1. Symbols for dash patterns

Symbol

Numeric

Description

. (dot)

(2,5)

A dot, followed by a short space.

- (dash)

(10,10)

A dash, followed by a space.

_ (underscore)

(20,10)

A long dash, followed by a space.

␣ (whitespace)

(...,10)

A space.

Figure 1.7. Top: Dashed lines resulting from various combinations of the standard dash symbols or given by an explicit numeric tuple (as shown on the left). Bottom: Effect of the different line cap styles. The actual line end and the added “cap” are shown in different colors.

Explicit tuple

The most general way to specify a dash pattern is to supply a tuple of numeric values. These values represent the length of subsequent solid and empty line segments. There can be at most eight elements in the tuple (so that a dash pattern can have at most four solid parts); the first entry in the tuple is interpreted as solid. Below are two examples (compare figure 9.7).

set dashtype 1 (2,2)             # dotted line
set dashtype 2 (20,10,2,4,2,10)  # long dash, followed by double dot

Lengths of line segments are measured in units of the linewidth; a tuple entry of 2 (not 1!) will result in a line segment that is as long as it is wide.

Or so it may appear. If you are using very short line segments, you are likely to run into an obscure subtlety: line caps. The cairo library defines three types of line ends: butt, round, and square (see figure 9.7, bottom)[7] which can be selected when choosing a cairo-based gnuplot terminal (see section 14.2). Unless the chosen style is butt, the line extends by half a linewidth past its actual end point. But the default style for the cairo-based terminals in gnuplot is square, meaning that all solid line segments in a dash pattern will appear longer by half the linewidth at both ends—and the empty segments will appear that much shorter. You now have two options: select the butt option when setting the terminal (for example, set terminal wxt butt) or take the extent of the line caps into account when setting up the dash pattern. (You may find it amusing to work out why the dash pattern (2,2) using butt line caps is equivalent to the dash pattern (0,4) when using square.)

[6] You may be able to use symbols beyond the ones provided by gnuplot itself by using a single character from a suitable font as point symbol—see section 6.3.1.

[7] http://cairographics.org/samples/set_line_cap/

9.3  Customizing color, dash, and point sequences

Let’s first establish when a predefined sequence of colors (and possibly of other line properties, such as dash patterns) is relevant: it comes into play only if you are plotting many functions (or data sets) and you want to let gnuplot automatically choose a different line color (or pattern) for each new curve. If you do not make use of gnuplot’s automatic style selection (because you specify the plotting style explicitly for each curve instead), then the existence of a predefined color sequence is immaterial. The topic of this section therefore applies only to a very limited purpose.

That being said, letting gnuplot choose line styles automatically is such a huge convenience (in particular when doing casual and exploratory work) that it can not simply be ignored. (It matters less when you are preparing presentation graphics, because in that case you naturally want to have full control over all aspects of the plot, anyway.)

Gnuplot’s automatic style selector chooses from a sequence of line types. Line types are gnuplot’s original method for specifying appearance options, and nowadays sit rather awkwardly next to the more modern paradigm (based on the set style facility; see section 9.4 later in this chapter). Line types are still essential for gnuplot’s automatic assignment of visual styles to different curves, but otherwise have lost much of their previous importance (in particular, line styles are no longer used as “containers” for general appearance options).

A line type is a combination of all graphic line attributes together with a point type and the other point attributes. By construction, every terminal provides an (in principle) infinite sequence of different line types. This guarantees that gnuplot never “runs out of” different styles, no matter how many curves you plot. (In practice, not all of these line types look different, of course—gnuplot uses only about a dozen different colors and dash patterns before it repeats itself. But every new line gets its own line type, with a unique numerical index.)

9.3.1  Customizing line types

You can customize the default sequence of colors and other properties using the set linetype command:

set linetype {int:idx} [ lineoptions ]

Here, the first argument is the index of the line type that you wish to modify; it is followed by specifications of graphical attributes (line color and width, point type and size, and so on—see section 6.2.3.2 for an explanation of the lineoptions). The terminal-specific default values prevail for those attributes that are not changed explicitly.

[Tip]
Line types are persistent

Line types are persistent: they prevail until changed explicitly. In particular, line types are not affected by calls to reset.

In practice, you will only want to redefine a few (the first few) line styles, and have gnuplot cycle through them repeatedly when plotting many curves. But because the number of available line types is always infinite, gnuplot has no way of knowing when to begin cycling: you have to tell gnuplot explicitly. That’s done with the command set linetype cycle:

set linetype cycle {int:period}

Let’s consider an example: you want curves to be drawn alternatingly with thin red and wide black lines (only two styles, in other words); the red line should use an empty square as point symbol and the black line a filled circle (that’s pt 4 and pt 7, respectively—see figure 9.6). The following commands will achieve this behavior. (Because there are odd precedence rules between explicit customizations and set linetype cycle it is necessary to wipe out previous settings first.)

unset for [i=1:32] linetype i            # erase any existing customizations
set linetype 1 lw 1 lc rgb "red" pt 4
set linetype 2 lw 2 lc rgb "black" pt 7
set linetype cycle 2

Unfortunately, the set linetype cycle facility applies only to line properties, but not to point types! It also obeys non-obvious precedence rules, which make its behavior surprising and hard to predict.[8] My recommendation, therefore, is to avoid it altogether, and instead set all line types explicitly:

set for [i=1:64:2] linetype i lw 1 lc rgb "red" pt 4
set for [i=2:64:2] linetype i lw 2 lc rgb "black" pt 7

This will give you the desired behavior for the first 64 curves you include in a graph (and, of course, you can always increase the upper limit in the loop as far as necessary.) The same method can be used, even if you want to use more than two distinct line types—just increase the step size in the loop accordingly!

9.3.2  Special line types

Every terminal defines two “special” line types, which cannot be modified by the user. See table 9.2.

The linetype 0 is frequently useful to add reference or grid lines to a plot. The linetype -1 can be used to add arrows or other decorations, having the guarantee that they appear solid black.

Table 1.2. Special line types defined by all terminals

Line type

Description

linetype 0

A minimally visible line, often a thin, dotted line in black or gray.

linetype -1

A solid black line.

[8] At the time of this writing (December 2014), this whole area of features appeared to be somewhat in flux. Check the gnuplot standard reference documentation for up-to-date details.

9.4  Global styles

All graphical aspects of plot elements can be controlled inline, by spelling out the desired appearance options, right there as part of the plot (or the set arrow or set object) command. But it is also possible to define global styles. There are three primary reasons for doing so:

Convenience—Instead of listing all the various desired options (possibly more than once, if you plot more than one data set or have more than one arrow or other decoration), you can simply refer to a style by its identifier.

Semantics—You can set up and refer to styles based on their function (such as “important”—maybe in bright red, or “reference”—possibly as a dotted hairline).

Display targeting—Imagine that you need graphs that look good in different contexts: clear and crisp on the screen, bold and bright in a presentation, and black-and-white in a print publication! Using collections of styles (“stylesheets”) for each intended use, this becomes much easier than having to redo all graphs by hand.

Styles are created using the set style command and can be defined for lines and points, arrows, objects, and so on. In addition to defining global styles, the set style is also used to fix the options on some special-purpose plots—we will not discuss these applications here but in their proper context (chapter 12).

9.4.1  Data and function styles

By default, functions are plotted with lines (with lines), files are plotted using both lines and points (with linespoints). These defaults can be changed using:

set style data ...
set style function ...

For example, to plot data using with lines, you would say: set style data lines.

9.4.2  Line styles

Line styles are defined using set style line: it takes an integer index of the style you want to create or change, followed by a combination of line options that define the aspects of the style (see section 6.2.3.2 for the lineoptions shorthand).

set style line {int:index} [ lineoptions ]

set style line {int:index} default

Unless explicitly overridden, all aspects of a line style default to the corresponding value of the line type of the same index. The keyword default restores a line style to the equivalent line type.

This begs the question: what’s really the difference between line styles and line types? In brief:

Line types are persistent: all of their properties prevail until explicitly changed. Every terminal provides an infinite number of line types and their properties. When assigning lines to curves automatically, gnuplot chooses from the available line types only.

Line styles are not persistent: they are deleted by calls to reset. The properties of line styles default to those of the line type with the same index, unless explicitly redefined. Line types are not available for automatic assignment to data sets and functions.

I use the following little mnemonic to remember which is which:

[Tip]
Distinguishing line types and line styles

Line-T-ypes are sys-T-em settings.

Line-S-tyles are u-S-er preferences.

My recommendation is to prefer line styles over line types whenever possible and to restrict the use line types to the automatic selection feature (where line types are required).

Here are some examples of line styles that give you an idea how line styles can be used:

solid     = 100
dotted    = 101
important = 102
reference = 103
strut     = 104

set style line solid lw 1 lc rgb "green" pt 5
set style line dotted lw 1 lc rgb "blue" dt '.'

set style line important lw 2 lc rgb "red" pt 0
set style line reference lw 0.5 lc rgb "grey" dt (2,4)

set style line strut lw 2 lc black

9.4.3  Arrow styles

You can predefine arrow styles using set style arrow. The available appearance options are summarized below—they should all be familiar from section 7.3.2 in chapter 7.

set style arrow {idx:index} default

set style arrow [{idx:tag}] [ nohead | head | heads | backhead ]
                            [ size {flt:length} [,{flt:angle}]
                              [,{flt:backangle}]
                              [fixed] ]
                            [ nofilled | empty | filled | noborder ]
                            [ front | back ]
                            [ lineoptions ]

Given all the different appearance options for arrows (in particular with regards to arrow heads), it may make sense to define common arrow types, for example like so:

nohead  = 100
filled  = 101
outline = 102
bold    = 103

set style arrow nohead nohead
set style arrow filled size screen 0.025, 30, 45 filled
set style arrow outline size screen 0.025, 30, 45 empty
set style arrow bold lw 2 lc rgb "red"

Of course, the use of symbolic names for the arrow styles is entirely optional. You can now create an arrow by simply referencing one of these styles:

set arrow 1 from 0,0 to 1,1 as filled   # use symbolic name
set arrow 1 from 0,0 to -1,0 as 103     # use explicit style index

Remember that it is a good idea to use the screen coordinate system to define the arrow head: this way, the head size will not change if the plot range changes!

9.4.4  Fill styles

Several plot styles and decorations can be filled with a color or pattern. In this section, we will introduce the required syntax. The three pertinent keywords are fillcolor (or fc), fillstyle (or fs), and border (or bo).

Two of these do not present surprises: fillcolor must be followed by a color specification (see 9.1 earlier in this chapter), border must be followed by any combination of line options (see 6.2.3.2). The keyword noborder suppresses the border. That leaves fillstyle, which does warrant some discussion.

There are three forms of fill specifiers:

fillstyle empty
fillstyle [transparent] solid [{flt:density}]
fillstyle [transparent] pattern [{idx:style}]

The first one leaves the area empty. The second one fills the area with color; an optional numerical argument between 0 and 1 is interpreted as density, with 1.0 standing for full intensity and 0.0 for an invisible fill color. If the optional transparent keyword is present, then the density value is interpreted as the value of the α channel for the fill color.

It is also possible to fill closed areas with cross-hatched patterns, that is what the pattern keyword does. All contemporary terminals define a set of eight different patterns (use the test command to see them—the test command will be formally introduced in section 14.1), you select one of them through an integer argument after pattern. If the transparent keyword is present, then the background remains visible through the grid pattern.

[Tip]
Use an explicit fill color

Areas may appear unfilled, even if an explicit fill style has been set using fillstyle, unless a fill color has also been selected using fillcolor.

Using set style fill, you can set a global default for many filled styles, such as with boxes, with candlesticks, with filledcurves (all chapter 6), and with histograms (chapter 12). It also becomes the default fill style for circles, ellipses, and polygons—but not for rectangles: for those, you have to use their own set style rectangle command. See the standard reference documentation for details.

9.4.5  Other global styles

Besides the applications just discussed, the set style command can be used to express other global appearance preferences. In particular, it is possible to create predefined styles for objects (or shapes), in a similar way as the one we have discussed for arrows. Check the gnuplot standard reference documentation for set style circle, set style ellipse, and set style rectangle.

The other available applications of set style are more specific. The command set style textbox applies only to boxed labels (see section 7.3.3). Again, the standard reference has the details.

Finally, set style boxplot and set style histogram apply only to special-purpose plotting styles that we have not yet introduced; we will do so in a chapter 12.

9.5  Overall appearance: aspect ratio and borders

Besides the graph elements that go onto a plot, there are some overall features that determine its general appearance: its size and aspect ratio, and the presence or absence of borders and margins.

9.5.1  Size and aspect ratio

We have to distinguish between the size of the plot and the size of the canvas. The latter determines the size of the output file (or screen), while the former affects the size of the graph on the canvas. The canvas size is specified using the

set terminal {enum:terminaltype} size {flt:x},{flt:y}

command, which we’ll discuss in chapter 14 on files and output devices. Here, we’re interested in the size of the plot on the screen or canvas, which is controlled by the set size command:

set size [ [no]square | ratio {flt:r} | noratio ]
         [ {flt:x} [,{flt:y}] ]

The numeric values x, y scale the plot with respect to the screen. For values less than 1.0, the graph won’t fill the entire screen; for values greater than 1.0, only part of the graph will be shown. If y is omitted, the plot will be scaled in both directions by x. For example, set size 0.5 will reduce the extent of the plot by half along both axes, while keeping the overall canvas size fixed.

If the size of the graph is smaller than the size of the canvas, it’s not clear where on the canvas the graph should be located. The graph’s position on the screen is fixed using the set origin option:

set origin {flt:x},{flt:y}

The arguments give the origin of the graph in screen coordinates (see figure 7.2 in chapter 7 for an overview of gnuplot’s coordinate systems).

The ratio option and its relatives (square and noratio) provide a shorthand to control the aspect ratio of the plot; that is, the ratio of the y axis length to the x axis length. Setting set size ratio 2 will therefore result in a plot that’s twice as tall as wide, and set size square is a synonym for set size ratio 1.

Negative values for the ratio have a special meaning: set size ratio -1 will scale the graph so that the length of the x axis unit is equal to the length of the y axis unit. For ratio -2, the y axis unit has twice the length of the x axis unit; for ratio -0.5, the x axis unit is twice the y axis unit’s length. Figure 9.8 demonstrates all these settings.

Figure 1.8. Controlling overall image size and aspect ratio with set size. Clockwise from top left: default settings, aspect ratio 10/7; set size 0.5, reduced plot on full-size canvas, default aspect ratio; set size ratio 1, full plot size, aspect ratio 1:1; set size ratio -1, full plot size, x unit with same apparent length as y unit.

With nosquare and noratio, you reset the aspect ratio of the graph to the default value of the terminal. Note that neither command resets the scale given by x, y.

It’s possible to give contradictory hints to gnuplot when prescribing both scale values in addition to the aspect ratio. In such cases, gnuplot attempts to maintain the desired aspect ratio without exceeding the minimum area given by the scale values. It’s always possible to avoid these situations by fixing only two of the values. My recommendation is to control the size of the plot on the canvas through the x or y scale values, and to control the aspect ratio using the ratio option. The effect of both of these options is intuitively clear, and no ambiguity is possible.

9.5.2  Borders

By default, gnuplot draws borders on all four sides of the plot. This is actually quite suitable if the graph is going to be used in a larger text document (because it gives the plot a “frame” separating it from surrounding text), but can feel cluttered for a standalone diagram. And it’s simply unsuitable for some special diagrams that don’t fit neatly into a rectangular box. For these reasons, gnuplot gives you the ability to control all parts of the border, using the set border option:

set border [ {int:mask} ] [ front | back | behind ] [ lineoptions ]

The only argument that isn’t familiar by now is mask (see figure 7.3 for layers section 6.2.3.2). This argument is a bit-mask in form of an integer, each bit referring to one side of the border (see figure 9.9). The bit positions are counted clockwise from the bottom, so that a mask value of 1 refers to the bottom, 2 to the left, 4 to the top, and 8 to the right, so that 15 (1+2+4+8) paints borders on all four sides. (There are additional bits used for three-dimensional plots using splot—we’ll discuss them in chapter 10.) To draw borders on some, but not all sides, we just need to switch on only those that we want to see:

set border 1+2  # Switch on bottom and left border
set border 3    # Same thing

If tic marks are drawn on the border (as is the default), unsetting the border does not unset the tic marks: they must be controlled separately using options discussed in chapter 8.

Figure 1.9. The set size command takes as first argument an integer that encodes the choice of borders to switch on in the form of a bit mask. The graph indicates which borders correspond to which bit (in decimal representation) and also gives the overall mask value for two common cases: borders on all sides and borders on the bottom and left.

Clipping data points close to the borders

The set clip option controls how gnuplot plots points that are too close to or outside the plot area:

set clip points
set clip [ one | two ]

The first version, set clip points, is only relevant when using a style that shows discrete plotting symbols (such as with points, with linespoints, and so on). If it’s active, symbols that would touch or overlap the borders of the plot are suppressed. Exactly how many points are clipped depends on the symbol size: for larger symbols, more points need to be clipped. (See chapter 6 for more detail on styles and ways to influence the symbol size.)

The second version controls how gnuplot plots line segments connecting points if at least one of the points falls outside the currently visible plot range. If set clip one is active, line segments are drawn if at least one of the end points falls into the visible plot range. If set clip two is active, line segments are drawn even if both end points are outside the current plot range, but a straight line connecting them crosses the visible range. In no case are parts of the line segment drawn outside the visible range. By default, set clip one is on, but set clip two is off.

9.5.3  Margins

The empty areas outside a plot’s borders make up the margins (see figure 7.2). By default, gnuplot calculates the size of the margins automatically, based on the presence or absence of tics, axis labels, the plot title, and other decorations placed outside the borders. If necessary, you can fix margins manually, using the set margin command.

Traditionally, there have been four different commands, for each of the margins separately. Gnuplot 5 also contains a version that combines all four into one:

set bmargin [ {flt:chars} | screen {flt:frac} ]   # bottom margin
set lmargin [ {flt:chars} | screen {flt:frac} ]   # left margin
set tmargin [ {flt:chars} | screen {flt:frac} ]   # top margin
set rmargin [ {flt:chars} | screen {flt:frac} ]   # right margin

set margin [ {left}, {right}, {top}, {bottom} ]

show margin                    # show current settings

By default, the arguments are interpreted as the desired margin in character widths (or heights). If preceded by the keyword screen, then the graph boundary will be placed at the specified absolute position, interpreted in screen coordinates: this will ignore the current values of set size and set origin. (See figure 7.2 for the meaning of screen; see section 9.5.1 for set size and set origin.) In any case, a missing or negative argument restores automatic margin sizing.

9.5.4  Internal variables

You can inspect gnuplot’s internal size and margin information by examining the values of several internal variables. (See section 5.5 for general information on internal variables.) The variables that are of interest in the current context all have the prefix GPVAL_TERM_, we can list them (and their values) as follows:

show variables GPVAL_TERM_

There are seven such variables; table 9.3 explains what they represent.

Frequently, the numerical value of these variables is less useful; what really matters are various ratios. For example, the position of the left border in screen coordinates (see figure 7.2 for the different coordinate systems) using (adding 0.0 is necessary to force floating-point evaluation of the fraction):

(0.0 + GPVAL_TERM_XMIN)/GPVAL_TERM_XSIZE

Table 1.3. Internal variables that contain information about the size of the canvas and the position of the borders.

Variable name

Description

GPVAL_TERM_XSIZE, GPVAL_TERM_YSIZE

Size of the entire canvas, in horizontal and vertical direction.

GPVAL_TERM_XMIN, GPVAL_TERM_YMIN

Positions of the bottom and left border with respect to the canvas. These are the lower limits of the the graph in both directions.

GPVAL_TERM_XMAX, GPVAL_TERM_YMAX

Positions of the top and right border with respect to the canvas. These are the upper limits of the the graph in both directions.

GPVAL_TERM_SCALE

TBD

9.6  Summary

In this chapter, we focussed less on individual graph elements and their properties, and instead discussed several topics that influence the overall appearance of a graph. Color, and the way it is made available in gnuplot occupied the first part of this chapter. Our discussion here was pretty technical; in chapter 11 we will return to the color topic, but then we will emphasize how to employ color for visualization tasks.

Next, we described how the appearance of lines and points can be controlled in minute aspects. Gnuplot 5 gives you the ability to define custom dash patterns for lines and we explained how to do that.

The last part of this chapter explained how to control the overall appearance of a graph, such as the presence or absence of graphical “borders” around a plot, or how modify the size and aspect ratio of the entire graph. Gnuplot’s defaults for all these properties are probably sufficient for the vast majority of your work, but there are situations where being able to go beyond the basics can give your graphs a very palpable “lift”.

10

Terminals in depth

This chapter covers:

Understanding terminals

Font selection and enhanced text mode

Standard graphics file formats

Mathematical formulas with gnuplot and LaTeX

Interactive terminals

Gnuplot can generate graphs in all common graphics formats (and some uncommon ones, too). The components within gnuplot that are ultimately responsible for creating all graphical output are called terminals. Different output formats require different terminals.

Because terminals are the final step in the creation of a graph, the ultimate appearance of a graph depends directly on the capabilities of the terminal that was used to generate it. Some of gnuplot’s terminals produce output of very high quality and are one of gnuplot’s primary assets. Unfortunately, terminal handling in gnuplot is a bit different than comparable functionality in other programs. We will therefore begin our discussion of terminals with a brief historical digression that explains the motivation behind gnuplot’s terminal abstraction.

An important capability that is provided by each terminal is the handling of fonts or typefaces: the selection of fonts and the available effects (italics, boldface, sub- and superscript) are determined by the choice of terminal. We will therefore discuss fonts and related topics in this chapter as well.

Next we discuss the most important terminals themselves: first, terminals that generate output in standard graphics file formats (specifically, PDF and PNG). Next, we will talk about ways to include mathematical notation and formulas on graphs, with and without leveraging LaTeX. We will introduce gnuplot’s svg and canvas terminals that generate graphics suitable for viewing in contemporary web browsers. And finally, we will quickly survey gnuplot’s interactive terminals.

10.1  The terminal abstraction

If you look into the standard gnuplot reference documentation, you will find that gnuplot supports some 70 different terminals! This may seem daunting, but the reality is that most of them are obsolete today—the number of terminals that are relevant is (thankfully) much smaller. But this leaves the question unanswered, where these 70 different terminals originally came from. For this, we need a bit of history.

10.1.1  Historical digression

Gnuplot was originally conceived and implemented at a time (late 1980s!) when graphics support was much less standardized than it is today. In fact, it wasn’t safe to assume that a computer even had an interactive graphics terminal at all: producing graphics usually required special hardware (such as an attached pen-and-ink plotter, or a dedicated graphics display).

The generic name for such devices that actually produced visible output was terminal. To keep gnuplot portable, all code specific to a single terminal was encapsulated into the terminal abstraction—today, we would call it the terminal driver. As graphic terminals developed and proliferated, gnuplot acquired a selection of such drivers. Nowadays, many of these legacy devices are obsolete, but the drivers are still part of the gnuplot distribution. Most of them can safely be ignored.

Nevertheless, the terminal abstraction has proven itself capable of adapting to changing circumstances. Today, the established architecture supports the two forms of graphical output that are still relevant: it drives interactive, screen-based displays (“interactive terminals”, such as the wxt and qt terminals), and it creates output files, using any one of a variety of standard graphics formats or command languages.

10.1.2  The terminal workflow

The primary reason terminal handling is often seen as obscure and confusing is that it requires multiple steps. Moreover, if the sequence of steps is interrupted, the gnuplot session may end up in a different state than at the beginning. This is confusing, because the near-unequivocal behavior of most contemporary programs is that an action “export to file” is both atomic and stateless (or, at least, it appears to be). In other words, you usually issue a single command to create a file, and then you are right back to where you started. Not so in gnuplot.

The complete workflow to create a graphics file from an interactive gnuplot session and to restore the interactive session again consists of the following six steps:

Save the current state of the terminal option with set terminal push.

Select a new terminal using set terminal.

Select desired output file or channel name with set output.

Generate a plot using plot, splot, replot, or refresh.

Restore the original terminal setting using set terminal pop.

Restore the original file or channel name with set output.

One reason for this multi-step process is that the same terminal abstraction is used for both the interactive session display as well as the printable export, requiring you to switch between different modes in a way that seems cumbersome today.

Two issues in particular tend to trip up the unwary. The first is the need to choose the file format (via the terminal) and the desired file name through two separate commands (namely set output and set terminal). The second is the requirement to restore the interactive session explicitly after the output file has been generated. (If this does not happen, subsequent plot commands continue to be sent to the file, clobbering its contents.)

Some day gnuplot will acquire an export command that will perform all the required steps automatically. Until then, you either have to make sure to complete all the necessary steps manually, or you use a macro of some form. (You can find implementations of macros for this purpose in sections "sec_practical_eval", “Scripts as subroutines”, “Worked example: export script”, and “Extending the command set”.)

[Tip]
Exporting to file is not atomic

To export a plot to a file from an interactive session requires multiple steps. Make sure you complete the process manually or use a macro.

As it turns out, there is a particularly nasty third problem, which is that output buffers are not necessarily flushed after a plot (or equivalent) command. This means that when the plot (or equivalent) command returns, the output file may still be incomplete and not yet ready to be used as input to other programs.

There’s a simple remedy, though: the next set terminal or set output command is guaranteed to complete the output file and flush any open buffers. Hence, even if you do not want to resume your interactive session, you still need to issue either a set output or set terminal command. Also be aware that quitting gnuplot does not necessarily flush open files!

[Tip]
Make sure to flush your output buffers

Output files may still not be complete, even after the plot command returns. To guarantee that all buffers are flushed and the file is completed and closed properly, it is necessary to issue either a set output or set terminal command after the actual plot command.

The pseudoterminals push and pop

The pseudoterminals terminal push and terminal pop can be very helpful in this context. The command set terminal push saves not only the current terminal type, but also all selected options. The command set terminal pop restores it. It is therefore possible to restore the previous terminal state, even without having explicit information about it.

10.1.3  Terminal capabilities and the test command

Output devices (“terminals”) obviously can differ widely in their capabilities—you can’t get color plots from a black-and-white printer. Gnuplot has a built-in command called test that generates a standard test image. The test image shows all available line styles and fill patterns, and also attempts to demonstrate more advanced terminal capabilities, such as the ability to rotate text through an arbitrary angle. To use the test command, we first need to select the terminal we are interested in and set the name of the output file (if it’s not an interactive terminal) like so:

set terminal pdfcairo
set output "test.pdf"
test

Note that the command is test—not plot test!

Along the right side of figure  10.1 we see the available line and symbol styles, fill patterns along the bottom, and line widths on the left side. We can also see what kinds of arrows the terminal supports, and whether it has the ability to rotate text. For the terminal as shown in the figure, all these features are supported.

Figure 10.1. The standard test image to demonstrate terminal capabilities, shown here for the pdfcairo terminal.

10.1.4  Gnuplot terminals today

In recent versions of gnuplot, terminal handling has become easier and more consistent than ever. In fact, the improvements in terminals and terminal handling alone justify upgrading to gnuplot 5 at the earliest possible moment!

A huge advance is the development of a family of terminals that are all based on a common set of high-quality libraries for graphics and font handling. The pngcairo, pdfcairo, epscairo, and wxt terminals all use the Cairo and Pango libraries for graphics and font rendering, and the fontconfig facility for font selection. These libraries provide advanced capabilities, enabling gnuplot terminals to produce high-quality graphics output. Moreover, because an entire family of terminals shares a common backend, it is now possible to generate bitmap (PNG), vector (PDF and EPS), and interactive (wxt) versions of the same graph, with only minor differences between them.[50]

Even outside this family of terminals sharing a common backend, terminals in gnuplot 5 exhibit more consistent behavior than ever before. A common set of options is available for basically all contemporary terminals. Terminals are guaranteed to offer at least 8 line types that have the same appearance (in regards to color, dash pattern, and point type) across all terminals. A background color can be selected as part of the set terminal command.

Font handling, in particular, has become much easier, because all contemporary terminals obtain their font support from the underlying font subsystem in the same way. Enhanced text mode (see section “Enhanced text mode”) is supported, and is enabled by default. At the same time, some traditional uses of enhanced text mode are no longer as relevant: thanks to Unicode and UTF-8 encoding, special symbols can now be requested without an explicit font change, and the font subsystem will do its best to provide a suitable glyph.

By default, terminals produce colored graphs using solid lines. The option mono is still available for all terminals to enforce graphs in black-and-white, but does not automatically turn on dashed lines, essentially rendering this option useless. (The options solid and dashed are still accepted by most terminals, but are ignored.) You therefore have to select dash patterns explicitly (either globally using set linetype or inline with dashtype), which means you are overall better of using a stylesheet (see section “Stylesheets”) to define a set of line types that have both the desired dash type and the required color when you want to create graphs in black-and-white.

[50] In particular, many of the configuration tricks and tweaks required by the bitmap terminals based on the GD library and the PostScript terminal are no longer necessary.

10.2  Font selection and enhanced text mode

Fonts aren’t handled by gnuplot itself—instead, gnuplot relies on capabilities of the installed system libraries (and, of course, on the selection of installed fonts). In this section, we investigate how to tell gnuplot to use a specific font from among the available ones, and we introduce enhanced text mode, which allows additional formatting options for text.

10.2.1  Font selection

There are three different ways to select a font, in increasing order of granularity:

Terminals have a default font that is used for all text labels (unless overridden). You can select a different default font using the font keyword in the set terminal command.

Most commands that generate textual output (such as set label or set key) let you select a font explicitly, again using the font keyword. The font will be used only for the specific label in question.

Using enhanced text mode, you can change the font inside a string itself. This makes it possible to render individual characters using a specific font if necessary or desired.

Another reason why the default font is important is that the average size of its characters is used to define the character coordinate system (see section "sec_decos_locations"), and that it is used to set the size of the graph’s margins (see section "sec_appearance_borders"). For variable-width fonts, gnuplot makes a best-effort attempt to determine an average character size.

Changing default font properties with set termoption

A special command exists to change certain terminal options, without having to reset the entire terminal. This can be useful when you want to modify some terminal aspect, regardless of the current choice of terminal. Five terminal aspects can be changed:

set termoption font "{str:name} [,{flt:size}]"
set termoption fontscale {flt:scale}
set termoption [no]enhanced

set termoption [ linewidth | lw ] {flt:factor}
set termoption [ dashlength | dl ] {flt:factor}

Be aware that only one aspect can be changed per invocation of set termoption; the different aspects cannot be combined into a single call.

10.2.2  Font resolution

In general, all terminals rely on some underlying system library for font resolution. Most contemporary terminals, and in particular the ones based on the Cairo library, use the fontconfig utility for this purpose. One important service this library provides is font substitution: given the request for a font, it selects that font from among the available ones that most closely matches the request, taking into account not just the name of the requested font, but also font characteristics such as style, slant, and weight. This “ensures that a font will always be returned, but doesn’t ensure that it is anything like the requested pattern” as the fontconfig user manual says.[51] The details of the matching process, and in particular the relative priority of the various attributes, are complicated and not necessarily transparent to the user. The substitution occurs glyph-by-glyph: I have encountered instances where each character in a string of special characters was rendered using a different font, depending on which font in the list of candidates first provided a glyph for the requested character.

One particular problem concerns the rendering of mathematical symbols and Greek characters. In general, textual labels on a graph should support the graph, but not distract from it. That calls for a clear and neutral font, like Helvetica or another linear sans-serif font with little line width variation. In contrast, mathematical symbols and Greek letters are traditionally drawn with varying stroke weight. If a sans-serif font is selected as default font for a terminal, then the font subsystem will attempt to find sans-serif glyphs for all requested characters, even for special symbols. In order to avoid this, I have found it necessary to switch fonts inside a string, in particular whenever I wanted to use letters from the Greek alphabet.

Figure  10.2 shows the actual results produced, on my system, when various free fonts are requested using the test script in listing  10.1 . (Depending on what fonts you have installed, your results may differ.) The test string consists of a set of regular characters chosen to exhibit the particulars of the font, and a selection of mathematical symbols. The first group of fonts consists of non-serifs, the second group of serif fonts, and the last group is made up of three special math fonts. (OpenSymbol is distributed with OpenOffice/LibreOffice, AsanaMath is intended as complement to Palatino, and the STIX project intends to provide a “comprehensive set of fonts” for scientific publishing.)

Listing 10.1. Commands for figure 10.2 (file: font-demo.gp)

unset border; unset tics; unset key

set t pdfcairo size 14cm,17cm
set output "fonttest.pdf"

s = "Quarges Gyt R17"
t = "∑αλ ∝ ∫ Γ(θ ⋅ π) dφ → ∞"

f = "Helvetica 'Nimbus Sans L' FreeSans TexGyreHeros Arial "
f = f . "Verdana 'DejaVu Sans' 'Liberation Sans' 'Droid Sans' ? "
f = f . "Times Palatino 'Bitstream Charter' Utopia 'Century Schoolbook L' ? "
f = f . "OpenSymbol STIXGeneral 'Asana Math'"

a = 0; b=2.75; c=5.75
n = words(f); m = n

do for[k=1:n] {
  m = m - 1
  w = word(f,k)
  if( w ne '?' ) {
    set label w at a,m font "Helvetica,12"
    set label s at b,m font sprintf("%s,16", w)
    set label t at c,m font sprintf("%s,16", w)
  }
}
plot [-0.25:9.25][-1:n] -2

Figure 10.2. Actual output produced, when various free fonts are requested (on my computer—your version may look different). The first part of the test string contains a selection of characteristic regular characters, the second part common mathematical symbols and Greek letters.

10.2.3  Enhanced text mode

All contemporary terminals support enhanced text mode, which allows additional formatting options. In enhanced text mode, some characters become control characters and indicate, for example, sub- and superscripts. As of gnuplot 5, enhanced text mode is now the default for all terminals. Enhanced text mode can be enabled or disabled for individual elements (such as text labels or the key) that contain strings. This may be useful if you want to use the special characters in regular text. Table  10.1, “Control characters for enhanced text mode” lists the special characters and their effects.

Table 10.1. Control characters for enhanced text mode

Control character

Example

Result

Description

{}

Grouping and scoping.

^

x^2

Superscript.

_

A_{ij}

Subscript.

@

x@^s_i

Alignment of sub- and superscripts.

x@_i^{-3/2}y

Put the shorter one first...

x@^{-3/2}_iy

... rather than last.

{/ }

{/Times Hello}

Change font.

{/= }

{/=24 A}

Select an absolute font size (size in printer points).

{/* }

{/*0.5 A}

Change font size relative to enclosing font.

{/:Normal }

{/:Normal Hello}

Select regular font characteristics.

{/:Italic }

{/:Italic Hello}

Select italics.

{/:Bold }

{/:Bold Hello}

Select boldface.

&

[&{abc}]

Space, corresponding to the length of the enclosed argument.

~

~B/

Overlay the two following characters or groups.

~x{.5-}

Overlay a - character on previous character, raised by 0.5 times the current character size.

\

\{ x \~ y \}

Escape a special character by preceding it with a backslash in single quoted strings.

\\

\\{ x \\~ y \\}

The backslash must be doubled to escape special characters in double quoted strings.

\NNN

\101

Select a symbol, using its octal code. (Single-byte characters only.)

\NNN\NNN

\316\261

Select a multi-byte character, using a sequence of octal numbers, one for each byte in the binary encoding of the character. (The Unicode character α at code point U+03B1 has the binary representation 0xCE 0xB1 in UTF-8, which translates to \316\261 in octal numbers.)

Here are some additional remarks on enhanced text mode:

Curly braces {...} are used both to group several characters together and to limit the effect of a font declaration to the contents of the braces only. They can be left off for single-character sub- and superscripts.

Phrases in curly braces nest, as in e^{-x^2} or a^{x_i}.

Font specifiers must be preceded by a slash / that immediately follows the opening curly brace.

Font specifiers can be combined into a single expression, for example {/Times:Italic*2 Hello}.

Font specifiers apply to everything within the enclosing braces: {/:Italic a_k} creates a label where both characters are italic. To prevent the subscript from being italic, you must either take it outside the braces ({/:Italic a}_k) or use an explicit :Normal ({/:Italic a_{/:Normal k}}).

A character within the single-byte ASCII range can be selected by supplying its ASCII code as a three-digit octal number, preceded by a backslash, and enclosed in curly braces.

Similarly, a multi-byte Unicode character can be selected by encoding each byte of its binary representation as a three-digit octal number. Remember that the Unicode codepoint U+xxxx does not equal the binary representation in UTF-8. Instead, you have to look up the binary representation, convert each byte separately to a three-digit octal number, and then supply the sequence of numbers thus obtained.

Special characters can be used when in enhanced text mode by preceding them with a backslash \ for single-quoted strings, and by preceding them with a doubled backslash \\ for double-quoted strings.

Overlays and the escaping of special characters is not available for the original postscript terminal. Both work normally for the pdfcairo and epscairo terminals.

Enhanced text mode vs. Unicode

Enhanced text mode used to be very important for two applications that today are better solved through Unicode. Overlays (using the ~ operator) were a way to create accented characters, and special characters (such as Greek letters and certain mathematical symbols) could be accessed by switching to a different font, like the PostScript “Symbol” font.

Today, both of these effects are better achieved through Unicode. Dedicated glyphs for accented characters will almost always look better than an ad-hoc solution using overlays, and system-wide font subsystems will do their best to return a suitable glyph when a special symbol is requested, without the user having to declare a specific font.

This leaves enhanced text mode as a solution for basic formatting tasks: sub- and super-scripts, local changes in font size and appearance (italics and boldface). Nevertheless, font changes are still available, and you are free to use them if you have reason to override the selection made by the font subsystem.

Worked example

Let’s look at some examples of enhanced mode (see listing  10.2 and figure  10.3 ). Most of the commands should be clear; I only want to point out a few details. Enhanced mode can be used anywhere that gnuplot expects a string: in the example, we use it in the labels, for the x tics, and for the entries in the graph’s key. The scope of a font selection can be limited using curly braces. We use this several times, for example in expressions such as {/OpenSymbol φ}(x), which prints the first character using the OpenSymbol font, but the last three using the regular font.

A word of warning: enhanced mode works best for simple applications, such as sub- and superscripts, boldface and italics, and simple changes to the font size of font face. But complicated, nested expressions may not always work out perfectly (in particular when using the PostScript terminal—see the footnotes in table  10.1, “Control characters for enhanced text mode”).

Listing 10.2. Commands for figure 10.3 (file: enhanced-mode.gp)

set label 1 "{/OpenSymbol φ}(x) = e^{-x^2/2} ∕ √2{/OpenSymbol π}" at 1.2,.25
set label 2
»   "{/OpenSymbol Φ}(x) = ∫@_{-∞}^x &{i} {/OpenSymbol φ}(t) dt" at 1.2,0.8

set label 3 "{/:Bold*2 Big and Bold}" at -2.5,0.6

set key top left title "{/:Bold Legend}"

set xtics ( "{/OpenSymbol π}/2" pi/2, "0" 0, "-{/OpenSymbol π}/2" -pi/2 )

plot [-3:3] exp(-0.5*x**2)/sqrt(2*pi) t "{/OpenSymbol φ}(x)",
»           norm(x) t "{/OpenSymbol Φ}(x)"

Figure 10.3. Enhanced text mode. See listing 10.2.

[51] http://www.freedesktop.org/software/fontconfig/fontconfig-user.html

10.3  Generating PNG and PDF with Cairo-based terminals

The pngcairo and pdfcairo terminals are the first choice when it comes to generating graphs in a standard bitmap (PNG) or vector (PDF) file format. (There is also the epscairo format, which is equivalent to the pdfcairo format, but produces encapsulated PostScript.) These terminals share a common set of options:

set terminal [ pngcairo | pdfcairo | epscairo ]
             [ size {flt:x}[ in|cm ],{flt,y}[ in|cm ] ]
             [ font "{str:face} [, {int:size}]" ] [ fontscale {flt:scale} ]

             [ background rgb {colorspec} ] [ [no]transparent ]
             [ [linewidth|lw] {flt:factor} ][ [dashlength|dl] {flt:factor} ]

             [ butt | square | rounded ]
             [ [no]enhanced ] [ [no]crop ] [ mono | color ]

The only option that requires some care is the size option, because it treats units differently depending on the output format.

For PNG, the size defaults to 640 × 480 pixels (aspect ratio 4:3). If no units are specified, pixels are assumed; if either cm or in are given, they are converted to pixels, assuming 72 pixels per inch.

For PDF and EPS, the size defaults to 5 × 3 inches (aspect ratio 5:3). Other sizes can be specified using either cm or in as units; if no units are given explicitly, inches is assumed.

Most of the other options are straightforward:

A typeface and size for the terminal’s default font can be chosen. The default is font "sans,12" (but see below for fontscale). It is possible to give only the size, leaving the font blank: font ",16".

Using fontscale, it is possible to adjust the size of the font separately, and relative to the current font size. Be aware that the default value of this option is 0.5!

A background color can be specified, using any one of the explicit color specifications (those using the rgb keyword. Using transparent, the background color can be made transparent.

Linewidths (for vector formats) default to 0.25 points (with 72 points per inch.) Both the line width and the dash length can be changed by a multiplicative factor. Note that it is not possible to change the point size in the same manner; you have to use the global set pointsize option instead.

The keywords butt, square, and rounded control how the Cairo library renders line ends and joins ). The default is butt.

Enhanced textmode is the default, you can disable it using noenhanced—for instance, because you want to use its control characters as regular characters.

When using crop, the terminal attempts to trim empty space around the graph before saving it. The plot is not rescaled, so that the resulting graph is in general smaller than the size specified in the size directive. (The crop option is only available for pngcairo.)

By default, terminals produce colored output. You can force monochrome output using mono, but be aware that this does not automatically turn on dashed lines—you have to do so explicitly for each line in your graph.

10.3.1  Scaling a complete plot up or down

It is occasionally necessary to scale an entire, finished graph up or down—for a presentation, say, or for inclusion in a publication. Simply increasing the size of the resulting graph using set terminal ... size is not enough, because the line widths and font sizes will not be scaled correspondingly. Instead, it is necessary to adjust the fontscale and linewidth suboptions to the set terminal command as well.

The set termoption command may be helpful in this context, because it can be used to adjust these two settings, separately from the set terminal command itself. There is no equivalent terminal option for the size of symbols; you must use the set pointsize command for this purpose instead.

10.4  Using gnuplot with LaTeX

One very common use of gnuplot graphs is to include them as illustrations in a LaTeX document. In this section, I give a couple of cookbook-style recipes. First, I describe how to include a regular graphics file as an image in a LaTeX document. Then we discuss gnuplot’s special cairolatex terminal, which allows us to combine gnuplot graphics with LaTeX text in the same illustration, so that we can use the full power of LaTeX for mathematical typesetting in gnuplot graphs.

10.4.1  Including a graph in a LaTeX document

Including an external figure in LaTeX is fairly easy. Here, we use the graphicx package for this purpose. This LaTeX package can handle both bitmap and vector formats, but for best results you want to use a vector format when available.[52]

As an example, let’s assume we want to include the graph from figure  10.3 in a LaTeX document. We’d have to export the graph to PDF, using the following commands:

... # commands from listing 10.2

set terminal pdfcairo
set output 'enhanced.pdf'
replot

The resulting file can then be included into a LaTeX document. The complete LaTeX document is shown in listing  10.3 .

Listing 10.3. Importing a PDF into LaTeX: see 10.4 (file: doc1.tex)

\documentclass{article}
\usepackage{graphicx}

\begin{document}

\section{The First Section}

Here is a very short paragraph. The plot will be included
after this paragraph.

\begin{figure}[h]
  \begin{center}
    \includegraphics[width=10cm]{enhanced}
  \end{center}
  \caption{A gnuplot graph, included in \LaTeX}
\end{figure}

And here is a second paragraph. The graph should have
been included before.

\section{The Second Section}

The second section really contains only a very short text.
\end{document}

The graphicx package provides the \includegraphics command, which takes the name of the graphics file to include as mandatory parameter. (The file name extension isn’t required and it’s recommended that you omit it.) The \includegraphics command takes a number of optional parameters as key/value pairs, which allow us to perform some useful operations on the image as it’s included: we can trim, scale, and rotate it. Here, we adjust its size ever so slightly (from 5 inches down to 10 cm).[53] The final appearance of the document after processing it with LaTeX is shown in figure  10.4 

Figure 10.4. The final appearance of the LaTeX document shown in listing 10.3. Note the labels using enhanced text mode in the included gnuplot graph.

10.4.2  Using the cairolatex terminal

In the previous example, we included a graphics file containing enhanced mode text in a LaTeX document. This seems inconvenient, to say the least: since LaTeX has such powerful capabilities to format text (and mathematical expressions specifically), we should find ways to use them to lay out our text, rather than dealing with the much more limited possibilities available through the enhanced text mode.

The cairolatex terminal does exactly that: it splits a gnuplot graph into its graphical and its textual components. The graph is stored as a PDF (or EPS) file, while the text is saved to a LaTeX file. We then include this LaTeX document, which in turn imports the graphics file, into our LaTeX master file.[54]

[Tip]
Don’t forget to close generated LaTeX files

When gnuplot generates a file that has to be postprocessed using LaTeX, it is particularly important to close the file properly by invoking set output after the plot is finished: otherwise, the generated file will not be complete, and LaTeX will fail. (This applies equally to the cairolatex terminal, as well as to the tikz terminal later in this section, and other comparable terminals.)

A worked example

An example will make this more clear. Let’s re-create the graph from figure  10.4 , this time using LaTeX formatting commands instead of enhanced text mode. See listing  10.4  (compare to listing  10.2  for a version of this graph using enhanced text mode).

Listing 10.4. Combining gnuplot and LaTeX using the cairolatex terminal (file: cairolatex.gp)

set label 1 '$\phi(x) = \frac{1}{\sqrt{2 \pi}} e^{-\frac{1}{2} x^2}$'
»   at 1.2,0.25
set label 2 '$\Phi(x) = \int_{-\infty}^x \phi(t) dt$' at 1.2,0.8

set label 3 '\textbf{\Large Big and Bold}' at -2.5,0.6

set key top left Left title '\textbf{Legend}'

set xtics ( '$-\pi/2$' -pi/2, '$0$' 0, '$\pi/2$' pi/2 )

plot [-3:3] exp(-0.5*x**2) /sqrt(2*pi) t '$\phi(x)$',
»           norm(x) t '$\Phi(x)$'

set terminal cairolatex pdf
set o "cairolatex.tex"    # Name of output LaTeX file with extension!
replot

The text for labels and tic marks that we’re adding to the graph now contains LaTeX directives. In particular, note the $...$, which indicate math-mode in LaTeX, and which are required for many of the special mathematical formatting commands.

All the strings in listing  10.4  use single quotes, and therefore the backslashes they contain do not need to be doubled. If we had used double quotes instead, then all the contained backslashes would have to be escaped. The third label, for example, would have to be written like so:

set label 3 "\\textbf{\\Large Big and Bold}" at -2.5,0.6

The cairolatex terminal generates two files: the actual graph (as PDF or EPS file) and the text (as LaTeX file). Gnuplot expects the full name (including extension) of the LaTeX file to be set using set output when using the cairolatex terminal (see the second-to-last line in listing  10.4 ).

Listing 10.5. LaTeX document that imports cairolatex.tex: see figure 10.5 (file: doc2.tex)

\documentclass{article}
\usepackage{graphicx}

\begin{document}

\section{The First Section}

This is a different document. We will include a \LaTeX\ file
containing the graph below.

\begin{figure}[h]
  \begin{center}
    \input{cairolatex}
  \end{center}
  \caption{A graphics file included in \LaTeX, with \LaTeX\ labels}
\end{figure}

And here is a second paragraph. The graph should have
been included before.

\section{The Second Section}

The second section really contains only a very short text.

\end{document}

The LaTeX master file is shown in listing  10.5 ”. Note how similar the document is to the one in listing  10.3 . We again must explicitly require the graphicx package at the beginning of the document. We then use \input (instead of \includegraphics—that’s the only difference) to include the LaTeX file written by gnuplot. Again, it’s recommended to omit the extension: LaTeX will look for a file with .tex extension. The appearance of the final document (after LaTeX processing) is shown in figure  10.5 

Figure 10.5. The final appearance of the LaTeX document shown in listing 10.5. Note the true LaTeX labels and tic marks on the graph. Compare to figure 10.4.

Options for the cairolatex terminal

Now that we’ve seen how the cairolatex terminal is used, we can take a closer look at its features. It shares most of its features with the pngcairo and pdfcairo terminals we already know:

set terminal cairolatex [ eps | pdf ] [ standalone | input ]
             [ header "{str:header}" | noheader ]

             [ size {flt:x}[ in|cm ],{flt,y}[ in|cm ] ]
             [ font "{str:latexfont}[,{int:size}]" ][fontscale {flt:scale}]

             [ background rgb {colorspec} ] [ [no]transparent ]
             [ [linewidth|lw] {flt:factor} ][ [dashlength|dl] {flt:factor} ]

             [ butt | square | rounded ] [ [no]crop ]
             [ blacktext | colortext ] [ mono | color ]

Only three options are really new: using the eps and pdf keywords, we choose the format of the graphics part of the output. Be aware that the default is eps (not pdf).

If we select standalone, the generated LaTeX document is complete and can be compiled by itself. In other words, it doesn’t have to be included into another LaTeX document for further processing. This way, we can create standalone graphics with LaTeX-style annotations.

In standalone mode, the name of the generated graphics (PDF or EPS) file is modified by appending -inc to the base name: the two files written by cairolatex would be called (for example) graph.tex and graph-inc.pdf (so that the resulting graph, which combines both plot and text, can simply be called graph.pdf). The default value for this option is input, which generates an incomplete LaTeX document for inclusion in a master file, as shown earlier.

The last new option is the header option. With it, an arbitrary set of LaTeX commands can be added into the LaTeX file containing the text component of the plot. We could use this, for instance like this:

set terminal cairolatex header "\\bfseries"

to print all text within the plot as boldface.

If the file gnuplot.cfg is found by LaTeX when processing a standalone file, it’s included into the preamble of the document. It’s included before the argument to the header option, so the latter can be used to override global settings from the gnuplot.cfg file.

Finally, the font option changes its meaning for the cairolatex terminal, compared to the pdfcairo and pngcairo terminals. Instead of the names of system fonts, cairolatex obviously expects names for LaTeX fonts. The font description consists of three parts: the font name, the font series, and the font shape. All three parts are optional, but the commas are mandatory. Two quick examples:

set terminal cairolatex font "cmr,b,it" # ComputerModern, Bold, Italic
set terminal cairolatex font ",,sc"     # Surrounding font, with Small Caps

The size of the font can be given after the font description: set terminal cairolatex "cmr" 14.

The cairolatex terminal provides additional ways to give you access to LaTeX features from within a gnuplot command script. Check the gnuplot standard reference documentation, and also your favorite LaTeX reference for more details.

10.4.3  Letting LaTeX generate the graph

When using cairolatex terminal, gnuplot draws the graph, and LaTeX provides the text labels. One can go a step further and let LaTeX draw: the text, and the graph itself. Gnuplot merely creates a set of LaTeX commands that accomplish both tasks. Several terminals exist that take this approach; here I will briefly introduce the tikz terminal, as an example for the overall concept. The context terminal follows a similar idea.

The tikz terminal uses the Lua language to create the required LaTeX commands. We will recreate the graph from listing  10.4 , but using the tikz terminal instead. See listing  10.6 for the complete listing.

Listing 10.6. Commands for figure 10.6 (file: tikz-demo.gp)

set label 1 '$\phi(x) = \frac{1}{\sqrt{2 \pi}} e^{-\frac{1}{2} x^2}$'
»   at 1.2,0.25
set label 2 '$\Phi(x) = \int_{-\infty}^x \phi(t) dt$' at 1.2,0.8

set label 3 '\textbf{\Large Big and Bold}' at -2.5,0.6

set key top left Left title '\textbf{Legend}'

set xtics ( '$-\pi/2$' -pi/2, '$0$' 0, '$\pi/2$' pi/2 )

plot [-3:3] exp(-0.5*x**2) /sqrt(2*pi) t '$\phi(x)$', norm(x) t '$\Phi(x)$'

set terminal tikz standalone
set o "tikz.tex"    # Name of output LaTeX file with extension!
replot

Notice that we invoke the tikz terminal with the standalone option: in this case, we are creating just the graph. Gnuplot produces a LaTeX file called tikz.tex that can be processed using pdflatex; the final plot is shown in figure  10.6 . Be aware that you need to have the pgf (TeX Portable Graphics Format) package installed, which is part of the LaTeX distribution, but may not be installed by default.

Figure 10.6. A graph drawn using the tikz terminal. See listing 10.6 for the commands.

How does this entire approach compare to the alternatives? Historically, one advantage of letting LaTeX generate the actual graph was that LaTeX allowed greater control over the appearance of lines, dash patterns, and the like. This aspect has become less important, as gnuplot itself has acquired some of these capabilities.

Terminals such as pdfcairo produce a separate (standalone) graph, which potentially has to be included into a document. Using the tikz or context terminals, it is possible to embed the commands that generate the graph itself into the document, so that there is only a single file that has to be processed. Whether this is a good idea is another question—personally, I prefer the “separation of concerns” that is provided by having standalone graphs and a document that references them. Also be aware that LaTeX may encounter resource (specifically memory) limitations when processing large graphs.

[52] Also keep in mind that you may have to use pdflatex when using graphics in PDF format, but plain latex for EPS graphics. Some newer TeX installations will perform the necessary format conversion automatically.

[53] There are many more options—check your favorite LaTeX reference for details. A good place to start is Guide To LaTeX (4th ed.) by H. Kopka and P. W. Daly, Addison-Wesley, 2004.

[54] The cairolatex terminal replaces the epslatex terminal.

10.5  Scalable graphics for the Web with SVG and HTML5

Traditionally, graphics to be displayed in web pages used to be bitmaps—and we can still do this, using the pngcairo terminal, for instance. But gnuplot can also generate graphs using scalable formats that can be rendered in a web browser. The svg terminal creates a graph using the SVG (Scalable Vector Graphics) format; the canvas terminal prepares JavaScript commands for use with the HTML5 canvas element.

10.5.1  The svg terminal

The svg terminal behaves similarly to the pngcairo or pdfcairo terminals, but it creates SVG code, which can be captured in a file using set output.

set terminal svg [ size {int:x},{int:y} [ fixed | dynamic ] ]
                 [ font "{str:name} [, {int:size} ]" ]
                 [ fname "{str:fontname}" ] [ fsize {int:fontsize} ]
                 [ fontfile {str:filename} ] [ [no]enhanced ]

                 [ rounded | butt | square ] [ background rgb {colorspec} ]
                 [ [linewidth|lw] {flt:fac} ] [ [dashlength|dl] {flt:fac} ]

                 [[ mouse ] [ standalone | jsdir "str:dirname" ]]
                 [ name "{str:title}" ]

A size can be specified in pixels; the default size is 600×480. By default, or if the keyword fixed has been given, the resulting graph has an absolute size; if dynamic has been given, then the resulting graph will adapt to the size of the viewer (typically filling the entire available area).

A font can be specified using the usual format, but it is also possible to specify only a font name or font size using the keywords fname and fsize. Using fontfile, a file with font information can be embedded into the SVG document; gnuplot will search for the requested file using the information in the GNUPLOT_FONTPATH environment variable. The font must be an SVG font, not a PostScript or TrueType font.

The keyword mouse enables mousing: mouse coordinates are continuously displayed when the mouse is over the image; and individual curves can be toggled on or off by clicking their entry in the graph’s key. Mousing requires an external JavaScript script, which is usually found in a standard location of the local gnuplot installation; the location can be changed by providing a path name to the jsdir option. The standalone option embeds the mousing code in the SVG file itself, rather than referring to an external document. Finally, an identifier can be given to the plot using name, by which it an be referred to by JavaScript or web page that includes the graph.

An SVG can be used by itself, or it can be included in a web page. It is recommended to use the <object> tag for this purpose, like so:

<object type="image/svg+xml" data="graph.svg">
  Your browser does not support SVG.
</object>

This also provides a fallback message to the user, in case the user’s browser does not support SVG.

10.5.2  The canvas terminal

The canvas terminal creates JavaScript commands that draw onto an HTML5 canvas element. It can be called in two different modes:

Using standalone, the terminal produces a web page (that is, HTML code), containing a <canvas> element and the JavaScript to draw on it. The set output option should therefore specify a file with extension .html (or equivalent).

The name option identifies the name of a JavaScript function that will be generated, and the terminal produces just the set of JavaScript commands that implement that function. The set output option should specify a file with extension .js.

The canvas terminal supports various options:

set terminal canvas [ size {int:x},{int:y} ]
                    [ font "{str:name} [,{int:size}] ] [ fsize {int:size} ]

                    [ [no]enhanced ]
                    [ title "{str:title}" ]
                    [ rounded | butt | square ] [ background rgb {colorspec} ]
                    [ [linewidth | lw] {flt:fac} ]
                    [ [dashlength | dl] {flt:fac} ]

                    [ standalone [mousing] | name "funcname" ]
                    [ jsdir "{str:url}" ]

A font can be specified using font. This setting is currently ignored, but a font size can be selected using fsize.

In standalone mode, the terminal creates a complete web page, which includes the JavaScript code for the graph. In addition, the web page refers to various other JavaScript files that are part of the gnuplot installation and that it expects to find in a standard location. A different directory can be specified using the jsdir option; this may be appropriate when the web page is not viewed locally, but served on the web. If the mousing keyword is used in conjunction with standalone, then some additional elements are included in the page that show the mouse coordinates and provide some further ways to interact with the graph (such as zoom and unzoom functionality).

When the name option is given, only the JavaScript is generated. The resulting code will contain the implementation of a JavaScript function with the desired name. For example, assume that we have created a JavaScript file called plotter.js, containing a function called drawplot() using the following commands:

... # plot commands

set terminal canvas name "drawplot"
set output "plotter.js"
replot

These files can be used in a web page like the one in listing  10.7 . Several details are important:

The header section references two external scripts, which are part of the gnuplot distribution. Make sure that the path to these files is correct in the header and that the files are visible.

The name of the function appears twice in the page: as value of the onload attribute of the <body> tag, and as value of the id attribute of the <canvas> tag.

The name of the generated JavaScript file is used in the <script> tag within the page’s body.

Listing 10.7. Using the canvas name terminal (file: canvas-demo.html)

<html>
  <head>
    <script src="/usr/share/gnuplot/5.0/js/canvastext.js"></script>
    <script src="/usr/share/gnuplot/5.0/js/gnuplot_common.js"></script>
  </head>

  <body onload="drawplot();">
    <script src="plotter.js"></script>
    <canvas id="drawplot" width=600 height=400>
      <div id="err_msg">Your browser does not support HTML5 canvas.</div>
    </canvas>
  </body>
</html>

10.6  Interactive terminals

In contrast to the large range of options available for file-based terminals, interactive terminals are rather straightforward.

10.6.1  Common options

Gnuplot can have multiple plot windows open at any given time, although only one of the windows will receive plot commands at any time. We can create a new plot window by specifying the terminal ID (which is just an integer expression) when choosing the terminal, like so:

set terminal wxt 1

To switch between open windows, use set terminal together with the terminal type followed by the window ID to select the active terminal. The keyword close in a set terminal command will close the identified window. (Depending on the window manager, the window may persist until it has received mouse or keyboard focus for the first time after the set terminal ... close command has been issued.)

To distinguish between multiple plot windows, we can give them a title, which will (depending on the window manager) be displayed in the window’s header bar. (This is the title of the window only and has nothing to do with the title that can be placed on the plot using set title.)

Plot windows are automatically raised and receive input focus when a new plot is drawn. This behavior can be disabled using the noraise option, so that keyboard focus stays with the command window even when a plot is issued.

By default, hitting the space bar while the plot window has focus raises the command window, and hitting the “q” key closes the plot window. Using the ctrl option, these bindings are replaced by combinations of the control key together with either the space bar or the “q” key (thus freeing up those keys to be rebound otherwise: see section “Custom hot keys”).[55]

When the persist option has been enabled, plot windows will stay up even when gnuplot is exited (or prevent gnuplot from terminating completely until all plot windows have been closed explicitly).

Three commands can be used to manipulate interactive terminals directly: raise, lower, and clear. The first two take an optional window identifier as optional argument. The clear command (which always acts on the most currently active terminal) is useful even for noninteractive terminals: it clears the plot area (and forces a form-feed or page break for noninteractive terminals). It is required, for example, in some applications of the multiplot mode (in section "sec_special_multiplot").

10.6.2  The wxt and qt terminals

The wxt and qt terminals are very similar in their functionality, but are built using different widget sets. Because both widget sets are cross-platform, both terminals are available on all current operating systems: Linux, Windows, and Mac OS X.

Both terminals share the most important options:

set term [ wxt | qt ] [ {int:id} ] [ title "{str:title}" ]
         [ size {int:width},{int:height} ] [ position {int:x},{int:y} ]
         [ font "str:face[,{int:size}]" ] [ [no]enhanced ]
         [ [no]raise ] [ [no]persist ] [ [no]ctrl ]
         [ close ]

In addition, both terminals also support the dashlength (or dl) option. The wxt terminal (but not the qt terminal) also offers the background (or backg) and linewidth (or lw) options.

10.6.3  Aqua

The aqua terminal is a native Mac OS X terminal for gnuplot. It relies on AquaTerm.app:

set term aqua [ {int:id} ] [ title "{str:title}" ]
              [ size {int:width},{int:height} ]
              [ font "str:face[,{int:size}]" ] [ [no]enhanced ]

10.6.4  Windows

The windows terminal is for use on Microsoft Windows platforms:

set term windows [ {int:id} ] [ title "{str:title}" ]
         [ size {int:width},{int:height} ] [ position {int:x},{int:y} ]
         [ font "str:face[,{int:size}]" ] [ [no]enhanced ]
         [ close ]

The windows terminal supports some additional GUI features, see the standard gnuplot reference documentation for details.

[55] Everything having to do with keyboard focus and the raising and lowering of windows depends on the interplay of the window manager and the widget set implementing the terminal. As of this writing, not all of this functionality works with all window managers.

10.7  Other terminals

As mentioned in the introduction to this chapter, gnuplot supports a lot of terminals—many more than I discussed in this chapter. While most of them are no longer relevant, there are some that may be of use in specialized situations..

Traditionally, the postscript terminal has been very important, and so have the gif, png, and jpeg bitmap terminals based on the GD library. These terminals have largely been replaced by the pngcairo and pdfcairo terminals discussed earlier in this chapter.

There are many more terminals that generate commands suitable for TeX or LaTeX processing than I described here. If you’re interested, you might want to check out the standard gnuplot reference documentation for eepic, latex, pslatex, pstricks, texdraw, and tpic.

Besides LaTeX, gnuplot supports some other graphical command languages. Check out the fig or tgif terminals.

You should also know about two terminals that are truly obsolete, but are still supported and may provide a safe fallback solution if necessary: the x11 and the dumb terminal.

The x11 is built on the X11 system without the benefit of a modern widget system. As an X11 application, it honors many of the standard X11 conventions, such as geometry, color resources, and app-defaults. The x11 terminal has been superseded by terminals such as wxt and qt that are built using contemporary widget sets.

Finally, a true classic: the dumb terminal (see figure  10.7 ). This is a terminal for character-oriented output devices. It uses characters to draw an ASCII-art rendition of the plot: it’s the fallback for console-oriented systems or over really slow network connections.

Figure 10.7. The dumb terminal.

For all of these terminals, see the standard gnuplot reference documentation.

10.8  Summary

Gnuplot uses the terminal facility to create output for other applications and devices. Using the pngcairo and pdfcairo terminals, we can export graphs in standard bitmap (PNG) and vector (PDF) formats. The svg and canvas terminals can generate vector graphics suitable for inclusion in web pages.

The appearance of strings and text labels depends on the available fonts and the capabilities of the chosen output format. Enhanced text mode allows simple formatting commands (such as sub- and super-scripts) to be embedded in text labels themselves.

For those who are familiar with LaTeX, the cairolatex terminal makes it possible to use LaTeX’s text (and in particular, math) formatting capabilities together with gnuplot’s plotting facilities to achieve particularly high-quality plots for publications or other printable documents.

Finally, we provided a brief overview over interactive terminals for various platforms.

11

Automation, scripting, and animation

This chapter covers:

Loops and conditionals

Command files and batch processing

Calling gnuplot from other programs

Animations

In the preceding chapters, we’ve studied most commands and options that can be used from within gnuplot. But up to this point, our discussions assumed that we’d be running gnuplot interactively: entering commands, viewing plots, and then entering more commands.

But sometimes that’s not what we want. For instance, we may have a large number of data sets and just want to convert all of them to plots: it would be great to automate this task through scripting. Or we may have a cron job running nightly to scan web server logs and want it to also generate some plots showing the number of hits per hour, without requiring human intervention: that’s batch processing. Or we might want to use gnuplot as a graphing backend, while tendering control to some other program.

All these situations involve tasks that have been automated to some degree. In this chapter, we study ways to do this, either using gnuplot’s own facilities, or using gnuplot in conjunction with an external programming language.

11.1  Loops and conditionals

Gnuplot has support for loops and conditionals: this makes it possible to automate repeated tasks. In contrast to the inline loops that we encountered earlier (see section "sec_practical_inlineloops"), the control structures introduced in this section are more general. They can be applied to any command, or any combination of commands—not just a single plot or set command as was the case for inline loops. In addition to two different loops and a conditional construct, there is also a special operator that calculates the sum of a set of numbers. Let’s look at all of them in detail; table  11.1, “Gnuplot loop constructs and conditionals” provides a summary.

The do for loop is a generalization of the inline loop concept. Loops can extend over a numeric range (start and end inclusive, the increment is optional and defaults to 1). Alternatively, the do for loop can also iterate over whitespace-separated tokens in a string.

The other loop is a while loop. It executes the statements in the loop body as long as the supplied condition does not evaluate to zero.

The if statement is used to construct conditionals. The else clause is optional; at present, there is no provision for repeated conditions (no elsif or similar).

Finally, the sum construct is an expression (meaning that it returns a value). It iterates over a set of integers, evaluates a given expression, and adds the expression’s value to a running total. Upon completion, the value of the running total is returned. It is not possible to give a step size, and the evaluated term need not include the iteration variable.

sum[var=start:end] expression

The following examples will make this more clear:

print sum[k=1:10] 1                         # prints 10.0
print sum[k=1:10] k                         # prints 55.0
plot "data" u 1:(sum[k=2:5] column(k)/4.)   # plots average of cols 2..5

Table 11.1. Gnuplot loop constructs and conditionals

Syntax

Example

do for [var=start:end:increment] {
    statement
    statement
    ...
}

do for[k=1:10] {
    print k
}

do for [var in tokenstring] {
    statement
    statement
    ...
}

do for [s in "abc uvw xyz"] {
    print s
}

while( condition ) {
    statement
    statement
    ...
}

x = 0
while( x < 10 ) {
    x = x + 1
    print x
}

if( condition ) {
    statement
    statement
    ...
} else {
    statement
    statement
    ...
}

x = 0
if( x < 0 ) {
    print "Negative"
} else {
    print "Not negative"
}

Limitations of gnuplot’s programming model

At the present time, gnuplot offers only a limited support for scripting: it has control structures (loops and conditionals), but neither data structures nor subroutines—and of course no such thing as local variables! This should not be a surprise: gnuplot was never intended as a programming environment, and the available scripting support is little more than an ad-hoc collection of convenience features, intended to take the tedium out of repeated tasks.

The problem is that control structures like loops require arrays and subroutines to work properly: they just don’t make much sense otherwise. But because gnuplot does not offer such facilities, existing features find themselves “press-ganged” into roles that they are not intended for: whitespace-separated strings, together with the word() and words() functions, are used to mimick iterable data structures, array indexes are faked through numeric suffixes on variable names, and individual command files have to stand in for subroutines. Depending on perspective, you may find this either hideous or ingenious.

You can’t very well have “a little” programming support: either you don’t have any, or you will find that each “small” feature will—before long—lead to the addition of yet another “small” feature, until you end up with a complete programming environment anyway, simply by following the law of unintended consequences.

Because of this, I doubt the wisdom of adding programming support to gnuplot simply through opportunistic accretion, and without clarity about the intended programming model. The evolution of programming languages over the last 30 years (and the last fifteen, in particular) has shown how difficult it is to create a programming environment that “wears well as one’s experience with it grows”.

11.1.1  Worked example: making graph paper

To kick off our list of examples for gnuplot’s looping constructs, let’s make some quad-ruled graph paper. This application demonstrates how repetitive tasks can be automated with loops. It also has an immediate graphical representation (see figure  11.1 ).

Figure 11.1. Graph paper: an easy application of gnuplot’s loops. See listing 11.1.

In commercially produced graph paper, every fifth line is printed a little wider than the remaining lines to impose some larger-scale, visual guidance to the user. In the graph in figure  11.1 I have instead made every fifth line a little darker than the other lines, using different shades of grey. For best effect, it is important to add the darker lines later to the plot than the other lines.

The commands for the graph paper in figure  11.1 can be found in listing  11.1 . The size of the entire graph is set to 12.5cm×12.5cm, but then the actual plot (the “canvas”) is scaled to 80 percent of that, leading to a finished square of 10cm×10cm, with millimeter spacing of individual lines.

Listing 11.1. Commands for figure 11.1 (file: graphpaper.gp)

set terminal pdfcairo size 12.5cm,12.5cm
set o 'graphpaper.pdf'

set size 0.8,0.8
set origin 0.1,0.1
set margins 0,0,0,0

unset key
unset tics

set style arrow 1 nohead lw 1

set for [i=1:99] arrow from i/10.,0 to i/10.,10 as 1 lc rgb 'light-grey'
set for [i=1:99] arrow from 0,i/10. to 10,i/10. as 1 lc rgb 'light-grey'

set for [i=5:95:10] arrow from i/10.,0 to i/10.,10 as 1 lc rgb 'grey'
set for [i=5:95:5] arrow from 0,i/10. to 10,i/10. as 1 lc rgb 'grey'

set for [i=10:90:10] arrow from i/10.,0 to i/10.,10 as 1 lc rgb 'dark-grey'
set for [i=10:90:10] arrow from 0,i/10. to 10,i/10. as 1 lc rgb 'dark-grey'

plot [0:10][0:10] 1/0
set o

11.1.2  Worked examples: iterating over files

It comes as no surprise that gnuplot’s looping facilities come in handy whenever we are dealing with a large number of similar files. Let’s consider two examples—one straightforward, the other one not.

Plotting several files

Imagine that you have a directory containing many data sets. Each data set is in a separate file, and the files are all named in a predictable fashion: data01, data02, and so on. You would like to generate a plot of every data set, each in its own file, with appropriately chosen filenames. Here is how to do it (the sprintf() function comes in handy to format the input and output filenames properly):

set terminal pdfcairo                        # choose output file format
do for[k=1:50] {
  set output sprintf( "graph%02d.pdf", k )   # create the output file
  plot sprintf( "data%02d", k ) u 1:2        # plot
  set output                                 # flush the output channel
}

Making data comparable

The next example is more complicated. In this case, we only have three files (data1, data2, and data3), but we don’t want to plot each data set in a separate graph. Instead, we want to combine them onto a single graph, so that they can be compared easily. To compare disparate data sets meaningfully, it is often necessary to normalize the data before plotting it. For now, and to keep the exposition simple, let’s assume that we just want to subtract the respective average value from each data set before plotting it. (We’ll see another example of normalizing data for the sake of comparison in section "sec_multivar_controlcharts".)

It is easy enough to loop over the files and use the stats command (see section "sec_practical_stats") to calculate the respective average (or mean) values. The problem is that we need to store the separate values temporarily, so that we can use them later in the plot command. Because gnuplot does not have an array data structure, we need to bend over backwards to accomplish this goal. I am going to show you two different ways to get there.

In the first implementation, I create variables with numeric suffixes to stand in as array entries. But because it is impossible to create variables dynamically, I instead build up a string containing the variable definition, which is then executed using eval. The subsequent plot command, using an inline loop, does the same thing: it builds up the variable name as a string, and obtains the value of the corresponding variable using the value() function (see "tbl_practical_stringfcts" for information about value()).

do for[file in "data1 data2 data3"] {
  stats file u 2 nooutput
  eval sprintf( "mean_%s=%f", file, STATS_mean )
}
plot for[file in "data1 data2 data3"]
»    file u 1:($2-value(sprintf("mean_%s", file)))}

The second implementation may be more straightforward, but personally I find it even more ominous. It fully embraces the idea that an array is nothing more than a sequence of values, and hence can conveniently be represented as a string. String operations are then used to extract the desired entry.

In the first loop, a string is built up from the values calculated by stats. (Do not miss the whitespace inside the sprintf() format specifier that indicates token boundaries.) The plot command contains an inline loop and uses the word() function to extract each appropriate element in turn from the string of values. (See table "tbl_practical_stringfcts" for information about the word() function.)

files = "data1 data2 data3"
means = ""
do for[file in files] {
  stats file u 2 nooutput
  means = means . sprintf( " %f", STATS_mean )
}
plot for[k=1:3] word(files,k) u 1:($2-word(means,k))

11.1.3  Worked examples: Taylor series and Newton’s method

In this section, I want to demonstrate two other applications of looping constructs. Both require some familiarity with Calculus—feel free to skip this section.[56]

Taylor series

According to Taylor’s theorem, every smooth function can be locally approximated by a polynomial: the higher order the polynomial, the better the approximation. We demonstrate this here for the function sin(x), which has the Taylor approximation:

\sin(x) \approx \sum_{k=0}{n} (-1)^{k} \frac{x^{2k+1}}{(2k+1)!}

Using gnuplot’s sum feature, together with the ! operator to indicate the factorial, this can be written very compactly in gnuplot (also see figure  11.2, “Successive approximations to sin(x) using the sum facility.”—the function s(n) has been introduced to fix the key entry when there is only a single term in the series):

f(x,n) = sum[k=1:n] (-1)**(k-1)*x**(2*k-1)/(2*k-1)!
s(n) = (n==1 ? "1 Term" : sprintf( "%d Terms", n ) )
plot [-8:8][-1.2:1.2] sin(x) lw 2, for[n=1:4] f(x,n) t s(n)

Figure 11.2. Successive approximations to sin(x) using the sum facility.

Newton’s method

The other example involves Newton’s method for finding the roots of an analytic function. Newton’s method consists of an iteration of the form:

x_{n+1} = x_n - \frac{f(x)}{f^\prime(x)}

This can be implemented using a while loop. The derivative f′(x) is evaluated through a finite-difference approximation:

f(x) = cos(x)
steps=0; x=0.5; eps=1e-6

while( abs(f(x)) > 1e-10 && steps < 15 ) {
  df = (f(x+eps) - f(x) )/eps
  x = x - f(x)/df
  steps = steps + 1
  print steps, x, f(x)
}

With the given choice of parameters, Newton’s method converges to the first positive root of the cosine in five steps. Here is the total output from running this snippet:

1 2.33048604607464 -0.688696509312027
2 1.38062453566562 0.189027591805359
3 1.57312250575387 -0.0023261768611035
4 1.57079632259625 4.19864216189357e-09
5 1.5707963267949 6.12323399573677e-17

[56] This is not the place to explain either Taylor series or Newton’s method; the purpose of this section is to demonstrate gnuplot’s looping features. For this reason, my presentation of the mathematical concepts is intentionally brief. The Wikipedia pages for both topics are reasonably accessible.

11.2  Command files

Gnuplot commands can not only be entered at the command prompt. It is also possible to capture commands in a script file. Such a file can be loaded into a gnuplot session, included in other command files, or simply processed by gnuplot in batch mode.

So far, we have encountered command files mostly as the product of the save command. The files generated by save capture the entire state of the gnuplot session at that moment: the values of all options and variables, all user-defined functions, as well as the most recent plot command. If you load such a file, it will recreate exactly the previous session state, and—in particular—it recreates exactly the previous plot.

It is worth emphasizing that this is not the only way to use command files; in particular, it is not necessary for command files to contain a plot or splot command. Command files without either of these commands obviously don’t generate visual output, but they can be very useful nevertheless:

As setup or configuration scripts: such scripts contain nothing but function definitions, customized line types and styles, or shorthands for commonly used color gradients. We will explore some these possibilities in the next chapter (chapter  12, Beyond the defaults: workflow and styles).

As subroutines: I mentioned earlier that gnuplot does not have subroutines: it is not possible to group several statements together into a single, callable unit. Command files provide a workaround for this limitation.

As batch files: the contents of a command file can be processed by gnuplot in the background, without an interactive user session.

11.2.1  Scripts as subroutines

Gnuplot does not have a way to group several commands or statements together into a single, callable unit. Gnuplot’s functions are expressions (they return a value), and their definitions must consist entirely of expressions as well. It is not possible to evaluate a gnuplot command within a gnuplot function, much less a sequence of commands.

Scripts provide a workaround. A script can contain any valid gnuplot code, and is callable as a single unit. The generic export script is a good example (compare section “The terminal workflow”):

set terminal push
set terminal pdfcairo
set output "graph.pdf"
replot
set output
set terminal pop

This sequence of commands can be run with a single invocation of load or call.

Passing parameters to scripts

The script in the previous example did not take any parameters: the name of the output file was hardwired. That’s not ideal; we would like to supply a file name at runtime. There are two ways to do this:

Prepopulate a gnuplot session variable and use it inside the script.

Use call to invoke the script and specify the parameters as part of the command.

To use the first method, you would rewrite the export script to use a variable:

set terminal push
set terminal pdfcairo
set output outfile      # outfile is a variable!
replot
set output
set terminal pop

and be sure to assign the proper value to this variable before invoking the script (assuming that the script is called export.gp):

outfile = "graph.pdf"
load "export.gp"

The alternative is to use the call command to invoke the script (instead of load). The call command accepts up to nine parameters on the command line, which will be available inside the script in variables called ARG1, ARG2, and so on. (The variable ARG0 is set to the name of the script file itself, and the variable ARGC holds the number of arguments supplied to call.) A version of the export script, suitable for call, would look like this:

set terminal push
set terminal pdfcairo
set output ARG1         # use special variable!
replot
set output
set terminal pop

and you would call it like this:

call "export.gp" "graph.pdf"

Personally, I prefer the call mechanism over prepopulating session variables, because it makes the parameter passing explicit. You need to be aware of an important limitation, though: parameters are passed by call as strings and therefore must be representable by strings (numbers are converted to strings on the fly). This makes passing functions inconvenient, whereas a function that exists in the current session can simply be used by the script. (We will see an example in a moment, when discussing a script that implements Newton’s method.)

Simulating local variables

Gnuplot does not have local variables: all variables that are created live in the global session space. This means that if we call a script that defines variables, then these variables will still be part of the session when the script finishes. This may be desirable (because it provides a way to retrieve values from the script), but more often, it will not. The problem also exists in reverse: if a variable existed in the session before the script was run, then it will have been clobbered afterwards.

The solution is to give the names of all strictly internal variables a unique prefix. We can even use the undefine command to clean up the session by removing all variables with a given prefix upon completion of the script.

Further possibilities with scripts

Scripts themselves may contain load or call commands. Their effect is to include the contents of the loaded file at the position of the load or call command. (Gnuplot safeguards against infinite recursive inclusion.)

When the command quit is used inside a script, it does not terminate the gnuplot session; instead, it terminates execution of the current script. It therefore acts similarly to a return statement in most programming languages, initiating an immediate return to the calling code (but without a return value, of course).

A worked example: Newton’s method (again)

Listing  11.2  is another implementation of Newton’s method that demonstrates some of the ideas we just learned. The script defines three internal variables, all of which are prefixed with NEWTON_ and are removed from the session on the last line of the script. Parameters are passed to the script implicitly: both the function f(x) and the starting value x must exist in the session before the script is called. Upon completion, the variable x holds the position of the found root. A sample use of this script might be (the last line is the final output from the script):

f(x) = cos(x**2); x=1.0
load "newton.gp"
f(1.253314137316) = -0.000000000001 in 4 steps

Listing 11.2. Newton’s method in gnuplot (file: newton.gp)

# Must prepopulate f(x) and x before call

NEWTON_steps = 0
NEWTON_eps = 1.e-6

while( abs(f(x)) > 1e-12 && NEWTON_steps < 15 ) {
  NEWTON_df = (f(x+NEWTON_eps) - f(x) )/NEWTON_eps
  x = x - f(x)/NEWTON_df
  NEWTON_steps = NEWTON_steps + 1
}
print sprintf("f(%.12f) = %.12f in %d steps", x, f(x), NEWTON_steps)

undefine NEWTON*

11.2.2  Worked example: export script

The script in listing  11.3  is intended to demonstrate how far one can push the “script-as-subroutine” concept in gnuplot. You decide whether this is a good idea.

The purpose of the script is my old standby: saving the commands of the last plot and exporting it in a graphics file format.[57] But this script does more:

It infers the desired file format by examining the extension of the target file.

The target file can include an absolute or relative path. The script will place both the command file and the graphics file into the specified directory.

It is possible to specify additional options when invoking this script. These options will be supplied to the chosen terminal when the graph is exported.

The script makes a number of (platform-specific) assumptions:

The filename extension indicating the desired graphics file format consists of exactly three letters.

The path separator is / (Unix convention—you need to change this if you are on a different platform).

The script itself is straightforward, but messy, because much of the string handling has to be done explicitly. Moreover, gnuplot does not follow the convention of most contemporary programming languages of using zero-offset strings and asymmetric bounds, thus necessitating the explicit conditional when splitting the argument into path and basename. (To reduce clutter, I have refrained from prefixing all internal variables to make them “private”.)

As I said, I regard this example as a “technology demonstrator”, that shows how one can encapsulate complex tasks in scripts and use them like subroutines. You decide whether you think this is the right approach.

Listing 11.3. Full-featured export script (file: mega-export.gp)

# Invoke as: call "mega-export.gp" "graph.png" "terminal options"
# File format MUST be a three-letter extension.

# Save arguments
arg = ARG1
opts = ARG2
len = strlen( arg )

# Find the last "/" in the filename
k = 0
do for [i = 1:len] {
  if( arg[i:i] eq "/" ) {
    k = i
  }
}

# Split the filename into path, file, and extension
if( k == 0 ) {         # No "/" found: use current dir
  path = ''
  file = arg[1:len-4]
} else {               # At least one "/" found
  path = arg[1:k]
  file = arg[k+1:len-4]
}
type = arg[len-2:]     # Assuming 3 letter extension

# Select a terminal based on the extension
term = ''
if( type eq 'eps' ) { term = 'epscairo' }
if( type eq 'pdf' ) { term = 'pdfcairo' }
if( type eq 'png' ) { term = 'pngcairo' }
if( type eq 'svg' ) { term = 'svg' }
if( term eq '' ) {
  print "Unknown file type - using pdf"
  term = 'pdfcairo'
}

save path . file . '.gp'

set t push
eval "set t " . term . " " . opts
set o path . file . "." . type
replot
set o
set t pop

print "Done!"

[57] If gnuplot should ever acquire a simple export command that bundles all the required steps into a single invocation, I’ll lose one of my best sources for examples.

11.3  Batch processing

Gnuplot can be run by itself as a standalone command interpreter, executing scripts from files—basically like a Perl or Python interpreter interpreting a program in its respective language. The only difference is that in gnuplot’s case, the final output is going to be a graph. Running gnuplot non-interactively as a command interpreter like this is known as batch mode.

[Tip]
Uses for batch mode

Batch mode is not terribly common, because we usually want to see and interact with the graph that was created. Batch mode is useful whenever a large number of file-based graphs need to be created using a predetermined set of commands, or when the same graphics file needs to be recreated again and again as the data set that is plotted changes.

Running gnuplot in batch mode is straightforward and equivalent to the way Perl or Python behave: any files listed after the gnuplot command itself are expected to contain gnuplot commands. They’re executed in the order specified, as if they had been loaded using load. Alternatively, gnuplot will read commands from standard input. There are thus three ways to execute a set of command files:

Using command-line arguments:

shell> gnuplot plot1.gp plot2.gp plot3.gp

Reading from standard input:

shell> cat plot1.gp plot2.gp plot3.gp | gnuplot

From within a gnuplot session:

load "plot1.gp"
load "plot2.gp"
load "plot3.gp"

Gnuplot doesn’t start an interactive session when invoked with command-line arguments: it just processes all commands (including any plot commands) and terminates. This implies that plots sent to an interactive terminal will usually not be visible, or rather, they’re visible for a tiny moment as gnuplot opens the terminal window, draws the graph, and immediately closes the plot window again and exits.[58] It’s a common mistake to forget to set appropriate (file-based) terminal and output options in gnuplot batch files!

[Tip]
Don’t forget to set the output channel when in batch mode

By default, gnuplot will use one of the interactive terminals for all graphical output. That’s great for interactive sessions, but makes no sense when using gnuplot in batch mode. Do not forget choose a meaningful output format and destination.

You can force an interactive session by using the special filename - (hyphen) on the command line. The following command will run a setup script before dropping you into an interactive session:

shell> gnuplot setup.gp -

11.3.1  Using gnuplot in shell pipelines

It is possible to use gnuplot as part of a command pipeline,[59] wherein gnuplot reads both commands and data from standard input. The special filename - (see section "sec_files_pseudofiles") tells gnuplot to read data from the current source; that is, from the same input device from which the most recent command was read. So if the command was read from standard input, then plot "-" will continue reading data from standard input as well. This makes it possible to build a pipeline like the following:

shell> cat script.gp data | gnuplot

Here, the command file script.gp may be as simple as:

set t pngcairo
set o "out.png"
plot "-" u 1:2 w lp

and the data file may contain any data set, as long as it has at least the two columns referenced in the plot command.

The pipeline concept can be taken a step further: remember that set output defaults to standard output (if no file name is given as argument). Now the script is even simpler:

set t pngcairo
plot "-" u 1:2 w lp

but in the pipeline we need to deal with the results that gnuplot sends to standard output. The easiest way is to capture them in a file:

shell> cat script.gp data | gnuplot > graph.png

but it would of course also be possible to pipe gnuplot’s output to some other postprocessing step instead.

A word of warning: to use gnuplot in a pipeline like that, both commands and data must be sent to standard input explicitly. It is not possible to just list them on the command line instead: gnuplot will not read a combination of command and data files from the command line.

[Tip]
Consider gnuplot as part of a pipeline

Using gnuplot in a shell pipeline may seem unusual at first, but it can be very convenient. Any pre- or post-processing can be done in one fell swoop. Moreover, sending the graph to standard output avoids all of gnuplot’s awkward file handling issues.

[58] You can also specify the command-line flag -persist when invoking gnuplot so that the plot remains visible on the screen.

[59] Provided the platform supports pipes, of course.

11.4  Calling gnuplot from other programs

Sometimes it is necessary or desirable to let gnuplot be controlled by another computer program. Three situations come to mind:

You need a level of automation and control that is difficult or impossible to achieve with gnuplot’s limited scripting capabilities.

You want to use gnuplot to visualize data that is generated by some other program without having to save the data to an intermediate file.

You would like to use gnuplot as a generic graphics backend for a system that has no graphics capabilities otherwise.

It is possible to use gnuplot in this way, and it is not even particularly difficult—but it isn’t especially convenient either. Gnuplot does not offer an API or a robust method of interprocess communication. Gnuplot has to be run as a subprocess, and all communication has to occur via pipes (where available), otherwise through the file system. Bidirectional communication (that is, sending data back from gnuplot to the controlling process) is particularly challenging.

[Tip]
Will batch mode do?

Before investing effort in one of the solutions sketched in the following sections, consider whether batch operations will be sufficient. In batch mode, your program creates a file with gnuplot commands (and a heredoc containing the data) on disk, and then invokes gnuplot on that file (in a subshell). The detour through the file system effectively decouples gnuplot from your program, which will make operations more robust and easier to debug.

In the following two sections, I demonstrate how to run gnuplot from either Perl or Python. The master process sends some data to gnuplot (via a pipe), and gnuplot creates a graphics file on disk—this is probably the most common case. The main differences between these examples concern the way Perl and Python handle subprocesses and have little to do with gnuplot itself. (The Python examples also makes use of a heredoc, whereas the Perl examples uses the - special file.)

11.4.1  Worked example: calling gnuplot from Perl

Perl, the “duct-tape of the internet”, is (still) a common choice for glueing processes together. It allows us to open a subprocess as a file handle—in the same way we’d open a file—provided the first character in the filename is a |. We can then write anything to this file handle using print. In the first line of listing  11.4  we open gnuplot as a subprocess and assign it to the file handle PROC. We then proceed to write some gnuplot options to this file handle, including the essential terminal information and output filename, followed by the plot command. The plot command is told to read from the special file -, which means that it expects the data to follow inline (see section "sec_files_pseudofiles"). The data stream is terminated by the single letter e, and then the file handle is closed, which terminates the gnuplot subprocess. If we forget to close the file handle explicitly, Perl won’t terminate while gnuplot is still running!

Listing 11.4. Calling gnuplot from Perl (file: driver.pl)

open PROC, "| /usr/bin/gnuplot " || die "Could not start gnuplot: $!";

print PROC "set xlabel 'Time'; set ylabel 'Amplitude';";

print PROC "set o 'graph.png';";
print PROC "set t pngcairo;";

print PROC "plot '-' u 1:2 t 'data' w l\n";

for( $x = 0.0; $x  <= 10.0; $x += 0.1 ) {
  $y = exp(-$x/5)*sin( $x );
  print PROC "$x $y\n";
}
print PROC "e\n";

close PROC;

11.4.2  Worked example: calling gnuplot from Python

The listing  11.5  shows a very simple Python program that uses gnuplot to create a graph. It uses the Popen facility from the subprocess module in the standard library. The constructor returns an object that provides a handle to the subprocess. The script sends commands to the subprocess via the stdin member of this handle. In contrast to the Perl example in the previous section, this script does not use the special - file name. Instead, it creates a heredoc named $d and references it in the plot command. The heredoc is particularly useful here, because we are reading two columns of data. If we had used the - file name, we would have had to loop through the data twice: once for the first column, and once for the second. The heredoc allows us to submit all the data to gnuplot in one fell swoop and then access it as needed.

Listing 11.5. Calling gnuplot from Python (file: driver.py)

import subprocess

data = [ [0, 12, 25],     # x, y1, y2
         [1, 14, 22],
         [2, 15, 23],
         [3, 11, 25] ]

gnuplot = "/usr/bin/gnuplot"

gp = subprocess.Popen( gnuplot, stdin=subprocess.PIPE )

gp.stdin.write( "$d << EOD\n" )
for row in data:
    for col in row:
        gp.stdin.write( "%f\t" % (col,) )
    gp.stdin.write( "\n" )
gp.stdin.write( "EOD\n" )

gp.stdin.write( "set t pngcairo; set o 'graph.png';\n" )
gp.stdin.write( "plot $d u 1:2 w l, $d u 1:3 w l\n" )

11.4.3  Helpful hints

Using gnuplot in such a way from another program works quite well, but can appear a bit fickle at first because we need to re-create exactly those conditions that are usually fulfilled by input coming from the interactive command-line environment. Diagnosing glitches in this area isn’t helped by error messages, which are intended for interactive use. Here’s a checklist of trouble spots to look for when things don’t work out at first:

Commands must be separated from one another by explicit semicolons or newlines. A common mistake is to write code like this (also see listing  11.4 

print PROC "set key";     # WRONG - missing newline or semicolon!
print PROC "set border";

The two set commands appear to be broken down onto two separate lines, but gnuplot will see them as one consecutive string. This will work:

print PROC "set key\n";   # RIGHT - explicit newline
print PROC "set border";

Instead of the newline, a semicolon could have been used as well.

The line containing the plot command must be terminated by an explicit newline. Gnuplot doesn’t parse a command line until it has encountered a newline. Because the plot command is the last command, it must end with an explicit newline.

You must use an explicit using directive when using the special filename - to tell gnuplot how to parse the incoming data stream.

For each occurrence of - in the plot command, there must be a separate data stream. Gnuplot will continue to interpret incoming data as data until it has encountered a corresponding number of end-of-file characters. (Another reason heredocs are probably more convenient in this situation.)

Keep in mind that some characters in gnuplot code may have a special meaning in a programming environment. For example the $ character, frequently used to indicate the value of a column in gnuplot, is also used by Perl and Unix shells. To prevent interpretation by the programming environment, it needs to be protected (that is, escaped). Alternatively, you can use the column() function, which eliminates the need for the $ character in gnuplot code.

Don’t forget to separate data lines from one another using newlines as well.

One final comment: when generating many graphs from the same program, it is usually a good idea to start gnuplot only once and use a single process instance for all of the graphs, rather than starting a separate gnuplot process for each of them. This doesn’t matter much when preparing two or three graphs, but when the number of graphs is large, the time savings are significant. Just make sure to reset all relevant options (and specifically the output filename) between invocations of the plot command.

Thoughts for the design of a gnuplot access layer

If you find that you need to drive gnuplot from some other program a lot, it might be worth investing in an access layer. A quick internet search will in fact turn up several gnuplot “wrappers” for a variety of languages. Most of them attempt to replicate gnuplot’s set of commands and options as API in the host language. This approach is fundamentally mistaken in my opinion, because whenever new features are added to gnuplot or existing ones change, the access layer is no longer up to date.

Instead, I would recommend building an access layer in the spirit of the “thin” database wrappers that exist in all common programming languages for interfacing to relational databases. They do not try to replicate SQL in the host language, they merely provide a way to submit well-formed SQL to the database and to move data back and forth.

I think such an access layer needs to have only three basic functions:

open(): Starts the subprocess and maintains a handle to it. Hides all the messy details.

exec(): Takes a string (or an array of strings) and sends them to gnuplot. It is the user’s responsibility to make sure that the strings contain valid gnuplot commands, but the access layer may apply some safeguards against common oversights, such as ensuring that each command is terminated with a semicolon or a newline.

data(): This takes the identifier for a heredoc (such as "$d") and a two-dimensional data structure in the host language. It turns the data structure into a sequence of strings, each string corresponding to one line in the data “file” and then sends commands to gnuplot that create a heredoc under the given name, containing the supplied data. Subsequent plot commands can then reference this heredoc by its identifier.

The primary idea here is to play on the strengths of the host language as much as possible, rather than trying to replicate gnuplot’s command set. For example, the user should be able to build up a native data structure and manipulate it in the host language—only at the last moment does the access layer turn it into strings that can be passed to gnuplot as a “data file”. The heredoc facility comes in handy, because it means that data need only be submitted to gnuplot once.

These are just considerations that you might find useful if you find yourself designing an access layer for gnuplot. Because every language is different, it does not make sense to provide a skeleton implementation here: in particular what is considered the most suitable “native” data structure is likely to differ so much from one host language to the next that little can be said in general. Also keep in mind that getting the edge cases right is going to require a disproportionate amount of effort (this is particularly true for the serialization of the data structure into strings). A one-off, specialized implementation that takes the specifics of the intended usage into account may be able to skirt some of these issues.

Finally, be aware of the false sense of security that such an access layer instills. A proper API would let you detect and recover from error conditions in the encapsulated code (by throwing exceptions or returning error codes). This is not true for gnuplot, because gnuplot has no way of notifying you when things go wrong: it just spews (human-readable) error messages to standard error. It is difficult to respond programmatically to that.

11.5  Animations

All the facilities we have encountered in this chapter support automation in some form: events occurring without human intervention. We can take this a step further and think about a series of graphs being created, by themselves, in sequence. In other words, we have all the ingredients for animation: moving pictures.

11.5.1  Introducing a delay

To achieve satisfactory animation, we will need one additional feature, namely a controlled delay between successive figures. This is where the pause command comes in. The pause command takes the number of seconds to wait (-1 will wait until a carriage return is encountered, or—if mousing is enabled—a mouse click has occurred) and an optional string argument, which will be printed to the command window to prompt the user:

pause {int:seconds} [ "{str:message}" ]

The pause command is usually used in conjunction with one of the loop constructs.[60] For a first example, let’s go back to the Taylor series example in section “Worked examples: Taylor series and Newton’s method” earlier in this chapter. Using pause, we can create an animated version of this example:

f(x,n) = sum[k=1:n] (-1)**(k-1)*x**(2*k-1)/(2*k-1)!

do for[n=1:8] {
  plot [-8:8][-1.2:1.2] sin(x) lw 2, f(x,n) t sprintf("%d Terms",n)
  pause 2
}

This will plot a new graph, each one representing a better approximation than the one before, every two seconds. (Try it yourself—there is no point of having a static graph of an animated effect.)

11.5.2  Waiting for a user event

There’s a second form of the pause command that waits until a specific user event has occurred:

pause mouse [ {eventmask} ] [ "{str:message}" ]

The event mask can contain any combination of the following keywords, separated by commas: keypress, button1, button2, button3, close (meaning that the plot window was closed) and any. If pause was terminated through a keyboard event, the ASCII value of the selected key will be stored in the gnuplot variable MOUSE_KEY and the corresponding character in the variable MOUSE_CHAR. If a mouse event occurred, the mouse coordinates will be stored in the variables MOUSE_X, MOUSE_Y or MOUSE_X2, MOUSE_Y2, respectively, and available for further processing. The command pause mouse (without an event mask) is equivalent to pause -1.

11.5.3  Further examples

Some of the most striking applications of animations involve three-dimensional graphics, which we will discuss in detail in appendix "app_threedim" and appendix "app_math", but I want to give to quick demos here (see listings  11.6  and  11.7 .[61]

The logic of both listings is straightforward: a parameter-dependent graph is continuously recreated inside a while loop. Because the parameter is slowly changed at the same time, no two successive graphs are exactly equal, giving the appearance of a smooth change.

It is essential to include a delay in the loop (using pause) and to provide an explicit break condition for the loop—gnuplot does not recover gracefully if the loop is stopped using Ctrl-C. The animations in the example scripts can be stopped by pressing the x key on your keyboard (we will formally introduce the bind command in chapter  12, Beyond the defaults: workflow and styles).

[Tip]
Include an explicit break condition in animations

Always include an explicit break condition; gnuplot does not recover if the loop is not stopped cleanly.

For another stunning example of what is possible using gnuplot animations, check out the demo/games directory of the gnuplot distribution. There you will find, among other things, a rather addictive implementation of a Tetris-like game—entirely in gnuplot!

Listing 11.6. Animation demo (file: pebble.gp)

unset key; unset border; unset tics

bind "x" "end=1"

end = 0; t = 0
while( end==0 ) {
  t = t + 0.1
  splot [][][-1:1] exp(-0.2*sqrt(x**2+y**2))*cos(sqrt(x**2+y**2) - t)
  pause 0.001
}

Listing 11.7. Animation demo (file: globe.gp)

unset key; unset border; unset tics
set hidden3d

set mapping spherical; set angles degrees
set parametric; set urange [-90:90]; set vrange [0:360]

bind "x" "end=1"

end = 0; t = 0
while( end==0 ) {
  t = (t + 1)%360

  set view 90-30*cos(t), 360-t, 1.5, 1.25

  splot cos(u)*cos(v),-cos(u)*sin(v),sin(u) w l lc rgb "grey",
» "world_110m.txt" w l lt 1

  pause 0.01
}

[60] In previous gnuplot versions, the same effect was achieved using the reread command. When included in a command file, the reread command instructs gnuplot to begin executing the current file from the beginning, resulting in an infinite loop.

[61] The file world_110m.txt required by the second listing can be found at http://www.gnuplotting.org/plotting-the-world-revisited/.

11.6  Case study: continuously monitoring a live data stream

Monitoring a live stream:[62] that means animated figures that are continuously changing as they are being updated with newly arriving data. This is not a situation that gnuplot was designed for, but—as it turns out—it can handle it quite gracefully.

In this section, I will present several different approaches to this problem. We will start with a very basic, ad-hoc solution; the final solution is more involved and can be regarded as the beginnings of a production-level real-time monitoring system that uses gnuplot as its graphics backend. Some of the initial ideas make use of Unix tools for simplicity, but the final solution is fully general and can, in principle, be adapted to any platform.

11.6.1  Using gnuplot to monitor a file

First, let’s assume that you would like to monitor the tail end of a log file. Messages are appended to the log file asynchronously by some other process (or processes). For demonstration purposes, let’s assume that each log entry consists of a timestamp, followed by the actual value. Our desire is to have a constantly updated plot of the last few entries in the log file.

The minimal gnuplot script to achieve this goal is just a few lines long:

while( 1 ) {
  plot "< tail -200 logfile" u 0:2 w lp
  pause 0.1
}

The script consists of an infinite loop, which is executed 10 times a second (pause 0.1). The plot command uses the Unix tail facility to extract the last 200 lines from the log file. The output from tail is piped directly to gnuplot, without accessing the file system. The pseudocolumn 0 (that is, the line number in the actual data set) is used for the x coordinate. (See chapter  4, Managing data sets and files for information on pseudofiles and pseudocolumns.)

This works quite well, although the result is not very pretty (we will remedy this next). But as a minimal-effort solution this script can’t be beat. It should be committed to memory for all situations where you have an ad-hoc need for real-time monitoring of data.

A data generator for testing

If you want to try the solutions described in this section and you don’t have access to a live data stream, you might find the script in listing  11.8, “A random data generator (file: datagen.py)” helpful. It generates data at random intervals; you must specify the average time between write events as command-line parameter. So, to have the script generate about 10 events per second, you would call it like this (depending on your needs, you may want to redirect the generator’s output to a file):

shell> python -u datagen.py 0.1

The -u option forces Python to use unbuffered IO. This is pretty important in this application, since we are interested in seeing “real-time” data. Don’t forget. The generated “data” here is simply a sine-wave. The script writes out about 20 points per period of the sine, regardless of the specified wait time between write events.

Listing 11.8. A random data generator (file: datagen.py)

import sys
import time
import math
import random

tmax = float( sys.argv[1] )

t = 0
while 1:
    dt = random.uniform( 0, 2.0*tmax )
    time.sleep(dt)
    t += dt

    print t, math.sin(0.1*math.pi*t/tmax)

A better monitor

Let’s improve the monitoring script given earlier. The data does contain time stamps, but so far, we are not using them. The graph therefore gives the impression that log entries are equally spaced in time—which is not true.

But if we try using the time stamp as x coordinate, the visual effect is unsatisfactory. The reason has to do with the way gnuplot handles the plot range: unless instructed otherwise, it will make sure that all data points are included and it will extend the plot range to the next tic mark. But because data points occur asynchronously (at random times), the range of seconds spanned by the points in the sample changes, as new points are added and old points are removed. This constant change in the plot range leads to the undesirable “flicker” effect.

We therefore need to fix the plot range. The script in listing  11.9, “A real-time data monitor (file: monitor.gp)” does just that.

Listing 11.9. A real-time data monitor (file: monitor.gp)

logname = "logfile"
tmplen  = 200
tmpname = "tmp"

bind "x" "end=1"
end = 0
while( end==0 ) {
  system sprintf( "tail -%d %s > %s", tmplen, logname, tmpname )
  stats [*:*][*:*] "tmp" u 1:2 nooutput
  set xrange [STATS_max_x-11.0:STATS_max_x+1]
  plot "tmp" u 1:2 w lp
  pause .01
}

The first few lines of the script are used to configure the script to local conditions: you must supply the name of the log file to monitor, as well as the name of a temporary file and the number of records that will be saved to it. (We will come back to the latter parameter later.)

The script then invokes the Unix tail utility via gnuplot’s system command to dump the configured number of data points from the end of the log file into a temporary file. The stats command is used to determine the value of the last (most recent) time stamp in this temporary file. The set xrange command then chooses a fixed plot range that is based on this value: the plot range extends one second into the future and eleven seconds into the past, measured from the last entry in the temporary file. (That the upper limit of the plot range is not made to coincide with the largest data point is sheerly for visual effect. Try it both ways: personally, I find it helpful to see explicitly where the available data ends.) The range indicators on the stats command are important: without it, stats would discard data outside the plot range, and hence fail to reflect changes to the input file.

The result works quite well in practice. (Try it!) Here are a few additional ideas you might want to explore:

The script contains three adjustable parameters: the wait time in the pause command, number of lines that tail extracts, and the size of the plot range in seconds. All three should depend on the frequency with which entries are added to the log file. Begin by deciding how many previous seconds you want to see in the graph. If you know at what rate new lines are appended to the log file, you can figure out how many records you need to extract. The wait time in pause should be smaller (by a factor of 2–10) than the time between log file updates to ensure smooth animation, but keep in mind that reducing this number drives resource consumption up.

Listing  11.9, “A real-time data monitor (file: monitor.gp)” uses the Unix tail command to extract a certain number of records from the log file to a temporary file, which is then read by both stats and plot. Both the tail command and the temporary file are optional: in principle, you could rely entirely on the plot range to restrict the data and run both stats and plot directly on the log file. But if the log file is long (and which log file is ever short?), then the tail command is likely to be faster and less resource-intensive than gnuplot’s own (and very complex) data file parser (which is invoked by both stats and plot). Furthermore, by creating a temporary file, it is guaranteed that both stats and plot see exactly the same data: if you simply let both commands run on the log file in sequence, you introduce a possible race condition wherein the log file is being updated between the two commands. You decide whether any of these precautions are necessary or desirable for your situation!

11.6.2  Using a driver to monitor arbitrary data sources

The solution in the previous section is nice, because it is implemented entirely using gnuplot’s own facilities. But it also has some drawbacks:

It’s a polling solution: gnuplot polls its input source constantly, whether there is new data or not. As a rule, polling architectures are more resource-intensive than other solutions.

It’s not flexible, because any processing of the data would have to be accomplished using gnuplot’s limited scripting capabilities.

The approach presented in this section is different, because it relies on an intermediate layer, written in a general-purpose programming language (Python in this case, but it could, of course, be anything). The script in listing  11.10, “A data monitor in Python (file: monitor.py)” acts as a filter: it gathers data from standard input, and passes it on to gnuplot only if new data is available. It therefore implements a “push” model, rather than the “pull” model of the previous solutions. The script also acts as a buffer, storing the most recent set of points and passing them to gnuplot. This is necessary, because we now skip the file system entirely and gnuplot itself does not retain data between plots.

Listing 11.10. A data monitor in Python (file: monitor.py)

import sys
import subprocess

buflen = 200                       # number of data points to buffer

gnuplot = "/usr/bin/gnuplot"

gp = subprocess.Popen( gnuplot, stdin=subprocess.PIPE )
gp.stdin.write( "set t wxt\n" )

buf = []
while 1:
    line = sys.stdin.readline()    # read unbuffered from stdin

    t, u = line.strip().split()
    buf.append( (float(t), float(u)) )

    buf = buf[-buflen:]            # drop oldest data points,
                                   # retaining at most buflen points

    mx = buf[-1][0]                # most recent time stamp
    gp.stdin.write( "set xrange [%f:%f]\n" % ( mx-10, mx+1 ) )

    gp.stdin.write( "plot '-' u 1:2 w lp\n" )
    for d in buf:
        gp.stdin.write( "%f %f\n" % d )
    gp.stdin.write( "e\n" )

The script runs gnuplot as a subprocess, and communicates with gnuplot by writing to gnuplot’s standard input via a pipe. The script itself reads incoming data from its own standard input (not from a file), splits and parses the intake, and pushes it onto a buffer, before passing it on to gnuplot. In this case, there is no need for gnuplot’s stats command, because now the determination of the plot range is done inside of the Python script.

The script is straightforward, only two Python-specific implementation details require a word of comment:

To achieve real-time behavior, it is necessary that all input and output is unbuffered. This is the reason that the script uses sys.stdin.readline() inside a while loop, instead of the more familiar idiom for line in sys.stdin:. Also, make sure to run this script using the -u command-line option.

The line buf = buf[-buflen:] retains the last buflen elements in the array. If the array contains fewer than buflen elements, then all of them are retained.

It should be clear that listing  11.10, “A data monitor in Python (file: monitor.py)” is only the beginning. Not only are all features missing that would make the program more robust (error handling and so on), but the function of the script could be extended in many ways:

Observe and combine multiple input sources. The script in listing  11.10, “A data monitor in Python (file: monitor.py)” listens to its own standard input—instead, it could keep tabs on a range of different logfiles, combining data from all of them, and generating a unified status plot.

Conversely, split a single, heterogeneous input source into multiple data streams, and maintain several gnuplot processes, each of which is used to plot only a single output channel.

Statistical manipulation of the data, such as outlier and trend detection. Add the output of a model to the raw output so that both can be plotted together and compared easily.

Anything at all you can think of...

The options are endless, but have little to do with our primary topic, which is gnuplot itself. Listing  11.10, “A data monitor in Python (file: monitor.py)” is only intended to get you started.

[62] This section was partially inspired by the blog post “Visualize real-time data streams with Gnuplot” by Thanassis Tsiodras. It can be found here: http://users.softlab.ece.ntua.gr/~ttsiod/gnuplotStreaming.html.

11.7  Summary

In this chapter we learned about ways to automate gnuplot tasks. Gnuplot itself provides some programming constructs, such as loops and conditionals, but for more complicated tasks, it may be more convenient to “drive” gnuplot from an external programming language as a subprocess.

Another way to automate tasks is to capture them all the required gnuplot commands in scripts that can be run by themselves, as batch processes or even as part of Unix pipelines. In chapter ???, we’ll learn about additional uses for script files.

Gnuplot’s automation capabilities enable the creation of animated graphics. We demonstrated the basic principles, and then used them in a particularly interesting application, namely the real-time monitoring of a live data stream.

12

Beyond the defaults: workflow and styles

This chapter covers:

The standard workflow and some variations

Using external editors and viewers

Hot keys and mousing

Configuring your workspace

Defining and using stylesheets

Overall, gnuplot's default's are pretty good—so good, in fact, that it's easy to forget that they are not set in stone, and that there there might be good reasons for changing them. This applies not only to the visual appearance of plots, but also to the way the user interacts with gnuplot itself.

In this chapter, we will explore some ways to customize your gnuplot experience. First, we consider workflow alternatives: ways to organize your interaction with gnuplot, beyond the regular, interactive session. We will discuss various shortcuts and conveniences, and also discuss the possibility of using an external editor for authoring gnuplot command files, rather than typing in commands interactively at the gnuplot prompt.

Next come various topics that are more typically associated with “user customization”: custom key (and mouse) bindings, and the definition of private option settings and configurations.

Finally, I introduce the idea of “stylesheets”: collections of appearance options for lines and points, that can be applied to graphs to give them a different appearance. In particular, defining and choosing stylesheets judiciously makes it possible to create graphs with specific purposes in mind, be it a computer screen, a black-and-white publication, or an overhead presentation.

12.1  The standard interactive workflow

Gnuplot was originally conceived as an interactive tool, and this is still the most “natural” way to use gnuplot: you start an interactive session, enter commands at the gnuplot prompt, examine the resulting graph, and iterate. Frequently, such an interactive session results in a saved command file that can be used to recreate the final state, as well as a final graph, exported in some graphics file format. In this section, I want to review some ideas for improving this straightforward workflow.

12.1.1  Extracting specifics from command files

I have pointed out several times before that it is not enough to just export the final graph as PNG or PDF—it is crucial to also save the final set of options (using save). Only then can you recreate the graph in gnuplot: to examine it again, to modify an aspect, or to update it with new data.

The files that the save command generates are long: they contain the values of all of gnuplot’s options,[63] as well as all user-defined functions and variables. Most of this information is redundant: options are written to file, even if they still have their default values. The advantage is, of course, that the resulting file is guaranteed to be self-contained. But the information that is specific to the current is hard to discern.

We can recover the plot-specific information by comparing the saved file against a file that contains only default values. This “baseline” file can be obtained by starting gnuplot, and—without doing anything else—issuing a save command.

Because the file format of gnuplot’s command files is so simple (line-oriented text files, with one command or option per line), it is easy to write a short program in one’s language of choice to compare the two files.[64] Wherever the GNU diff utility is available, it can be used to good effect as well (the cryptic command-line options instruct diff to print only lines that are new or different in the second file):

shell> diff --new-line-format="%L" --old-line-format=""
»           --unchanged-line-format="" default.gp graph.gp

The resulting file is short and neat, but it is less “self-contained” than the original file was, because it does not contain a complete set of options. If the shortened file is loaded into a gnuplot session that is configured differently from the baseline file, the resulting graph will be different than the original. Also, if gnuplot’s defaults change, the graph created by the shortened file will be affected. This is something to be aware of.

12.1.2  Extending the command set

Sometimes, it takes several gnuplot commands to achieve a common task (exporting to file, for instance), sometimes the syntax is awkward and hard to remember (setting up a palette for false-color plots), and sometimes both (time series scaling for the x axis). It would be nice if we could define our own commands to take the drudgery out of it. There are basically two mechanisms available to us:

Build up a string of commands and execute it using eval (see section "sec_practical_eval").

Capture commands in a command file and run it using either load or call (see section “Scripts as subroutines”).

Both methods are almost equivalent—the main difference between them lies in the way they are called. Let’s look at some examples. (Either method can also be bound to a hot key—see section “Custom hot keys” later in this chapter.)

Using scriptlets

It is important to remember that command files do not need to be “complete”: a command file that only defines some functions or sets some options is perfectly legitimate and often useful. Listings  12.1  and  12.2  show two examples that do just that: listing  12.1  merely defines two distributions that I frequently need in my work, whereas listing  12.2  switches on time series mode, using a specific timestamp format. The real benefit of the latter example is that now I don’t need to remember the conversion specifiers! (I can never remember whether it’s minutes or months that go with an uppercase “M” as opposed to a lowercase one.)

It is also possible to collect style definitions and appearance options (line widths and colors) together to form “stylesheets”. This idea is so important that it is given its own section (section “Stylesheets”) later in this chapter.

Gnuplot looks for scripts invoked with load or call first in the current working directly, and then in the directories listed in the loadpath option.[65] The loadpath option can be set using set loadpath; if the GNUPLOT_LIB environment variable is set, its contents are appended to loadpath (see section “Startup and initialization files”). It is therefore possible to create a library of useful scripts in a personal collection.

Listing 12.1. Scriptlet defining common functions (file: normdist.gp)

sqrt2pi = sqrt(2.0*pi)

gauss(x,m,s) = exp(-0.5*((x-m)/s)**2)/(sqrt2pi*s)
lognorm(x,m,s) = exp(-0.5*((log(x)-m)/s)**2)/(sqrt2pi*x*s)

Listing 12.2. Scriptlet to activate time-series mode (file: timeseries-mode.gp)

set xdata time
set xtics format "%T"
set timefmt "%H:%M:%S"

Using eval

Strings containing valid gnuplot commands can be executed using eval. Furthermore, we can build strings up dynamically for a specific purpose, or—even better—write functions to do this for us.

Here is a really simple example. Switching to polar mode usually requires several commands (see section "sec_math_polar"). Combining them all into one string, we can achieve the same effect with a single call:

polar = "set polar; set size sq; unset border; unset tics; set rtics; set grid polar; set key outside"

Now, eval polar will not only turn on polar mode, but also set up suitable appearance options.

Switching on time series mode can be accomplished in a similar fashion. If the following strings exist in the current gnuplot session:

times = "set xdata time; set xtics format '%T'; set timefmt '%H:%M:%S'"
dates = "set xdata time; set xtics format '%F'; set timefmt '%Y-%m-%d'"

then time series mode can be turned on simply by using eval times or eval dates. (Of course, the format of the timestamps in the input file must match the format specified in these strings.)

Finally, and as an example for a function that generates a string dynamically, consider again the export function introduced in section "sec_practical_eval" (see listing  12.3 . It generates a string that—when executed using eval—exports a graph, using a user-supplied file name and terminal. In fact, the function can be called directly from within the eval command:

eval export( "graph.pdf", "pdfcairo" )

Listing 12.3. A function that generates a command-string dynamically (file: export.gp)

export( file, terminal ) = sprintf(
»       "set t push; set t %s; set o '%s'; replot; set o; set t pop",
»       terminal, file )

Additional considerations

On startup, gnuplot reads an initialization file (see section “Startup and initialization files” later in this chapter) and it may seem natural to place as many definitions as possible into this file so that they are always available. Over time, I have moved away from this, and instead would suggest to use many smaller files each containing only closely related definitions, which can be loaded as needed, in order to avoid polluting the gnuplot session with all kinds of rarely used functionality.

In fact, this is one of the main differences between scriptlets and dynamically evaluated strings. The strings (or functions that generate them given a set of arguments) must exist in the gnuplot session, whereas scriptlets do not. Furthermore, anything that lives in the gnuplot session may be overwritten at any time and without warning; scriptlets, by contrast, offer a degree of “encapsulation”.

Finally, it is possible to “bind” either scriptlets or strings to be evaluated with eval to keyboard events. In that case, simply pressing the appropriate key invokes the desired action. (We will discuss this later in this chapter, in section “Custom hot keys”.)

12.1.3  Session variables, loops, and macros

Don’t forget that it is often possible to streamline your workflow using the techniques introduced in chapter  5, Practical matters: strings, loops, history: session variables, string macros, and inline loops. Inline loops don’t need any further explanation (we have seen plenty of examples in the meantime), but here are some ideas for the the use of session variables and macros.

Using variables

Imagine that you want to work with a data set that is in a file with an inconveniently long file name (possibly not even in the current directory). Simply assigning the name to a variable helps to keep command lines much shorter:

FILE = "../path/to/directory/long-and-painful-filename.dat"
stats FILE u 2 nooutput
plot FILE u 1:($2/STATS_mean) w lp

Another possibility is to assign just the base name of the file to a variable, and add extensions on the fly:

BASE = "long-and-painful-filename"

plot BASE . ".dat" u 1:2

save BASE . ".gp"

set terminal pdfcairo
set output BASE . ".pdf"
replot

You get the idea.

Using macros

Macros are another way to shorten command lines (see section "sec_practical_macros"). Imagine that you want to plot several files using the with xyerrorbars style. That style requires six columns to be named in the using clause. A macro may come in handy:

cols = "using 1:2:3:4:5:6"
plot "data1" @cols w xyerrorbars, "data2" @cols w xyerrobars

In fact, you could have made the with xyerrorbars part of the macro itself! Macros can also help straighten out complicated inline transformations:

trsf = "using ($1-$2/$3):(sqrt((abs($4-$5)/$6)))"
plot "data1" @trsf, "data2" @trsf

Yet a third idea is to use macros to shorten appearance options, in particular colors (see section "sec_appearance_color"). Compare this command:

plot "data" u 1:2 lc rgb "red", "" u 1:3 lc rgb "green",
»    "" u 1:4 lc rgb "blue"

with this:

r = "lc rgb 'red'"
g = "lc rgb 'green'"
b = "lc rgb 'blue'"

plot "data" u 1:2 @r, "" u 1:3 @g, "" u 1:4 @b

Defining line styles is an alternative, of course.

[63] With three exceptions: the set output, set terminal, and set linetype settings.

[64] One such program can be found among the contributed scripts on the gnuplot homepage at www.gnuplot.info.

[65] Gnuplot not only uses loadpath for scripts, but also for data: it searches first in the current directory, then in the directories listed in loadpath, for data files named in calls to plot or splot.

12.2  Using external editors and viewers

In everything so far, I have assumed that you were using gnuplot interactively: that you have started gnuplot and are typing commands at gnuplot’s own command prompt and that you viewed the generated plots in one of gnuplot’s interactive terminals (such as the wxt or qt terminal if you are working on Linux). But it turns out that there are other ways to interact with gnuplot, even for interactive, non-batch work. (Batch-oriented operations are treated in chapter  11, Automation, scripting, and animation.)

Rather than entering commands at gnuplot’s prompt, you may find it convenient to use your favorite text editor to type commands into a command file, and then let gnuplot run this file as a script.

One way to do this is to run an interactive gnuplot session in another window.[66] You then load the script using load and examine the graph in one of the interactive terminals.

Alternatively, you may choose to work with one of the file-based terminals and export to graphs in PNG or PDF format directly. If you do this, you will need to specify the output terminal and filename in your script before the plot command:

set terminal pdfcairo
set output "graph.pdf"
plot sin(x)

Now you don’t even need to start an interactive session; instead you run gnuplot at the shell prompt, giving the name of your script as command-line argument to gnuplot:

shell> gnuplot script.gp

and view the resulting graph in a separate viewer (such as okular or evince). Modern viewers usually will detect when the file they are displaying changes on disk, so that they will automatically reload and redisplay the graph after you made some change to your script and ran gnuplot again, leading to a comfortable workflow experience.

Alternatively, you can load the command file into a gnuplot session using load, but in this case your script must contain the command set output after the plot command to ensure that all output buffers are flushed. Running gnuplot from the shell prompt avoids this need, because now the gnuplot process itself terminates after completion of the script, and that forces all open files to be flushed.

Using an external editor (and, possibly, an external viewer) has some advantages compared to running gnuplot in an interactive session:

All commands and option values are simultaneously visible in the command file.

Editing might be more convenient in a proper editor.

You can never forget to save your commands (not just your graph).

For file-based formats: there can be no discrepancy between the appearance of the graph during the interactive session and the final graph saved to disk. (This can be a problem when preparing a graph using an interactive terminal, but then exporting the final version as PNG or PDF.)

Because multiplot mode (see section "sec_special_multiplot") does not permit you to change the output and terminal settings, the command-file based approach is the only practical way to create a multiplot graph.

The primary disadvantage of using an external editor is the need to run a separate window (which can be mitigated if there is a gnuplot plugin available for your editor). When using file-based output formats and external viewer, you lose the ability to interact with the graph using the mouse—which often does not matter, but can be quite painful at times.

[Tip]
External editor and viewer

Consider using an external editor and viewer. You might like it!

[66] Depending on your text editor, you may not even need a separate window. If you use Emacs, for instance, an Internet search will turn up not one, but two gnuplot-modes that let you run gnuplot from within your Emacs window.

12.3  Invoking shell commands from gnuplot

Gnuplot can’t do everything alone. It is occasionally very useful to invoke other programs within a gnuplot session. There are four ways to do this:

The system command executes a command in a subshell, directing its output to the interactive gnuplot terminal. For example, you can list the contents of the current directory with system "ls".

The system() function is similar to the system command. It takes a command as argument, executes it as a subprocess, and returns its output as a string (in contrast to the system command, which always directs any output from the subprocess to the interactive terminal).

Back ticks or back quotes (decimal ASCII code 96—typically bound to the leftmost key in the top row on the keyboard) may be familiar from Perl and some Unix shells. The command in back quotes is executed, and its output replaces the back-quoted text (including the back quotes). Back quote substitution can be used anywhere, except inside of single-quoted strings.

The shell command suspends the current gnuplot session and starts an interactive shell. Quitting the shell (typically by typing exit or Control-D) resumes the gnuplot session. (The shell command is somewhat of a legacy from days before windowing systems—today, we’re more likely to pop up a shell in another terminal window, rather than suspending the process in the current window.)

In general, the system() function is more versatile than back ticks, and the preferred way to capture output from a subprocess. But back ticks have the advantage that they can occur inside of double-quoted strings, whereas the output from the system() function would have to be concatenated explicitly to the string. In other words, the following two commands are equivalent:

set label 1 "Today: `date`" at 0,0
set label 2 "Today: " . system( "date" ) at 0,1

For simple string substitutions such as these, back ticks can be a convenient short-hand. For more complicated operations, the system() function is the better choice.

12.3.1  Worked example: plotting each file in a directory

Shell commands can be combined with loops. To plot every file in the current directory, either of the following two commands would work:

files = system( "ls" )            # using the system() function
plot for[f in files] f u 1:2

plot for[f in "`ls`"] f u 1:2    # using back ticks

12.4  Hot keys and mousing

When we issue a plot command, gnuplot pops up a new window containing the plot (assuming of course that you are working with one of the interactive terminals, and not currently exporting graphs to file). The new window containing the graph is automatically raised or active.

When a graphics window is active like this, we can invoke gnuplot commands through keyboard shortcuts or mouse clicks. In this section, we look at some of the default bindings provided by gnuplot, and will also learn how to define our own keyboard or mouse shortcuts.

12.4.1  Default hot keys

Hitting the spacebar when the plot window is active raises the gnuplot command window. This is convenient when doing interactive work: you can iteratively continue working on a single graph without ever having to take your hands off the keyboard!

Grid lines can be toggled on or off through the hot key g. Logarithmic plotting is bound to l (y axis only) and L (in this case, gnuplot scales only the axis closest to the mouse pointer logarithmically). The + and - keys can be used to zoom in and out of the plot. Hitting q closes the current plot window.

These and some other interesting default bindings are listed in table  12.1 , and we can get a display of all currently defined key bindings by pressing h while the plot window is active.

I don’t find most of the default bindings all that useful (and some outright annoying), but custom hot keys (see section “Custom hot keys”) can be very useful indeed.

Table 12.1. Selected default hot key bindings

Key

Function

Spacebar

Raise command window and switch keyboard focus to it

q

Close the current plot window

g

Toggle grid lines on the plot

r

Toggle crosshair (ruler) at current mouse position

l (lowercase letter L)

Toggle y axis logarithmic scale

L

Toggle logarithmic scaling for the axis closest to the mouse pointer

+

Zoom in

-

Zoom out

u

Unzoom (after zooming using the mouse)

h

Help: show all key bindings

Arrow keys

Scroll the graph in the indicated direction.

12.4.2  Mousing

In interactive terminals, we can use the mouse to navigate the graph. By default, mouse actions should be enabled, but in case they aren’t, we can enable them using the command

set mouse

When the mouse is active, the current coordinates are always displayed at the bottom of the plot window. We can place a temporary crosshair at the current mouse position by hitting the r key. Hitting r again switches the crosshair off. While the crosshair is active, its location and the relative distance of the mouse pointer from the crosshair coordinate are shown in the plot window, together with the absolute coordinates of the mouse pointer.

Clicking a key entry toggles the corresponding curve in the graph on or off.[67]

We can use the mouse to zoom in on a graph by dragging the mouse while holding mouse button 3 (often the right mouse button). Click into the graph with mouse button 3 and drag while holding the button down, then click with mouse button 1 to replot only the section of the graph in the indicated region. Hit u (unzoom) to return to the previous setting. When using the mouse to set the plot range in this way, both the x and the y range are fixed according to the mouse coordinates (in particular, the y range is no longer autoscaled).

The set mouse command takes some additional options that mostly control the display format of the mouse coordinates in the plot window. Check the standard gnuplot reference for details.

12.4.3  Custom hot keys

The standard hot key bindings are nothing to write home about, and neither is the default mouse behavior. But we can also define our own hot key and mouse bindings, and that allows for some very exciting possibilities.

The bind command is the essential facility to do so. It is used to bind gnuplot commands to key strokes or mouse clicks. Here is a trivial example:

bind "r" "replot"

This “binds” the r key to the replot command. Now hitting the r key (while the plot window has focus) will redraw the most recent plot.

The bind command in detail

The bind command takes two string arguments: the first is the key (possibly in combination with one or several of the control keys), the second is the command to be executed. (When invoked without arguments, the bind command simply shows all currently defined bindings.)

bind                                             # show current bindings
bind [allwindows] "{str:keys}" "{str:command}"   # define a new binding

It is possible to bind commands to special keys—see listing  12.4, “Symbolic names of some of the function keys available” for the symbolic names of some function keys you might want to know, and the standard gnuplot reference documentation for the bind command to see the complete list.

Listing 12.4. Symbolic names of some of the function keys available

# Control keys:
ctrl-   alt-

# On main key block:
"Tab"   "Escape"   "Return"

# On arrow-block:
"Home"   "End"   "PageUp"   "PageDown"
"Left"   "Up"    "Down"     "Right"

# Function keys:
"F1" ... "F12"

# Window events:
"Button1"
"Close"

Usually, the commands bound to the hot keys will only be executed if the plot window has keyboard focus; hot keys don’t apply when the command window has focus. This behavior can be overridden by the optional allwindows option to bind: if it has been given, then the binding applies to all plot windows, active or not. In this case, the variable MOUSE_KEY_WINDOW is set to the ID of the window generating the event.

Here are some additional points to watch out for:

Neither the spacebar nor the q key can be rebound from their default settings.

Only mouse button 1 can be bound.

Running bind without arguments displays a list of current bindings. So does show bind or hitting the h key while the plot window has focus.

The command reset bind returns all key bindings to their defaults.

Worked examples

Some of the “extensions” of the standard command set that were discussed in section “Extending the command set” earlier in this chapter can be bound to keys. Assuming that the string variables polar, times, and dates exist in the gnuplot session and have the values shown earlier, we can bind them to keys, like so:

bind "F1" polar
bind "F2" times
bind "F3" dates
bind "F4" "reset session"

For convenience, I have bound these commands to function keys here; of course, you can bind them to any other key as well. The binding for F4 resets gnuplot to its state at the beginning of the session; presumably resetting the configuration changes brought about by any of the other keys.

Of course it is also possible to bind the loading of a scriptlet to a key. In this case, the effective command is load:

bind "F5" "load 'normdist.gp'"

One word of warning: if you use string variables in bind (as in the bindings for F1, F2, and F3 in the example above), then the variables must contain the string to evaluate at the time the bind command is executed (early binding). It is not possible to bind a key to a variable first, and then dynamically change the value of that variable later.

12.4.4  Capturing mouse events

Whenever a mouse or keyboard event occurs while the graph window has focus (and provided mousing is enabled using set mouse), then gnuplot populates several variables with information regarding the details of the most recent events. The names of these variables all begin with the prefix MOUSE_. See table  12.2, “These variables are set whenever a mouse or keyboard event occurs when a plot window is active, provided that mousing has been enabled using set mouse.” for a complete list.

We can use the information contained in these variables, in combination with the bind command, to write gnuplot code that responds to user actions. In this way, it is possible to achieve a limited form of programmable interactivity.

Table 12.2. These variables are set whenever a mouse or keyboard event occurs when a plot window is active, provided that mousing has been enabled using set mouse.

Variable name

Description

MOUSE_X

x coordinate at the time of the mouse event (measured in first coordinate system)

MOUSE_Y

y coordinate

MOUSE_X2

x coordinate (measured in second coordinate system)

MOUSE_Y2

y coordinate

MOUSE_BUTTON

ID of the mouse button clicked (1, 2, or 3)

MOUSE_SHIFT

Nonzero if the Shift key was pressed when the event occurred

MOUSE_CTRL

Nonzero if the Control key was pressed when the event occurred

MOUSE_ALT

Nonzero if the Alt (Meta) key was pressed when the event occurred

MOUSE_KEY

ASCII code of the key that was pressed

MOUSE_CHAR

The character value of the key that was pressed

MOUSE_KEY_WINDOW

ID of the plot window that received the event

Worked example

Here is an intentionally simple example, mostly to demonstrated the principle. We will present a much more interesting and useful application in the following section.

If you execute the following command:

bind "Button1" "set label at MOUSE_X, MOUSE_Y center point pt 1 ps 4; set label at MOUSE_X, MOUSE_Y center point pt 6 ps 3; replot"

and then click into the plot window, a point symbol will be placed at the current mouse position, for example to draw attention to an oddity in the data. To make things a little bit more interesting, it’s a symbol that gnuplot does not provide as a built-in: a “cross-hair”, here generated from a separately drawn cross and circle. To remove the symbols, you need to find their tags first using show label, and then remove them with unset label, followed by the tag.

This code snippet is a straightforward application of the bind command. Let me draw your attention to the replot command at the end of the command string: this is common practice when using bind and ensures that the result of the action is visible immediately. We will see more examples of this technique in the next section.

12.4.5  Case study: placing arrows and labels with the mouse

Truth be told, I don’t find most key bindings terribly useful: the whole purpose of gnuplot is graphs, and so I want to interact with them using the mouse, not the keyboard (for a stunning exception to this rule, see section "sec_heatmaps_explorer"). Fortunately, the bind command can be used to create some special mouse bindings as well.

Wouldn’t it be nice if we could place arrows and labels onto the graph with the mouse: just point-and-click? Turns out we can. Here are some ideas.

Labels

The problem with placing labels is that we don’t have a way to prompt the user for the text of the label. Hence, we must rely on the user to prepopulate an agreed-upon variable, or we must rely on the user to call an appropriate function later.

The following snippet follows the former approach:

bind "ctrl-Button1" 'eval sprintf( "set label %d \"%s\" at %f,%f; replot",
» labeltag, labeltext, MOUSE_X, MOUSE_Y )'

It assumes that the variables labeltext and labeltag have been populated appropriately. Then clicking into the plot while pressing the Control-key will place the label at the current position of the mouse pointer.

Two notes on implementation: in the second argument to bind, it is necessary to escape the innermost quotation marks. Also note the replot command, which ensures that the plot automatically updates when the mouse button is clicked. This is common practice when using bind.

The next code snippet takes the other approach: you click first, then you supply the text:

bind "ctrl-Button1"
»    "label_x=MOUSE_X; label_y=MOUSE_Y;
»     set label 1000 at label_x,label_y point; replot"

label( tag, text ) =
»    sprintf( "set label %d '%s' at %f,%f; unset label 1000; replot",
»    tag, text, label_x, label_y )

Now, if you control-click anywhere in the plot window, a point symbol is placed at the position of the mouse cursor: to remind you to supply the label text. You do the latter using the label() function, which removes the point from the graph and replaces it with the supplied text. Do not forget to use eval with this function:

eval label( 2, "Hello, World!" )

Arrows

Arrows are trickier, because we need to capture two positions. The problem is that the only accessible mouse event is the click: this means that we can’t use “drag-and-drop” as in drawing program. Instead, we have to capture two clicks separately. This leads to a problem: what if we change our minds between the first click and the second?

Listing  12.5, “Commands to place arrows using the mouse (file: mouse-arrows.gp)” shows the solution. We bind both the mouse click and the Escape key. When the mouse key is clicked once, a temporary marker is placed onto the plot, upon a second click, the marker is removed and an arrow is drawn. If we change our minds after the first click, we can hit the Escape key, which removes the temporary marker and leaves gnuplot in a clean state. Pretty cool! (Try it!)

Listing 12.5. Commands to place arrows using the mouse (file: mouse-arrows.gp)

bind "Button1" \
"if(exists('arrow')==0) { \
   arrow=1; arrow_x=MOUSE_X; arrow_y=MOUSE_Y; \
   set label 1000 at arrow_x,arrow_y point; replot; \
 } else { \
   unset label 1000; \
   set arrow from arrow_x,arrow_y to MOUSE_X, MOUSE_Y; replot; \
   undefine arrow,arrow_x,arrow_y \
}"

bind "Escape" "unset label 1000; replot; undefine arrow,arrow_x,arrow_y"

A crucial element of the logic in listing  12.5, “Commands to place arrows using the mouse (file: mouse-arrows.gp)” is the variable called arrow: if it exists in the gnuplot session, then we know that the first point has been selected and the temporary marker has been placed; if it does not exist, then the first point has not been selected yet. We can test for the existence of a variable with this name using the exists() function. The reason that we test for the existence of this variable (rather than for its value) is that a test for value would result in an error if the variable has not even been initialized yet. The existence test is more robust.

[Tip]
Try it—it’s awesome

The snippet in listing  12.5, “Commands to place arrows using the mouse (file: mouse-arrows.gp)” works extremely well. It entirely takes the pain out of gnuplot’s set arrow facility.

[67] This feature is new with gnuplot 5.

12.5  Startup configurations and initialization

It is possible to pass commands and information to gnuplot upon startup. All three typical ways of initializing a program are available: initialization files, environment variables, and command-line flags. Let’s consider them in turn.

12.5.1  Startup and initialization files

On startup, gnuplot attempts to read the file gnuplotrc from a “standard” location (such as /usr/share/gnuplot/ on Linux—run show loadpath to see the configured location). By default, this file is empty, but it can be used for customizations that should be global to your installation.

Gnuplot also looks for a personal initialization file, called .gnuplot on Unix (and Mac OS) and GNUPLOT.INI on Windows. Gnuplot searches for this file first in the current directory, then in the current user’s the home directory (as defined by the corresponding environment variable). Once a file has been located, searching doesn’t continue. The file is executed (using load) before any other files or commands are read from the command line, standard input, or an interactive terminal. In general, settings in the initialization file override environment variables.

You can suppress reading of both initialization files by giving the -d command-line option when invoking gnuplot. This may be useful when you want to all gnuplot’s internal settings to assume their “default” values.

What to put in your initialization file

I recommend keeping the default initialization file relatively short, in order to prevent pollution of the gnuplot session (and all command files subsequently saved with save). I think it is a better idea to group related settings in individual command files that can be loaded as needed, and to limit the default file to only those settings that you really want to have effect at all times. Also keep in mind that there is only one initialization file that is read automatically by gnuplot. With individual command files or “stylesheets” (see section “Stylesheets”), you have greater control over the state of your gnuplot session. (You can either load them after gnuplot has started using load or specify them on the command line itself.)

Listing  12.6, “A typical .gnuplot initialization file (file: dog.gnuplot)” shows an initialization file that is relatively short, mostly intended to fix instances where gnuplot’s defaults do not work well.

Listing 12.6. A typical .gnuplot initialization file (file: dog.gnuplot)

# Improve default settings
set encoding utf8

set samples 3000

set isosamples 30
set hidden3d

# Local library and style
set loadpath "/home/janert/lib/gnuplot/"

load "default-style.gp"

# Color management functions
hsv( h, s, v )  = hsv2rgb( h - floor(h), s, v )
pack( r, g, b ) = 2**16*r + 2**8*g + b

# Set arrows with mouse
bind "Button1" \
"if(exists('arrow')==0) { \
   arrow=1; arrow_x=MOUSE_X; arrow_y=MOUSE_Y; \
   set label 1000 at arrow_x,arrow_y point; replot; \
 } else { \
   unset label 1000; \
   set arrow from arrow_x,arrow_y to MOUSE_X, MOUSE_Y; replot; \
   undefine arrow,arrow_x,arrow_y \
}"

bind "Escape" "unset label 1000; replot; undefine arrow,arrow_x,arrow_y"

The first few lines enable UTF-8 (section "sec_practical_unicode"), and increase the sampling density for plot and splot (sections "sec_plot_plotting" and "sec_threedim_specialoptions"). Next, I define the path to my local library of gnuplot scripts and load my default style sheet directly from that directory. Finally, I define two functions for handling color (see section "sec_appearance_explicitcolor") and define bindings that allow me to set arrows with the mouse (see listing  12.5, “Commands to place arrows using the mouse (file: mouse-arrows.gp)”). That’s it—everything else I will load only as required.

12.5.2  Environment variables

Gnuplot examines several environment variables (see table  12.3, “The environment variables that gnuplot reads on startup. Unless otherwise specified, the variables have the same name on all platforms.”). Not all of them are still important; some of them provide information for graphics terminals that are no longer current. In general, I don’t like to rely on environment variables and instead prefer explicit initialization in a configuration file or scriptlet.

Table 12.3. The environment variables that gnuplot reads on startup. Unless otherwise specified, the variables have the same name on all platforms.

Name (Unix and Mac OS X)

Name (Windows, if different)

Description

GNUTERM

The name of the terminal to be used. (Compare the set terminal command.)

GNUHELP

The path to the online help file gnuplot.gih.

GNUPLOT_LIB

Additional directories to be searched for data and command files. (Compare the set loadpath facility.) [a]

GDFONTPATH and GNUPLOT_DEFAULT_GDFONT

Search path used by terminals based on the GD library (PNG, GIF, and JPG) to locate font files, and a default font to be used with these terminals, respectively.

GNUPLOT_FONTPATH

Search path used by the PostScript and SVG terminals to locate font files. (Compare gnuplot’s set fontpath facility.)[b]

GNUPLOT_PS_DIR

Used by the PostScript terminal to locate custom prologue files.

GNUFITLOG

The name of the log file maintained by the fit command (discussed in section "sec_math_fit" ); defaults to fit.log. If a directory is named, then gnuplot will create a file with the default name in this directory.

HOME

GNUPLOT and USERPROFILE

The directory to search for a .gnuplot initialization script if none is found in the current directory.

PAGER

n/a

The name of pager program to use.

SHELL

COMSPEC

The name of the shell to spawn in response to the gnuplot shell command.

[a] Path entries are separated from one another using a colon “:” on Unix, and a semicolon “;” on Windows.

[b] Path entries are separated from one another using a colon “:” on Unix, and a semicolon “;” on Windows.

12.5.3  Gnuplot command-line flags

In general, gnuplot executes the contents of command files named on the command line when gnuplot is first started (see section “Batch processing”). Remember that the special file name - suspends processing of the command line and begins reading interactive commands from standard input.

In addition, gnuplot recognizes six command-line flags. See table  12.4, “Command-line options recognized by gnuplot”.

Table 12.4. Command-line options recognized by gnuplot

Short form

Long form

Description

-V

--version

Prints version information.

-h

--help

Prints a short usage message.

-p

--persist

Continues to display the display window for interactive terminals, even after the main gnuplot has terminated.

-d

--default-settings

Suppresses loading of both the global (gnuplotrc) and the personal (.gnuplot) initialization files.

-c ...

Treats the rest of the command line as if invoked using call. This option should be followed by the name of a command file and, optionally, its arguments.

-e "..."

Executes the commands given as string argument before processing the rest of the command line.

12.6  Stylesheets

The concept of a stylesheet is probably most familiar in the context of web development, but it is, in fact, more general: wherever a document is prepared using markup, the stylesheet principle can be applied. The markup language is used to indicate semantics, whereupon the stylesheet is applied to fix the visual appearance.

This notion can be carried over to gnuplot, once you realize that a command like:

plot "data" u 1:2 w lp lt 1, "" u 1:3 w l lt 2

constitutes a form of markup. The actual appearance of the plot depends on the styles that are in effect when the plot is actually rendered by gnuplot. By supplying suitably chosen definitions for line types and other properties (such as arrow styles and text fonts), we can therefore create a family of graphs, each with a different visual appearance.

The notion of using “stylesheets” like this with gnuplot may seem unfamiliar (at least, I personally have never seen this done before), but it is an extremely powerful concept. It enables us to create graphics for specific circumstances and purposes. At its most basic, stylesheets allow us to create both color and black-and-white versions of graphs, but the idea can be taken further. Graphs for interactive work (on a computer screen) place different demands on colors, line widths, point size, and so on, than graphs prepared for (say) a PowerPoint presentation (see the sidebar).

To be clear: I am not saying that you can simply apply a different stylesheet to an existing command file and obtain a satisfactory graph without tweaking (as you can with CSS or, for example, LaTeX). But by using different sets of predefined line types and other visual appearance options, you can begin to create graphs that take the specifics of the intended presentation channel or medium (screen, print, overhead) into account. If nothing else, you can use a personal stylesheet to escape the “standard” look of gnuplot graphics (although I don’t think you should—see section the sidebar in section "sec_axes_ticplacement" ). But it’s nice to have the option!

[Tip]
Use stylesheets

Customize styles to adapt the appearance of graphs to their intended use and medium. Collect related styles together to create stylesheets for different purposes.

12.6.1  Worked example: stylesheets

What does the “stylesheet” concept look like in practice? In this section, I will present the same graph in four different ways, using four different stylesheets—with no changes to the plot command itself! The plot includes six curves (so that there is a need for a variety of distinguishable line styles), and both sparse and dense data sets (so that ther is an opportunity for both the with lines and the with linespoints plot styles).

The plot command is shown below in listing  12.7, “Plot command for figures 12.1 through 12.4 (file: style-cmd.gp)”. It is used, without modification, to generate all of the next four graphs. Only the stylesheets differ.

Listing 12.7. Plot command for figures 12.1 through 12.4 (file: style-cmd.gp)

set xlabel "Input"
set ylabel "Response"

plot "sparse" u 1:2 t "Sparse 1" w lp, "" u 1:3 t "Sparse 2" w lp,
»    "" u 1:2:(1000) t "Smooth" s acs w l,
»    "dense" u 1:2 t "Dense 1" w l, "" u 1:3 t "Dense 2" w l,
»    "" u 1:4 t "Dense 3" w l

The stylesheets must be loaded (using load) before the plot command is issued. You should also run reset or reset session before loading the next stylesheet (otherwise previous styles will sometimes interfer with current ones).

Interactive computer work 1: thin and bright

The stylesheet in listing  12.8, “Stylesheet for figure 12.2 (file: style1.gp)” is intended as a replacement for the old gnuplot default appearance, and is intend for interactive use at a computer screen. It is characterized by relatively thin lines, in bright colors. In order to make thin lines visually distinguishable, it is necessary to choose relative bright colors, but in contrast to the old gnuplot default, the colors are toned down a bit. A dark-green has been used in place of the bright green, and the color yellow (traditionally the sixth style in the default sequence), which is too difficult to see against a white background, has been replaced with a drab goldenrod. The choice of point type also warrants a remark: instead of the spidery cross and saltire, this stylesheet prefers filled and open circles and squares. The resulting plot is shown in figure  12.1, “The example graph using the stylesheet from listing 12.8: thin lines, bright colors, small symbols.”. (The set termoption command changes just one aspect of a terminal, without requiring the terminal itself to be specified—see chapter  10, Terminals in depth.)

Listing 12.8. Stylesheet for figure 12.2 (file: style1.gp)

set linetype  1 pt 7 lc rgb '0xee0000' # red2
set linetype  2 pt 5 lc rgb '0x008b00' # green4
set linetype  3 pt 6 lc rgb '0x0000cd' # mediumblue
set linetype  4 pt 4 lc rgb '0xff3fb3' # maroon1
set linetype  5 pt 2 lc rgb '0x00cdcd' # cyan3
set linetype  6 pt 7 lc rgb '0xcd9b1d' # goldenrod3
set linetype  7 pt 5 lc rgb '0x8968cd' # mediumpurple3
set linetype  8 pt 6 lc rgb '0x8b8b83' # ivory4
set linetype  9 pt 4 lc rgb '0x00bb00' # (custom green)
set linetype 10 pt 2 lc rgb '0x1e90ff' # dodgerblue1
set linetype 11 pt 7 lc rgb '0x8b2500' # orangered4
set linetype 12 pt 5 lc rgb '0x000000' # black

set termoption font "Helvetica,11"
set termoption fontscale 0.5

Figure 12.1. The example graph using the stylesheet from listing 12.8: thin lines, bright colors, small symbols.

Interactive computer work 2: thick and pastel

The second stylesheet (listing  12.9, “Stylesheet for figure 12.2 (file: style2.gp)”) is also intended for interactive use on a computer screen, but behold how different the resulting graph appears! (See figure  12.2, “The example graph using the stylesheet from listing 12.9: thick lines, pastel colors. Compare figure 12.1.”.) Gone are the brighly colored hairlines, and instead we use thick and wide lines in soft pastel colors. (We also use a boldface font, in keeping with the thick line widths.) If you are used to graphs like the previous figure (figure  12.1, “The example graph using the stylesheet from listing 12.8: thin lines, bright colors, small symbols.”), then this style takes some getting used-to, but I recommend that you give it a fair try. After a while, you may notice that the muted colors and soft edges have a welcome, soothing effect. But it is quite different than what we saw before.

Listing 12.9. Stylesheet for figure 12.2 (file: style2.gp)

set linetype 1 lw 3 pt 7 lc rgb '0x87ceeb' # skyblue
set linetype 2 lw 3 pt 5 lc rgb '0x7ccd7c' # palegreen3
set linetype 3 lw 3 pt 6 lc rgb '0xab82ff' # mediumpurple1
set linetype 4 lw 3 pt 4 lc rgb '0x6ca6cd' # skyblue3
set linetype 5 lw 3 pt 7 lc rgb '0xfa8072' # salmon
set linetype 6 lw 3 pt 5 lc rgb '0xd2b48c' # tan

set pointsize 0.5

set termoption font "Helvetica-Bold,11"
set termoption fontscale .5

Figure 12.2. The example graph using the stylesheet from listing 12.9: thick lines, pastel colors. Compare figure 12.1.

Print publishing: black-and-white

For print publishing, it may still be necessary to submit graphs in black-and-white. For line art, this means that we have to resort to dash patterns to distinguish lines from each other. This is what the stylesheet in listing  12.10, “Stylesheet for figure 12. 3 (file: style3.gp)” is for. It uses custom dash patterns that are intended to recreate (and improve upon) the dash patterns from gnuplot’s PostScript terminal. In keeping with the preference for hairlines in scientific drawing, it does not use filled point symbols, and instead gives preference specifically to the cross and saltire—but with a twist! It uses gnuplot’s new pointintervalbox feature so that the continues line does not actually touch, and thereby interfere with, the actual point symbol. The resulting graph is shown in figure  12.3, “The example graph using the stylesheet from listing 12.10: black-and-white, dashed hairlines, cross-shaped symbols.”.

Listing 12.10. Stylesheet for figure 12. 3 (file: style3.gp)

set dashtype 1 (2,0)
set dashtype 2 (15,7)
set dashtype 3 (10,12)
set dashtype 4 (5,5)
set dashtype 5 (20,6,2,6)
set dashtype 6 (12,5,2,5,2,5)
set dashtype 7 (4,2,4,7,2,7)
set dashtype 8 (2,2)

set pointsize 0.5
set pointintervalbox 1.75

set linetype 1 pi -1 pt 1 dt 1 lc black
set linetype 2 pi -1 pt 2 dt 2 lc black
set linetype 3 pi -1 pt 6 dt 3 lc black
set linetype 4 pi -1 pt 4 dt 4 lc black
set linetype 5 pi -1 pt 1 dt 5 lc black
set linetype 6 pi -1 pt 2 dt 6 lc black
set linetype 7 pi -1 pt 6 dt 7 lc black
set linetype 8 pi -1 pt 4 dt 8 lc black

set termoption font "Helvetica,11"
set termoption fontscale 0.5

Figure 12.3. The example graph using the stylesheet from listing 12.10: black-and-white, dashed hairlines, cross-shaped symbols.

A walk on the wild side

The last stylesheet has a two-fold purpose: I want to demonstrate some intentionally questionable choices for the appearance of a graph—these are not recommendations! More seriously, I also want to show how to make a graph look entirely different from the usual gnuplot appearance. Here are some of the differences: the graph has no frame, but a couple of arrows indicating the coordinate axes; the aspect ratio is square; the font is a serif type; the lines are wide and bright; despite color-coded, they are also dashed; the frame around the key unnecessarily clutters and partially obscures the graph; the text is either too large or too small. Not all of these choices are poor, but some definitely are. (See listing  12.11, “Stylesheet for figure 12.4 (file: style4.gp)” and figure  12.4, “The example graph using the stylesheet from listing 12.11: as different from the gnuplot defaults as possible.”.)

Listing 12.11. Stylesheet for figure 12.4 (file: style4.gp)

unset border
set xtic nomirror
set ytic rotate by 90 center nomirror
set arrow 1000 from graph 0,0 to graph 1.075,0 head filled lw 2
set arrow 1001 from graph 0,0 to graph 0,1.075 head filled lw 2

set size square

set lmargin 7
set rmargin 5
set tmargin 1.5
set ylabel offset 3,0
set xlabel offset 0,0.5

set key opaque title "Legend" box lw 2 font ",10"

set linetype 1 lw 2 pt  9 lc rgb '0xff4500'          # orangered
set linetype 2 lw 2 pt 11 lc rgb '0x00ff00' dt '-'   # green
set linetype 3 lw 2 pt 13 lc rgb '0x0000ee' dt '_'   # blue1
set linetype 4 lw 2 pt  9 lc rgb '0xff1493' dt '.'   # deeppink
set linetype 5 lw 2 pt 11 lc rgb '0x00eeee'          # cyan2
set linetype 6 lw 2 pt 13 lc rgb '0x9400d3' dt '-'   # darkviolet

set termoption font "Times,20"
set termoption fontscale 0.5

Figure 12.4. The example graph using the stylesheet from listing 12.11: as different from the gnuplot defaults as possible.

But is it beautiful?

Gnuplot’s defaults owe some to scientific tradition, and some to technical limitations on early computer systems. Few of these limitations still remain, which means we are free to choose from the available options for best results.

The first two of gnuplot's symbols come from a scientific tradition: on graph paper, where accuracy needed to be maintained manually, two thin, short lines intersecting in a point provided a good way to fix a location on the graph precisely. Today, when the data is available numerically and the computer takes care of the accuracy, this is much less of a concern, and we can choose symbols that are more aesthetically pleasing.

For scientific plots of medium-sized data sets, both cross and saltire may still be adequate. The combination of the two (the third symbol in gnuplot's default ordering) is bulky and should not be used.

For presentation graphics, in particular for sparse data sets, I recommend the filled circle and the filled square. These two symbols, together with a choice of colors, should be enough to distinguish several data sets in a single graph. If you need even more symbols, I recommend the unfilled circle and unfilled square, followed by cross and saltire, and possibley the tip-down triangles (filled and unfilled). The "diamond" (four-cornered symbol, standing on its tip) does not reproduce well and should probably, though regretfully, be avoided.

For large data sets consisting of many points, the unfilled circle has a special advantage: even if points are spaced densely enough that symbols overlap partially, unfilled circles provide the best chance of being recognizable as separate symbols. Where filled symbols simply occlude each other, and symbols made up of straight lines (such as crosses and squares) blend together as indivdual edges align, unfilled circles can still be distinguished from each other.

Traditionally, thin lines have been used in scientific publishing: again, because a thin line fixes a location on a graph more precisely. This is still a good tradition to follow, in particular for print publishing: thin lines often give a publication a “classy” appearance. (Provided they are not too thin to render properly—as publishers switch from photo-offset to print-on-demand technology, sturdier lines might be a good idea!.)

For other publication channels, this is not true. In particular for slides and “overhead” presentations, thicker, rounded lines work better: they can still be seen at the back of the room, and they tend to fit better with the overall style of the medium. Graphs for the Web fall between these two extremes.

In general, I recommend to use non-serif fonts (such as Helvetica) for labels on a graph, because such fonts are the most “neutral”. Make sure that the font size is chosen properly: it is surprisingly common to see graph labels that are too large, leading to an imbalanced appearance of the entire graph.

12.7  Summary

In this chapter, we explored different ways to go beyond gnuplot’s default appearance and behavior. First, we considered ways to streamline your gnuplot workflow through scripts and customized key bindings (“hot keys”). You may want to personalize your work environment when using gnuplot, and I showed you how to do that.

In the last part of the chapter, I introduced the concept of stylesheets. Stylesheets are bundles of appearance options that you can use to create graphs that are optimized for their intended use and medium—be it an interactive computer session, a black-and-white print publication, or a presentation to a live audience.

13

Basic techniques of graphical analysis

This chapter covers:

Investigating relationships

Using logarithmic scales

Representing point distributions

Visualizing ranked data

Organizing your work

In this chapter and the next, I’ll largely take gnuplot and its features for granted, and instead concentrate on applying gnuplot to specific goals that arise commonly when graphing data. At the same time, I’ll take the opportunity to show you how certain tasks can be accomplished using gnuplot’s features. In this chapter, I’ll discuss some basic activities that are generally useful in graphical analysis; the next chapter will cover some more specialized tasks in greater depth.

When faced with a new data set, there are two questions that usually dominate. The first one is, how does one quantity depend on some other quantity—how does y vary with x? The second question (for data sets that include some form of statistical noise) asks, how is some quantity distributed—where are data points located, and what’s the character of the randomness? These are the two dominant questions for this chapter. In the course of it, we will also revisit logarithmic plots and their uses, and see more applications of smooth approximations to noisy data sets—topics we first encountered in chapter 3.

13.1  Representing relationships

One of the conceptually easiest and most familiar types of graphs is the one that shows the dependence of one quantity on another—so much so that creating a plot of this type is what is commonly meant by “graphing data”. We have seen many examples in this book, like figure "essential_prices01", figure "plot_records01", and many more.

13.1.1  Scatter plots

For many data sets, we’re naturally interested in the question of whether one quantity depends on another, and if so, how: does y grow as x grows, or does it fall, or does y not depend on x to begin with? A scatter plot is the first step in finding the answer. In a scatter plot, we just show unconnected symbols, located at the position given by x and y. It’s an easy way to get a feeling for an otherwise unknown data set.

A simple example: car data

Listing  13.1  shows the first few lines from a sample data set, containing 26 attributes for 205 different car models that were imported into the US in 1985.[1]The 14th column gives the curb-weight in pounds, and the last (26th) column the price (in 1985 dollars). A scatter plot like the one in figure  13.1 shows us how weight varies as a function of price.

Listing 13.1. Incomplete data for figure 13.1 (file: imports-85.data)

1,158,audi,gas,turbo,four,sedan,fwd,front,105.80,192.70,71.40,55.90,
0,?,audi,gas,turbo,two,hatchback,4wd,front,99.50,178.20,67.90,52.00,
2,192,bmw,gas,std,two,sedan,rwd,front,101.20,176.80,64.80,54.30,2395
0,192,bmw,gas,std,four,sedan,rwd,front,101.20,176.80,64.80,54.30,239
0,188,bmw,gas,std,two,sedan,rwd,front,101.20,176.80,64.80,54.30,2710
...

Figure 13.1. Curb weight versus price for 205 different cars. See listing 13.1.

In this case, the input file isn’t whitespace separated, but comma separated. Instead of transforming the input file to space separated, it’s more convenient to use gnuplot’s set datafile separator option to plot this data file. The open circle (pt 6) is used because it remains easiest to distinguish even when data points overlap:

set datafile separator ","
plot [2500:] "imports-85.data" u 26:14 pt 6

We can clearly see that weight goes up as the price increases, which is reasonable. We should also note that there are many more low-price/low-weight cars than heavy, premium vehicles. For budget cars, weight seems to increase in step (linearly) with price for a while, but for higher-priced vehicles, the gain in weight levels off. This observation may have a simple explanation: the price of mass market vehicles is largely determined by the cost of materials, so that a car that’s twice as big (as measured by its overall mass) is also twice as expensive, whereas the price of luxury cars is determined by higher quality (fancier materials such as leather seats, and additional options such as more electronics), rather than by sheer bulk.

Using scatter plots

This example demonstrates what to look for when examining a scatter plot. The first question usually concerns the nature of the relationship between x and y. Does y fall as x grows or vice versa? Do the points fall approximately onto a straight line or not? Is there an oscillatory component? Whatever it is, take note of it.

The second question concerns the strength of the relationship, or, put another way, the amount of noise in the data. Do the data points jump around unpredictably as you go from one x value to the next? Are there outliers that seem to behave differently than the majority of the points? Detecting outliers is important: gone unnoticed, they’ll mess up most statistical quantities (such as the mean) you may want to calculate later. And sometimes outliers indicate an interesting effect—maybe some subgroup of points follows different rules than the majority. Outliers in scatter plots should never go uninvestigated.

A third aspect to look out for in a scatter plot is the distribution of points in either dimension. Are points distributed rather uniformly, or do they cluster in a few locations? If so, do we understand the reason for the clustering, or is this something we need to investigate further? There may be a lot of information even in a humble scatter plot!

13.1.2  Highlighting trends

The primary reason for preparing scatter plots is to determine whether there is any relationship between the plotted quantities, and what the nature of this relationship might be. In particular when the data is noisy, it may be difficult to recognize the dominant behavior from the individual data points alone, and so we are searching for graphical tools to highlight any trends that may exist in the data. Gnuplot offers weighted cubic splines for this purpose. Let’s study an example, before discussing splines in more detail.

Worked example: Marathon winning times

Figure  13.2  shows the finishing times of the winners in a marathon event from when the event was first conducted until 1990.[2] In general, we see that the finishing times have decreased over time—the top athletes are getting better every year. The changes are particularly dramatic for the women’s results since they started competing in 1966.

Figure 13.2. Finishing times (in minutes) for the winner of a marathon (up to the year 1990), together with the best straight-line fit. Will women overtake men in the coming years?

Also shown are the “best-fit” straight-line approximations to the data (as found by the stats command), and they seem to represent the data quite well. The only issue is that according to those fits, women should overtake men sometime in the early ’90s—and then continue to get dramatically faster. Is this a reasonable conclusion?

This example attempts to demonstrates two important points when working with data. The first one is the need to be sensitive to the structure and quality of the data. For the data in figure  13.2 , fitting a straight line provides only a very coarse—and as we’ll see, misleading—approximation of the real behavior.

You need to remember that by fitting a straight line, you’ve chosen one specific model to represent the data. At this point, you’re no longer analyzing the data with the intention of revealing the information contained in it, but are making a very specific statement about its surmised behavior. Before making such a strong statement, you should have sufficient evidence for the applicability of the particular model selected. And coming back to the current example, there certainly doesn’t seem to be any strong theoretical reason why athletic performance should follow a straight line as a function of time.

To understand the structure of the data, we might instead attempt to represent the data by some “soft” local approximations, such as weighted splines. Some experimentation with the weights will tell us much about the structure of the data: does the overall shape of the approximation curve change drastically as we vary the weighting? Which features are the most robust, and which disappear most quickly? Typically, significant features tend to be rather robust under transformations, while less relevant features are more transient.

Figure  13.3  shows the same data as figure  13.2 , but instead of a straight line, a soft spline has been used to represent the data, like so:

plot [1890:2017.5][115:215] "men-women" u 1:2 t "Men" w p pt 4,
»                           "" u 1:3 t "Women" w p pt 6,
»                           "" u 1:2:(1e-2) t '' s acs lt 1 lw 2,
»                           "" u 1:3:(1e-2) t '' s acs lt 2 lw 2

This approximation suggests that women’s performance starts to level off in the late ’80s, and the results from years after 1990 corroborate this observation. Note that the spline approximation in the graph is based only on years up to and including 1990, but not on later data points.

Figure 13.3. The same data as in figure 13.2, together with a weighted-splines fit. The fit is based only on points prior to 1990, but the actual finishing times for the following years are also shown. The softer spline clearly reveals the leveling off of the women’s results well before 1990.

Digression: splines

Splines are a way to provide a smooth approximation to a set of points. The points are called knots.[3]

Splines are constructed from piece-wise polynomial functions, which are joined together in a smooth fashion; the joints are also referred to as “knots”. In the case of interpolating splines, the knots are made to coincide with the actual data points, the resulting curve is therefore forced to pass exactly through all data points. In the case of smoothing or approximating splines, the vertical position of each knot is left to vary, so that the resulting curve will in general not pass through the individual data points. Because in the latter case the curve doesn’t have to pass through any data points exactly, it can be less wiggly.

Both interpolating and approximating splines must fulfill the same smoothness conditions, but in addition, the approximating spline must strike a balance between the following two conditions:

Passing close to the data points

Not being too wiggly

These conditions are expressed in the following functional, which is minimized by the approximating spline s(x) :

where (xi, yi) are the coordinates of the data points, and the wi are the weights attached to each data point, and the prime indicates a derivative with respect to x. In this functional, the first term is large if s(x) is wiggly, and the second term is large if s(x) doesn’t pass close to the data points. (The form of the first term comes from a physical analogy: if the spline were made out of a real material, such as a thin strip of wood or metal, the first term would be related to the total bending energy of the strip.)

The balance between these two terms is controlled through the weight parameters wi : if the wi are small, the first term dominates, and the resulting spline approaches a straight line (which happens to coincide with the least-squares linear regression line for the data set). If the weights are large, the second term dominates and the spline approaches the interpolating spline (which passes exactly through all data points).

Another way to think about the weights is to write wi = 1/di² , where di is a measure for the uncertainty in the data of point i (such as the standard deviation in this point). We’d expect that the spline will pass through the interval [yi-di, yi+di] at xi . The higher our confidence in one point, the smaller we can choose this interval, and therefore the larger the weight wi will be. By choosing di = 0 for one of the points, we can even force the curve to pass through this point exactly, although we can let the spline float more freely for the other points. Although we may choose different weights for each point, we can also use the same weight for all points, if we know all points to the same accuracy. This is what was done for all examples in this book.

One important remark: the way J[s] is written in our example, the size of the second term depends on the number of data points—if you double the size of the data set, the size of the second term will be (approximately) twice as large. By contrast, the first term does not depend on the number of data points. If the number of data points grows, the second term will therefore become larger relative to the first one, and the resulting spline will be more wiggly.

To maintain the original balance between wigglyness and closeness of approximation, the weights must be increased accordingly for data sets containing a larger number of points. Equivalently, you might want to take the number of points into account explicitly by writing wi = ui/N , where ui is the actual weight and N is the number of knots. With this choice for wi , the balance between both terms will be maintained regardless of the number of knots in the data set. [4]

[1] This example comes from the “Automobile” data set, available from the UCI Machine Learning Repository: Asuncion, A. and Newman, D.J. (2007). UCI Machine Learning Repository [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, School of Information and Computer Science.

[2] This example was inspired by the book Graphic Discovery by Howard Wainer, Princeton University Press (2005).

[3] I’d like to thank Lucas Hart and Clark Gaylord for helpful correspondence regarding this topic.

[4] More information on splines can be found in chapter 1 of Handbook on Splines for the User by E. V. Shikin and A. I. Plis (CRC Press, 1995).

13.2  Logarithmic plots

Logarithmic scales are one of the most versatile tools in the graphical analyst’s toolbox. I introduced them already in section "Log plots" and discussed how they work. Now let’s put them into action.

Logarithmic scales serve three purposes when plotting:

They rein in large variations in the data.

They turn multiplicative deviations into additive ones.

They reveal exponential and power-law behavior.

13.2.1  Large variations in data

To understand the meaning of the first two items, let’s study the daily traffic pattern at a web site. Figure  13.4 shows the number of hits per day over approximately three months. There’s tremendous variation in the data, with alternating periods of high and low traffic. During periods of high traffic, daily hit counts may reach close to half a million hits, but then fall to very little traffic shortly thereafter. On the scale of the graph, the periods of low traffic seem barely different from zero, with little fluctuation.

Figure 13.4. Traffic patterns at some web site. Daily hit count versus day of the year. Note the extreme variation in traffic over time.

In figure  13.5  we see the same data, but now on a semi-logarithmic scale. The logarithmic scale helps to dampen the extreme variation of the original data set (two orders of magnitude), so that we can now see the structure both during the high- and the low-traffic season. That’s the first effect of logarithmic plots: they help to make data spanning extreme ranges visible, by suppressing high-value outliers and enhancing low-value background.

Figure 13.5. The same data as in figure 13.4, but on a semi-logarithmic scale. Note how the high-traffic outliers have been suppressed and the low-traffic background has been enhanced. In this presentation, data spanning two orders of magnitude can be compared easily.

Furthermore, we can see that the relative size of the day-to-day fluctuations is about equal during both phases. The absolute size of the fluctuations is quite different, but their size as a percentage of the average value is roughly the same (very approximately, during low season, traffic varies between 2,000 and 20,000 hits a day, a factor of 10; whereas during high season it varies between 30,000 and 300,000 hits a day, again a factor of 10). That’s the second effect of logarithmic plots: they turn multiplicative variations into additive ones.

Figure  13.6  tries to demonstrate the last point in a different way. The bottom panel shows the web traffic on consecutive days (like figure  13.4 , displaying great seasonal variance, but the top panel shows the ratio of the difference in traffic on consecutive days divided by the actual value— (current day - previous day)/current day —which does not exhibit a seasonal pattern: further proof that the daily fluctuation, viewed as a percentage of the overall traffic, is constant throughout. The calculation for the top panel can be performed as inline transformation, by the way:

prv=40000
plot [0:365][-1:1] "webtraffic" u (y=($2-prv)/($2+prv), prv=$2, y) w l

Figure 13.6. Bottom panel: hits per day over time (as in figure 13.5); top panel: change in traffic between consecutive days, divided by the total traffic. Note how the relative change (top panel) doesn’t exhibit any seasonal pattern, indicating that the relative size of the variation is constant.

13.2.2  Power-law behavior

Finally, let’s look at a curious example that brings together two benefits of logarithmic plots: the ability to display and compare data of very different magnitude, and the ability to turn power-law behavior into straight lines.

Mammals come in all shapes and sizes, from tiny rodents (the smallest known land mammal is the Pygmy Shrew, which weighs only a few grams, but some bats found in Thailand are apparently smaller still) to the largest of whales (weighing several hundreds of tons). It’s a curious empirical fact that there seem to be fixed relationships between different metabolic quantities—basically, the larger an animal is, the slower its bodily functions progress. Figure  13.7  shows an example: the duration (in seconds) of a single resting heartbeat, as a function of the typical body mass. The regularity of the data is remarkable—spanning eight orders of magnitude for the mass of the animal. What’s even more amazing is how well the data is represented by the simple function T ~ m1/4 . This law isn’t limited to the examples shown in the graph: if you added additional animals to the list, they’d also fall close to the straight line (I didn’t just pick the best ones).

Figure 13.7. Allometric scaling: the duration of a average resting heartbeat as a function of the typical body mass for several mammals. Note how the data points seem to fall on a straight line with slope 1/4.

The existence of such scaling relations in biological systems has been known for a long time and seems to hold generally. For example, it turns out that the typical lifetime of a mammal also obeys a quarter-power scaling law relation against the body mass, leading to the surprising conclusion that the total number of heartbeats in the life of a single organism is fixed—no matter what the typical resting heart rate is. (In case you care, the number comes out to about 1.5 billion heartbeats during a typical lifetime.)

Recently these observations have been explained in terms of the geometrical constraints that must exist in the vascular networks (the veins and arteries), which supply nutrients to all parts of the organism.[5] As it turns out, you can derive the quarter-power scaling laws starting from only three simple assumptions, namely that the support network must be a space-filling fractal, reaching all parts of the organism; that the terminal capillaries where nutrients are actually exchanged are the same size in all animals; and finally that organisms have evolved in such a way that the energy required for the transport of nutrients through their bodies is minimized. I think it’s amazing how such a powerful result can be derived from such simple assumptions, but on the other hand, we shouldn’t be surprised: generally applicable laws (such as the quarter-power scaling in this example) must stem from very fundamental assumptions disregarding any specifics.

Let’s come back to figure  13.7 . The double-logarithmic scales make it possible to follow the data over eight orders of magnitude. (Had we used linear scales, all animals except for the whale would be squished against the left side of the graph—literally crushed by the whale.) So again, logarithmic scales can help to deal with data spanning a wide range of values. In addition, the double-logarithmic plot turns the power law relationship T ~ m1/4 into a straight line and makes it possible to read off the exponent from the slope of the line. I explained how this works in detail in section "sec_plot_logplots" and won’t repeat it here.

Finally, figure  13.7  is a nice example for the power of gnuplot’s with labels plot style. The graph was generated using

plot "mammals" u 2:3 w points, "" u 2:(1.1*$3):1 w labels

The first part of the command draws the symbols (with points); the second adds the labels. All the labels are shifted a bit upward so as not to obscure the symbols themselves. In this example, the vertical offset is multiplicative, because of the logarithmic scale of the graph (remember: logarithms turn multiplicative offsets into linear ones).

[5] The original reference is the paper “A General Model for the Origin of Allometric Scaling Laws in Biology” by G. B. West, J. H. Brown, B. J. Enquist in the journal Science (Volume 276, page 122 (1997)). Additional references can be found on the web.

13.3  Point distributions

Besides detecting relationships between quantities, we may want to understand how data points that are somehow random are distributed. Are data points spread out evenly or are they clustered in a few spots? Are distributions symmetric or are they skewed? How much weight is contained in the tails of a distribution, compared to its center?

Let’s say we have a file containing a set of measurements—these can be anything: interarrival times for requests at a web server, completion times of database queries, weights of potatoes, heights of people—whatever. What can we say about them?

13.3.1  Summary statistics and box plot

Two commands will give a first answer: the stats command and the with boxplot style. Both share the advantage that they have no adjustable parameters—they are as simple as can be. Both also share the disadvantage that they lose a lot of information about the distribution of individual points, because both aggregate the entire distribution into just a handful of summary statistics. (A box-and-whisker plot is after all nothing more than a graphical representation of the results from running the stats command.)

In short: to get a first idea how the values in a new data are distributed, both of these commands are convenient, because of their extreme simplicity. But both hide too much information to develop a detailed understanding.

13.3.2  Jitter plots and histograms

One easy way to get a visualization of a collection of random points is to generate a jitter plot, which is really a one-dimensional scatter plot, but with a twist (see the bottom part of figure  13.8 ).

This graph was created by shifting each data point vertically by a random amount. (The rand(0) function returns a random number in the range [0:1].) If we’d just plotted the data in a true, one-dimensional fashion, too many of the points would’ve overlapped, making it difficult to detect clustering. Such jittering by a random amount is a good trick to remember whenever creating scatter plots of larger data sets!

plot "jitter" u 1:(0.25*rand(0)-.35)

We can see that the distribution of points is skewed. It’s strictly bounded by zero on the left, with points clustering around one, and as we move to the right, points become increasingly sparse. But it’s hard to say something more definite by just looking at the jitter plot. For instance, is there a second cluster of points between three and four? This does seem possible, but it’s hard to tell for sure using this representation.

The next step when investigating the properties of a distribution usually involves drawing a histogram. To create a histogram, we assign data points to buckets or bins and count how many events fall into each bin. It’s easiest to make all bins have equal width, but with proper normalization per bin, we can make a histogram containing bins of differing widths. This is sometimes useful out in the tails of a distribution where the number of events per bin is small.

Gnuplot doesn’t have an explicit histogramming function, but we can use the smooth frequency functionality (see section "sec_plot_unique") to good effect. Recall: smooth frequency sorts the x values by size, and then plots the sum of y values per x value. That’s what we need to build a histogram.

In the following code, I introduce a function bin(x,s) of two parameters. The first parameter is the x value we’d like to bin, and the second parameter is the bin width. The return value of the function is the position of the left edge of the bin—you can use the binc(x,s) function for the center of the bin value.

The smooth frequency feature forms the sum of all y values falling into each bin. If all we care about is the overall shape of the histogram, we may supply any constant, such as (1), but if we want to obtain a normalized histogram (one including a total surface area equal to unity), we need to take into account the number of points in the sample and the bin width. You can convince yourself easily that the proper y value for a normalized histogram is

1/(bin-width * number-of-points-in-sample)

We can use the with boxes style to draw a histogram (see figure  13.8 , but we want to fix the width of the boxes in the graph to coincide with the bin width. (By default, the boxes expand to touch their neighbors, which leads to a faulty graphical representation if some of the internal bins are empty.) We choose a bin width of 0.1 and use the stats command to find the number of points in the data set. The plot command uses the binc() function, because the with boxes style positions its boxes centered at the supplied position.

bin(x,s)  = s*floor(x/s)
binc(x,s) = s*(floor(x/s)+0.5)

set boxwidth 0.1

stats "jitter" u 1 noout
plot "jitter"
»    u (binc($1,0.1)):(1./(0.1*STATS_records)) smooth frequency with boxes

Figure 13.8. Three ways to represent a distribution of random points: jitter plot (bottom), histogram (with boxes), and cumulative distribution function (dashed line)

13.3.3  Kernel density estimates and rug plots

The apparent simplicity of the histogramming method hides some pitfalls. The first concerns the width of the bins: make them too narrow and the resulting histogram will be bumpy; make them too wide and you lose relevant features. There’s also ambiguity in regard to the placement of the bins: is the first bin centered at zero (or any other value) or flush left there? The overall appearance of the histogram can depend quite sensitively on these details!

A better method to generate distribution curves from individual data points goes under the name kernel density estimation. Rather than counting how many data points fall into each bin, we place a strongly peaked, but smooth function (a kernel) centered at the location of each data point. We then sum the contributions from all these curves and plot the result. Mathematically, the kernel estimate f(x) for a data set consisting of N points xi is

Here, K(x) is any smooth, peaked, normalized function, and h is the bandwidth: a measure of the width of the kernel function. A popular example is the Gaussian kernel:

As we already saw in section "sec_plot_kdecdf", gnuplot can generate such curves, using the smooth kdensity functionality. It works in much the same way as the smooth frequency feature we saw earlier:

plot "jitter" u 1:(1./STATS_records) smooth kdensity bandwidth 0.05

The first column specifies the location; the second gives the weight each point should have. For a normalized histogram, this should be the inverse of the number of data points—because the kernel functions are normalized themselves, you don’t have to worry about the bandwidth at this point as you did for histograms using smooth frequency. The bandwidth parameter (preceded by the bandwidth keyword) is optional. If it’s omitted (or negative), gnuplot calculates a default bandwidth, which would be optimal if the data were normally distributed. This default bandwidth tends to be quite conservative (that means, rather broad).

Figure 13.9. An alternative to histograms: kernel density estimates using smooth kdensity. Curves for three different bandwidths are shown. A bandwidth of 0.3 seems to give the best trade-off between smoothing action and retention of details. Note how it brings out the secondary cluster near x=3.5. Individual data points are represented through the “rug plot” along the bottom.

Figure  13.9  shows several curves drawn using kdensity for the same data set we’ve already seen in figure  13.8 , for a variety of bandwidth parameters. Studying this graph carefully, we may conclude that there’s indeed a second cluster of points, located near 3.5. Note how the choice of bandwidth can hide or reveal features in the distribution of points.

One more thing: figure  13.9  uses a different representation for the individual data points: instead of using symbols as in the jitter plot included in figure  13.8 , this figure uses a rug plot along the bottom edge. Each data point is represented through a short, vertical line. In gnuplot, this can be accomplished through the with vectors style:

plot [:11][-0.1:0.8] "jitter" u 1:(-0.1):(0):(0.05) t '' w vec nohead

I prefer jitter plots, because lines in a rug plot tend to obscure each other, making it difficult to identify individual data points. But rug plots may give a somewhat “neater” appearance to your graphs and appear less noisy.

13.3.4  Cumulative distribution functions

Histograms and density estimates have the advantage of being intuitive: they show us directly the probability for a certain value to occur. But they have some disadvantages when it comes to making quantitative statements. For example, based on the histogram in figure  13.8 , it’s hard to determine how much “weight” is in the tail of the distribution: how likely are values larger than 4 to occur? How about values larger than 6? We can guess that the probability will be small, but it’s hard to be more precise. To answer such questions, we need to know the area under the histogram within certain bounds. In other words, we want to look at the cumulative distribution function (or simply distribution function for short).

The value of the cumulative distribution function at position x gives us the fraction of events that have occurred with xi less than x . In figure  13.8 , I already showed the distribution function together with the histogram. To repeat: the value of the cumulative distribution function at position x is equal to the area under the (normalized) histogram from its left border to the position x .

Cumulative distribution functions are accessible using smooth cumulative. The smooth cumulative feature is similar to smooth frequency: first, all points are sorted in order of ascending x value, then the sum of all y values to the left of the current position is plotted as smoothed value. To obtain a normalized distribution function, we must supply 1/number-of-points as y value. In contrast to histograms or density estimates, distribution functions don’t depend on a width parameter.

stats "jitter" u 1 noout
plot "jitter" u 1:(1./STATS_records) smooth cumulative

Cumulative distribution functions can be a little unintuitive at first, but they’re well worth becoming familiar with. They make it easy to answer questions such as those raised at the beginning of this section. From figure  13.8  we can immediately see that there’s a 3 percent chance of finding a point at x > 6 and about a 15 percent chance for x > 4 . We can also find more proof for the second cluster of points between 3 and 4: at this point, the distribution function seems to make a jump, indicating an accumulation of points in this interval.

The cumulative distribution function is also very important theoretically and often the basis for further tests and calculations. The discussion of probability plots in section "sec_math_probplot" is a good example, and also shows how the results from smooth cumul can be captured into a heredoc for further processing.

13.4  Ranked data

Imagine I give you a list of the countries in the European Union, together with their land area (in square kilometers) and population numbers. How would you represent this information? How would you represent it graphically?

The particular challenge here is that the independent variable has no intrinsic ordering. What does this mean?

Given the name of a country, the value of the area measure is fixed; hence the name is the independent variable and the area is the dependent variable. We’re used to plotting the independent variable along the x axis and observing the behavior of the dependent variable with it. But in this case, there’s no natural ordering of the independent variable. Sure, we can order the states alphabetically by their names, but this ordering is entirely arbitrary and bears no relationship on the data. (We wouldn’t expect the size of a state to change if we gave it a different name, would we?) Also, the ordering would change if we were to translate the names to a different language. But the information that we want to display depends on the areas, and shouldn’t be affected by the spelling of the country names in any way.

For data like this, the only ordering that’s intrinsic to the data itself is in the values of the dependent variable. Therefore, a graphical representation of this data should be ordered by the dependent variable, not the independent one. Such plots are often called dotplots, but I tend to think of them as rank-order plots. Figure  13.10  shows an example.

Figure 13.10. A rank-order plot. Because there’s no natural ordering in the independent variable (in this case, the country names), we sort the data by the dependent variable to emphasize the structure in the data.

If the input file is sorted by the appropriate quantity, we can generate such plots easily using gnuplot’s facility for reading tic labels from the input file. Given an input file containing the names and areas in two columns, such as this:[6]

Malta               316
Luxembourg         2586
Cyprus             9251
Slovenia          20273
...

the entire plot can be generated using the following command:

plot "europeanunion" using ($2/1000):0:ytic(1) w p pt 7

The ytic(1) function selects the values in column 1 as tic labels for the y axis (see section "sec_axes_ticsfromfile"), and the pseudocolumn 0, which evaluates the line number in the input file, is used as the corresponding vertical coordinate (see section "sec_files_pseudocolumns"). The inverted y range places the first line in the file at the top of the graph instead of the bottom.

This is the basic idea. We could’ve plotted the state names along the x axis instead, but then we’d need to rotate the labels, to make sure they don’t overlap. Unfortunately, rotating the labels by 90 degrees (so that they run vertically) makes them hard to read. A good trick is to rotate them by some angle so that they run diagonally, but the initial layout, with the names running down the y axis, is the easiest to read.

What if we want to show and compare multiple data sets, such as the land area and the population? The best strategy is to declare a primary data set, which determines the ordering for all others. In figure  13.11 , we can see an example. The points of the secondary data set (the population in millions) have been connected by lines to make them stand out more. Additionally, the x axis has been scaled logarithmically, which is often useful with dot-plots of this sort. We can see that overall the population count follows the area, but there are some notable outliers: the northern Scandinavian countries Sweden and Finland are thinly populated, whereas the so-called Benelux countries (Belgium, Netherlands, and Luxembourg) have an exceptionally high population density.

Figure 13.11. A rank-order plot displaying a primary and a secondary data set for comparison. The country names are sorted according to the primary data set (the area); the points in the secondary data are connected by lines to make them easier to distinguish. Note the logarithmic scale for the horizontal axes.

Dot- or rank-order plots are useful whenever the dependent variable has no natural ordering. On the other hand, if the dependent variable can be ordered, even if it’s nonnumeric (such as the categories Strong Dislike, Dislike, Neutral, Like, Strong Like), we should use the information in our graphs and order data points by the independent variable.

[6] The data can be found on the Wikipedia page for the European Union.

13.5  Pie charts

Pie cha.5 rts are a way to visualize how some entity breaks down into its constituent parts. Their strongest point (and the only reason to use them) is that, by construction, they represent all components of the overall entity, without any possibly “missing pieces”. Aside from that, pie charts have many well-known problems. Specifically, comparing the size of different components with similar magnitudes is difficult; as is the representation of more than a handful of components. For such tasks, some variant of rank-order plot (see section “Ranked data”) is usually a better solution.

Gnuplot does not have pie charts built in, but they can be constructed through judicious use of existing gnuplot features. Given the following data set:

# Name  Percent
A       55
B       25
C       10
D       7
E       3

you can generate the pie chart in figure  13.12, “A pie chart” using the following commands:

set size square
set angles degrees
unset key; unset tics; unset border

beg(x) = (b=a, a=a+x, 360.*b)
end(x) = (c=c+x, 360.*c)
med(x) = (u=v+x/2, v=v+x, 360.*u)

a=b=c=u=v=0
plot [-1:1][-1:1]
»  "pie-data" u (0):(0):(1):(beg($2/100.)):(end($2/100.)) w circ,
»  "" u (0.75*cos(med($2/100.))):(0.75*sin(360.*u)):1 w labels

The challenge is that the with circles style requires the beginning and end angle of each pie segment (each “wedge”) . But the data set only gives the percentage for each component, which corresponds to the angle enclosed (or subtended) by the wedge. The function end(x) therefore keeps a running total of all percentages encountered so far: this is the leading edge of the current slice. The function beg(x) does the same, but always reports the most recent value: this is the trailing edge of the current slice. The function med(x) finally reports a value half-way between the leading and the trailinge edge: this is where the label is placed. Do not forget to set (or reset) all variables to zero before each plot!

Figure 13.12. A pie chart

13.6  Organizational issues

The basic techniques of graphical analysis do not just range over types of graphs and when to use them, but also include some recurring administrative tasks and organizational questions, mostly relating to the handling of data sets and the files containing them. In this section, I will make some recommendations on that.

I also want to stress that graphs are prepared for different purposes. Graphs that are prepared merely for personal use and discovery require little polishing, but this is not true for presentation graphics that are intended to communicate results to an audience. Presentation graphics are important: they document one’s work. I will close this section with some recommendations on for the preparation of attractive and valuable presentation graphics.

13.6.1  The lifecycle of a graph

It’s helpful to have a sense for the life expectancy of your graphs: short (seconds to days, for interactive exploration and ongoing work), intermediate (days to weeks, for intermediate results), and long (weeks to infinity, for final results and public consumption).

Treat graphs differently, based on their expected lives: for short-lived graphs, ease of use is the predominant concern. Nothing should stop you from redrawing the graph, changing the range or plotting style, or a transformation. Any amount of “polishing” is too much.

For graphs in the intermediate bracket, polishing is still not required, but contextual descriptions (title, units, key) become important: not necessarily to communicate such details to others, but to serve as reminder to yourself, should you find yourself coming back to your work after a few days or weeks of absence. (It’s amazing how quickly even essential details can be forgotten.)

For long-lived graphs, and those published or presented publicly, different rules apply. Such graphs belong to “presentation graphics” proper, and I’ll have a bit more to say about that topic in section “Presentation graphics”.

13.6.2  Input data files

Data files should be reasonably self-contained and self-explanatory. Let me explain.

When I was looking for data sets to use as examples for this book, I checked out quite a few publicly accessible data set libraries on the web. On one of them, I found a data set that, according to the description on the web site, contained annual sunspot numbers for a certain specified range of years. I downloaded the corresponding file together with several others data sets from the web site, and only then started to examine the contents of each file in detail.

Each file consisted of only a single column, containing the dependent variable—and nothing else! Looking at these files alone, it was no longer possible to determine which one contained what, be it sunspot numbers or the price of pork bellies, or some other data set. Since the independent variable was missing as well, I couldn’t even tell whether I was looking at monthly or yearly data, and in any case, for what time frame.

In other words, the sheer act of downloading a file turned it into instant garbage, by irrevocably separating it from the information that gave it meaning!

To avoid such embarrassments, it’s generally a good idea to keep the basic information that’s necessary to understand the data contained in the data set as part of the data file itself, typically as a block of comments near the beginning of the file. (I find it more convenient to have this information at the top of the file than at the end.) The information contained in such a header is more or less the same information that we’d put onto a graph itself in the form of textual labels and descriptions. Here are some pointers:

Most important is a description of the file format. It should at least include a description of the content of all columns, together with the units used. It’s nice if it also contains a brief overall “headline” description of the file. If there’s ancillary information that would be required to recreate the data set, it should also be included: things such as parameter settings for the experimental apparatus, or starting values and the version number for the simulation program that was used to calculate the data. If the data was downloaded from the web, I always make sure to keep a reference to the source URL in the file itself.

I also recommend to be rather generous when it comes to the inclusion of redundant information. The sunspot data I mentioned earlier is an interesting example. The lack of data for the independent variable made it more difficult to use and understand the contents of the file. Given the starting value and the increment, it’s trivial to reproduce it, but it’s generally much more convenient to have all this information available already.

The ability to reproduce a data file if necessary is critical. Should you combine data from several sources into a single data set, or manually edit a data set to remove glitches, keep the originals. And stick a note in the combined file explaining its origins. More likely than not, you’ll have to come back and do it all over again at some point. Remember: if you delete the originals, they’re gone forever.

Unless there are extenuating circumstances (and even in most cases when there are), prefer plain text files over binary formats. The advantages of plain text are just too numerous: plain text is portable across all platforms (despite the annoying newline character issue), and can be manipulated using standard tools (editors, spreadsheets, scripting languages). A corrupted text file can be repaired easily—not so for a binary file. A text file is by construction more self-explanatory than a binary file will ever be. And finally, text files compress nicely, and therefore don’t have to take up much disk space.

Some statistics packages keep their data sets in proprietary binary formats, and I’ve found the constant need to invoke a special, proprietary program just to view the contents of a file a major inconvenience, and the dependence on the operations provided by the package for all data manipulation tasks an unwelcome hindrance to my work.

If you have a legitimate need for a binary file format, at least use an existing standard, for which quality libraries are easily available. Ad-hoc binary formats are almost certainly a bad idea.

Something we don’t tend to think of often are clerical errors: typos, incorrectly named files, data entered into the wrong column, wrongly recorded units—that sort of thing. They’re apparently more common than you might think: Cleveland and Wainer[7] give interesting examples of odd entries in well-known data sets and provide convincing arguments that explain these entries as due to clerical errors, such as the interchange of digits, or the inadvertent switching of columns for the data points in question.

Finally: make sure critical data is backed up frequently. (I once accidentally deleted data that had taken weeks to compute. Praise your system administrators!)

13.6.3  Output files

The most important advice (again) is to make plots reproducible. Don’t just export to a printable format and move on. It’s almost guaranteed that you’ll want to redraw the graph, with minor modifications, as more data becomes available or the understanding of it grows. Always save the plotting commands and options to a file before moving on.

Use an appropriate file format. Bitmaps (such as PNG) are simple and portable, but cannot be resized without loss of quality. Whenever there is a chance that you (or whoever is using your graphs) might need to resize a graph, make sure to use a vector format (such as PDF). This is particularly true for print publishing: not only will graphs routinely be resized, but only vector formats will take advantage of the higher resolution offered by printing devices.

Decide whether you prepare your graphs in color or not. Consider the use of stylesheets "sec_config_stylesheet" to target graphs to different display channels. I routinely create different versions of the same graph (using different stylesheets), so that I’m all set for all possible applications. Adopting a naming convention helps.

[7] See section 6.4 in Visualizing Data by W. S. Cleveland, Hobart Press (1993) and the introduction to Graphic Discovery by H. Wainer, Princeton University Press (2005) for details.

13.7  Presentation graphics

This book isn’t primarily about presentation graphics, but about graphical analysis. There’s already plenty of advice on presentation graphics, and you’ll have no difficulty finding it, but not much of it appears to be based on rigorous studies. Nevertheless, the advice is often worded assertively, if not emotionally, and there’s an unfortunate (and ultimately unhelpful) tendency toward the derision of work considered inadequate. Given the lack of rigorous evidence, tastes and personal opinions naturally play a large role.

I don’t intend to add to this debate. Instead, I’d like to present a list of reminders concerning details that are easily (and inadvertently) forgotten when preparing a graph for publication. Most of them concern the inclusion of contextual information, which can’t be inferred from the graph itself, but which is nevertheless important for understanding. By “publication” I mean any form of wider distribution of the graph: anything leaving the immediate group of coworkers who were involved in its creation, and in particular any use with a long expected lifetime.

These aren’t commandments, but reminders. Use your own good judgment.

Axes should be labeled. The labels should describe the quantity plotted and should include the units of measurement. Don’t leave this information to a separate figure caption. Keep in mind that the graph may be separated from its context, so that information contained only in the caption may be lost. The caption should be used to describe salient features of the graph, not to give vital information about the plotted data itself.

Choose meaningful, self-explanatory labels. If you must use abbreviations that aren’t in common use, explain them, ideally on the graph itself, not in the caption (see previous item). (In a recent book on data graphing, I found a figure of a histogram in which the buckets were labeled Married, Nvd, Dvd, Spd, and Wdd. I marveled about the possible meaning of Nvd for quite a while. The abbreviations were explained neither in the text nor in the figure caption.)

If there’s more than one line in a graph, explain what each of the lines represent; either through a key, or using an explicit label referring to each line. If using a key, make sure the key entries are meaningful.

If there’s ancillary information, consider placing it onto the graph itself, rather than supplying it only in the caption.

When publishing a false-color plot, always include the associated color scale in the graph. No matter how intuitive the chosen palette may appear to you, remember that there’s no universal and unambiguous mapping from numbers to colors and vice versa.

Describe the meaning of errorbars. Do they show the calculated standard deviation of the sample population? Do they represent interquartile ranges? Or do they indicate the limits of resolution of your experimental apparatus? This information can’t be inferred from the graph, but must be explained through textual information.

Use an appropriate measure of uncertainty. Don’t show standard deviations for highly skewed or multimodal populations just because they’re easy to calculate.

Don’t forget the basics. Choose meaningful plot ranges. Make sure that data isn’t obscured by tics, keys, labels, or arrows.

Don’t be shy about choosing a different font if the default font looks ugly. Given the importance of textual information on a graph (labels, tic marks, keys), make the necessary effort to ensure that all text is clearly readable, even after the graph has been reproduced and possibly reduced in size a few times. (On the other hand, making labels too big or too bold can easily ruin the overall appearance of a plot. Experiment!)

In general, sans-serif fonts (such as Helvetica) are preferred for standalone pieces of text, whereas serif fonts (such as Times Roman) are considered more suitable for body text. Since labels on a graph tend to be short, this suggests using a good sans-serif font in plots. (I also find that sans-serif fonts enhance the visual clarity of a graph, whereas serif fonts don’t, but others may disagree. Judge for yourself.)

Use an appropriate aspect ratio. Human perception seems to have an affinity to figures with an aspect ratio of roughly 10:7, in “landscape” orientation (that is, wider than tall). In particular, it is generally not a good idea to use a square graph, unless the symmetry of the data demands it.

Don’t use bit-mapped graphics formats (PNG, GIF, JPG) in print publications. Use vector formats such as PDF or PostScript, EPS instead.

Proofread graphs. Common spots of errors include typos in textual labels, switched data sets or interchanged labels, and omitted qualifiers (milli-, kilo-, and so on) for units.

13.8  Summary

Understanding data with graphs typically involves certain recurrent tasks and questions. How one quantity depends on another is one such question; how a bunch of data points are distributed is another. Other recurrent problems have to do with data that spans many orders or magnitude, or with data that is not strictly numeric.

In this chapter, we discussed the standard types of graphs to investigate such questions, and showed how to create them using gnuplot. All these methods and plots are very general—in the next chapter we will concentrate on some more specific problems and explore them in depth.

14

Topics in graphical analysis

This chapter covers:

Techniques for time-series plots

Techniques for multivariate data sets

Techniques to improve visual perception

The previous chapter discussed some fundamental graphical methods that are very generally useful. In this chapter, we will select a few tasks that are more specific, and explore them in greater depth.

Because of their immense practical importance, our first topic will be the handling of time-series. In particular, we will learn some clever ways to smooth a time series using only gnuplot commands. The next topic concerns multivariate data sets: data sets with more than two or three different attributes. A variety of different techniques is available for such data, and we will see how some of gnuplot’s relatively new features facilitate the visual exploration of such data.

Next, we will discuss ways to enhance the visual perception of graphs. We can aide analysis and discovery by preparing graphs such that noteworthy features are more easily recognized. That’s not a topic that’s frequently discussed, but some awareness of the issues involved can greatly improve results.

14.1  Techniques for time-series plots

Time-series plots are probably the most common graph in practice: whenever we want to monitor a quantity, we are dealing with a time-series plot of some form. In this section, I want to introduce some operations that are frequently useful when dealing with time series, and I also want to show how to coax gnuplot into performing those operations.

14.1.1  Plotting an Apache webserver log

Technically, a time series consists of measurements at equally spaced time intervals. The requirement that all time steps are of the same size is important: several time series operations and transformations rely on this property.

But raw data is frequently not of that form. Web server logs (and other event logs, quite generally), simply record the timestamp when an event occured—they are asynchronous. Turning this into a time series means counting the number of events per (equally-sized) time interval.

In general, I strongly suggest that you do this kind of data munging outside of gnuplot, using the programming language of your choice. But in a pinch, it can be done with gnuplot. Listing  14.1 shows the beginning of a Apache web server log file (lines are truncated on the right).[8]

Listing 14.1. Part of a webserver log. Incomplete data for figure 14.1.

64.242.88.10 - - [07/Mar/2004:16:05:49 -0800] "GET /twiki/bin/edit/Main/Doub
64.242.88.10 - - [07/Mar/2004:16:06:51 -0800] "GET /twiki/bin/rdiff/TWiki/Ne
64.242.88.10 - - [07/Mar/2004:16:10:02 -0800] "GET /mailman/listinfo/hsdivis
64.242.88.10 - - [07/Mar/2004:16:11:58 -0800] "GET /twiki/bin/view/TWiki/Wik
64.242.88.10 - - [07/Mar/2004:16:20:55 -0800] "GET /twiki/bin/view/Main/DCCA
...

The timestamp is in the fourth column, and we need gnuplot’s time series mode to parse it properly. Then we need to truncate the timestamp to the interval that we want to use for our time series, and count the number of events in each interval. We can use smooth frequency (see section "sec_plot_logplots") for the latter. When truncating the timestamp, keep in mind that gnuplot treats it as a string, and hence you need to use string operations for all manipulations. In listing  14.2  , I simply truncate the timestamp to the hour: that will be the step size of the resulting time series (see figure  14.1 ).

Listing 14.2. Commands for figure 14.1 (file: access-log.gp)

set xdata time
set timefmt "[%d/%b/%Y:%H:%M:%S"   # The left bracket is part of the format!
set xtics format "%d%b\n%H:00" rotate

plot "access_log" u (strcol(4)[1:15] . ":00:00"):(1) s f w l

Figure 14.1. Webtraffic (hits per hour). Compare listing 14.2.

Because this method relies on gnuplot’s timestamp parsing, it is rather brittle, and does not deal well with missing or malformed entries in the log file. If you encounter such problems, it’s probably easiest to preprocess the raw data in a general programming language.

14.1.2  Smoothing and differencing

Probably the transformation most frequently applied to a time series is some form of smoothing, in order to remove the amount of noise and to bring out the underlying trend more clearly. Another often useful operation is the differencing of a time series, wherein the original data is replaced by the difference between any two consecutive observations. To handle these tasks, we will make use of session variables, and the comma operator.

Exponential smoothing in gnuplot

Probably the most familiar smoothing methods are the floating and trailing averages, which replace each data point with the average over some number of adjacent data points. In the first case, the averaging interval is symmetrical, which means that the smoothed curve can never extend all the way to the most recent value in the time series; for this reason, the trailing average uses a one-sided interval. Both methods have some problems, not the least of which being that they need to keep track of the all the data points in the averaging interval individually.

An often favorable and much simpler smoothing method is known as exponential smoothing or Holt-Winters method. It replaces each data point not with the average, but with the following quantity:

s[t] = a * x[t] + (1-a) * s[t-1]

Here, s[t] is the smoothed value at time step t, and it is calculated as a mixture between the observed value x[t] at that time, and the most recent smoothed value s[t-1]. The smoothing parameter a is some number between 0 and 1: the closer to 1, the less the data will be smoothed.

Exponential smoothing can be implemented as an inline transformation in gnuplot:

s=0; a=0.5
plot "data" u 1:(s=a*$2 + (1-a)*s) w l

This works, because the value of an assignment expression is the assigned value. It may be convenient to bundle the operation into a function, as in listing  14.3 ” (also see figure  14.2 ).

Listing 14.3. Single exponential smoothing: commands for figure 14.2 (file: smoothing.gp)

hw(x,a) = a*x + (1-a)*s
s=1; plot "unsmoothed" u 1:2 w l, "" u 1:(s=hw($2,0.2)) w l lw 2

Figure 14.2. A noisy time series and a smoothed trend line. The trend was calculated using single exponential smoothing: s=a*x + (1-a)*s, with a smoothing parameter of a = 0.5.

It is important that the variable s is initialized before the plot command. For best results, it should be initialized to a value that is similar to the first value in the data set.

[Tip]
Become familiar with exponential smoothing

Exponential smoothing is an exceedingly simple, yet robust and versatile method to smooth a time series and to bring out an underlying trend. It can be implemented using a convenient inline transformation in gnuplot, but don’t forget to initialize the temporary variable before invoking the plot command.

Double exponential smoothing

Single exponential smoothing (as in listing  14.3 ) works best for stationary time series that do not exhibit a long-term trend. If there is a trend present, the smoothed curve tends to lag behind and not constitute a good approximation to the raw data. For data sets that include a trend, one therefore uses a smoothing scheme that explicitly includes a trend. One such method is double exponential smoothing:

s[t] = a*x[t] + (1-a)*(s[t-1] + u[t-1])
u[t] = b*(s[t] - s[t-1]) + (1-b)*u[t-1]

Here, s[t] is the smoothed value at time step t and u[t] is the smoothed trend at the same time step. There are now two smoothing parameters, a and b, for the value and the trend, respectively. This method, too, can be implemented as an inline transformation in gnuplot:

a=0.5; b=0.5
s=0; u=0; q=0

plot "data" u 1:(s=a*$2+(1-a)*(s+u), u=b*(s-q)+(1-b)*u, q=s) w l

Obtaining a good approximation now requires finding suitable values for both smoothing parameters, a and b. This may involve a fair bit of trial-and-error experimentation!

Again, the formulas can be wrapped into a function:

hw2(x,a,b) = (s=a*x+(1-a)*(s+u), u=b*(s-q)+(1-b)*u, q=s)

s=0; u=0; q=0
plot "data" u 1:(s=hw2($2,0.5,0.5)) w l

Keep in mind that a gnuplot function does not create a scope for local variables: all variables are global. The function finds the variables it writes to (such as s, u, or q) as session variables in the current gnuplot session. It is a mistake to supply such variables as arguments to the function. But because the function expects to find these variables in the gnuplot session, it is essential that the variables have been created (that is, initialized) before the function is invoked.

Differencing a time series in gnuplot

Another transformation that is often useful is to difference the time series. This amounts to replacing each entry in the series with the difference between two consecutive values; the resulting series is shorter by one record.

In gnuplot, this can be accomplished through an inline transformation as well:

u=0
plot "data" u 1:(v=$2-u, u=$2, v) w l

This works, because gnuplot treats the comma , as an operator, similar to the way the C programming language does. It separates different expressions, evaluating each in turn (left to right), and returns the value of the last (right-most) one.

Using the comma operator, it is possible to define gnuplot functions comprising multiple statements. Traditionally, the body of a gnuplot function had to consist of a single expression, but the comma operator provides a way to circumvent this limitation. We have already seen an example in the previous section on double exponential smoothing, in the definition of the function hw2(x,a,b) = (s=a*x+(1-a)*(s+u), u=b*(s-q)+(1-b)*u, q=s).

[Tip]
Consider the comma operator

The comma operator lets you combine multiple statements into a single expression. Statements are evaluated left-to-right, the value of the right-most statement is returned as the value of the entire expression.

14.1.3  Monitoring and control charts

One of the most common applications of time series and time series graphs is the monitoring of some metric over time—usually in order to detect whether the observed system is behaving as expected. In such applications, some tasks tend to arise: data needs to be normalized, so that one can compare the respective behavior across different metrics. Furthermore, there is an interest in identifying unexpected behavior, such as outliers and drifts. Let’s discuss them in turn.

Making data comparable

Imagine you’re in charge of some operation. It may be a factor or production plant, it may be a data center, it might even be a sales department. What matters is that you want to monitor the most relevant metrics constantly.

To keep things simple, let’s assume a manufacturing operation, where there are just three parameters that really matter: the overall productivity (units per hour), the completion time for each unit (in minutes), and the defect rate (the ratio of defective units to the total number of units produced). You might want to plot them together on a control chart, so that you can immediately see if one of them starts running out of the allowed zone. But most of all, you want to be able to compare them against each other: is one parameter consistently performing better than the other ones? Is one of the parameters on a slippery slope, getting worse and worse, relative to the other ones? And so forth.

A naive way to achieve this effect is to just plot the three parameters in a single chart. The result is shown in figure  14.3  and is probably not what you wanted!

Figure 14.3. A control chart, showing three very different quantities simultaneously. What’s wrong with this picture?

The problem is that the three parameters assume very different values: productivity is typically around 10,000 units per hour, assembly time is on the order of an hour, and the defect rate should be very, very small. Before you can compare them, you therefore need to normalize them.

Making metrics comparable by normalizing them usually involves two steps: first, they are shifted by subtracting a constant offset (such as a long-term average), then they are rescaled by dividing them by their typical range of values (as expressed in the standard deviation, for example). The resulting normalized metrics are all centered on or near zero and their fluctuations are of roughly unit size. (Also see figure  14.4 .)

More generally, normalizing metrics always involves a linear transformation of the form:

where m is a measure for the typical value of the metric in question, and s is a measure for the typical spread or range of values. If the metric is well-behaved, with relatively regular and symmetric fluctuations, then the mean and standard deviation are popular choices for m and s, but if the metric is strongly affected by outliers, then the median and the interquartile range might be more robust choices. Another possibility is to use the smallest possible value for m and the range between the greatest and the smallest value for s: the resulting normalized quantity will not be centered on zero, but restricted to the unit interval.

Because normalization involves rescaling by the typical range of values, the resulting quantities are all dimensionless and are therefore directly comparable. (In contrast to the original data, whose numerical values depended on the units of measure that were used.)

Figure 14.4. A control chart showing normalized metrics. The data is the same as in figure 14.3.

Identifying outliers

Identifying outliers is in some ways a complementary activity to normalizing data, as it amounts to finding those data points that are different than the rest. When dealing with time series, there is one additional issue: the baseline itself is changing constantly!

In the context of time series analysis, an outlier is a data point that is further away from the trend than is typical. In the absence of other information about the data, the smoothed data is our best guess at the true “trend”, and that’s what we’ll use as a baseline.

Identifying outliers in a time series therefore involves three steps:

Establish a baseline by finding a smooth trend for the time series.

Find the range of typical deviations of the raw data from the trend.

Tag those records that differ by more than the typical range from the trend as outliers.

Figure 14.5. Number of daily calls to a call center. Shown is the raw data and the smoothed trend line.

Figure  14.5  shows the situation after the first step. The data consists of the daily number of calls made to a call center, and a trend has been added using single exponential smoothing:

s=0; a=0.25
plot "callcenter" u 1:2 w l, "" u 1:(s=a*$2+(1-a)*s) w l lt 3 lw 2

The next step involves finding the typical range of the fluctuations of the raw data around the trend. We can find this using the stats command:

s=0; a=0.25
stats "callcenter" u 1:(s=a*$2+(1-a)*s, $2-s)

It is important that the last expression in the using expression is $2-s: this is the momentary deviation of the actual data from the smoothed trend. For simplicity, I will use the standard deviation (as reported by stats) as measure or the range—for this data set, it does not differ very much from the interquartile range. The value of the standard deviation is roughly 21, and I choose to label all records that differ by more than twice that as “outliers”.

Figure  14.6  finally shows the data together with the trend, as before. Outliers as identified by the foregoing procedure are marked with symbols, and the typical range of fluctuations is indicated using a grey band. The complete plot command is:

s=0; a=0.25

plot "callcenter" u 1:2 w l,
»    "" u 1:(s=a*$2+(1-a)*s) w l lt 3 lw 2,
»    "" u 1:(s=a*$2+(1-a)*s, abs($2-s)>2*STATS_ssd_y?$2:NaN) w p lc 2 pt 7,
»    "" u 1:(s=a*$2+(1-a)*s, s+2*STATS_ssd_y):(s-2*STATS_ssd_y)
»       w filledc fs transparent solid 0.5 lc rgb "dark-grey"

Figure 14.6. Same data as in figure 14.5. In addition to the smoothed trend line, the region of normal fluctuations is indicated (shaded); outliers outside this region indicated through symbols. (See text for details.)

Cusum plots

Occasionally, we will find ourselves monitoring a quantity that has a well-defined target value (or setpoint in control-theory lingo): for instance, you may want to keep a power-supply at a precisely determined temperature. In such cases, it is essential to detect early whether the metric has drifted away from its setpoint. This may be difficult, because any noise in the data is likely to obscure a small, but systematic change quite effectively. Consider figure  14.7  Would you say that the system has undergone any change from its setpoint at y = 5 during the observed interval? (The horizontal lines at y=3 and y=7 are intended to indicate the “permissible range” for the observed metric.)

Figure 14.7. A control chart. The target value for the observed metric is 5. The horizontal lines indicate the permitted range around the setpoint.

The trick is to amplify the effect by keeping an accumulating sum of all deviations from the intended setpoint. As long as the system is centered on the correct value, the deviations will cancel each other out. But a persistent deviation, even if it is small, will quickly lead to the build-up of noticeable error in the cumulative sum. Plotting this quantity in a cusum chart provides a low-tech, yet remarkably sensitive diagnostic instrument to spot even small, but persistent deviations from the setpoint.

Figure 14.8. Same data as in figure 14.7, but shown now together with the cumulative deviation from the intended setpoint of 5. The sudden and persistent increase in the cumulative error indicates that the system is shifted away from its target value—even though this is not immediately visible in the raw data itself.

In gnuplot, you can obtain a cusum chart through a simple inline transformation. The following commands were used to create figure  14.8 , which clearly shows that something changed near time step 15:

setpoint = 5
s=0; plot [][-1:11] "cusum" u 1:2 w lp pt 7, "" u 1:(s=s+($2-setpoint)) w l

In this case, the data has been created synthetically as Gaussian noise with mean μ=5 and standard deviation σ=1 for the first 15 observations, but with mean μ=5.5 for the subsequent data points. Looking at the raw data in figure  14.7  does not reveal this permanent shift; even comparison with the reference lines above and below does not suggest any trouble. But the cumulative error in figure  14.8 is a dead giveaway!

When using cusum charts, then the cumulative error should not be calculated since the “beginning of time”. Even when the system is perfectly centered on its setpoint, the cumulative error will exhibit arbitrarily large deviations in the long run. For this reason, the cumulative error should be reset frequently and calculated over only a relatively small number of recent observations (a few dozen).

14.1.4  Changing composition and stacked curves

As a final example of a time series problem, let’s consider a situation where the composition of an aggregate changes over time, while at the the same time, the magnitude of the entire aggregate also changes. The total human population, for example, changes over time—but so does its breakdown by continent. Representing both aspects (overall size and composition) simultaneously is a difficult problem. In this section, we will introduce four complementary solutions. This will also give us an opportunity to demonstrate gnuplot’s for and sum constructs. (See listing  14.4 .)

Listing 14.4. Commands for figure 14.9 (file: composition.gp)

set xtics 26
set mxtics 2

set xrange [0:150]

set multip layout 2,2 rowsfirst margins 0.1,0.95,0.15,0.9 spacing 0.1,0.05

# top left: individual product numbers
set key horizontal samplen 2 at screen 0.55,0.95 width -7 center
set xtic format ""
set ylab "Absolute" offset 1,0
tx(k) = (k==2 ? 'Product 1' : ''.k-1)
plot for[k=2:5] "composition" u 1:k t  '' . tx(k) w l
unset ylab
unset key

# top right: stacked product numbers: use sum
plot for[k=2:5] "composition" u 1:(sum[j=2:k] column(j)) w l

# bottom left: individual fractions: divide by sum over all products
set xtic format "%g"
set xlab "Individual"
set ylab "Fractional" offset 1,0
plot [][0:.6] for[k=2:5] "composition"
»    u 1:(column(k)/(sum[j=2:5] column(j))) w l
unset ylab

# bottom right: stacked fractions: divide sum by sum over all
set xlab "Stacked"
plot [][0:1.025] for[k=2:5] "composition"
»    u 1:((sum[j=2:k] column(j))/(sum[j=2:5] column(j))) w l

unset multiplot

Figure 14.9. Four different ways to present changes in composition. Each line corresponds to the number of units produced for one of four different products. The top row shows absolute numbers of units produced; the bottom row shows fractions, relative to the total production. In the left column, each line is drawn separately; in the right column, the lines are “stacked”, giving a cumulative view into the data. See listing 14.4.

Let’s consider a company that manufactures four products, numbered 1 through 4. The absolute weekly production numbers (in 1,000 of units), for each product separately, are shown in figure  14.9 , top left. Such a plot makes it easy to follow the behavior of each product indvidually, but as the number of products grows, such a plot can become quite cluttered. Moreover, it is not easy to see how the total production changes over time. One could add a curve, but it would obviously be on a very different scale than the individual contributions. Using a second vertical scale (one for the individual numbers, and one for the totals) is possible, but is going to lead to even more clutter.

For these reasons, one occasionally finds the data represented in a stacked plot, as in figure  14.9 , top right. Here, the numbers for all products have been stacked on top of one another, so that the curve for (say) product 2 comprises the total manufactured output for products 1 and 2 combined. By construction, lines in such plots never cross, and hence they always have a “neat” appearance—which makes them popular in the general interest publications. Nevertheless, their problems are significant: only two curves are easy to interpret: the bottom one (product 1) and the top one (total production). It is exceptionally hard in such a stacked graph to estimate how the contributions from the individual components change over time. For example, how much of the sudden increase in total production past week 130 is due to product 4? It is basically impossible to tell from this graph—you have to go back to the previous panel. Similarly, from this graph it is not easy to see that the double-hump configuration starting in week 100 is due exclusively to product 2. (Judging the distance between closely spaced curves that vary a lot over the plot range is exceptionally difficult—see section  “Judging lengths and distances”).

If we are interested in the relative changes in composition, then—following the principle to “plot what’s relevant”—we should not plot the absolute numbers but their fractional contribution to the total. That’s what the panel in the bottom right of figure  14.9 ” does. This panel is also a “stacked” graph, but it shows the relative contribution of each component, not the absolute numbers. It shares with the previous panel that it is guaranteed to look “clean”, and that the individual contributions are hard to estimate. In addition, this diagram completely hides any information about the total production. In particular, the dramatic increase towards the end of the observation time frame is entirely invisible! Although diagrams of this form have a lot of intuitive appeal, they convey information poorly.

Finally, the panel in the bottom left shows the relative contributions, separately for each component again. It shares with the first panel (in the top left) that it appears cluttered, but that the information on individual products is easy to discern.

Figure 14.10. A stacked graph of absolute production numbers. The curves have been sorted in such a way that lines showing the smallest variation are lower in the stack, leading to a more stable baseline to which the subsequent products are added.

The appearance and information content of stacked graphs (like the ones on the right side of figure  14.9 ) depends strongly on the order in which the individual curves (here: product lines) are stacked. If there is some natural ordering, then such stacked graphs may, in fact, be the best way to represent the information. Figure  14.10 shows one such attempt. In this figure, the products have been ordered by their contribution to the total output, based on the (reasonable) assumption that products that contribute more are more important. Moreover, it makes sense to include curves that show less variation and are flatter earlier in the graph, to retain a more stable baseline for the curves that are added later to the stack. For this reason, product 1 has been included before product 2, although it contributes overall slightly less: in this way, the infamous double-hump affects fewer curves in the overall stack.

Because the components for figure  14.10  had to be picked individually, the sum facility was much less useful:

plot [0:150] "composition" u 1:5 t "Product 4" w l lt 4,
»            "" u 1:($5+$2) t "1" w l lt 1,
»            "" u 1:($5+$3+$2) t "2" w l lt 2,
»            "" u 1:($5+$3+$2+$4) t "3" w l lt 3

[Tip]
Stacked graphs

Stacked graphs look appealing, but often convey information poorly, unless you can find a natural order in which curves are added to the stack.

[8] The complete file can be found at http://www.monitorware.com/en/logsamples/apache.php.

14.2  Graphical techniques for multivariate data sets

Data sets with more than two (or maybe three) columns are called multivariate data sets. In principle, the number of columns can be arbitrarily large. Multivariate data sets pose special challenges for graphical methods: it is simply difficult to represent more than two dimensions in a necessarily flat graph!

There are basically two approaches to overcome the limitation posed by the two-dimensional nature of the canvas:

Juxtaposition of several graphs, each representing only a small number of the available attributes.

Use of complex symbols or glyphs that convey information not only through their position on the graph, but also through their appearance (shape or color).

In this section, I will demonstrate some of the most common graphical techniques for multivariate data sets. My intent here is twofold: I want to describe these graphical methods, but I also want to show how to realize them in gnuplot. Given that we are dealing with a multivariate problem, this will be an exercise in the use of loops and multiplot mode!

14.2.1  Introduction

Our case study for exploring multivariate graphics will be a data set describing the composition of a little more than 200 individual samples of glass.[9]

The data set consists of a total of eleven columns: the first is an integer to identify the record, followed by nine columns of actual measurements, and finally a column that specifies the type of glass as an integer between 1 and 7.[10] Hence, this data set has nine (or, if you count the glass type as well) ten attributes. The data set is therefore neither particularly large, nor excessively high-dimensional—just the right size to introduce our topic.

The data file is comma-separated, we therefore need to use set datafile separator ','. Listing  14.6 does this, and also includes a bunch of other definitions that will turn out useful. In all subsequent listings, this file with common definitions will be loaded first.

Listing 14.5. Common definitions for the “glass” data set (file: glass.def)

set datafile separator ','

data = "glass.data"                               # Name of input data file
names = "ID RfIx Na Mg Al Si K Ca Ba Fe Type"     # Column names
types = "Bldg1 Bldg2 Vhcl1 Vhcl2 Cont Tbl Lamp"   # Types of glass (col 11)

14.2.2  Distribution of values by attribute

The first question concerns the distribution of values for the various attributes. The nine observed values are numerical, but the glass type is categorical—it can take on only seven distinct values. Figure  14.11  shows a histogram of the latter; the commands are in listing  14.6 .

Figure 14.11. Number of data points for each type of glass. Compare listing 14.6.

Listing 14.6. Commands for figure 14.11 (file: glass1.gp )

load "glass.def"

set boxwid 1
set xtic format ""
set for[k=1:7] xtic add ( word(types,k) k )
plot data u 11:(1) smooth frequency w boxes

Clearly, window glass as used in buildings dominates this data set; moreover, there are no records in this data set of type 4 (vehicle windows of variety 2). It is for this reason, that it is important to fix the boxwidth: otherwise, the two adjacent boxes would extend to fill the gap.

Next we investigate the distribution of the nine numerical attributes. Keep in mind that, at this point, we have no idea at all what their numerical ranges are. That suggests box-and-whisker plots, because they do not require any adjustable parameters. See figure  14.12 and listing  14.7 

Figure 14.12. Distribution of values for each attribute. Compare listing 14.7.

Listing 14.7. Commands for figure 14.12 (file: glass2.gp )

load "glass.def"

unset xtics; unset key
set ticscale 0; set grid lt 1 lc rgb "grey" lw 2
set ytic offset 0.5,0         # Move tic marks closer to their boxes
set style fill solid 0.5

set multiplot layout 1,9 margin 0.075,0.975,0.1,0.95 spacing 0.075,0
do for[k=2:10] {
  set xlabel word(names,k)
  plot data u (1):k:(0.5) w boxplot pt 6
}
unset multi

The graph is a bit cluttered, mostly because each panel requires its own set of tic marks. For this reason, individual tic marks have been replaced with grey grid lines for the benefit a quieter visual appearance. Empty, instead of filled, circles are used for the outliers, because those are easier to distinguish when they overlap partially.

14.2.3  Distribution by level

It is natural to ask whether different glass types can be distinguished by the observed properties of the sample. Figure  14.13  therefore investigates the distribution of values separately for each type of glass, whicn in the present case constitutes the “level”. (Also see listing  14.8 .) The situation is certainly not as clearcut as it was in section "sec_special_serial"!

Figure 14.13. Distribution of values for each attribute, separately for each type of glass. The sequence of glass types is the same as in figure 14.12; type “Vhcl2” has been ignored, because there are no records for it. Compare listing 14.8.

Listing 14.8. Commands for figure 14.13 (file: glass3.gp )

load "glass.def"

unset xtic; unset key

set multi layout 3,3 margin 0.075,0.99,0.015,0.975 spacing 0.055,0.035
do for[k=2:10] {
  set label word(names,k) at graph 0.95,0.9 right
  plot data u (1):k:(0.5):11 w boxplot pt 6
  unset label
}
unset multi

Sample size in a serial box-and-whisker plot

Figure  14.13 is great to get an overall view, but a graph showing only a single attribute allows greater detail—see figure  14.14 

One question when looking at a serial box-and-whisker plot concerns the number of data points for each level: that would be good information to have in order to evaluate the relative importance of each level. One way to represent this quantity is via the width of the central box for each level, and that’s what has been done in figure  14.14 .

Gnuplot’s with boxplot style does not support this feature “out of the box”, but because it does support level-specific box widths, we can accomplish this goal with a little effort—see listing  14.9 .

The stats command is used to count the number of records for each level. A string is built up holding all the counts as whitespace-separated tokens; the word function is used to extract the appropriate token when needed. (See section "sec_scripting_fileloops" for a more detailed discussion of this technique.)

Figure 14.14. Distribution of values for attribute “Si”, separately for each type of glass. The width of the central box is proportional to the number of data points for each level. Compare listing 14.9.

Listing 14.9. Commands for figure 14.14 (file: glass4.gp )

load "glass.def"

col = 6                          # Column to display
get( s, k ) = word( s, int(k) )  # Access k-th element in string s

unset key
set title "Attribute: " . get(names,col)

s = ''
do for[k=1:7] {
  stats data u ($11==k?1:0) noout
  s = s . sprintf( " %d", STATS_sum )
}
plot data u (1):col:(0.01*get(s,$11)):(get(types,$11)) w boxplot

14.2.4  Scatter plot matrix

Box-and-whisker plots help us understand the distribution of data points for each attribute, but they tell us nothing about the relationships between different attributes. That’s what the scatter-plot matrix is for.

A scatter-plot matrix very simply consists of scatter plots for all possible pairs of attributes in a data set. In the present case, we have nine numerical attributes, and hence the scatter-plot matrix forms a 9×9 array (see figure  14.15 ). We can inspect this collection of graphs for correlations between attributes. For example, the index of refraction grows quite regularly with the Ca content of the sample.

The upper and lower half of a scatter plot matrix are not independent; in fact, each pairing of attributes appears twice, with horizontal and vertical axis interchanged. For this reason, I use the bottom half of the array to visualize clusters of points. The symbols are larger than in the top half, but the color is so transparent to be almost invisible. Only where many points overlap is the density high enough to form recognizable spots. (See section "sec_appearance_alpha" for information about transparency and alpha shading.)

You can find the commands to generate figure  14.15  in listing  14.10 . The command set size square ensures that each individual panel of the 9×9 matrix is square—for best results, you may want to make sure that the shape of the overall canvas forms a square as well, using set terminal ... size ... with your preferred terminal.

Figure 14.15. Scatter plot matrix of all attributes, ignoring the type of glass. Upper half of the matrix shows individual points, lower half shows smeared out and partially transparent symbols to facilitate the recognition of clusters. Compare listing 14.10.

Listing 14.10. Commands for figure 14.15 (file: glass5.gp )

load "glass.def"

unset key; unset tics

set size square

set multi layout 9,9 margin 0.01,0.99,0.01,0.99 spacing 0.01
do for[i=2:10] {
  do for[j=2:10] {
    if( i!=j ) {
      if( i<j ) {
        plot data u i:j w p ps .25 pt 7
      } else {
        plot data u i:j w p ps .5 pt 7 lc rgb "#e60000ff"
      }
    } else {
      unset border
      set label word(names,i) at 0,0 center
      plot [-1:1][-1:1] 1/0
      unset label
      set border
    }
  }
}
unset multi

14.2.5  Parallel-coordinates plot

The scatter-plot matrix is a method to visualize the relationship between any two attributes, but it provides no help at all when it comes to relationships across more than two attributes at the same time. For that, we turn to parallel-coordinate plots. They have been introduced before (in section "sec_special_parallel"), and I have already pointed out that gnuplot’s current implementation is subject to some limitations: in particular, the number of attributes is constrained to a small integer number. Among other things, we will now learn how to work around this issue.

The trick is to use multiplot mode (again)! Using multiplot mode, we can include as many attributes as desired, simply by placing several graphs side-by-side, each of which does not exceed the maximum number of columns supported (see listing  14.11 ).

Multiplot mode helps us cope with the number of attributes; to cope with the number of records we again resort to alpha shading. In the present case, the number of records is sufficiently high that lines simply obscure each other, making it impossible to discern any dominating structure in the graph. By using partially transparent lines, isolated lines fade into the background, but areas where lines cluster stand out strongly. (See figure  14.16 .)

Figure 14.16. Parallel-coordinates plot of the entire “glass” data set. Because of the relatively large number of records, all lines are drawn partially transparent, so as not to obscure each other. To accomodate all attributes, this graph has been composed using multiplot mode (see listing 14.11).

Listing 14.11. Commands for figure 14.16 (file: glass6.gp )

load "glass.def"

unset key; unset ytics
set xtics format ""

set style parallel lw 1 lc black

set linetype 1 lc '#ddff0000' lw 1

set multi layout 1,2 margin 0.05,0.9,0.1,0.95 spacing 0

# Left side
set for[k=1:5] xtic add ( word(names,k) k )
plot [1:6] data u 1:2:3:4:5:6 w parallel lt 1

# Right side
set for[k=1:6] xtic add ( word(names,k+5) k )
set paxis 6 tics format ""
set for[k=1:7] paxis 6 tics add ( word(types,k) k ) offset 7,0
plot [1:6] data u 6:7:8:9:10:11 w parallel lt 1

unset multi

Highlighting a subset of records

It is often desirable to highlight a subset of records in a parallel-coordinates plot, based on some condition. In figure 14.15, records were highlighted based on their “class”. In the present case, we will highlight all records that belong to a particular cluster. If you examine figure  14.16 , you will notice that there is a reasonably well-separated cluster of records with intermediate concentration of calcium (chemical symbol: Ca). In figure  14.17  those lines are highlighted, and we can observe two things: records with an intermediate calcium concentration tend to have unusually high or low aluminum (symbol: Al) concentration. And they tend to correspond to glass made for lamps or tableware, but not for containers or windows. That’s and interesting finding.

Highlighting makes use of data-dependent coloring (see section "sec_appearance_datacolors"): an additional column is specified at the end of the using declaration, and its value is evaluated to the linetype to use. Listing  14.12  defines the function c() that selects the linetype, depending on whether the argument to the function falls into the desired numerical range. (Tic marks have been turned on for the attribute in question.)

The listing defines two line types in addition to the default: one for highlighting, and one that is completely transparent and therefore invisible. Because it does not obscure any of the existing graphical elements, it is used for the records that are not selected for highlighting.

Figure 14.17. Same as figure 14.16, but with a subset of records highlighted. Records with a Ca content between 9 and 10 have been selected for highlighting. Compare listing 14.12.

Listing 14.12. Commands for figure 14.17 (file: glass7.gp )

load "glass.def"

unset key; unset ytics
set xtics format ""

c(x) = 9<x && x<10 ? 2 : 3

set style parallel lw 1 lc black

set linetype 1 lc '#ddff0000' lw 1   # Default style
set linetype 2 lc '#aa0000ff' lw 1   # Highlight style
set linetype 3 lc '#ff000000' lw 1   # Invisible (transparent) style

set multi layout 1,2 margin 0.05,0.9,0.1,0.95 spacing 0

# Left side
set for[k=1:5] xtic add ( word(names,k) k )
plot [1:6] data u 1:2:3:4:5:6 w parallel lt 1,
»          data u 1:2:3:4:5:6:(c($8)) w parallel lc var

# Right side
set for[k=1:6] xtic add ( word(names,k+5) k )
set paxis 3 tics 1                   # Turn on tic marks for one col
set paxis 6 tics format ""
set for[k=1:7] paxis 6 tics add ( word(types,k) k ) offset 7,0
plot [1:6] data u 6:7:8:9:10:11 w parallel lt 1,
»          data u 6:7:8:9:10:11:(c($8)) w parallel lc var

unset multi

Further suggestions

Figure  14.17  only explores one aspect of the data set; many similar clusters can be found and traced across all attributes. You may also do what we did in figure "fig_special_parallel03" and highlight all records belonging to a particular type of glass. Does this reveal any interesting structure in the data set? Another possibility is to draw all lines belonging to the same type of glass in the same color, using six different colors simultaneously.

Another aspect affecting the appearance of a parallel-coordinates plot is the order or sequence in which attributes are plotted. In figures  14.16  and  14.17 I used the (arbitrary) arrangement of columns in the original data file, but different permutations may give quite different results. Unfortunately, gnuplot does not make it easy to shuffle columns in a parallel-coordinates plot: you have to create different using declarations to rearrange attributes. Here are two permutations that you may want to try out:

1,4,6,2,5,3,8,7,10,9,11
1,4,6,3,5,8,2,10,7,9,11

[9] This example comes from the “Glass Identification” data set, available from the UCI Machine Learning Repository: Asuncion, A. and Newman, D.J. (2007). UCI Machine Learning Repository [http://www.ics.uci.edu/~mlearn/MLRepository.html]. Irvine, CA: University of California, School of Information and Computer Science.

[10] The measurements comprise the refractice index (RfIx) and the relative concentration of various chemical elements in each glass sample. The types of class include two types of glass each for building and vehicle windows, glass for containers, tableware, and headlamps.

14.3  Visual Perception

Even if we’ve found the most appropriate combination of quantities to look at, the amount of information we extract from a graph still depends on the way the data is presented. The way we perceive a plot depends on our human, visual-psychological perception of it. To get the most out of a graph, we should therefore format it to improve the way it will be perceived.

Whether we’re aware of it or not, in looking at a plot we always tend to engage in comparisons. Is this line longer than that? Is this part of the curve steeper than that? Is the area underneath this function larger than the area underneath that function? And so on.

To gain the maximum insight from a plot, we therefore want to find a representation of the data that facilitates such comparisons as much as possible. In this section, I want to give you some ideas how this can be achieved.

14.3.1  Banking

The idea of banking (or banking to 45 degrees) has been proposed by W. S. Cleveland, who demonstrated in a series of controlled experiments that our ability to compare angles is greatest for angles that are close to 45 degrees.[11] In other words, if we want to assess and compare the slopes of lines on a graph, we’ll do best if the graph is sized in such a way that most of the lines are roughly going diagonally across it.

In a way, this is just a confirmation of something that you probably have been doing intuitively already. Whenever you’ve found yourself adjusting the plot ranges of a graph to get its aspect ratio right (as opposed to narrowing the plot ranges to “zoom in” on some region), you’ve probably been “banking.” Let’s consider an example: figure  14.18  shows the number of sun spots observed annually, for the 300 years from 1700 to 1999.[12] The number of sun spots observed varies from year to year, following an irregular cycle of about 11 years.

Figure 14.18. Annual sunspot numbers for the years 1700 through 2000. What can you say about the shape of the curve in this representation?

Because of the large number of cycles included in the graph, vertical (or almost vertical) line segments dominate the graph, making it hard to recognize the structure in the data. In figure  14.19 , the plot range has been split into thirds, and each third is displayed in a separate panels. The aspect ratio of the panels has been chosen in such a way that the raising and falling edges of the curve are close to 45 degrees. The graphs alone (without the labels and decorations) can be created very easily:

set multiplot layout 3,1 margins 0.1,0.95,0.1,0.95 spacing 0
plot [0:100] "sunspots" u ($1-1700):2 w l
plot [0:100] "sunspots" u ($1-1800):2 w l
plot [0:100] "sunspots" u ($1-1900):2 w l
unset multi

We can now easily recognize an interesting feature in the data, namely, that the number of sun spots during each cycle rises quickly, but tends to fall more slowly. (There’s the visual comparison, referred to earlier!) We wouldn’t have been able to spot this from the representation of the data in figure  14.18 .

Figure 14.19. A different representation of the sunspot data from figure 14.18. The aspect ratio of each segment in the cut-and-stack plot is such that it banks lines to 45 degrees

Banking is a valuable tool. In particular, I find it helpful because it draws our attention to the importance of the apparent slopes of lines on a graph. Nevertheless, it must be used with judgment and discretion. Taken by itself, it can lead to graphs with strongly skewed aspect ratios. Figure  14.19  is a good example: if we hadn’t stacked the three segments on top of each other, the resulting graph would have a very awkward shape (extremely long and flat).

[Tip]
Banking

To improve the perception of slopes, choose the plot range and the aspect ratio of your graphs such that most of the curves are at approximately 45 degrees to the horizontal. If necessary, chop up the data set into segments and arrange them in a cut-and-stack plot, to keep the overall aspect ratio of the plot at a convenient value.

14.3.2  Judging lengths and distances

Look at figure  14.20 . It shows the flows to and from a storage tank over time. For the interval considered here, the inflows are always greater than the outflows, so that the tank tends to fill up over time, but that’s not our concern right now. (Let’s say the tank is large enough.)

Figure 14.20. Inflow and outflow to and from a storage tank. What’s the net flow to the tank?

Instead, let’s ask for the net inflow as a function of time—the inflow less the outflow at each moment. Could you draw it? Does it look at all like the graph in figure  14.21 ? In particular, did your graph contain the peak between 6 and 7 on the horizontal axis? How about the relative height of the peak?

Figure 14.21. Net flow to the storage tank. This is the difference between the inflow and the outflow (see figure 14.20).

This example shows how hard it is to estimate accurately the vertical distance between two curves with large slopes. The eye has a tendency to concentrate on the shortest distance between two curves, not on the vertical distance between them. The shortest distance is measured along a straight line perpendicular to the curves. For nearly horizontal curves, this is reasonably close to the vertical distance, but as the slopes become more steep, the difference becomes significant. (Because we’re interested in the difference between the two flow rates at the same point in time, we’re looking specifically for the vertical distance between the two curves.)

You may also want to compare the observations made here with the discussion in section   “Changing composition and stacked curves”—the problems are quite similar. As a general rule, it is difficult to estimate changes against a baseline that itself is not constant, and the more more rapidly the baseline changes, the harder it is to use.

[Tip]
Distance between curves

The eye has difficulties accurately judging the vertical distance between two curves with large slopes. If the nature of the difference between two quantities is important, then plot it explicitly. Do not rely on observers to recognize it from the contributing components.

14.3.3  Plot ranges and whether to always include zero

Plots should have meaningful plot ranges, showing those parts of the data that the viewer is most likely going to be interested in at a reasonable resolution. It may not be possible to convey all the meaning appropriately with a single choice of plot ranges, so don’t be shy about showing two or more views of the same data set, for instance an overview plot and a detailed close-up of only a small section of the entire data set.

A long-standing, and largely unfruitful discussion concerns the question of whether zero should be included in the range of a graph. The answer is simple: it all depends.

If you’re interested in the total value of a quantity, you probably want to include zero in your range. If you care only about the variation relative to some baseline other than zero, then don’t include zero.

Figure  14.22  demonstrates what I mean. Both panels of the graph show the same data. The main graph tells us that the total variation is small, compared to the overall value. The inset tells us that there has been a steady increase from left to right. Both views are valid, and each gives an answer to a different question.

Figure 14.22. Two different views of the same data set. The outer graph includes 0 in the vertical plot range, the inner graph displays only that part of the vertical axis that corresponds to data points. Both views are valid: the outer graph shows that the relative change in the data is small to its absolute value; the inner graph shows that the variation increases dramatically from left to right.

Plot ranges are a bit more of a concern when several graphs need to be compared against each other. In such a situation, all graphs should have the same scale to facilitate comparison; in fact, using different scales for different graphs is a guaranteed path to confusion (because the difference in scales will go unnoticed or be conveniently forgotten). And if one of the graphs legitimately includes zero, then all of them will have to do the same.

[11] See The Elements of Graphing Data by William S. Cleveland, Hobart Press (1994) and references therein.

[12] The data was obtained from WDC-SILSO, Royal Observatory of Belgium, Brussels, http://www.sidc.be/silso/datafiles.

14.4  Summary

In this chapter, we choose three recurring topics and discussed them in greater depth. The first was the graphical analysis of time series, and we paid particular attention to finding a smooth approximation to the raw data’s overall trend. We then described a variety of methods for the graphical treatment of multivariate data sets and how to realize them in gnuplot. Finally, I pointed out the importance of visual perception and described some ways to take advantage of it.

15

Coda: Understanding data with graphs

But in the end, the most important thing is to draw the right conclusions from the available evidence.

During World War II, a program was launched to explore the possibility of equipping fighter planes with additional armor as protection against ground-based anti-aircraft fire.[13] To determine where on the airplane to place the armor, fighter planes returning from combat missions were investigated for bullet holes. In figure  15.1 , the areas where bullet holes were found are shown shaded; areas without bullet holes are left white.

Figure 15.1. Schematic outline of a fighter airplane. Areas where bullet holes were found on machines returning from combat missions are shaded. Where would you recommend additional armor to be placed?

Given this evidence, where would you put additional armor?

The not-so-obvious obvious answer is to add the armor in those areas where no bullet holes were found. Why? Because airplanes are subject to hits everywhere, but if the hits strike in the white areas in figure  15.1 , the airplane doesn’t come back from its mission. (Statisticians speak of survivorship bias.) Therefore, those are the most vital areas of the machine and should receive the best possible protection.

So, let this be our final lesson. Evidence, be it graphical or otherwise, is just that: mere data. But actual insight arises only through the correct interpretation of those facts.

[13] The inspiration to this story stems from the book Graphical Discovery by Howard Wainer, Princeton University Press (2005).