[Copyright Information] [Table of Contents] [Que Home Page]
[Prev Chapter] [Next Chapter]

19 - What is CGI?

Chapter Index

CGI, or Common Gateway Interface, is the standard programming interface between web servers and external programs. It is almost one of the most exciting and fun areas of programming today. The CGI standard lets web browsers pass information to programs written in any language. If you want to create a lightning-fast search engine, then your CGI program will most likely be written in C or C++. However, most other applications can use Perl.

The CGI standard does not exist in isolation, it is dependent on the HTML and HTTP standards. HTML is the standard that lets web browsers understand document content. HTTP is the communications protocol that, among other things, lets web servers talk with web browser.

Note
If you are unfamiliar with HTML, you might want to skip to the HTML introduction in Chapter 20, "Form Processing," before continuing. Otherwise, take the HTML references in this chapter at face value.

Almost anyone can throw together some HTML and hang a "home page" out on the web. But most sites out there are, quite frankly, boring. Why? The fact is that most sites are built as a simple series of HTML documents that never change. The site is completely static. No one is likely to visit a static page more than once or twice. Think about the sites you visit most often. They probably have some interesting content, certainly, but more importantly, they have dynamic content.

So what's a webmaster to do? No webmaster has the time to update their web site by hand every day. Fortunately, the people who developed the web protocol thought of this problem and gave us CGI. CGI gives you a way to make web sites dynamic and interactive.

Each word in the acronym, "Common Gateway Interface," helps to understand the interface:

CGI applications can perform nearly any task that your imagination can think up. For example, you can create web pages on-the-fly, access databases, hold telnet sessions, generate graphics, and compile statistics.

The basic concept behind CGI is pretty simple, however, actually creating CGI applications is not. That requires real programming skills. You need to be able to debug programs and make logical connections between one idea and another. You also need to have the ability to visualize the application that you'd like to create. This chapter and the next, "Form Processing," will get you started with CGI programming. If you plan to create large applications, you might want to look at Que's "Special Edition, Using CGI".

Why use Perl for CGI?

Perl is the de facto standard for CGI programming for a number of reasons, but perhaps the most important are:

The advantage of an interpreted language in CGI applications is its simplicity in development, debugging and revision. By removing the compilation step, you and I can move more quickly from task to task, without the frustration that can sometimes arise from debugging compiled programs. Of course not any interpreted language will do. Perl has the distinct advantage of having an extremely rich and capable functionality.

There are some times when a mature CGI application should be ported to C or another compiled language. These are the web applications where speed is important. If you expect to have a very active site, you probably want to move to a compiled language because they run faster.

CGI Apps vs Java Applets

CGI and Java are two totally different animals. CGI is a specification that can be used by any programming language. CGI applications are run on a web server. Java is a programming language that is run on the client side.

CGI applications should be designed to take advantage of the centralized nature of a web server. They are great for searching databases, processing HTML form data, and other applications that require limited interaction with a user.

Java applications are good when you need a high degree of interaction with users; for example, games or animation.

Java programs need to be kept relatively small because they are transmitted through the Internet to the client. CGI applications on the other hand can be as large as needed because they reside and are executed on the web server.

You can design your web site to use both Java and CGI applications. For example, you might want to use Java on the client side to do field validation when collecting information on a form. Then once the input has been validated, the Java application can send the information to a CGI application on the web server where the database resides.

Should You Use CGI Modules?

I encourage you to use the CGI modules that are available on the Internet. The most up-to-date module that I know about is called cgi.pm - but you must be using Perl v5.002 or an even newer version in order to use it. cgi.pm is very comprehensive and covers many different protocols in addition to the basic CGI standard.

You might find that CGI.pm is overkill for simple CGI applications. If so, look at cgi-lite.pl. This library doesn't do as much as CGI.pm but you'll probably find that it is easier to use.

You can find both of these scripts at one of the CPAN web sites that are mentioned in Chapter 22, "Internet Resources."

However in this book, I have purposely not used these scripts. I feel it is important for you to understand the mechanisms behind the protocols. This will make debugging your applications easier because you'll have a better idea what the modules are doing behind the scenes. You will also be able to make better use of pre-existing modules if you can make educated guesses about what a poorly documented function does.

How does CGI Work?

CGI programs are always placed on a disk that the web server has access to. This means that if you are using a dial-up account to maintain your web site, you need to upload your CGI programs to the server before they can be run.

Tip
You can test your scripts locally as long as you can use Perl on your local machine. See the "Debugging CGI Programs" section later in this chapter.

Web servers are generally configured so that all CGI applications are placed into a cgi-bin directory. However, the web server may have aliases so that "virtual directories" exist. Each user might have their own cgi-bin directory. The directory location is totally under the control of your web site administrator.

Tip
Finding out which directory your scripts need to be placed in is the first step in creating CGI programs. Since you need to get this information from your web site administrator, send an email message right now requesting this information. Also ask if there are any CGI restrictions or guidelines that you need to follow.

Calling Your CGI Program

The easiest way to run a CGI program is to type in the URL of the program into your web browser. The web server should recognize that you are requesting a CGI program and execute it. For example, if you already had a CGI program called test.pl running on a local web server, you could start it by entering the following URL into your web browser:

http://localhost/cgi-bin/test.pl
The web server will execute your CGI script and any output is displayed by your web browser.

The URL for your CGI program is a virtual path. The actual location of the script on the web server depends on the configuration of the server software and the type of computer being used. For example, if your computer is running the Linux operating system and the NCSA web server in a "standard" configuration then the above virtual would translate into /usr/local/etc/httpd/cgi-bin/test.pl. If you were running the webSite server under Windows 95, the translated path might be /website/cgi-shl/test.pl.

If you have installed and are administering the web server yourself, you probably know where to place your scripts. If you are using a service provider's web server, ask the server's administrator where to put your scripts and how to reference them from your documents.

There are other ways to invoke CGI programs besides using a web browser to visit the URL. You can also start CGI programs from:

Interestingly enough you can pass information to your CGI program by adding extra information to the standard URL. If your CGI program is used for searching your site, for example, you can pass some information to specify which directory to search. The following HTML hyperlink will invoke a search script and tell it to search the /root/document directory.

<A HREF="cgi-bin/search.pl/root/document">Search the Document Directory</A>
This extra path information can be accessed through the PATH_INFO environment variable.

You can also use a question mark to pass information to a CGI program. Typically a question mark indicates that you are passing keywords that will be used in a search.

<A HREF="cgi-bin/search.pl?Wine+1993">Search for 1993 Wines</A>
The information that follows the question mark will be available to your CGI program through the QUERY_STRING environment variables.

Using either of these approaches will let you create canned CGI requests. By creating these requests ahead of time, you can reduce the amount of typing errors that your users might otherwise have. Later in this chapter, the "CGI and Environment Variables" section discusses all of the environment variables you can use inside CGI programs.

Note
Generally speaking, visitors to your web site should never have to type in the URL for a CGI program. A hypertext link should always be provided to start the program.

Your First CGI Program

You can use any text editor or word processor in the world to create your CGI programs because they are simply Perl programs that are invoked by a URL instead of the command line. Listing 19.1 contains a five line CGI program - nearly the smallest program you can have.

Pseudocode

Turn on the warning option.

Turn on the strict pragma.

Send the HTTP header to the web browser.

Send a line of text to the web browser.

Listing 19.1-19LST01.PL - A Very Small CGI Program


#!/usr/local/bin/perl -w use strict; print "Content-type: text/plain\n\n"; print "Hello, World.\n";

The file that contains this CGI program should be placed in your web server's cgi-bin directory. Then, the URL for this program will be something like http://localhost/cgi-bin/test.pl (change localhost to correspond to your web server's hostname). Enter this URL into your web browser and it should display a web page saying "This is a test."

Note
You may wonder how the web server knows that a CGI program should be executed instead of being displayed. This is an excellent question. It can be best answered by referring to the documentation that came with your particular server.

When the web server executes your CGI program, it automatically opens the STDIN, STDOUT, and STDERR file handles for you.

The web server will also make some information available to your CGI program through environment variables. You may recall the %ENV hash from Chapter 12, "Using Special Variables." Details about the environment variables that you can use can be found in the "CGI and Environment Variables" section later in this chapter.

Why Are File Permissions Important in UNIX?

File permission controls who can access files in UNIX systems. Quite often I hear of beginning CGI programmers that try to write files into a directory in which they do not have write permission. UNIX permissions are also called rights.

UNIX can control file access in a number of ways. There are three levels of permissions for three classes of users. To view the permissions on a file use the ls command with the -l command-line option. For Example:

C:indyunix:~/public_html/pfind>ls -l
total 40
-rw-r--r--   1 dbewley  staff        139 Jun 18 14:14 home.html
-rwxr-xr-x   1 dbewley  staff       9145 Aug 14 07:06 pfind
drwxr-xr--   2 dbewley  staff        512 Aug 15 07:11 tmp
Each line of this listing indicates a separate directory entry. The first character of the first column is normally either a dash or the letter d. If a directory entry has a d it means that the entry is a subdirectory of the current directory.

The other nine characters are the file permissions. Permissions should be thought of in groups of three, for the three classes of user. The three classes of user are:

Each of the classes can have one or more of the following three levels of permission:

If a permission is not allowed to the user that ran the ls command, its position is filled with a dash. For example:

ls -l hform.html
-rwx------   1 dbewley  staff      11816 May  9 09:19 slideshow.pl
The owner, dbewley, has full rights - read, write, and execute for this file. The group, staff, and everyone else have no rights.

Tip
Perl scripts are not compiled, they must be read by the perl interpreter each time they are run. Therefore, perl scripts unlike compiled programs must have execute and read permissions.

Here is another example:

ls -l pfind.pl
-rwxr-x---   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
This time, the owner has full access while the group staff can read and execute the file. All others have no rights to this file.

Most HTML files will have permissions that look like this:

ls -l schedule.html
-rw-r--r--   1 dbewley  staff       2439 Feb  8 1996  schedule.html
Everyone can read it, but only the user can modify or delete it. There is no need have execute permission since HTML is not an executable language.

You can change the permissions on a file by using the chmod command. The chmod command recognizes the three classes of user as u, g, and o and the three levels of permissions as r, w, and x. It grants and revokes permissions with a + or - in conjunction with each permission that you want to change. It also will accept an a for all three classes of users at once.

The syntax of the chmod command is:

chmod <options> <file>
Here are some examples of the chmod command in action:

ls -l pfind.pl
-rw-------   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
chmod u+x pfind.pl
ls -l pfind.pl
-rwx------   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
The first ls command shows you the original file permissions. Then, the chmod command added execute permission for the owner (or user) of pfind.pl. The second ls command displays the newly changed permissions.

To add these permissions for the both the group and others classes use go+rx as in the following example. Remember, users must have at least read and execute permissions to run perl scripts.

ls -l pfind.pl
-rwx------   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
chmod go+rx pfind.pl
ls -l pfind.pl
-rwxr-xr-x   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
Now, any user can read and execute pfind.pl. Let's say a serious bug was found in pfind.pl and we don't want it to be executed by anyone. To revoke execute permission for all classes of user use the a-x option with the chmod command.

ls -l pfind.pl
-rwxr-xr-x   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
chmod a-x pfind.pl
ls -l pfind.pl
-rw-r--r--   1 dbewley  staff       2863 Oct 10 1995  pfind.pl
Now, all users can read pfind.pl, but no one can execute it.

HTTP Headers

The first line of output for most CGI programs must be an HTTP header that tells the client web browser what type of output it is sending back via STDOUT. Only scripts that are called from a server-side include are exempt from this requirement.

Table 19.1 - A List of HTTP Headers
Response Type HTTP Header
Text Content Type: text/plain
HTML page Content Type: text/html
gif graphic Content Type: image/gif
Redirection to anther web pageLocation: http://www.foobar.com
Cookie Set-cookie: ...
Error Message Status: 402

All HTTP headers must be followed by a blank line. Use the following line of code as a template:

print("Content Type: text/html\n\n");
Notice that the HTTP header is followed by two newline characters. This is very important. It ensures that a blank line will always follow the HTTP header.

If you have installed any helper applications for Netscape or are familiar with MIME types, you already recognize the text/plain and text/html parts of the Content Type header. They tell the remote web browser what type of information you are sending. The two most common MIME types to use are text/plain and text/html.

The Location header is used to redirect the client web browser to another web page. For example, let's say that your CGI script is designed to randomly choose from among 10 different URLs to order to determine the next web page to display. Once the new web page is choosen, your program outputs it like this:

print("Location: $nextPage\n\n");
Once the Location header has been printed, nothing else should be printed. That is all the information that the client web browser needs.

Cookies and the Set-cookie: header are discussed in the "Cookies" section later in this chapter.

The last type of HTTP header is the Status header. This header should be sent when an error arises in your script that your program is not equipped to handle. I feel that this HTTP header should not be used unless you are under severe time pressure to complete a project. You should try to create your own error handling routines that display a full web page that explains the error that happened and what the user can do to fix or circumvent it. You might include the time, date, type of error, contact names and phone numbers and any other information that might be useful to the user. Relying on the standard error messages of the web server and browser will make your web site less user friendly.

CGI and Environment Variables

You are already familiar with environment variables if you read Chapter 12, "Using Special Variables." When your CGI program is started, the web server creates and initializes a number of environment variables that your program can access using the %ENV hash.

Table 19.2 contains a short description of each environment variable. A complete description of the environmental variables used in CGI programs can be found at

http://www.ast.cam.ac.uk/~drtr/cgi-spec.html
Table 19.2 - CGI Environment Variables
CGI Environment Variables Description
AUTH_TYPE Optionally provides the authentication protocol used to access your script if the local web server supports authentication and if authentication was used to access your script.
CONTENT_LENGTH Optionally provides the length, in bytes, of the content provided to the script through the STDIN file handle. Used particularly in the POST method of form processing. See Chapter 20, "Form Processing," for more information.
CONTENT_TYPE Optionally provides the type of content available from the STDIN file handle. This is used for the POST method of form processing. Most of the time this variable will be blank and you can assume a value of application/octet-stream.
GATEWAY_INTERFACE Provides the version of CGI supported by the local web server. Most of the time this will be equal to CGI/1.1.
HTTP_ACCEPT Provides a comma-separated list of MIME types the browser software will accept. You might check this environmental variable to see if the client will accept a certain kind of graphic file
HTTP_USER_AGENT Provides the type and version of the user's web browser. For example, the Netscape web browser is called Mozilla.
HTTP_FROM Provides the user's email address. Not all web browsers will supply this information to your server. Therefore, only use this field to provide a default value for an HTML form.
QUERY_STRING Optionally contains form information when the GET method of form processing is used. QUERY_STRING is also used for passing information like search keywords to CGI scripts.
PATH_INFO Optionally contains any extra path information from the HTTP request that invoked the script.
PATH_TRANSTLATED Maps the script's virtual path (i.e. from the root of the server directory) to the physical path used to call the script.
REMOTE_ADDR Contains the dotted decimal address of the user.
REMOTE_HOST Optionally provides the domain name for the site that the user has connected from.
REMOTE_IDENT Optionally provides client identification when your local server has contacted an IDENTD server on a client machine. You will very rarely see this because the IDENTD query is slow.
REMOTE_USER Optionally provides the name used by the user to access your secured script.
REQUEST_METHOD Usually contains either "GET" or "POST" - the method by which form information will be made available to your script. See Chapter 20, "Form Processing," for more information.
SCRIPT_NAME Contains the virtual path to the script.
SERVER_NAME Contains the configured hostname for the server.
SERVER_PORT Contains the port number that the local web server software is listening on. The standard port number is 80.
SERVER_PROTOCOL Contains the version of the web protocol this server uses. For example, HTTP/1.0.
SERVER_SOFTWARE Contained the name and version of the web server software. For example, webSite/1.1e.

URL Encoding

One of the limitations that the WWW organizations have placed on the HTTP protocol is that the content of the commands, responses, and data that are passed between client and server should be clearly defined. It is sometimes difficult to tell simply from the context whether a space character is a field delimiter or an actual space character between to add whitespace between two words.

To clear up the ambiguity, the URL encoding scheme was created. Any spaces are converted into plus (+) signs to avoid semantic ambiguities. In addition, special characters or 8-bit values are converted into their hexadecimal equivalents and prefaced with a percent sign (%). For example, the string Davy Jones <dj@planet.net> is encoded as Davy+Jones+%3Cdj@planet.net%3E. If you look closely, you see that the < character has been converted to %3C and the > character has been coverted to %3E.

Your CGI script will need to be able to convert URL encoded information back into its normal form. Fortunately, Listing 19.2 contains a function that will convert URL encoded.

Pseudocode

Define the decodeURL() function.

Get the encoded string from the parameter array.

Translate all plus signs into spaces.

Convert character coded as hexadecimal digits into regular characters.

Return the decoded string.

Listing 19.2-19LST02.PL - How to Decode the URL Encoding