Batchftp Version 1.02 October 1990 Public Domain TABLE OF CONTENTS Section title Page I. INTRODUCTION II. GENERAL FEATURES III. COMPILATION IV. NEW RELEASES V. QUICK REFERENCE VI. COMMAND LINE OPTIONS VII. IMPORTANT CONSTANTS VIII. REGULAR MODE IX. FREE-FORM MODE X. RANDOM DELAY XI. SESSION ALARM XII. FILES XIII. COMPATIBILITY XIV. TESTING XV. SUPPORT I. INTRODUCTION Batchftp is a C program which automates the ftp process on Berkeley UNIX systems. A user is able to upload and download files from almost any remote computer without monitoring the transfer. A user simply enters the commands or files to be retrieved in an input file. Then, the program will log onto the specified computer (not necessarily a ftp site) and execute the given commands. Since Batchftp can run in the background, a user is free to do other things while the transfer sessions are taking place. More importantly, it will not give up until a successful connection has been established with the remote site. However, execution will be temporarily suspended after a given number of failures; connection attempts will resume after the extended delay. Batchftp is based on the package AutoFtp30, a Bourne shell script and three C programs by Mingqi Deng (July 1989). II. GENERAL FEATURES 1. Batchftp tries repeatedly to log onto a remote computer and execute the given commands. After a given number of failures, it will temporarily suspend execution and sleep for an extended time. 2. The program supports a shorthand notation (similar to the notation in AutoFtp30) which simplifies file requests. In addition, Batchftp has a free-form mode which allows the user to execute any ftp command. Shorthand notation is also supported in free-form mode. 3. If the user desires, Batchftp will attempt to validate the given hostname or Internet address of the remote computer. Also, if required, it will substitute the Internet address for the hostname. Furthermore, it obtains as much information as possible on a user's login session and the local machine from which the ftp requests are originating. 4. The adjustable alarm tries to minimize some of the problems caused by silent connections. If a transfer has not terminated by the time the alarm "sounds", Batchftp will check the size of the output file and determine if the session is producing any output. If the file is empty, the ftp process is killed, and the program attempts to reconnect. Otherwise, the ftp process is allowed to proceed until completion and terminate naturally. 5. All string handling functions are implemented in the program to increase compatibility. III. COMPILATION Batchftp consists of a single C program, and it does not need to linked with additional libraries. The name of the executable file does not matter. An example would be: cc batchftp.c -o batchftp Note: The executable file may be usually large in some cases. Batchftp compiled on Pyramid 9815 running OSx 4.4 (4.3 BSD) was approximately 140K. However, the same program was compiled on a SUN 3/160 running SunOS 4.1, and it was only 32K. IV. NEW RELEASES (Original Release September 1990) Version 1.02 (October 1990) 1. The variable h_errno is included in the source code. On some systems it is not defined in . 2. A option for a recursive directory listing has been added (-r switch). The suggestion was made by Mark A. Horstman (mh2620@sarek.sbc.com). 3. Some redundancies have been eliminated in ftp script files. In free-form mode the same local and remote directories were listed for each file request. 4. The error handling and process control is improved to help ensure that processes are killed after "silent connections." 5. The overall file-handling efficiency has been inproved. 6. The -i switch has been added to the ftp command line to disable interactive prompting during a session. It was an oversight on my part not to include it in the earlier release. Some people may say that I should have also included the -v switch to enable verbose mode. However, if the transfer session produces output before a connection is made, the session may continue indefinitely. V. QUICK REFERENCE This section briefly describes the input file commands and other conventions of Batchftp. For additional information, see the sections later in this document. Constants: _TENEX Check the man page for your local ftp server to determine the command for 36 to 8 bit transfer; the default is "type tenex". Command-line options: Batchftp supports two switches and a variable number of arguments on the command line (In this case switches are not considered to be arguments). The three different command lines are: batchftp input_file remote_host [username password] No address mapping -d: (no database address translation) batchftp -d input_file remote_host [user_name password] No ftp script initialization: -i batchftp -i input_file Note: The -i option also disables address mapping. Shorthand notation: -a ASCII transfer -b binary transfer -t tenex mode (36 to 8) -l get a directory listing -r recursive directory listing -c change local directory -d change remote directory These options require the following arguments; file specifications enclosed in brackets are optional. This notation is valid in both regular and free-form modes. Regular mode constitutes using only these commands to request files. -a, -b, -t file1 [file2] -l remote_directory output_file -r initial_remote_dir output_file -c local_directory -d remote_directory Example: -d /mirrors/msdos/sysutl -c ~/archives/local -a sysinfo.txt sysinfo.dat -b vmix225.arc -l /mirrors/msdos/dskutl disk.lst Free-form mode: The beginning and end of a free-form transfer are signified by '{' and '}'. Once one of these symbols is read on an input line, no other characters will be recognized. All shorthand notation described above is valid in free- form mode. Any number of file requests (shorthand symbols) may appear within a pair of brackets. All of these commands will be executed during one ftp transfer session. (In regular mode each file request is a separate session). Example: { -d /mirrors/msdos/sysutl -c ~/archives/local -a sysinfo.txt sysinfo.dat put info.dat info.txt -b vmix225.arc -l /mirrors/msdos/dskutl disk.lst } VI. COMMAND LINE OPTIONS Batchftp supports a variable number of switches and command line arguments. The rather cryptic usage message that a user gets is as follows: Usage: batchftp [-di] input_file [remote_host username pass] The switches determine which command line arguments are necessary. A. (No switches) If no switches are used, the required arguments are the user's input file and the remote host (official name or Internet address). The program provides default values for any other arguments which the user does not specify. The default username is obtained from the local password file by userid. The default password is "user@host". If no information can be obtained about the local host from the network database, the secondary default is "guest". Usually, however, the local call to "gethostname" succeeds. The name returned is then used by "gethostbyname" to get the network name if possible (host.domain.network). It seems that the network database only recognizes the full "network name" of a host. Therefore, if you would rather use an alias, specify the "-d" switch to suppress address validation and mapping. A username and a password may be specified on the command line. For security reasons it is not recommended that you specify a password in this manner. The most secure way is to enter it in your input file using free-form mode. B. (-d switch) The "-d" switch suppresses address mapping and validation. All other conventions outlined in part A apply here also. C. (-i switch) The "-i" switch disables ftp script initialization. Without this specification Batchftp begins each ftp script file with the following three lines (with appropriate substitutions). open remote_host user login_name password verbose If the switch is used, the user most provide this information in the input file for each transfer session (explained in more detail in the section on free-form mode). Note: If the program cannot open the user's input file, it will terminate without printing an error message or producing an output file. VII. IMPORTANT CONSTANTS The user may want to change the value of the following constants. These along with others are defined at the "top" of Batchftp. Asterisks indicate the most important constant(s). Constant Value Description ** _TENEX "type tenex" 36 to 8 transfer mode do "man ftp" to find out the command on your ftp server _ALARM_TIME 900 (sec) time before output analysis--transfer will terminate if it is a silent connection _MAX_ATTEMPT 25 maximum number of failures before an extended delay _EXTENDED_DELAY 3600 (sec) length of extended sleep _MOD_NUMBER 31 (sec) used for calculating random delay before connection _DEF_SLEEP 15 (sec) minimum time to sleep before connection attempt added to random number _LBUF 501 length of buffer for input file (using fgets) _DIR_PATH 200 directory path length can be MAXPATHLEN usually 2048 VIII.REGULAR MODE (Shorthand notation) The quick reference section above gives a brief explanation of the shorthand notation. This section will attempt to explain the option more fully. Also, it covers the slight differences in the commands for regular and free-form mode. The commands once again are as follows: -a ASCII transfer -b binary transfer -t 36 to 8 bit binary (tenex mode) -l directory listing default transfer (regular mode only) Note: Batchftp does not support the "-8" command (equivalent to "-t"). -c change local directory -d change remote directory The arguments for these "shorthand" commands are as follows: (Arguments in "[]" are optional) -a,-b,-t file1 [file2] (default) file1 [file2] -l directory output_file -r initial_remote_dir output_file The first file in ASCII, binary, tenex, or default transfer modes is the remote file the you are requesting. Be certain that the file name is in the proper case; the ftp server from which you are requesting files may be case sensitive. The second file name (optional) is the name of the file on the local machine after it has been transferred. The "-l" option allows you to obtain a listing of a remote directory. The name of output file is required. The directory specification on this line does not affect the current remote directory. The "-r" allows option allows you to obtain a recursive directory listing. The first argument you provide is the initial remote directory; you will obtain a listing of all of the files in that subtree. As with the "-l" option, the output file is required. Unlike the "-l" option, the directory specification does change the current remote directory. Unfortunately, some non-UNIX ftp servers may not support recursive directory listings. Default transfer mode is a special option only available in regular mode. If no transfer mode is given on the input line, Batchftp will use the most recently specified mode (a, b, or t -- -l is NOT supported). For example, if you had the following input file: -a FILE.TXT file.txt -b exec.arc run.arc In this case the default transfer mode is binary "-b". Therefore, "run.arc" would be transferred in binary mode. The required arguments for the directory commands are as: -c local_directory -d remote_directory Directory commands may appear anywhere in an input file. However, in regular mode, only the most recent local and remote directory commands are effective. Note 1: The one-letter "shorthand" commands must be preceded by a hyphen. But this hyphen may appear anywhere on an input line as long as it is before the command (white space before or after "-"). Also, the commands may be in upper or lower case, and they are assumed to be only one character. Therefore, the first non white space character after the command is interpreted as being the beginning of the first argument. Furthermore, any amount of white space may appear between arguments (at least one). In addition, you may insert blank lines in an input file. Example: (equivalent input lines) -a file1.txt file.txt - A file1.txt file.txt - afile1.txt file.txt - Afile1.txt file.txt Note 2: In regular mode each file request (a,b,l,t) is a separate transfer session. For example, if you request three files, Batchftp will connect to the remote site a total of three times and transfer one file per session. This option is useful when all of your files are large. IX. FREE-FORM MODE In free-form mode a user is able to enter any ftp or shorthand command. A free-form transfer session begins with a '{' (left brace) and ends with a '}' (right brace). If a free-form transfer session is not terminated by a '{', Batchftp assumes that the remainder of the input file is a single session. No other characters will be recognized on a line after a '{' or a '}' is detected; a brace may appear anywhere in an input line. The program will append a "quit" command to each session as in regular mode. A user's input file may consist of any number of free-form and regular mode transfer sessions. If an input line does not contain a shorthand command, the line is copied directly into the ftp script file. As such, default transfer specifications are not supported in free-form mode. The "a", "b", "l", and "t" commands are interpreted exactly as they are in regular mode. However, the directory commands -- "c" and "d'-- are treated somewhat differently. Each local or remote directory command is written to the script file as it is encountered. In contrast, only the last directory command is effective in regular mode. The default local and remote transfer directories are only included in the ftp script file once per session. Unfortunately, this information is not included in the script file automatically. It is difficult to determine where to insert the information in the script file. For instance, if the directory commands are included at the brginning of a session, these extra commands may interfere with a connection. As such, the user may insert a hyphen "-" where the directory information should be included. Of course, if any of the shorthand commands are used, the directory data will be included by default. Example: (Comments are included for clarity.) -a zmodem.dat /* regular mode */ { /* start of free-form */ -c local_dir -d /pc -a info.txt type binary put basic.exe } /* end of free-form */ /* Can be followed by other regular or free- form sessions */ Free-form mode must be used in all cases in which the "-i" option is specified on the command line. This option disables ftp script initialization (described in an earlier section). As such, every script file Batchftp generates will not be initialized. Initialization entails prepending a script file with the name of the remote computer, a user's login name, and password. More importantly, the "verbose" command is also specified. Batchftp analyzes a transfer session by interpreting its "verbose" output; the program will not report accurate results if verbose mode is off. Therefore, the user must provide the initialization information for each transfer session in an input file. In effect, simple regular mode sessions are not valid if the user specifies the "-i" option. Example: { open remote_host user login_name password verbose . /* any number of commands */ . . } { open remote_host user login_name password verbose . . . } In this case since the user provides the initialization information, the only required and recognized command line argument is the name of the input file. Of course, please remember that free-form mode is valid without the "-i" option. Without this option a user may specify any number of regular and free-form sessions. Regular mode sessions are legal with the "-i" option; however, the program will never connect to a remote computer. X. RANDOM DELAY Batchftp will "sleep" for a random number of seconds before it attempts to connect to a remote computer. The default delay is from 15 to 45 seconds per session attempt; the total delay before an attempt will accumulate with successive failures. The "seed" for the random number generator is the current system time. After _MAX_ATTEMPT successive failures (default 25) Batchftp will sleep for an EXTENDED_DELAY (default 3600 seconds) before attempting another connection. The random delay and number of attempts are both reset to zero after an extended delay or a successful connection. The random delay is an attempt to ensure that two or more ftp processes will not become synchronized and prevent each from connecting to a single remote computer. In addition, the accumulated and extended delays improve the program's overall performance during "heavy traffic." XI. SESSION ALARM If a transfer session is not proceeding "well" after a specified time, the alarm will sound, and then Batchftp will terminate the session. A valid session is one that is producing output -- output file is larger than zero. In some cases a connection is successful, but no files are transferred, and no output is produced (silent connection). If the alarm sounds and the output file is larger than zero, Batchftp allows the session to terminate normally. The default alarm time is 900 seconds; however, the program has transferred multiple files successfully with alarm times as short as 15 seconds. XII. FILES During execution Batchftp generates five files, but only one of the five remains after execution is complete. The process id (pid) is appended to each file name. In this way multiple copies of the program may be run concurrently. The files are: f_stdout -- output file for a single session ftp_script--ftp script for session input -- copy of user's input(name varies) sh_script one line shell script (ftp) msgfile permanent output file for entire program. The output of each session is copied to this file. Note: The user's input file need not be in the same directory as the executable for Batchftp; the user may include a pathname on the command line. The copy of the input file will be in the same directory as the original. All other files will be in the same directory as the executable. XIII.COMPATIBILITY Batchftp will currently run on Berkeley UNIX systems only. It has been tested on machines running two different operating systems -- SunOS 4.1 (SPARC-1 and 3/160) and Pyramid OSx 4.4 (4.3 BSD/System V on a Pyramid 9815). If I had had access to other UNIX systems, I would have used them also. This program uses network database and process control functions which are not available in System V. These network functions are "gethostbyname", "inetaddr", and "inetntoa". In this program they are used to obtain information about the local and remote hosts. Unfortunately, "gethostbyname" does not always return the complete domain name of the local host. The amount of information returned may depend on the constants defined for the local system such as HOSTALIASES and LOCALADMIN. The user's login name and host (user@host) is the default password for an ftp login. If these functions were eliminated, the program would be more compatible. An alternative would have been to use the nameserver routines to obtain the same information. However, this choice may have caused additional compatibility problems. At least one system -- Sun-- requires a user to link a program with the external library "nameserv." A third option would have been to read the output of the shell level command "domain." The most serious compatibility problem is with the process control function "killpg." Killpg sends a signal (in this case SIGKILL) to a group of processes. Batchftp exec's a one-line Bourne shell script which executes the ftp command; it generates an additional process (ftp -In). Since both of these child processes must be signalled (in case of a silent connection), a simple "kill" command would not be adequate. The kill command in System V may be the solution to the problem. The statement "kill(0,SIGKILL) will terminate all processes with the same group id except for the caller. Also the setpgrp command is not necessary in this case. Note: Batchftp uses the UNIX shell level commands "cat", "cp", and "system" to manage files. XIV .TESTING As stated above, Batchftp has been tested on two different systems. But I will never say with confidence that a program has been "fully tested." Every conceivable situation can not realistically be tested. I am sure that the program will fail under certain conditions on some systems. To combat this problem, I have prepared this document and commented the source code to help you diagnose the problem. I would understand if you would not want to spend the time to correct the error. Furthermore, it is always difficult to understand how and why someone else has coded a program in a certain way. Technically, it is my responsibility to "correct all errors" (a miracle) before I release a program. In any case please report all errors that you discover, and I will either incorporate your changes or do my best to fix the problem. My address and phone number are listed below. Because "silent connections" do not occur very often, it is difficult to test if a program handles them properly -- kills all child processes and attempts to reconnect. One may say that they can be fabricated by using the "non_verbose" mode omitting the "quit" statement at the the end of the ftp script file. However, some ftp servers normally exit after all files have been transferred. I have tested this feature. However, how well the program recovers from silent connections will always be a point of uneasiness and uncertainty for me. XV. SUPPORT Please send comments, reports of problems, and suggestions for improvements to any one of the addresses below. Any information you could possibly provide about a problem and the conditions under which it occurred would be extremely helpful. If possible, when reporting a problem, please send me one or more of the following pieces of information: the machine, the operating system, the section of source code, and a short description of the problem. Internet: U.S. Mail: cooper@rex.cs.tulane.edu Shawn Cooper cooper@comus.cs.tulane.edu 3708 Melissa Drive Harvey, LA 70058 UUCP: cooper@rex.uucp Phone: cooper@comus.uucp (504)-341-0348 My computer accounts may be permanently chopped at any time. If you are on another UNIX system, you may want to "finger" me on rex before you send email.