Chapter 5: Data Transfer, Import, Handling, and Formatting

Section 5.1: Transfer of Data in between Computers

[ Previous chapter ][ This chapter ][ Next chapter ] In order to work with sequence data that you have transfered from another computer, you need to know what a sequence format is. The procedure to type sequence data manually into files is described in section "Sequence Editing" .

Note that word processors on personal computers quite frequently store data in non-ASCII format. Make sure that the file you want to transfer is really plain text. If needed, use the <save as> option and add <printed text> or <printed text with line breaks>.

Subsection 5.1.1

'ftp'

The file transfer program (ftp) requires that the computers which shall exchange data run the same protocol, the TCP/IP (Internet Protocol) suite. Your computer has to be configured accordingly with the appropriate software .

Personal Computer Setups

To find the 'ftp' program on your desktop computer, search for ftp, NCSA, or fetch on your PC's or Macintosh's hard disk. Ask your system manager if you have questions about the configuration of your network . The big advantage of the personal computer setups is the user-friendly interface. All 'ftp' applications follow the same scheme:

Open communication as usual : supply host name, user name, and password (for anonymous ftp see below you will ALWAYS need these three.)

Select the working directory at the remote host

Select the file(s) to be transferred

Select the receiving directory at the local host

Take care to transfer files in the correct mode. Sequence data files are usually text-only files and, therefore, need to be transferred in text mode (also called ASCII mode).

NOTE: Personal computers with modern operating systems (Macintosh, Windows 95, OS/2) allow file names which cannot be used on UNIX or VMS systems (which are usually installed at the host of interest).

Bear in mind the following rules for file names in order to facilitate working:

Use common case in all names (e.g., all-lowercase).

Do not use special characters and blanks; useful separators of letters are hyphen (-) and underscore (_).

Do not use numbers as first character, rather start with an alphabetic character.

Use file names of sensible length. More than 8 characters are difficult to type if you work with command line interfaces.

Use extensions as suggested by the program package you work with. E.g., seq and pep for sequences, and dat or txt for data and text files.

From and to other Hosts via 'ftp'

The details of this procedure depend on the implementations of the 'ftp' program in UNIX, VMS, or other operating systems. In general, the following points are important:

The style of the remote file names reflects the operating system of the remote host.

UNIX-like navigation is generally possible, but the navigation of the local host is usually better supported.

File names may not violate the conventions of either local or remote operating system characters, case, length and other details. Specifically, the "version numbers" of the VMS operating system and special characters (symbols, blanks, etc.) of personal computer systems must be considered as potential traps.

The following table gives an overview on the most important 'ftp' commands:

 
 
  
what                           | 'ftp' command 
  
-------------------------------+------------------- 
  
see where you are              | pwd 
  
-------------------------------+------------------- 
  
look for files                 | ls  
  
-------------------------------+------------------- 
  
change to subdirectory "test"  | cd test 
  
-------------------------------+------------------- 
  
go one level up                | cd .. 
  
-------------------------------+------------------- 
  
set file type binary           | bin 
  
-------------------------------+------------------- 
  
set file type text             | ASCII 
  
-------------------------------+------------------- 
  
get file "t1.seq"              | get t1.seq 
  
-------------------------------+------------------- 
  
get all files "t1.*"           | mget t1.* 
  
-------------------------------+------------------- 
  
transfer "t1.seq" to remote    | put t1.seq 
  
-------------------------------+------------------- 
  
toggle question in 'mget' mode | prompt 
  
-------------------------------+------------------- 
  
print progress during transfer | hash

Note that the use of FTP requires that you provide user name and password in order to access the remote directory.

SECURITY ADVICE: You should never leave your terminal or PC unattended if you are logged in to the computer. To ensure data security, you should avoid using other people's accounts. FTP access to a remote computer implies full read/write access to the remote data and is as sensitive as login via telnet or similar.

The option to use anonymous ftp is important for data retrieval and access to programs such as the JAMF code which is used to customize this Biocompanion. It requires that the server of choice is specifically configured for this option:

 
                host name              username           password   
  
                -------------------------------------------------------------
  
normal use      any multi-user         your user name     your passowrd
  

  
anonymous ftp   ftp server             anonymous          your eMail address

NOTE: Not all sites will allow that you configure 'anonymous FTP' yourself.

SECURITY ADVICE: If you configure your PC or Macintosh for anonymous FTP usage, make sure that the directory which is accessible for the user 'anonymous' is NOT THE MAIN HARD DISK directory (on DOS, typically C:, on Macintosh, typically HD-500 or whatever the name of the hard disk is). If you fail to ensure that 'anonymous' gets an own, authorized directory , you risk data security and license ramifications.

SECURITY ADVICE: Many PC or Macintosh setups will automatically configure anonymous FTP for you. Turn it OFF unless you need it for scientific data exchange.

SECURITY ADVICE: Make sure that anonymous FTP directories are never writable unless you apply very tight security on permission based on the host name. You are responsible for anonymous FTP file server contents and must not allow the deposition of illegal or volumninous material on your desktop computer.

Subsection 5.1.2

VMS Import from other VMS Hosts via DECnet

The file name specification in VMS is

node::device:[directory.subdirectory]file.extension;versionnumber

If you do not have a so-called "proxy account", you need to type your user name (e.g., doelz) and password (e.g., gar34rwq) in the following way to copy the file in your current directory:

$ copy yogi"doelz gar34rwq"::d$bio:[doelz.sequence]t.seq []

 
 %S, YOGI"doelz password"::D$BIO:[DOELZ.SEQUENCE] T.SEQ1 copied to D$D:[DOELZ]T.SEQ1 (7 blocks)

SECURITY ADVICE: You should never use this mechanism in automaic procedures to avoid that your password becomes visible in stored files unless you use the PROXY mechanism, which is a potentioal security breach as well. Make sure that noone peeks at your screen while using DECnet copy - the ftp program (see above) is your only safe option.

Subsection 5.1.3

Remote UNIX Copy

The option to copy files via the command 'rcp' is not described here for the sake of brevity.

SECURITY ADVICE: You should never use the 'trusted hosts' mechanism to avoid that you need to supply your password. Unless you run a 'yellow page' or 'network information system' (NIS) the ftp program (see above) is your only option.

Subsection 5.1.4

'kermit'

If you are connected via a serial line (i.e., rather old networking or modem lines), you may need to use the program 'kermit'. On the remote computer you must give the following command to receive a data file:

% kermit -r test.seq

The procedures are reversed if you want to transfer a data file from the remote to the local computer. On the remote computer you need to type the following command to send a data file:

% kermit -s test.seq

The local options are reversed. On MS-DOS you must start with the local escape character which is shown at the bottom of the screen.

<CTRL><A> c

Then, you can give the command to send (or receive) a file, e.g.,

KERMIT-MS> send test.seq

To get back, type

KERMIT-MS> connect

NOTE: The local escape character might vary (e.g., <CTRL><[> c), but is usually shown at the bottom line of the screen.

On Macintosh and other Graphical User Interfaces there are usually options in the <File> or similar menu.

Subsection 5.1.5

ZMODEM

This method is known from bulletin boards and other servers, but rarely used in molecular biology environments. It covers file transfer with built-in compression and its use is similar to the 'kermit' program.

[ previous chapter ],[ this chapter ][ next chapter ] , [next page/section] , or [overview] , or [table of contents]