fmtseq - A File Conversion Program

This program is a reimplementation and extension of Don Gilbert's readseq program for performing file format conversions. Its main purpose is to convert biological sequence files from one format to another, although it's interactive mode is robust enough that it can serve as a simple sequence viewer.

The program runs in two modes, interactive and non-interactive. The program runs non-interactively if an input file or the `-pipe' option is specified on the command line. In that mode, the program simply performs the conversions of the given input files (or standard input if `-pipe' is used) and then exits. Using the `-verbose' option in non-interactive mode will cause the program to output progress messages, but otherwise the program runs silently. The interactive mode of the program is described below.

Differences from readseq

• `-verbose' is no longer the default option in non-interactive mode. It is the default in interactive mode.
• `-all' is now the default sequence selection method.
• The file formats supported are different, and the format numbers used to refer to those file formats have changed.
• The purpose of `-degap' has changed slightly. See the Sequence Transformations section below for how `-degap' and `-raw' affect the sequence characters output.

`-ask', `-bigalign', `-gapchar', `-gapin', `-gapout', `-idprefix', `-long', `-mode', `-no...' (see below), `-raw', `-split', `-informat', `-interleave', `-skipempty'

Interactive Mode

• The commands you enter should look just as if you were writing a command line (except for any of the shell operations, like quoting, variable substitutions, and so on). And yes, you can enter more than one option at a time.

• Remember, it's `-r' (`-run') and `-q' (`-quit') to perform a conversion or quit the program. If you just type `r' or `q', that just sets the input file.

• Any option that is not displayed is an unset option, except that the pretty-print options are only displayed when the output format is set to `Pretty'.

• When an output file other than standard output is specified and a file conversion has been performed, the output file stays open, and it is only closed when the output file or output format is changed or when the program exits. The reason for this is so that you can convert several input files into one large output file interactively.

  The disadvantage with this occurs when the output format is the PHYLIP, Clustalw, MSF or Pretty format. These formats delay the actual writing of the output until the file is closed (they need to know how many sequences will appear in the file before they can start producing the output). So, you need to close the output before you can look at it. The easiest way to do that is to enter the command `-noo' or `-nof' (short for `-nooutput' and `-noformat').

Program Options

Program Options (text in [...] is optional):
  -al[l]            select all sequences
  -as[k]            ask whether to select each sequence
  -b[igalign]       convert FASTA program output to big alignment
  -c[aselower]      convert to lowercase
  -C[ASEUPPER]      convert to uppercase
  -d[egap]          remove gaps from sequences
  -gapch[ar=-]      set the gap symbol for both input and output
  -gapi[n=-]        set the gap symbol for the input
  -gapo[ut=-]       set the gap symbol for the output
  -id[prefix]=gb    set identifier prefix for input
  -i[tem=]2,3,4     select sequences by position in input
  -li[st]           only list sequence information
  -lo[ng]           long form conversion (input header included as comment)
  -mo[de]=pretty1   run program in specified mode (listed in BIOSEQ entry)
  -p[ipe]           read from standard input
  -no...            unset any program option (eg `-noitem')
  -o[utput=]out.seq specify an output file
  -ra[w]            leave gaps in sequences
  -re[verse]        reverse-complement each sequence
  -sp[lit]=ext      split output to separate files
  -v[erbose]        output progress messages
  -f[ormat=]name    set output format by name
  -f[ormat=]#       set output by number
  -inf[ormat]=name  set input format by name
  -inf[ormat]=#     set input format by number
       1. Raw                      12. NBRF                  
       2. Plain                    13. NBRF-old              
       3. EMBL                     14. IG/Stanford  (ig)     
       4. Swiss-Prot  (sprot)      15. IG-old                
       5. GenBank  (gb)            16. GCG                   
       6. PIR  (codata)            17. MSF (gcg-msf)         
       7. ASN.1  (asn)             18. PHYLIP                
       8. FASTA  (Pearson)         19. PHYLIP-Int  (phylipi) 
       9. FASTA-old                20. PHYLIP-Seq  (phylips) 
      10. FASTA-output (fout)      21. Clustalw  (clustal)   
      11. BLAST-output (bout)      22. Pretty                

Pretty-print Options:
  -interle[ave]     output interleaved sequences
  -w[idth=#]        sequence line width
  -t[ab=#]          indent sequence
  -co[lspace=#]     add space columms in sequence lines
  -gapco[unt]       count gap chars in sequence numbers
  -namel[eft=#]     print name to left of sequences
  -namer[ight=#]    print name to right of sequences
  -namet[op]        print names at top of output
  -numl[eft]        print position numbers to left of sequences
  -numr[ight]       print position numbers to right of sequences
  -numt[op]         print position numbers above sequences
  -numb[ottom]      print position numbers below sequences
  -ma[tch=.]        replace matches to first sequence
  -interli[ne=#]    add blank lines between sequence blocks
  -sk[ipempty]      don't output lines with only gap characters

Option Format and Use

There are a couple exceptions to this, in order to maintain compatibility with readseq. First, `-caselower' and `-CASEUPPER' must be given using the appropriate case, and their unique prefix is determined by the case of the letters instead of the string itself (so `-c', `-C', `-CASE' and `-cas' are all valid options).

Second, there are shortcut options `-ivalue', `-fvalue' and `-ovalue' for the `-item', `-format' and `-output' options. In these shortcuts, the value is not separated by a '=', but specified immediately after the 'i', 'f' or 'o'. Note that this shortcut is only valid for `-i', `-f' and `-o', and not any longer prefix.

Any option can be unset by prepending the option name with the string "no", as in `-nooutput', `-nocase' or `-nointerleave'. For the options that correspond to program flags, unsetting the option will disable that flag. For options with values, unsetting the option will cause its value to revert to a default value (for `-nooutput', the output reverts to standard output, for `-nogapchar' or `-nogapin', the program will assume that no gap symbols occur in the sequence). Any option can be set or unset as needed, either interactively or on the command line (except for `-pipe' which cannot be set interactively).

In addition to setting and unsetting options from either the command line or the interactive mode command line, the program allows the use of "run modes" for setting many different options at once. In order to use this run mode option, a BIOSEQ file must be created and the environment variable "BIOSEQ" must include the name of that file. (See file "user.doc" for complete information on creating BIOSEQ files.) If a BIOSEQ entry is created in that file with the name "fmtseq", the information lines in that BIOSEQ entry can specify the various run modes of the program. For example, if the following BIOSEQ entry occurred in the BIOSEQ file:

>fmtseq
>blast: -informat=blast-out -fpretty -nogapcount -nameleft -numleft -nametop
>fasta:  -fpretty  -nametop  -nameleft=11  -width=60  -nocolspace
>        -gapout=  -interline=3 
>dbconvert:  -fig/stanford  -split=ig  -long  -C  -verbose

   # Remember, every BIOSEQ entry must have at least one non-> line.

Program Input and File Formats

The program can handle a number of different file formats, including the ones listed above in the program options plus special GCG-* formats for no loss conversions of GenBank, PIR, EMBL, Swiss-Prot, FASTA, NBRF and IG/Stanford entries to and from the GCG format. See below for details about this special set of formats.

All of the formats supported by the program are interconvertable, with the exceptions that the FASTA-output and BLAST-output are input-only formats and the Pretty format is an output-only format. By default, the format of the input is automatically determined from the input text. The format determination should work correctly for all of the supported formats except the Raw format, and the program uses the Plain format if no match is found to a support format. The option `-informat' can be used to specify the input's format, if either the automatic determination fails (or you just want to make sure it's correct).

Most of the file formats are the common formats used to store biological sequences. The more unusual formats are Raw, Plain, FASTA-old, FASTA-output, BLAST-output, NBRF-old, IG-old, GCG-* and Pretty. The Raw and Plain formats both specify that the complete contents of the file contain a single sequence. The only difference between the two is that the Plain format omits any whitespace and digits from the sequence, whereas the Raw format takes every character in the file as part of the sequence.

The FASTA-old, NBRF-old and IG-old formats are actually just variations of the FASTA, NBRF and IG/Stanford formats, where the form of the output in these variations differs slightly. The basic formats all have a place where one or more comment lines can appear in the sequence entry. The *-old formats all limit the entry output so that they contain a single line beginning the entry, a one-line sequence description (in NBRF-old and IG-old only), and then the sequence. No other lines will appear in the entry. These limited formats are included to produce output compatible with other commonly used programs. The FASTA-old format output, for example, can be used as the input to the "pressdb" and "setdb" programs in the BLAST release.

The GCG-* formats are actually a set of format names (GCG-GenBank, GCG-PIR, GCG-EMBL, ...) used to distinguish the GCG forms of GenBank, PIR, EMBL, Swiss-Prot, FASTA, NBRF and IG/Stanford entries from the generic GCG format (which treats all of the header lines as unstructured comments). Any of the alternate names for the formats can replace the `*', so gcg-gb, gcg-sprot and gcg-igold are all valid formats. The program distinguishes these formats, because it is able to convert between the non-GCG and GCG forms of these formats without losing any of the header information (most of the program's conversions lose the references, features and other things in the header).
(Note: When you are specifying a conversion, you can use just the simple "GCG" format name, and the program will automatically detect and convert to or from one of the GCG-* formats whenever it can.)

The FASTA-output and BLAST-output formats are read-only formats which take as input the output from one of the FASTA programs (FASTA, TFASTA, SSEARCH, LFASTA, LALIGN or ALIGN) or BLAST programs (BLASTN, BLASTP or BLASTX). The program should be able to read the output from any of the BLAST programs (and possibly even the TBLAST* programs, but that hasn't been tested yet). It does, however, have some trouble automatically determining the BLAST-output format, since many e-mail servers include some descriptive text at the beginning of the output file and that confuses the determination program.

For the FASTA-output format, the program can handle alignment output produced with a MARKX value (or '-m' option value) of 0, 1, 2, 3 or 10. The automatic format determination also has some trouble with some of the variations of the FASTA output, so for best results, the FASTA program output should be created in "non-interactive" mode, where the program header like this one

 SSEARCH searches a sequence database
 using the Smith-Waterman algorithm
 version 2.0u4, Feb. 1996
Please cite:
 T. F. Smith and M. S. Waterman, (1981) J. Mol. Biol. 147:195-197; 
 W.R. Pearson (1991) Genomics 11:635-650
.
.
.

The Pretty format is just the same as in the readseq program, except that I changed the look of the `-numtop' and `-numbottom' numbering a little bit, added the `-interleave' option to explicit specify the interleaved format and added the `-skipempty' option to shorten the output of alignments containing large gaps in a number of sequences (like a big alignment of the BLAST output). As mentioned in the readseq documentation, the best way to figure out the pretty-print options is to try them out in various combinations and see which you like. A favorite set of options of mine is "-nametop -nameleft=11 -width=60 -nocolspace -interline=3 -gapout= ".

Finally, the main database identifier in every sequence's entry (if they exist) is given an "identifier prefix" specifying which database the identifier refers to, such as "gb" for GenBank or "sp" for Swiss-Prot. The option `-idprefix' can be used to specify that identifier prefix, if the program is unable to correctly determine it. See file "user.doc" for more information about identifier prefixes.

Program Output Variations

Sequence Selection and Transformation

The final method for selecting sequences is by using the `-item' option to specify sequences by their position in the input. The value to `-item' is a list of positions, and the sequences in those positions in the input are selected for conversion. The positions in the list can be given in any order, however the input sequences will always be output in the order they appear in the input. (There currently is no way to reorder the sequences using fmtseq, but this will be corrected in the next version.)

Once the input has been selected, one or more transformations may be applied to those sequences. The simple transformations occur when the options `-caselower', `-CASEUPPER' or `-reverse' are set, or when the value of `-gapin' differs from `-gapout'. The `-caselower' and `-CASEUPPER' options convert the sequence to all lowercase or all uppercase. The `-reverse' option reverses the sequence and, if the sequence is DNA or RNA, complements the sequence characters.

The `-gapin' and `-gapout' options define what the gap symbol should be in the input and output. If `-gapin' is set to a character, and either `-gapout' is not set or set to a different character, then all of the input gap symbols will be removed (if `-gapout' is not set) or changed to the `-gapout' character. The `-gapchar' option simply sets `-gapin' and `-gapout' to the same value.

The complex tranformations involves the `-degap', `-raw' and `-bigalign' options and the format of the input and output. The general rule for the three options is that `-degap' is used to remove gaps from the sequences, `-raw' is used to retain the gaps (or other notational symbols), and `-bigalign' is used only with the FASTA-output format. However, the application of these three options changes depending on the input and output file formats.

If the input format is FASTA-output or BLAST-out, then the default action of the program (and remember that `-bigalign' is set by default) is to find the pairwise alignments specified by the input file and to construct a big alignment. where all of the pairwise alignments are combined using the query sequence as the reference point. The output consists of the sequences in the big alignment. By default, only the characters forming the actual alignment are used in the big alignment (the context strings output by the FASTA programs are ignored). If the `-raw' option is specified, then the context strings around the pairwise alignments are added to the big alignment. If the `-degap' option is specified, then after the big alignment is constructed, all gaps are removed (although why you would want to do this, I don't know).

Finally, the operations listed above only occur when `-bigalign' is set. If `-bigalign' is not set, then the input is treated just as if it were a FASTA input file (see below), and all of the sequences listed in the FASTA program output will be used as the input sequences. (NOTE: In this case, the input sequences will consist of many occurrences of the query sequence, one for each pairwise alignment.)

If the input format is not FASTA-output or BLAST-output, or `-bigalign' is not set, then the program tries to be intelligent about the default conversions between formats. It divides all of the file formats into two types:

1. Formats that typically only store database sequences, and do not typically store aligned sequences (i.e., sequences with gaps). These formats are GenBank, PIR, EMBL, Swiss-Prot, their GCG-* formats, and ASN.1.
2. Formats that can store aligned sequences. These formats are Raw, Plain, FASTA, FASTA-old, NBRF, NBRF-old, IG/Stanford, IG-old, their GCG-* formats, PHYLIP, Clustalw, GCG and MSF.

• If the input is a "database" format, then by default, the program only reads the alphabetic characters from the sequence entries. Any gap symbols or notational characters are ignored. If the `-raw' option is specified, then the gap symbols and notational characters will be included as part of the sequence (i.e., all characters except whitespace and digits). If the '-degap' option is also specified, then gap symbols will be removed from the "raw" sequences (although this will leave any notational symbols).

• If the output is a "database" format, then all non-alphabetic characters are removed from the sequence, unless the `-raw' option is set. Also, if the `-degap' option is set (or `-gapout' is unset), then any gap symbols are removed from the sequence.