The NoSQL system comprises a set of programs called Operators.
Each operator is a separate program module that performs a unique function on the data. Operators can be grouped into data movers, report generators, and utilities.
The data movers are operators that extract or rearrange the data in some way. They each read an rdbtable via STDIN and write a rdbtable via STDOUT, and so are frequently connected using the UNIX pipe function to form a larger task. Each operator in such a "pipeline" style of operation gets its input from the output of the previous operator in the "pipeline". The data movers include:
Picks columns by name, outputs columns in listed order.
Computes an arbitrary expression using column names.
Generates and replaces (or removes) the header of an rdbtable.
Natural or "Master/Detail" join of two rdbtables.
Merges two like rdbtables.
Selects rows based on arbitrary expressions.
Selects rows based on a multi-column key of a sorted or indexed rdbtable.
Sorts a datafile by one or more columns.
Lists subtotals of specified columns.
Safely [over]writes an rdbtable.
Makes an rdbtable unique on specified columns.
The report generators each read an rdbtable via STDIN and produce a report on STDOUT, so when they are in a "pipeline" of operators they will be the operator at the end. The report generators are:
Quick and easy printing of output formatted from information in the header.
Best form of output, with definable format.
Summary/Statistical information about data values in an rdbtable.
Verifies the structure of an rdbtable.
The utilities are used for manipulating the structure and content of rdbtables and are generally used as separate tasks,i.e. they do not read STDIN. The utilities are:
NoSQL interactive terminal interface.
Uses an editor to allow modifications to an rdbtable.
Generate rdbtable index files to be used by 'nsq-search'.
Trivial table locking program, for serializing access to rdbtables.
Attempts to repair candidate NoSQL datafiles.
Updates/Creates a unique record ID on each row of an rdbtable. The field must be the first column in the table.
All operators take a '-h[elp]' option to show details of operation online. Following is a section for each operator, in alphabetic order.
Usage: nsq [options] [rdbtable]
Interactive analysis tool for NoSQL data tables (rdbtables). This is especially useful for rdbtables that were made from spreadsheet data.
Uses nine NoSQL modules: nsq-col, nsq-ed, nsq-pr, nsq-report, nsq-row, nsq-sort, nsq-t2l, nsq-summ, nsq-valid.
Specific module options are available using the '-help' option of individual modules. This utility uses the PAGER environment variable.
Usage: nsq-col [options] list
Selects ("projects") columns by name (and order) and outputs an rdbtable with these columns. Can effectively select, order, add, delete, or duplicate columns.
The value 'list' is normally a list of column names. If 'list' contains a triplicate of the form '-c NAME NEW' then column name 'NAME' will be changed to 'NEW'. If 'list' contains a triplicate of the form '-a NAME DEFN' then a new (null) column is added, at that point in the list of column names, with name 'NAME' and definition 'DEFN'.
This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.
As an example using the sample rdbtable from the DATA section (named sample), to select columns named 'NAME' and 'COUNT' the command would be:
nsq-col NAME COUNT < sample
To select all columns except column 'NAME' the command would be:
nsq-col -v NAME < sample
To add a new column named 'LENGTH' with a size of 10 the command would be:
nsq-col -v -a LENGTH 10 < sample
Note that to include documentation with the new column definition the command would be:
nsq-col -v -a LENGTH '10 length in meters' < sample
The '10 length in meters' must be quoted so that it will be treated as a single token.
Usage: nsq-compute [options] [statements]
Computes values for data fields based on arbitrary statements using column names. Any characters that are special to the UNIX shell must be quoted.
Comparison operators may be of the form: gt, ge, lt, le, eq, ne. For example 'NAME eq Hobbs'. Logical constructors 'or' and 'and' may be used; as well as 'null' to indicate an empty data value. The supplied statements may be essentially any valid PERL statements.
All of the Comparison operators and Logical constructors are reserved and should not be used as column names (they are all lower case and four characters or less).
This operator reads a rdbtable via STDIN and writes a rdbtable via STDOUT. Options may be abbreviated.
If a file is used to contain the statements any line in the file that starts with a sharp sign (#) is treated as a comment and ignored. Also if there is a sharp sign preceded by a space character anywhere on the line the rest of the line is also treated as a comment.
Since column names and reserved words are parsed by the program, do not put the entire expression in a single pair of quotes as that will prevent the parsing. Also note that column names and reserved words need to be surrounded by blank spaces if they are not individually quoted. For example either form below is fine:
nsq-row NAME eq "L Brown" < sample
nsq-row "NAME" "eq" "L Brown" < sample
but do not use this form:
nsq-row "NAME eq L Brown" < sample
Example rdbtable (named cfile):
name count type amt
6 5N 4 5N
Bush 3 A 133
Hansen 39 A 23
Newton 8 E 8
Hobbs 42 B 144
Hart 2 C 55
Jones 4 B 244
Smith 5 D 1111
The command:
nsq-compute count += 100 if type lt D < cfile | nsq-pr
gives the output:
name count type amt
------ ----- ---- -----
Bush 103 A 133
Hansen 139 A 23
Newton 8 E 8
Hobbs 142 B 144
Hart 102 C 55
Jones 104 B 244
Smith 5 D 1111
Example file of commands named 'XXX':
if( type eq A ){
name = NEW ;
amt = count * 2 ;
type = 'AAA' ;
}
else{
name = OLD ;
amt = count + 1000 ;
type = 'ZZZ' ;
}
Output from command:
nsq-compute -fXXX < cfile | nsq-pr
would be:
name count type amt
------ ----- ---- -----
NEW 3 AAA 6
NEW 39 AAA 78
OLD 8 ZZZ 1008
OLD 42 ZZZ 1042
OLD 2 ZZZ 1002
OLD 4 ZZZ 1004
OLD 5 ZZZ 1005
Usage: nsq-ed [options] rdbtable [col_spec] [line_spec] [pat_spec]
This utility calls an editor to allow the editing of selected lines and/or columns of (or the entire) rdbtable. Options may be abbreviated.
A "col_spec" is a list of column names.
A "line_spec" is a list of line numbers, of increasing value, optionally separated by a dash to specify a range, e.g. "10-20". The form "N-" means from line N to end of file. The header is always included, so do not specify lines 1 or 2 (except as the first part of a larger group, e.g. "1-10").
A "pat_spec" is a single pattern (of the form: /pat/ ) optionally followed by one or more column names, and may be preceded with the reserved word 'ne' to negate the meaning (e.g. the pattern should NOT match).
The order of "col_spec", "line_spec", and "pat_spec" is significant only to the extent that "col_spec" must precede "pat_spec" in the command line if both are given.
If none of "col_spec", "line_spec", or "pat_spec" are given then the entire rdbtable will be edited. If one or more of the three above options are given then the selected subset of the rdbtable will be edited. The option "col_spec" identifies which columns of the rdbtable are to be edited, and options "line_spec" and "pat_spec" determine which lines will be selected for editing, either by direct reference ("line_spec" given "col_spec" not given) or by pattern matching ("col_spec" given "line_spec" not given). If both "line_spec" and "pat_spec" are given then only lines within the bounds of "line_spec" will be considered for selection by pattern matching.
If "pat_spec" does not include column names then the pattern (any PERL regular expression) is matched against each entire row; a row is selected if there is a match anywhere in the row. If column names are included the pattern is matched against only the specified columns. In this case a row is selected if a match is found in any specified column. If the "ne" option precedes the "pat_spec" without column names then an entire row is selected if the pattern does not match anywhere in the row, and if column names are given then the row is selected if the pattern does not match in any specified column.
The form of the file to be edited is either "column" with visible column delimiters (the default) or "list" format where the column names are on the left and the data is on the right. The default editor is specified by the environment variable EDITOR if set, otherwise the editor 'vi' is used.
In either form of editing the delimiter is a "pipe" symbol (|). Care should be taken when editing not to use any "pipe" symbols in the data, or to delete any existing pipe symbols in the file. Also, in the case of "list" form, one or more blank lines must separate each record.
The rbdtable may be an existing file, or it may be automatically checked out from RCS. In the latter case it will be checked back into RCS after the editing is complete. The default action is that if the rdbtable does not exist an attempt will be made to find the rdbtable under RCS (the '-RCS' option may be used to force the use of an RCS file).
Afterward, except in the RCS case, the original contents of the rdbtable will be left in a file of the same name preceded with a comma, e.g. "sample" will be ",sample".
Uses NoSQL operators: nsq-col, nsq-pr, nsq-mktbl, nsq-t2l, nsq-l2t.
WARNING: If line_spec is given the number of columns must not be changed by editing, or if col_spec and/or "pat_spec" is given the number of lines must not be changed by editing, otherwise the results may be unpredictable.
An example command to edit the rdbtable (named sample) from the DATA section would be:
nsq-ed sample
which would edit the entire rdbtable. The file as it is ready to edit is shown in Table 3. The pipe character '|' must not be removed during the editing process, although it may be moved left or right if necessary and the spaces around the pipe character may be deleted if desired.
This form of editing is fine if the rdbtable is not large. If it is large then editing only those parts that need changes is faster and less error prone. To edit only columns 'NAME', 'COUNT', and 'AMT', the command would be:
nsq-ed sample NAME COUNT AMT
To edit only lines five thru seven the command would be:
nsq-ed sample 5-7
Table 3
RDBTABLE (SAMPLE) READY TO EDIT, COLUMN FORM
NAME | COUNT | TYP | AMT | OTHER | RIGHT
6 | 5N | 4 | 5N | 8 | 8>
Bush | 44 | A | 133 | Another | This
Hansen | 44 | A | 23 | One | Is
Jones | 77 | X | 77 | Here | On
Perry | 77 | B | 244 | And | The
Hart | 77 | D | 1111 | So | Right
Holmes | 65 | D | 1111 | On | Edge
To edit only lines five thru seven of only columns 'NAME', 'COUNT', and 'AMT' the command would be:
nsq-ed sample NAME COUNT AMT 5-7
and the file to edit would look like:
..>>> 1 2 CONTROL LINE, DO NOT TOUCH <<<
NAME | COUNT | AMT
6 | 5N | 5N
..>>> 5 3 CONTROL LINE, DO NOT TOUCH <<<
Jones | 77 | 77
Perry | 77 | 244
Hart | 77 | 1111
Note that whenever a line_spec is given, control lines (starting with '..>>>' are inserted into the file to edit. They must not be modified during the editing process. They are used to reconstruct the rdbtable after editing.
If the rdbtable has data fields that are long, i.e. longer than convenient to edit in the column form shown above, the 'list' form is the preferred method. The usage of line_spec and col_spec are unchanged but the form of the file to edit is different. For example consider an rdbtable (named sample3) which is shown in Table 4, where the TAB characters are represented by '<T>and the newline characters are represented by <N>This small rdbtable looks very incoherent in raw form, and a file of any real size with long data fields is even more so. The command to edit the file sample3 in 'list' form would be:
Table 4
RDBTABLE (SAMPLE3) ACTUAL CONTENT
name<T>datatype<T>agencysrc<T>dbms<T>contact<T>contents<T>notes<N>
46<T>15<T>60<T>15<T>21<T>530<T>600<N>
ACAS (Air Combat Assessment)<T>BDA<T>Bigplace AFB<T>File<T>Starr<T>Air
Combat Assessment BDA data. Duplicates data under ACAS (Air Combat
Assessment) BDA Sorties, diskettes nr 1,2,3.<T>On two 3.5 inch
diskettes.<N>
ACAS (Air Combat Assessment) BDA Sorties<T>BDA<T>Sawyer AFB<T>File<T>
Hobbs/Emerson<T>85 files, 2 per day containing 12 and 24 hour reports.
This data is different from that under ACAS (Air Combat Assessment) BDA
Data, from diskettes 1,2,3.<T>Received 5/6/91.<N>
ATO (Air Tasking Orders) Original<T>ATO<T>HQ USAF, Universal AFB<T>
File<T>Marshall<T>Original ATO messages. Both sets are incomplete.<T>
To be joined into single file and edited. Missing sections not yet
ordered. May be parsed completely, or only for key comments. Much data
to be processed.<N>
ABC Original<T>ABC<T>HQ USAF, Universal AFB<T>File<T>Marshall<T>Original
ATO messages. Both sets are incomplete.<T>To be joined into single file
and edited. Missing sections not yet ordered. May be parsed completely,
or only for key comments. Much data to be processed.<N>
The command to edit the file sample3 in 'list' form would be:
nsq-ed -list sample3
which would produce a file to edit as shown in Table 5. Note that each section holds information relating to one row in the rdbtable and that the first section holds information relating to the header of the rdbtable. Also note that each section is separated by a blank line (it could be any number of blank lines).
Each row in a section relates to a single data value. The pipe character '|' must not be removed during the editing process, although it may be moved left or right if necessary. Only one pipe character is to be in the information relating to one data value, although that information may be physically on more than one line in the section if the data value is long. The spaces on both sides of the pipe character as well as the spaces around the column names are only for readability; they may be moved or even deleted if desired.
Table 5
RDBTABLE (SAMPLE3) READY TO EDIT, LIST FORM
name | 46
datatype | 15
agencysrc | 60
dbms | 15
contact | 21
contents | 530
notes | 600
name | ACAS (Air Combat Assessment)
datatype | BDA
agencysrc | Bigplace AFB
dbms | File
contact | Starr
contents | Air Combat Assessment BDA data. Duplicates data under ACAS
(Air Combat Assessment) BDA Sorties, diskettes nr 1,2,3.
notes | On two 3.5 inch diskettes.
name | ACAS (Air Combat Assessment) BDA Sorties
datatype | BDA
agencysrc | Sawyer AFB
dbms | File
contact | Hobbs/Emerson
contents | 85 files, 2 per day containing 12 and 24 hour reports. This
data is different from that under ACAS (Air Combat
Assessment) BDA Data, from diskettes 1,2,3.
notes | Received 5/6/91.
name | ATO (Air Tasking Orders) Original
datatype | ATO
agencysrc | HQ USAF, Universal AFB
dbms | File
contact | Marshall
contents | Original ATO messages. Both sets are incomplete.
notes | To be joined into single file and edited. Missing sections
not yet ordered. May be parsed completely, or only for key
comments. Much data to be processed.
name | ABC Original
datatype | ABC
agencysrc | HQ USAF, Universal AFB
dbms | File
contact | Marshall
contents | Original ATO messages. Both sets are incomplete.
notes | To be joined into single file and edited. Missing sections
not yet ordered. May be parsed completely, or only for key
comments. Much data to be processed.
The advantage of this form of edit file is that even with very large data values most, if not all, of the information from each row of an rdbtable will be visible on the screen at once.
Usage: nsq-headchg [options] file.tpl
Replaces the header (first two rows) of an rdbtable with a header generated from information in the template file 'file.tpl'. Options are available to add, copy, or delete the header, or to generate a template file from an existing rdbtable.
Each line of the Template file contains info about a column, in order. The lines contain: (optional) index number (starting at 0 or 1), column name, definition, and (optional) comments or documentation, white space separated. If column name contains spaces it must be enclosed in double quotes. Names containing space characters are not recommended, however, as it is generally troublesome and error prone. A good substitute is the underscore character (_).
Lines that start with a sharp character '#' are skipped, as are blank lines. To start a column name with a sharp character '#' the name must be enclosed in double quotes. (but this is not recommended).
The number of columns in the header is normally reported on STDERR. This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated. This operator uses the NoSQL operator: nsq-valid.
As an example, to generate a template file named 'new.tpl' from the rdbtable (named sample) from the DATA section, the command would be:
nsq-headchg -templ < sample > new.tpl
The contents of file 'new.tpl' would then be:
0 NAME 6
1 COUNT 5N
2 TYP 4
3 AMT 5N
4 OTHER 8
5 RIGHT 8>
To change the header of rdbtable 'sample', the procedure is to edit the file 'new.tpl', and then run 'nsq-headchg' using the modified file. For example, to change the names so that only the first letters are upper case and to make column 'OTHER' numeric, edit file 'new.tpl' so it looks like the following:
0 Name 6 All names are first letter upper case.
1 Count 5N
2 Typ 4
3 Amt 5N
4 Other 8N Now numeric.
5 Right 8>
Note the index in the zeroth column and the documentation in the fourth column, both of which are optional, but recommended. The command to change the header of rdbtable 'sample' and make a new rdbtable called 'new.sample' would be:
nsq-headchg new.tpl < sample > new.sample
Usage:
nsq-index [options] rdbtbl column ...
or
nsq-index [options] -update [ index_file ... ]
The first form of usage of this utility generates an index file for the column(s) given, that refers to the specified rdbtbl. An index file is actually another (smaller) rdbtable containing only the column(s) given plus a column for index information. An index file can be used by the operator 'nsq-search' to quickly locate rows of data in the referenced rdbtable.
Index files are named by appending an 'x' and the column name(s) (seperated by a dots) to the base name of the rdbtable it refers to. For example an index file for the rdbtable "area.rdb" on column "strip" would be "area.x.strip". An index file that was also on column "depth" would be "area.x.strip.depth".
The second form of usage of this utility updates the index file(s) given. If no files are given all index files in the current directory are updated. An update of an index file is necessary when the rdbtable it refers to has been modified.
This utility writes or rewrites rdbtables with defined names in the current directory. Options may be abbreviated.
Usage: nsq-join [options] col.name[=col.name_2] rdbtable_2 < rdbtable_1
where rdbtable_2 is the "secondary" rdbtable, while rdbtable_1 is the the "primary" one.
Does a join of two rdbtables on the column(s) specified. The default is a "natural" join, with optional "Master/Detail" or cartesian (cross-product) type joins. Options may be abbreviated.
The Table from STDIN is the master. A natural join produces a new rdbtable that contains only rows from the input rdbtables that match on the specified columns (key columns). A master-detail join produces a new rdbtable that contains all rows from the master rdbtable and those rows from the secondary rdbtable that match. A cartesian join produces an rdbtable that contains all rows of both input rdbtables.
Each item in the list of column(s) may specify column names that are different in the two rdbtables, i.e. '=column_2', if given, refers to a name in rdbtable_2 that corresponds to 'column' in rdbtable_1. If '=column_2' is not given it means that the corresponding column name in both rdbtables is the same.
If different column names are specified, the name of the join columns in the output rdbtable will be from rdbtable_1.
Note that the two rdbtables must be sorted on the columns specified in order for a join operation to function correctly.
The order of columns in the output rdbtable will be: first the join columns, then the other columns from rdbtable_1, then the other columns from rdbtable_2.
This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT.
If we have the rdbtable (named samplej) here:
name nr typ amt
6 2 4 4
Bush 1 A 133
Bush 2 A 134
Hansen 3 A 143
Hobbs 4 B 144
Hobbs 5 B 144
Jones 6 C 155
Perry 7 D 244
Perry 8 D 311
and the rdbtable (named samplej2) here:
name cnt typ amt
6 5N 4 5N
Hobbs 41 A 141
Hobbs 42 BB 142
Hobbs 51 BB 144
Hobbs 43 CC 143
then the command to do a natural join of samplej and samplej2 on column name is:
nsq-join name samplej2 < samplej
and the result is shown in Table 6.
Table 6
NATURAL JOIN OF RDBTABLES SAMPLEJ AND SAMPLEJ2
name nr typ amt cnt typ amt
6 2 4 4 5N 4 5N
Hobbs 4 B 144 41 A 141
Hobbs 4 B 144 42 BB 142
Hobbs 4 B 144 51 BB 144
Hobbs 4 B 144 43 CC 143
Hobbs 5 B 144 41 A 141
Hobbs 5 B 144 42 BB 142
Hobbs 5 B 144 51 BB 144
Hobbs 5 B 144 43 CC 143
The command to do a "masterdetail" join of the same two rdbtables on column name is:
nsq-join -md name samplej2 < samplej
and the result is shown in Table 7.
Table 7
MASTER-DETAIL JOIN OF RDBTABLES SAMPLEJ AND SAMPLEJ2
name nr typ amt cnt typ amt
6 2 4 4 5N 4 5N
Bush 1 A 133
Bush 2 A 134
Hansen 3 A 143
Hobbs 4 B 144 41 A 141
Hobbs 4 B 144 42 BB 142
Hobbs 4 B 144 51 BB 144
Hobbs 4 B 144 43 CC 143
Hobbs 5 B 144 41 A 141
Hobbs 5 B 144 42 BB 142
Hobbs 5 B 144 51 BB 144
Hobbs 5 B 144 43 CC 143
Jones 6 C 155
Perry 7 D 244
Perry 8 D 311
Usage: nsq-l2t [options]
Converts a file in "list" format to an rdbtable. Long data fields may be folded. This operator is mainly used by other operators. Options may be abbreviated.
Usage: nsq-merge [options] < old_table column ... merge_table
This operator merges and/or deletes rows of 'old_table' based on data values in 'merge_table' in the specified column(s). Both tables should be sorted on the specified column(s).
In the normal case, one or more rows in 'merge_table' either replace one or more existing rows in 'old_table' if the key column(s) match, or are inserted in order if the key column(s) do NOT match.
If the delete option is specified on the command line, one or more existing rows in 'old_table' will be deleted if there is a key column(s) match and the data in the delete column is equal to the delete string, ">>DEL<<" (without the quotes) by default. The delete column is the first non-key column in 'merge_table'.
Both tables should have similar data structures. The header for the new rdbtable is taken from 'merge_table', thus allowing a change of header information to be made.
This operator writes an rdbtable via STDOUT. Options may be abbreviated.
Usage: nsq-mktbl [options]
Makes a file of data in columns (with visible column delimiters) into an rdbtable. The column delimiter is the pipe symbol (|). This operator is mainly used by other operators.
This operator reads a file via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.
Usage: nsq-pr [options]
This operator used for quick and easy printing of an rdbtable, in a simple but useful form. It prints an rdbtable using formatting information from the header.
The printing of each row of data will be on one line if possible, but when multiple lines are necessary the second and later lines are indented for readability. Also when multiple lines are necessary a simple space availability algorithm is used to minimize the number of lines printed for each row of data. This may result in the order of some data values being rearranged from their order in the rdbtable. The '-b0' option can override this algorithm and force the same printing order as in the rdbtable.
This NoSQL operator reads an rdbtable from STDIN and writes a formatted report on STDOUT. Options may be abbreviated.
As an example using the sample rdbtable from the DATA section (named sample), the command to view this rdbtable would be:
nsq-pr < sample
which would produce the output shown in Table 8. The same command with a page heading for printing:
nsq-pr -PP < sample
produces the output as shown in Table 9. Using an rdbtable (named sample4) that has long data values, shown in Table 10, the command to print the rdbtable using the truncate option is:
nsq-pr -t < sample4
Table 8
PRINTING RDBTABLE (SAMPLE) USING NSQ-PR
NAME COUNT TYP AMT OTHER RIGHT
------ ----- ---- ----- -------- --------
Bush 44 A 133 Another This
Hansen 44 A 23 One Is
Jones 77 X 77 Here On
Perry 77 B 244 And The
Hart 77 D 1111 So Right
Holmes 65 D 1111 On Edge
Table 9
PRINTING RDBTABLE (SAMPLE) WITH PAGE HEADING USING NSQ-PR
Page 1 Mon Dec 2 16:56:43 PST 1991
NAME COUNT TYP AMT OTHER RIGHT
------ ----- ---- ----- -------- --------
Bush 44 A 133 Another This
Hansen 44 A 23 One Is
Jones 77 X 77 Here On
Perry 77 B 244 And The
Hart 77 D 1111 So Right
Holmes 65 D 1111 On Edge
Table 10
RDBTABLE WITH LONG DATA VALUES (SAMPLE4) ACTUAL CONTENT
name<T>type<T>contact<T>contents<N>
10<T>4<T>21<T>20<N>
Hansen<T>AAA<T>R. Starr at the UCLA & USC<T>Duplicate data under
processing order number 55-7.<N>
Hart<T>CCC<T>Hobbs/Emerson at RAND Corporation<T>85 files, 2 per
day containing 12 and 24 hour reports.<N>
Hobbs<T>EEE<T>Marshall at Universal AFB<T>Original PAF messages.
Both sets are incomplete.<N>
Bush<T>KKK<T>General USAF personnel<T>Duplicate ATO messages,
incomplete.<N>
Lender<T>RRR<T>Army base in Nevada<T>Nothing.<N>
Emerson<T>UUU<T>Navy at Washington DC<T>More than we thought at
first.<N>
which will produce output with the data values truncated to the defined column width as in Table 11. Using the same rdbtable with the fold option:
nsq-pr -fold < sample4
produces output with the long data values 'folded' within their defined column widths as shown in Table 12. Note that each line is repeated until the entire data value for each column is completely shown. This makes this type of output variable length.
If you need a quick and easy way to look at the data in an rdbtable use the -win option. This option will cause nsq-pr to list as many columns as possible in single line records that will fit in the current window or terminal width. Note that you do not have to type the column names (or even know them) to use this option.
Table 11
PRINTING RDBTABLE (SAMPLE4) WITH NSQ-PR -TRUNC OPTION
name type contact contents
---------- ---- --------------------- --------------------
Hansen AAA R. Starr at the UCLA Duplicate data under
Hart CCC Hobbs/Emerson at RAND 85 files, 2 per day
Hobbs EEE Marshall at Universal Original PAF message
Bush KKK General USAF personne Duplicate ATO messag
Lender RRR Army base in Nevada Nothing.
Emerson UUU Navy at Washington DC More than we thought
Table 12
PRINTING RDBTABLE (SAMPLE4) WITH NSQ-PR -FOLD OPTION
name type contact contents
---------- ---- --------------------- --------------------
Hansen AAA R. Starr at the UCLA Duplicate data under
& USC processing order
number 55-7.
Hart CCC Hobbs/Emerson at RAND 85 files, 2 per day
Corporation containing 12 and 24
hour reports.
Hobbs EEE Marshall at Universal Original PAF
AFB messages. Both sets
are incomplete.
Bush KKK General USAF Duplicate ATO
personnel messages,
incomplete.
Lender RRR Army base in Nevada Nothing.
Emerson UUU Navy at Washington DC More than we thought
at first.
It may be combined with the -t option to increase the number of columns of data shown on each line at the expense of some column width.
For example the command 'nsq-pr < d11c' on an 80 character wide window or terminal produces the following:
name count type amt n1 n3
------ --------------- ---- ----- ------------------------ ------------
n2 n4 n5
------------------------ ------------------------ ------------
n6 n7
------------ ------------
Bush 3 A 133 alpha22.307 117722
baker DDBBx17 other
124567 8GGXXH17
Hansen 39 A 23 beta222.307 117723
charlie DDBBx18 data
1239870 GGXXH17
Newton 8 E 8 gama22.333 117724
dog DDBBx19 exists
1239870 GGXXH17
Hobbs 42 B 144 delta3.3.118 117725
echo DDBBx20 here
1239870 GGXXH17
Hart 2 C 55 epslion33.118 117726
foxtrot DDBBx21 also
1239870 GGXXH17
This is readable, but not very nice to look at, and even worse if there are more columns. The command
nsq-pr -win < d11c
produces:
name count type amt n1 n3
------ --------------- ---- ----- ------------------------ ------------
Bush 3 A 133 alpha22.307 117722
Hansen 39 A 23 beta222.307 117723
Newton 8 E 8 gama22.333 117724
Hobbs 42 B 144 delta3.3.118 117725
Hart 2 C 55 epslion33.118 117726
Not all the data is listed, but the first few columns (sometimes the most important) are easier to view. The command
nsq-pr -win -t6 < d11c
shows even more of the data, at the expense of some data width:
name count type amt n1 n2 n3 n4 n5 n6 n7
------ ------ ---- ----- ------ ------ ------ ------ ------ ------ ------
Bush 3 A 133 alpha2 baker 117722 DDBBx1 other 124567 8GGXXH
Hansen 39 A 23 beta22 charli 117723 DDBBx1 data 123987 GGXXH1
Newton 8 E 8 gama22 dog 117724 DDBBx1 exists 123987 GGXXH1
Hobbs 42 B 144 delta3 echo 117725 DDBBx2 here 123987 GGXXH1
Hart 2 C 55 epslio foxtro 117726 DDBBx2 also 123987 GGXXH1
Usage: nsq-repair [options] file ...
Attempts to repair candidate NoSQL datafiles, e.g. files that have been ported from a MacIntosh or PC (MSDOS computer) in spreadsheet form but that do not yet have valid rdbtable structure. Generates definition lines (second line of header). The width of all data values is checked and the maximum width for a column is used as the column width in the definition line for that Table.
It also works with existing rdbtables ('-exist' option) and is convenient for removing leading and trailing space characters from data values (-blank option).
Adds fields as necessary to rows (null), or to header (DUM1, DUM2, ...) to make the Table structure valid.
The new rdbtables will be in the current directory (even if the input files are not) and will have the suffix changed (or added) to '.rdb' by default.
Options may be abbreviated.
Usage: nsq-report [options] file.frm
Formats and prints an arbitrary style report, with the format specified in the file "file.frm". A page header may be specified.
This NoSQL operator reads an rdbtable from STDIN and writes a formatted report on STDOUT. Options may be abbreviated.
The "file.frm" file (or form file) shows pictorially one 'record' of output, which will contain data from one row of an rdbtable. An optional page header may be defined as well.
The form file contains regular text, picture fields, and associated column names. Regular text prints just as given. Picture fields define the width and justification for printing a data value from a column. The names of the associated columns are listed on the line following the picture fields and in the same order. Note that this file should not contain any TAB characters; space characters should be used instead.
Picture fields start with either '@' or '^' and are followed by one of three primary characters to define the width of the field. The three characters are '<', '>', or '|' to specify left, right, or center justification respectively. There is also an alternate right justification character for printing numeric data, with optional decimal point. The character is the sharp sign '#', and a period specifies the decimal point placement, as in '@########.##'.
A numeric picture field has the following features:
Numeric Data may be in the form of integers, fixed point, or scientific notation' e.g. 12345, 4567.345, or 1.678E17.
Normally picture fields start with the '@' character. That means to put the referenced data value into the defined picture field, or as much of the data as will fit into the field, if the data is larger than the field. If the field starts with the '^' character it means to repeat the field on as many lines as necessary in order to print the entire data value. This is useful for large data fields, such as comments or free text.
Instead of a column name there are some special names that can be used to have other information inserted. This are especially useful if there is a page header. The special names and what they mean are:
An example of a form file for use with rdbtable 'sample' is shown in Table 13.
The first and last lines (that start with 'format' or a single period) define the pictorial records and must be as shown. The first record defines the header and is optional. If this form file (named sample.frm) were used in the command:
nsq-report sample.frm < sample
it would produce the one page report as in Table 14.
Table 13
FORM FILE FOR RDBTABLE (SAMPLE)
format top =
Page @>, The Page Header @<<<<<<<<<<<<<<<<<<<<<<<<<<<
_pgnr_ _date_
.
format =
Name: @<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< Type: @>>>>
NAME TYP
Total: @<<<<<<<< Other: @<<<<<<<<<<<<<<<
AMT OTHER
.
Table 14
PRINTING RDBTABLE (SAMPLE) WITH NSQ-REPORT
Page 1, The Page Header Mon Dec 2 16:21:18 PST 1991
Name: Bush Type: A
Total: 133 Other: Another
Name: Hansen Type: A
Total: 23 Other: One
Name: Jones Type: X
Total: 77 Other: Here
Name: Perry Type: B
Total: 244 Other: And
Name: Hart Type: D
Total: 1111 Other: So
Name: Holmes Type: D
Total: 1111 Other: On
For another example, one might want to have a date on a report in other than the standard date output format, and an idea of who executed the program, and have a reference count of the records being produced. The form file might be as in Table 15. which could produce:
Table 15
ANOTHER FORM FILE
format top =
Run By: @<<<<<<< The Date/Time is @<<<<<<<<<<<<<<<<<<<<<<<<
_`whoami`_ _`date "+%m/%d/%y %H:%M"`_
.
format =
RecordNr: @>> @<<<<<<<<<<<<<<<<<<<<
_rcnr_ OTHER
.
Run By: hobbs The Date/Time is 10/15/91 09:43
RecordNr: 1 Other data here
RecordNr: 2 and here,
RecordNr: 3 and so on.
... ... ...
Another example shows how longer data values can be handled. If the form file (named sample2.frm) looks like Table 16, and if the following command is used:
nsq-report sample2.frm < sample2
then the output would be as shown in Table 17.
Table 16
FORM FILE FOR RDBTABLE (SAMPLE2)
format top =
Page @>, Page Header here @<<<<<<<<<<<<<
_pgnr_ _`date "+%m/%d/%y %H:%M"`_
Executed by @<<<<<<< on: @<<<<<<<<<
`whoami` _`hostname`_
format =
Name: @<<<<<<<<<<<<<<< Other: @<<<<<<<<<< Type: @<<<<<<<<
NAME OTHER TYP
Comment: ^<<<<<<<<<<<<<< Long: ^<<<<<<<<<<<<<< Right: @>>>>>>>
COMMENT LONG RIGHT
=======================================================================
Table 17
PRINTING RDBTABLE (SAMPLE2) WITH NSQ-REPORT
Page 1, Page Header here 12/02/91 16:32
Executed by hobbs on: id
Name: Bush Other: Another Type: A
Comment: A comment Long: This a long Right: This
here. message for
test.
=======================================================================
Name: Hansen Other: One Type: A
Comment: A longer Long: This a long Right: Is
comment here. message for
test.
=======================================================================
Name: Jones Other: Here Type: X
Comment: A longer, Long: Short test. Right: On
longer comment
here.
=======================================================================
Name: Perry Other: And Type: B
Comment: A short comment Long: This a long Right: The
here. message for
test.
=======================================================================
Name: Hart Other: So Type: D
Comment: Little here. Long: Here too. Right: Right
=======================================================================
Name: Holmes Other: On Type: D
Comment: A comment here Long: A short Right: Edge
that is a message.
little ongoing,
so to speak.
=======================================================================
Note that since there were two picture fields that started with the '^' character on one line the length of output records varies according to the length of the two associated data values.
Usage: nsq-row [options] expression
Selects rows from the input rdbtable based on an arbitrary expression using column names. Characters that are special to the UNIX shell must be quoted.
Logical constructors 'or' and 'and' may be used; as well as 'null' to indicate empty data values. Comparison operators may be of the form: gt, ge, lt, le, eq, ne, mat, nmat. The first six are the usual operators, E.g 'name eq Hobbs' or 'COUNT gt 100'. The last two stand for 'match' and 'non-match' and are used for pattern matching. They are exactally the same as using the PERL operators '=~' or '!~' respectively, except that pattern matching can be specified easier in expressions, as in:
NAME mat /[Hh]obbs/ <<< First letter either case
NAME mat /hobbs/i <<< any combination of case
NAME nmat /[aeiou]/i <<< names without vowels
where 'NAME' and 'COUNT' are column names, of course. A warning message is produced on STDERR if either of 'mat' or 'nmat' is used with a numeric type column, but the execution continues. It does not check the '= ' or '! ' forms.
All of the Comparison operators and Logical constructors are reserved and should not be used as column names (they are all lower case and four characters or less).
Since column names and reserved words are parsed by the program, do not put the entire expression in a single pair of quotes as that will prevent the parsing. Also note that column names and reserved words need to be surrounded by blank spaces if they are not individually quoted. For example either form below is fine:
nsq-row NAME eq "L Brown" < sample
nsq-row "NAME" "eq" "L Brown" < sample
but do not use this form:
nsq-row "NAME eq L Brown" < sample
This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.
As an example using the sample rdbtable from the DATA section (named sample), to select rows that have the NAME column equal to 'Hansen' the command would be:
nsq-row NAME eq Hansen < sample
which would produce:
NAME COUNT TYP AMT OTHER RIGHT
6 5N 4 5N 8 8>
Hansen 44 A 23 One Is
to select rows that have the TYP column equal to 'A' or that have the AMT column greater than 1000 the command would be:
nsq-row TYP eq A or AMT gt 1000 < sample
producing:
NAME COUNT TYP AMT OTHER RIGHT
6 5N 4 5N 8 8>
Bush 44 A 133 Another This
Hansen 44 A 23 One Is
Hart 77 D 1111 So Right
Holmes 65 D 1111 On Edge
Note that in some rare cases there could be a column name that is identical to a data value specified in an expression using another column name that might cause a problem (this actually happened). For example if two column names are 'N' and 'T', and column 'N' has a data value of 'T', to select all rows where column 'N' is equal to 'T' the normal command would be:
nsq-row < table N eq T
Unfortunately the 'T' in the expression gets translated to 'column name T', not used as 'data value T'. That is, the expression askes for all rows where the data value of column N equals the data value of column T, legal, but not what was wanted. There is a simple workaround however. The 'T' in the expression can be escaped with a backslash to prevent the translation to a column name, as in the revised command:
nsq-row < table N eq '\T'
Thus either meaning can be specified, as desired.
Usage:
nsq-search [options] rdbtbl < keytbl
or:
nsq-search [options] -ind index_file [rdbtbl] < keytbl
This operator does a fast search of 'rdbtbl' (or index_file) using a binary search on a key of of one or more columns. The 'rdbtbl' (or index_file) must be sorted on the key columns. Each column in the key may be of type string or type numeric (but be carefull with numeric data and exact matches). In the second form of usage for this operator if 'rdbtbl' is not given its name will be inferred from the name of index_file. For example if index_file is 'skb.x.typ' then the rdbtbl name inferred will be 'skb.rdb'.
The column(s) in the file 'keytbl' specify both the key column name(s) and the argument values to search for. File 'Keytbl' is in rdbtable format.
Normally an argument value and a data field must compare exactally for a match to occur (exact match). If the paritial match otpion (-part) is selected, and if the argument value compares with the initial part of the data field it is considered a match. This applies to string type data only. Note that for numeric type data an exact match is always necessary.
Normally all rows that match will be written to the new rdbtable, in the same order as in the old rdbtable. If only a single row key match is appropriate some execution time can be saved by specifing the '-sgl' option.
This operator writes an rdbtable via STDOUT. Options may be abbreviated. Returns the number of non-finds at exit.
Usage: nsq-sort [options] [-r] column [[-r] column] ...
Sorts an rdbtable on one or more columns. Each column may be sorted in normal (ascending) or reverse (descending) order. Also a column of monthnames (Jan, Apr, ...) in any case letters, may be sorted.
This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated. Uses the UNIX 'sort' routine.
For example, using the sample data file from the DATA section (named sample) in the following command:
nsq-sort COUNT TYP < sample
would produce:
NAME COUNT TYP AMT OTHER RIGHT
6 5N 4 5N 8 8>
Bush 44 A 133 Another This
Hansen 44 A 23 One Is
Holmes 65 D 1111 On Edge
Perry 77 B 244 And The
Hart 77 D 1111 So Right
Jones 77 X 77 Here On
Of course it would look better if it was piped through 'nsq-pr'. The command:
nsq-sort COUNT -r AMT < sample
would produce:
NAME COUNT TYP AMT OTHER RIGHT
6 5N 4 5N 8 8>
Bush 44 A 133 Another This
Hansen 44 A 23 One Is
Holmes 65 D 1111 On Edge
Hart 77 D 1111 So Right
Perry 77 B 244 And The
Jones 77 X 77 Here On
Usage: nsq-subtot [options] B_column ... -s column ...
This operator lists subtotals of specified column(s) whenever the value of specified break columns(s) (B_column(s)) changes. A single break column will produce a sub-total of all specified columns on each line. If there is more than one break column given then in addition whenever the value of the first break column changes an additional line will be printed showing the sub-total for that group.
If no break column is given the first column is used; if no subtotal column is given then all columns of type numeric are sub-totaled.
This operator reads an rdbtable via STDIN and writes an rdbtable via STDOUT. Options may be abbreviated.
Example rdbtable (named small):
name amt typ count n
6 5N 4 5N 2
Hansen 39 A 23 3
Hansen 9 A 3 3
Hansen 9 B 3 4
Jones 42 B 144 5
Jones 4 B 14 5
Hart 9 C 3 5
Hart 2 C 55 6
Hart 2 D 55 6
Hobbs 57 X 7 4
Hobbs 5 X 57 4
The output from the command:
nsq-subtot name -s amt < small | nsq-pr
would be:
name amt
------ -----
Hansen 57
Jones 46
Hart 13
Hobbs 62
The output from the command:
nsq-subtot name typ -s amt count < small | nsq-pr
is shown in Table 18.
Table 18
OUTPUT FROM THE NSQ-SUBTOT OPERATOR
name typ amt count
------ ---- ----- -----
Hansen A 48 26
Hansen B 9 3
57 29
Jones B 46 158
46 158
Hart C 11 58
Hart D 2 55
13 113
Hobbs X 62 64
62 64
Usage: nsq-summ [options] [column ...]
Produces "summary" information about the rdbtable. If no columns are given then information about all columns is produced. A Count of the data rows is always shown.
This operator reads an rdbtable via STDIN and writes a summary report via STDOUT. Options may be abbreviated.
Usage: nsq-t2l [options]
Converts an rdbtable to "list" format. Long data fields are folded. This operator is mainly used by other operators.
This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.
Usage: nsq-tee [options] rdbtable
Reads an rdbtable via STDIN and [over]writes the rdbtable given as a command line argument, abiding by the NoSQL table locking scheme. This operator is named after the UNIX utility tee.
This operator can be safely used in constructs like :
nsq-compute ... < rdbtable | nsq-tee rdbtable | ...
without worring that the original input rdbtable be destroyed before the first pipeline has completed.
If no output stream is desired on STDOUT then the latter can be directed to /dev/null, as usual:
nsq-compute ... < rdbtable | nsq-tee rdbtable > /dev/null
This operator is especially meant to be used in programs that need to modify rdbtables 'in place'. It can optionally take a backup copy of the target rdbtable, which will have the same name as the destination table, prefixed by a comma (,).
The above locking mechanism is fine when only one user at a time is modifying a given rdbtable. However, if multiple users have write access to the table, then a race condition may still occur. Consider two users, User A and User B, who execute the above command ad the same time:
User A: nsq-compute ... < rdbtable | nsq-tee rdbtable
User B: nsq-compute ... < rdbtable | nsq-tee rdbtable
Now, consider the folowing sequence of events:
To prevent such race conditions, any pipeline of commands that end up with changing the contents of an rdbtable should lock the output table before reading it, i.e.:
${NSQLOCKER:-nsq-lock} rdbtable.LCK && \
nsq-compute ... < rdbtable | nsq-tee -N -U rdbtable
This ensures that either the pipeline can set the lock right from the beginning, or it will not even start reading until it can (or until it times-out on the lock file).
Note: the command shown follows the usual Bourne shell sintax. If you do not understand it, please refer to your shell documentation.
Usage: nsq-uniq [options] column ...
Reads the input rdbtable and compares adjacent rows. The second and succeeding copies of repeated rows, considering only the selected columns, are removed. That is, adjacent rows are considered equal if the data values in all of the selected columns are equal. The remaining rows are written to the output rdbtable.
Note that repeated rows must be adjacent in order to be found. Normally this means that the input rdbtable should be sorted on the selected columns for this capability to work properly.
This NoSQL operator reads an rdbtable from STDIN and writes an rdbtable to STDOUT. Options may be abbreviated.
Usage: nsq-valid [options] [file ...]
Validates the structure of one or more rdbtables. Checks number of data fields per line, max width of column names and data values, and checks numeric data type values. Reports errors by line number and column name.
Reads from STDIN if filenames are not given. Writes diagnostic information on STDOUT. Options may be abbreviated.
If there is more than one file given each file will be identified on the output.
The '-size' option has proven very useful as it shows the actual size of the largest data value for each column, in addition to the template information. The command:
nsq-valid -size sample
shows the following output:
0 NAME 6 6
1 COUNT 5N 2
2 TYP 4 1
3 AMT 5N 4
4 OTHER 8 7
5 RIGHT 8> 5
Columns: 6, Rows: 6, File format valid sample
The last two columns above show the defined size of each column in the rdbtable, and the actual maximum size of the data values for each column in the rdbtable.
In addition to the operators explained above, which are mostly from the original RDB package, I have added a few new ones, but they are not yet documented here. They all take a '-help' option, so please refer to that for more information on what each operator does. These new operators are :
In the ./src/
subdirectory of the NoSQL distribution
you will find a few extra operators, which are not yet
part of the main set, as they are still experimental. They also
take a '-help' option, so refer to that for details of what they
do. You should read the README file in that directory before you
try to use them. These operators are: