1. Foreword and Introduction

1.1 Copyright

Copyright (c) 1998,1999,2000,2001,2002,2003 Carlo Strozzi.

Parts of this manual come from the RDB documentation, written by W.Hobbs and included here with permission of the author.

Permission is granted to copy, distribute and/or modify this document under the terms of the GNU Free Documentation License, Version 1.2 or any later version published by the Free Software Foundation; with no Invariant Sections, no Front-Cover Texts, and no Back-Cover Texts. A copy of the license is included in the section entitled "GNU Free Documentation License".

You should have received a copy of the GNU Free Documentation License along with this work; if not, write to the Free Software Foundation, Inc., 675 Mass Ave, Cambridge, MA 02139, USA.

1.2 Preface

This working draft describes, and provides instructions for the use of, NoSQL (I personally like to pronounce it noseequel), a derivative of the RDB DataBase system. The RDB system was (and still is) developed at RAND Organization by Walter W. Hobbs. Part of the NoSQL code, as well as large parts of the text of this document, have been taken directly from RDB, so a good share of the credit goes to the original author.

Other major contributors to the original RDB system, besides W. Hobbs, were:

Chuck Bush

Don Emerson

Judy Lender

Roy Gates Rae Starr

People who helped with turning RDB into NoSQL:

David Frey

Maurizio (Masar) Sartori

Vincenzo (Vicky) Belloli

Giuseppe Paternò

Paul Lussier

Seth LaForge

The NoSQL.png logo has been kindly provided by Kyle Hart.

NoSQL tends to be biased in favour of Linux. This means that, wherever it matters, NoSQL makes use of the GNU versions of the various UNIX commands, as those are the ones that are usually packaged together with the Linux O.S. And among the many Linux "distributions" available nowadays, NoSQL has been developed entirely on Debian, although it should run just as fine on any other common Linux distributions. NoSQL is Free Software, released under the terms of the GNU General Public License. As such, it fully qualifies as Open Source Software.

1.3 Introduction

A good question one could ask is "With all the relational database management systems available today, why do we need another one ?". The main reasons are:

1. Several times I have found myself writing applications that needed to rely upon simple database management tasks. Most commercial database products are often too costly and too feature-packed to encourage casual use. There is also plenty of good free databases around, but they too tend to provide far more than I need most of the times, and they too lack the shell-level approach of NoSQL. By contrast, NoSQL takes a very simple approach (even simplistic, as some may argue :-), but that is exactly its distinguishing feature. Admittedly, having been written mostly with interpretive languages (Shell, Perl, AWK), NoSQL is not the fastest DBMS of all, at least not always (a lot depends on the application).
2. NoSQL is easy to use by non-computer people. The concept is straightforward and logical. To select rows of data, the 'row' operator is used; to select columns of data, the 'column' operator is used.
3. The data is highly portable to and from other types of machines, like Macintoshes or DOS computers.
4. The system should run on any UNIX machine that has perl(1) and mawk(1) installed.
5. NoSQL has very few arbitrary limits, and can work where other products can't. For example there is no limit on data field size, the number of columns, or file size (the number of columns in a table may actually be limited to 32.768 by some implementations of the AWK interpreter, including mawk I think).

Note: NoSQL has only been tested with mawk(1), that is Mike Brennan's implementation of the AWK programming language, and will most likely not work out-of-the-box with any other AWK, including gawk(1). While getting NoSQL to work also with gawk(1) should not be difficult, making it work with other AWKs may prove hard, if at all possible.

As its name implies, NoSQL is not an SQL database. The rationale behind this is well explained in the accompanying paper 4gl.ps (Postscript), or 4gl.txt (ASCII).

The data is contained in regular UNIX ASCII files, and so can be manipulated by regular UNIX utilities, e.g. ls, wc, mv, cp, cat, more, less, editors like 'vi', head, RCS, etc.

The form of each file of data is that of a relation, or table, with rows and columns of information.

To extract information, a file of data is fed to one or more "operators" via the UNIX Input/Output redirection mechanism.

There are also programs to generate, modify, and validate the data. A thorough discussion on why this type of relational database structure makes sense is found in the book, "UNIX Relational Database Management", Reference #2.

It is assumed that the reader has at least a basic familiarity with the UNIX Operating System, including knowledge of Input/Outout redirection (e.g., STDIN, STDOUT, pipes).

Again, the key feature of NoSQL (and other similar packages mentioned in this manual), is its close integration with UNIX. Real-world problems are tipically more complex than the data models provided by many DBMS'es. Actual applications, and Web-based ones make no exception, are complex puzzles made up by many small pieces, several of which are data-related. Unlike other fourth generation systems, NoSQL is an extension of the UNIX environment, making available the full power of UNIX during application development and usage. NoSQL was designed with the UNIX shell language as its user interface. This level of integration removes the need to learn yet another set of commands to use and administer the database system. A database is just a file, and can be maintained like all other files that the user owns or has access to. Because NoSQL commands are executable programs, the UNIX shell is inherited as the primary command language of the database; no other proprietary database scripting language, to my knowledge, is as powerful and flexible as the UNIX shell. The shell-level nature of NoSQL encourages casual use of the system, and successful casual use leads to familiarity and successful formal use. This concept is much more thoroughly explained in the paper "The UNIX Shell As a Fourth Generation Language", included in the NoSQL documentation tree with the file name 4gl.ps (Postscript) or 4gl.txt (ASCII), that shows why the UNIX shell is an excellent tool for scripting database access.

NoSQL cannot certainly be seen as a ``Big Name Database''. But its extreme generality and absence of restrictions make it more convenient and effective for many everyday data management tasks than supposedly more powerful products.

1.4 Perl and the Operator/Stream Paradigm

As stated in the Abstract, NoSQL uses the Operator/Stream DBMS Paradigm. The main reason why I decided to turn the original RDB system into NoSQL is that the former is entirely written in Perl. Perl is a good programming language for writing self-contained programs, but Perl's pre-compilation phase and long start-up time are worth paying only if once the program has loaded it can do everything in one go. This contrasts sharply with the Operator/Stream model, where operators are chained together in pipelines of two, three or more programs. The overhead associated with initializing Perl at every stage of the pipeline makes pipelining somewhat inefficient. A better way of manipulating structured ASCII files is using the AWK programming language, which is much smaller than Perl, is more specialized for this task and very fast at startup (on my Pentium II Linux /usr/bin/mawk (POSIX AWK) is just 99K. Perl 5 is almost 500K. You get the point).

1.5 Bug reports

There is a mailing list for discussions related to NoSQL. The address is noseequel(at)strozzi.it. To subscribe simply send a message to minimalist(at)strozzi.it with the word "subscribe" (without the quotes) in the message subject.

Please send bug reports (fixes are most welcome) to the same list noseequel(at)strozzi.it. Always include as much information as possible, especially the content of file nosql.version, which is created in the NoSQL installation directory during install. By 'bug reports' I mean not just errors in the code, but also grammatical mistakes, typos and bad English constructions in the documentation, as English isn't my native language.