SPREAD: A Reliable Multicast and Group Communication Toolkit ----------------------------------------------------------- /===========================================================================\ | The Spread Group Communication Toolkit. | | Copyright (c) 1993-2002 Spread Concepts LLC | | All rights reserved. | | | | The Spread package is licensed under the Spread Open-Source License. | | You may only use this software in compliance with the License. | | A copy of the license can be found at http://www.spread.org/license | | | | This software is distributed on an "AS IS" basis, WITHOUT WARRANTY OF | | ANY KIND, either express or implied. | | | | Spread is developed at Spread Concepts LLC and the Center for Networking | | and Distributed Systems, The Johns Hopkins University. | | | | Creators: | | Yair Amir yairamir@cs.jhu.edu | | Michal Miskin-Amir michal@spreadconcepts.com | | Jonathan Stanton jonathan@cs.jhu.edu | | | | Major Contributors: | | Cristina Nita-Rotaru crisn@cnds.jhu.edu - group communication security.| | Theo Schlossnagle jesus@omniti.com - Perl, skiplists, autoconf. | | Dan Schoenblum dansch@cnds.jhu.edu - Java interface. | | John Schultz jschultz@d-fusion.net - contribution to process | | group membership. | | | | Contributors: | | Ryan Caudy wyvern@cnds.jhu.edu - Group membership | | Ben Laurie ben@algroup.co.uk - FreeBSD port and warning fixes | | Daniel Rall dlr@finemaltcoding.com - Java & networking fixes, | | configuration improvements | | Marc Zyngier - Windows fixes | | | | Special thanks to the following for providing ideas and/or code: | | Ken Birman, Danny Dolev, Mike Goodrich, Ben Laurie, | | David Shaw, Robbert VanRenesse. | | | | Partial funding provided by the Defense Advanced Research Projects Agency | | (DARPA) and The National Security Agency (NSA) since 2000. The Spread | | toolkit is not necessarily endorsed by DARPA or the NSA. | | | | WWW : http://www.spread.org and http://www.cnds.jhu.edu | | Contact: spread@spread.org | | | | Version 3.17.1 Built 20/June/2003 | \===========================================================================/ June 20, 2003 Ver 3.17.1 ---------------------------- Bugfixes: *) Fix memory corruption and crash with groups of large size. *) Correct make install so it installs header files. *) Fix syntax error in build.xml file for Java/Ant. *) Cleanup prototypes to remove compiler warnings. *) Fix parser to correctly recognize upper, lower, and mixed case command options. *) During make install, remove old symlinks. *) Change setgroups call to be more portable. (fixes MacOSX) *) Change name of r and s to sprecv and spsend, and add as make targets. They can be built by "make testprog" (not built by default). *) Work on making long group names possible. *) Increase listen backlog for accepting client connections. *) Fix Win32 project files to have correct path to source files. (note CVS was always ok, but 3.17.0 release had incorrect path) *) Fix bug where large groups overflow Mess_buf in groups.c. *) Fix memory corruption bug when a message header is received in several separate packets in session.c. Thanks to Ryan Caudy for many, many hours tracking this down. *) Change order of build in Makefile so binaries are built before documentation. *) Fix Java bug where connection objects cannot be disconnected and then reconnected, but must be created anew. They can now be reused. *) Fix compile error on AIX for struct if_info. *) Fix security issue with buffer checks in the C library. *) Fix obscure off-by-one buffer error with the parser. SOURCE INSTALL: --------------- The source install uses autoconf and make to support many Unix and unix-like platforms as well as Windows 95+. Windows is supported by a set of Visual Studio desktop and project files in the win32 subdirectory. These files build the daemon, libraries, and user programs. Any not-specifically supported platform that has reasonably close to normal sockets networking should also be easily portable. See the file PORTING for hints on porting. From the directory where you unpacked the Spread source distribution do the following: 1) Run "./configure" 2) Run "make" 3) Run "make install" as a user with rights to write to the selected installation location. (/usr/local/{bin,man,sbin,include,lib} by default) 4) Create a spread.conf file appropriate to your local configuration. For more info on how to do this look at the sample file "sample.spread.conf" or below in the binary install instructions. BINARY INSTALL: -------- We recommend that if you are experimenting with spread you create a special 'spread' directory (for example /usr/local/spread or /opt/spread) and keep all the files together there so it is easy to find stuff. That also makes it easier to run multiple architectures as the binaries for each are in their own subdirectory. This is not necessary though. You can create that directory anywhere (e.g. your own directory). If you are installing spread for active use it is probably easier to just install the correct version of the binaries, headers, man pages, and libraries into your standard locations. The directions below assume you are doing this. 1) Unpack the spread.tar.gz file into a temporary directory 2) Look at the Readme for any updates 3) Select the appropriate architecture: arch-bsdi arch-sgi arch-sunos arch-sunsol arch-pcsol arch-linux 4) Type 'make arch-????' with your architecture as the option to make 5) Now you need to copy the files, I will assume you use /usr/local/{bin,include,lib,man} Replace "ARCH" with the directory for your architecure. cp -p include/* /usr/local/include/ cp -p ARCH/libspread.a /usr/local/lib/ cp -p ARCH/libtspread.a /usr/local/lib/ cp -p ARCH/spread /usr/local/bin/ cp -p ARCH/spmonitor /usr/local/bin/ cp -p ARCH/spuser /usr/local/bin/ cp -p ARCH/sptuser /usr/local/bin/ cp -p ARCH/spflooder /usr/local/bin/ cp -p docs/*.3 /usr/local/man/man3/ cp -p docs/*.1 /usr/local/man/man1/ 6) To run spread you need to modify the 'spread/sample.spread.conf' file for your network configuration. Comments in the sample file explain this. Then cp -p sample.spread.conf /usr/local/bin/spread.conf (Notice the name change. The config file must be named that to be found. Alternatively you can run spread -c ) You can run "spread usage" to see the spread running options. To use the Java classes and examples you need to have a copy of the main 'spread' daemon running. Then the sp.class file gives you the equivelent of the libsp.a as a java class. The user.java, user.html, and user.class files give you a demonstration applet and source code. The tree.html AllNames.html and packages.html give some documentation for the java interface. For Windows (95/NT) systems use the spread.exe daemon and the libsp.lib to link with your programs. OVERVIEW: --------- Spread consists of two parts, an executable daemon which is executed on each machine which is part of the heavyweight 'machine' group, and a library which provides a programming interface for writing group multicast applications and which is linked into the application. The daemon, called "spread", should be run as a non-priveledged user (we created a 'spread' user) and only needs permissions to create a socket in /tmp and read its config file which should be in the same directory as the executable. By default the daemon binds to and runs off the non-priveledged port 4803 unless the config file indicates otherwise. Each daemon can be started independently and if it does not find any other members of its defined configuration currently active it runs as a machine group of 1 machine. When other daemons are started they will join this machine group. The machines which are running a daemon with a common config file form a 'machine group' (in contrast to a 'process group'). The daemons handle multicasting mesages between each other,providing ordering and delivery guarantees, detecting and handling failures of nodes or links, and managing all applications which are connected to each daemon. Each application utilizing the spread system needs to link with the 'libspread.a', 'libtspread.a', 'libtspread.lib' or 'libspread.lib' library and needs to use the calls defined in the 'sp.h' and 'sp_func.h' header files. Documentation for the interface is currenly incomplete, but a basic application must do at least the following: 1) Before using any other spread calls you need to 'connect' to a daemon by calling the SP_connect(...) call. This will set a mbox which is an integer representing your connection to the daemon. You use this when making other spread calls. You CAN connect to daemons multiple times from the same application (and we know of times when this is very useful). 2) If you want to receive messages from a group you need to call SP_join(). Groups are named with standard alphanumeric strings. 3) To send to a group you do NOT need to join it, just call SP_multicast() with an appropriate group name. You will not receive the messages back if you are not a member of the group. 4) To be nice to everyone else you should call SP_disconnect() when your program is finished using the spread system, if you do not your program's disconnection will eventually be detected. Some important observations when writing spread programs: 1) All the messages for a particular connection are received during an SP_receive() call. This includes messages from different groups and membership messages(like group leave and join), so you must demultiplex them yourself. This is a feature. You can opt to avoid getting membership messages all together by indicating that to SP_connect(). 2) Spread handles endian differences correctly for metadata and our headers but does NOT convert your data for you. It DOES tell you in the SP_receive call whether or not the data originated at machine with a different endianness then yours so you can easily use this to convert it yourself if necessary. 3) Each connection to a daemon has a unique 'private group' name which can be used to send a message to a particular, and only to a particular, application connection. 4) Spread supports the following ordering/delivery guarantees for messages: Unreliable (least) Reliable (will get there, no ordering) Fifo (reliable and ordered fifo by source) Causal (reliable and all mesg from any source of this level are causally ordered) Agreed (reliable and all mesg from any source of this level are totally ordered) Safe (Agreed ordering and mesg will not be delivered to application until the mesg has reached ALL receipients' daemons) For more detail, much of which IS important, see papers on Extended Virtual Sychrony, Transis and Totem. UTILITY PROGRAMS: ----------------- Spread comes with several demonstration programs and useful tools. 1) spmonitor: This program has a special interface to spread which allows it to control a machine group. It can terminate all the spread daemons in the group, change flow control parameters, and monitor the stats for one or all of the machines. It needs to know the spread.conf file used for a particular set of spread daemons. 2) spuser: This program is provided with source and uses all the functions of the spread interface. It also acts as a simple client which is useful for testing. Reading the source code to this program should show you how to use all the features of spread, and answer many questions about syntax details. 3) simple_user: This is just about the simplest spread program possible and is provided with source. It sends and receives text strings. 4) spflooder: This is used as one type of performace test. It 'floods' a spread group with data messages. It is provided with source. CONTACT: -------- If you have any questions please feel free to contact: commedia@cnds.jhu.edu yairamir@cs.jhu.edu jonathan@cs.jhu.edu