Description

Coordinator
Oct 9, 2008 at 3:02 PM

 

TABLE OF CONTENT

1.     Introduction. 2

1.1.      Purpose. 2

1.2.      Definitions, acronyms and abbreviations. 2

1.3.      References. 2

1.4.      Document overview.. 2

2.     General description. 2

2.1.      Canals/Ports paradigm.. 2

2.2.      Ports. 2

2.2.1.       Output ports. 2

2.2.2.       Input ports. 3

2.3.      Canals. 3

2.4.      Connection Scheme Description Language (CSDL) 3

2.5.      General Architecture. 3

3.     Platform support 4

4.     Functional Requirements. 4

4.1.      General Implementation Remarks. 4

4.2.      Ports/Canals Data Exchange Protocol 4

4.2.1.       General Packet Structure. 4

4.2.2.       CM_DATA message. 4

4.2.3.       CM_EOS message. 4

4.2.4.       CM_READY message. 4

4.2.5.       CM_CLOSE. 5

4.3.      Base Port class. 5

4.4.      PortOut class. 5

4.5.      PortIn class. 5

4.6.      Canal Manager Server 6

4.6.1.       General Remarks. 6

4.6.2.       Canal Manager Administrative Console. 7

4.7.      Canal Manager Local Service. 7

4.8.      Component 7

4.8.1.       General Remarks. 7

4.8.2.       Static Ports. 7

4.9.      General CSDL Implementation Remarks. CSDL Editor 8

5.     Examples. 8

5.1.      “Plus/Minus” Example. 8

5.2.      “Chat” Example. 9

5.3.      “SortMerge” Example. 11

6.     Further Extensions. 12

6.1.      Auto-uploading of components. 12

6.2.      Port - arrays. 12

6.3.      Sub-applications. 12

6.4.      Starting CM applications from local computer 12

 

 

 

 

 

Coordinator
Oct 9, 2008 at 3:03 PM

1.                Introduction

1.1.                Purpose

Canal Manager Framework is intended to develop distributed applications based on Canals/Ports paradigm.

1.2.                Definitions, acronyms and abbreviations

UI – user interface.

CM – Canal Manager

1.3.                References

http://www.jpaulmorrison.com/fbp/2001paper.htm

1.4.                Document overview

This document contains the result of requirements investigation. One of the purposes assigned to the document is description of the external requirements, discovering internal requirements and developing technical decisions for Canal Manager Framework implementation. The document should allow provide reliable estimation and decrease possible risks on the implementation stage. During implementation this document serves as base to track requirements in accordance with implemented functionality.

2.                General description

The section describe in general Canal Manager Framework features.

2.1.                Canals/Ports paradigm

The general idea of the paradigm is building distributed application from components using canals and ports. Each component is an executable (running as separate process), that exposes several ports. Components can be run on different computers across network. Canal Manager Framework provides passing data from an output port of a component to several input ports of other components. Passing data from an output port to input ports provided with canals, connected with ports. Each canal connected with an output port and several input ports. The connection scheme of components is described with special Connection Scheme Description Language (CSDL).

2.2.                Ports

Each executable component has several input/output ports. A port can be input or output (bi-direction ports is not supported). Each port has a name.  Each port has a type and only data of the type can be input/output from/to the port.

2.2.1.        Output ports

Output ports are used to pass data from a component to a canal. The main method of the port is:

        Put (<type>  value)

Put method waits until all port-recipients processed previous value and as a result the canal became ready to accept new data.

Also output port has the event:

NextRequest - This event is raised if canal is ready for accepting new value (So, all ports – recipients have processed previous value)

 

2.2.2.        Input ports

Input ports are used to get data from the connected canal.

The main method of the port is:

 <type> Get()

The method waits until the new value is available and returns the received value.

(So, the port just waits until source port put new value into canal).

 

Also, input port has event:

          ValueReceived – is raised when new value is available.

2.3.                Canals

Canals are used to pass data from a source port (output port connected to the canal) to several destination ports (input ports, connected to the canal).

Coordinator
Oct 9, 2008 at 3:05 PM

1.1.                Connection Scheme Description Language (CSDL)

The CSDL language is used to describe connection scheme between ports of component.

The syntax is the following:

<scheme element1>,…, <scheme elementN>

 

where <scheme element> has the following syntax:

[<computer name>:] <component name> (<a port name1>=<a canal name1>,…,<a port nameN>=<a canal nameN>)

 

 

1.2.                General Architecture

The Canal Manager Framework consists of the following element:

         Canal Manager Server (CMS) – main element of the Framework. It accept connection scheme (in CSDL), ask the corresponding Canal Manager Local Services to run necessary component on local computers and to establish the necessary connection between canals and ports.

         Canal Manager Local Service (CMLS) – should be run on each local computer where components are located. Processes request from Canal Manager Sever to run component and to establish connection between ports and canals.

         Canal Manager Component (CMC) – an executable that runs by CMLS, input data through the input ports and output data through output ports.

 

2.                Platform support

Canal Manager Framework implementing only for Windows NT/2000/XP platform with .NET Framework installed.

 

3.                Functional Requirements

3.1.                General Implementation Remarks

1.                                      All data passed between elements of Canal Manager Framework via Berkeley sockets (System.Net.Sockets.Socket).

2.                                      Only serializable data types (marked with SerializableAttribute) can be passed via ports/canals (see System.Runtime.Serialization.Formatters.Binary.BinaryFormatter class for details)

Coordinator
Oct 9, 2008 at 3:06 PM

1.1.                Ports/Canals Data Exchange Protocol

The following protocol is to communicate between canals and ports.

1.1.1.        General Packet Structure

Each packet (message) in the protocol has the following structure:

MessageID       (Int32)  - id of the message (see the following sections for the possible values

SizeOfData       (Int32) – optional – size of passed Data element of the structure

Data                 (byte[]) – optional  - passed data

 

(Now only CM_DATA message contains SizeOfData, Data elements of the structure)

1.1.2.        CM_DATA message

The message is used to pass data between ports/canals and canals/ports. Data section of the packet contains serialized data from the source port.

 

1.1.3.        CM_EOS message

The message is used in canals/ports data exchange to mark end of stream.

 

1.1.4.        CM_READY message

The message is used in the communication destination ports – canals and canal – source port

1.                                      Destination port – canals: in this case the message means that destination port has processed the previous value and is ready to process next value.

2.                                      Canals – source port: in this case the message means that all destination ports connected to the canal are ready. And the canal is ready to accept new value from the source port.

 

 

1.1.5.        CM_CLOSE

Means CM should close the whole application.

The message can be send by any port. As a result the CM send the message to all ports of all components involved in the application. If all ports of a component receive the message (i.e. all ports of the component are closed) it usually results terminating of component’s instance.

1.2.                Base Port class

The class contains common functionality for PortIn and PortOut classes.

Protected class Port

{

    public Port(String name);

   public void QueryClose(); //send CM_CLOSE message

   public bool IsClosed; // true if CM_CLOSE message was received

   public event EventHandler Close;// fire when CM_CLOSE message was received

}

1.3.                PortOut class

Public class PortOut <T> :Port

{

      public PortOut <T>(String name);

      public Put(T val);    //Put the next value from the port; waits  until canal become ready ( CM_READY message);

      public bool PutEOS() ;//Puts CM_EOS message; the synchronization logic as for Put method

 

      public T Value   //Return the current value in the port

            {

                        get;                              

            }

       public void QueryClose(); //send CM_CLOSE message

 

      public event EventHandler  NextRequest; //fire when a CM_READY received

     public bool IsClosed; // true if CM_CLOSE message was received

     public event EventHandler Close;// fire when CM_CLOSE message was received

}

 

1.4.                PortIn class

This generic type represents input port.

Public class PortIn <T> :Port

{

      public PortIn <T>(String name);

       public T Get();    //Get the next value from the port; send CM_READY and wait for the next value

      public T Value   //Return the current value in the port

            {

                        get;                              

            }

      public bool IsEOS() ;//CM_EOS message was received

      public void QueryNext(); //send CM_READY message to canal

      public void QueryClose(); //send CM_CLOSE message

 

      public event EventHandler  ValueReceived; //fire when a next CM_DATA or CM_EOS received

     public bool IsClosed; // true if CM_CLOSE message was received

     public event EventHandler Close;// fire when CM_CLOSE message was received

}
Coordinator
Oct 9, 2008 at 3:09 PM

1.1.                Canal Manager Server

1.1.1.        General Remarks

 

In principle, there are two variants of implementation of canals in CM Framework.

Variant A : All canals resided  on CM Server. The variant is simpler, but provides less performance in the case if, for example, two communicated components are resided on two local (not server) computer. In this case they could pass data directly from one computer to another, but in the variant they will pass data through sever.

Variant B: CM Local Services are responsible for managing canals. The variant is more effective, but more complex.

In 1 version the variant A is proposed to implement. So, in the text below we will consider just variant A.

The CM Server is first started element of CM Framework. After starting of CM Server CM Local Services are started and establish connection with CM Server.

In general, the CM Server provides the following:

1.                                      Establishing connection with CM Local Services and registering the name of these services.

2.                                      Start CM application. This process included the following:

·         Parsing CSDL scheme of the application

·         Creating necessary canals for the application

·         Ask CM Local Services for run necessary components

·         Establish connections canals-ports for these components

3.                                      Support execution of the CM application: in general it includes passing data between canals and ports (see  Ports/Canals Data Exchange Protocol for more details)

 

4. Terminating CM application by user request

 

1.1.2.        Canal Manager Administrative Console

The Administrative Console provides the following functionality

1.         Show all CM Local Services connected to the CM Server

2.                   Start application

3.  Show all active applications

4.  Show all canals for given application

 

5. Show all ports connected to the given canal

6. Terminate application

 

1.2.                Canal Manager Local Service

CM Local Service is running on local computers where running components are located. Each CMLS has a unique name in the system. The name is used in CSDL to refer to the corresponding computer.

In general a CM Local Service provides the following functionality:

1.                                      Establish connection with CM Server just after starting

2.   Run components (by request from CM Server) and pass to components data that necessary to establish connection “component – CM Server” (more strictly “ports of components  - canals on CM Server”)

1.3.                Component

1.3.1.        General Remarks

 

Components are building blocks in CM Framework. The main feature of components: they contain ports.

In general a component provides the following functionality:

1.                                      Establish connection between ports and canals

2.                                      Processing data passed to in-ports and output data through out-ports

 

There are two types of components: single-instance and multi-instance.

Single-instance - means that a new process is started when the component required.

Multi-instance – means all instance of the component are resided in the single process.

1.3.2.        Static Ports  

Usually, in a multi-instance component each port has several instances (an instance for each instance of component). But in some cases it’s necessary to have only one instance of a port – common to all instances of component. Such port is called static port.

 

1.4.                General CSDL Implementation Remarks. CSDL Editor

XML is used for inner representation of CSDL schemes (so, the schemes are stored in XML files).

To create CM application with CSDL the special editor is used. The CSDL scheme in the editor is presented in the standard CSDL notation (see above). Also the editor contains list of all CM components installed with information about available ports for each component. With the editor user can choose a component from the list and place it on the editor window.

Coordinator
Oct 9, 2008 at 3:11 PM

1.                Examples

1.1.                “Plus/Minus” Example

The following example uses C# notation.

//Component Plus

Class Plus

{

            PortIn<int>  A;

             PortIn<int>  B;

             PortOut<int> C;

             Void Main()

            {

                        A.Get();

                        B.Get();

                        while (!A.IsEOS && !B.IsEOS) // check for end of stream

{

     C.Put(A.Value+B.Value)   // Put A+B into C port

                             A.Get();

                             B.Get();

}

C.PutEOS();

              }

 

}

//Component Minus

Class Minus

{

            PortIn<int>  A;

             PortIn<int>  B;

             PortOut<int> C;

             Void Main()

            {

                        A.Get();

                        B.Get();

 

                        while (!A.IsEOS && !B.IsEOS) // check for end of stream

{

     C.Put(A.Value-B.Value)  //Put A-B into port C

                             A.Get();

                             B.Get();

 

}

C.PutEOS();

              }

}

 

And connection scheme (fragment, in CSDL) for CM in the following form:

 

 

PLUS(A=X,B=Y,C=S),MINUS(A=U,B=S,C=RES)

 

(Here X,Y,S,U,S,RES are canals)

It means run process PLUS, MINUS, connect port A with canal X, port B with Y etc. We can see that canal S is used in C=S and B=S: it means that out port C of process PLUS is connected with input port B of process MINUS. De facto, this fragment calculates expression RES=U-(X+Y). So, if we send stream of int data to canals U,X,Y the result will be calculated for every element of these streams and produced new stream in result canal  - RES. This RES canal can be connected to some input port of some other component etc.

1.2.                “Chat” Example

The sample consists of the ChatServer and ChatClient components.

ChatServer accept messages from several clients (ChatClient), merge them into single stream and pass the stream back to all clients.

ChatServer

ChatServer is multi-instance component with two ports:

            CliMsg in-port – accept message from a client

            SrvMsg out-port – static port (common for all instances of the component). It used to pass back merged stream of messages

Public class ChatServer

{

    static PortOut<String> SrvMsg;

             PortIn<String>   CliMsg;

Void MainLoop()

            {

                        CliMsg.Get();

                        while(!CliMsg.IsEOS())

                        {

                                    SrvMsg.Put(CliMsg.Value);

                                    CliMsg.Get();

                        }                                  

            }

}

ChatClient

ChatClient component is used to pass a next message from the client and display messages from all clients (result of merging all client messages on the server)

The component has two ports:

OutMsg – out-port –message typed by client

InMsg –in-port – message to display

Public class ChatClient

{

            PortOut<String>  OutMsg;

            PortIn<String>  InMsg

            //called when the user input a message text (press OK button)

            Void OnInput(String msg)

            {

                        OutMsg.Put(msg)

            }

            void MainLoop()

            {          

                        InMsg.Get();

                        While(!InMsg.IsEOS())

                        {

                                    DisplayMsg(InMsg.Value) ;//the function displays the message on the screen                                

InMsg.Get();

                        }

            }

}

 

CSDL scheme

Here is scheme with three clients:

A:ChatClient(InMsg=All,OutMsg=Out1), ChatServer(SrvMsg=All,CliMsg=Out1),

B:ChatClient(InMsg=All,OutMsg=Out2), ChatServer(SrvMsg=All,CliMsg=Out2),

C:ChatClient(InMsg=All,OutMsg=Out3), ChatServer(SrvMsg=All,CliMsg=Out3),

 

Remark: As ChatServer is multi-instance component only one process started for ChatServer

 

1.3.                “SortMerge” Example

This example of a component that merges two sorted stream into the sorted stream (all stream sorted in ascending order).

There two input ports In1 and In2 and one output port Out.

Public class SortMerge

{

     PortIn<int> In1;

     PortIn<int> In2;

     PortOut<int> Out;    

     //Pass through rest of a stream to Out port

     Void PassRest(PortIn<int> src)

     {

          while(!src.IsEOS())

          {

              Out.Put(src.Value);

               Src.Get();

           }

           Out.PutEOS();

      }

      void MainLoop()

      {

            In1.Get();

             In2.Get();

            While(!In1.IsEOS() && !In2.IsEOS())

            {

                 if(In1.Value > In2.Value)

                  {

                        Out.Put(In2.Value)

                        In2.Get();

                   }

                  else

                 {

                        Out.Put(In1.Value)

                        In1.Get();

 

                 }                                         

            }          

           if(In1.IsEOS())

          {

               PassRest(In2);

          }

          else

          {

               PassRest(In1);

          }

       }

}

 

2.                Further Extensions

2.1.                Auto-uploading of components

All potentially necessary components are located on the server computer (where CM Server is run). If a component is required, but is not present on a local computer, then the mechanism supports uploading the necessary components from server computer to the corresponding local computer.

2.2.                Port - arrays

The extension provides possibility to use port-array. The port-array is special type of port that, de facto, contains several ports referred by index. The element of such array can be referred in CSDL scheme with using the following notation:

            <name of port>[<index>]

 The actual number of elements in such array equal to maximum index +1 used for the corresponding component in CSDL scheme.

2.3.                Sub-applications

The extension introduces possibility to create CM sub-application (or sub-scheme). Sub-scheme is a scheme with parameters that can be used in general scheme as like as components.

For example:

P(In , Out): C(InPort=In),D(OutPort=Out).

….

   In the example we declare sub-scheme P with parameters In and Out that uses two components C and D.

Now we can use the sub-scheme in another scheme using following syntax.

…P(In=Canal1, Out=Canal1)…

 

Thus, sub-scheme provides a way to simplify developing of complex scheme as like as procedures in usual language.

2.4.                Starting CM applications from local computer

[TBD]

In the basic variant CM applications are started from sever computer via CM Administrative Console. The starting from local computer provides a way to start CM application from local computer without direct using CM Administrative Console.