<?xml version="1.0" encoding="ISO-8859-1"?>

<!DOCTYPE book SYSTEM "/home/logilab/docbook/dtd/docbookx.dtd"
    [
     <!ENTITY legalnot SYSTEM "../legal_notice.en.xml">
    ] >


<!-- ############################################################ --> 
<!-- ###  Third Book : Extension Programmer Handbook          ### -->
<!-- ############################################################ -->


<book lang="en" id="mph.en">

  <bookinfo>

    <title><application>Narval</application> - Extension Programmer Handbook</title>

    <titleabbrev>EnAlModuleProg</titleabbrev>

    <authorgroup>
      <author>
	<firstname>Alexandre</firstname> <surname>Fayolle</surname>
      </author>
      <author>
	<firstname>Olivier</firstname> <surname>Cayrol</surname>
      </author>
     <author>
      <firstname>Sylvain</firstname>
      <surname>Thénault</surname>
     </author>
    </authorgroup>

    <abstract>
      <para><application>Narval</application>'s kernel is able to
	manipulate plans that can perform almost any task, provided that the
	required actions and transformations are available. The purpose of this
	document is to explain how actions and transformations are
	written.</para>
    </abstract>

    <copyright>
      <year>2000-2005</year>
      <holder>Logilab</holder>
    </copyright>
    <copyright>
      <year>2004-2005</year>
      <holder>DoCoMo Euro-Labs GmbH</holder>
    </copyright>

    &legalnot;

    <revhistory>
      <revision>
	<revnumber>$Revision: 1.4 $</revnumber>
	<date>$Date: 2001/10/16 12:59:35 $</date>
	<authorinitials>$Author: alf $</authorinitials>
      </revision>
    </revhistory>
    
  </bookinfo>

<!-- *********************************************************************** -->

  <preface id="mdh-preface.en"><title>Intended audience</title>

    <para>This document is aimed at programmers who want to code
    modules for <application>Narval</application>.  The preference
    language for coding modules is Python, and some knowledge of the
    python programming language is assumed. Having a previous
    experience in python programming and a good acquaintance with the
    python standard library definitely
    helps. <application>Narval</application> represents the data it
    manipulates using python objects. Usually those objects should be
    exportable/importable to/from XML. <!-- A short introduction to
    XML can be found in <xref linkend="mdh-dom.en"/>
    .--></para>

  </preface>

<!--
     ***********************************************************************
     -->

  <preface id="mdh-amw.en">

    <title>Introduction to narval extensions programming</title>

    <para><application>Narval</application> is a powerful system. Yet, left
    alone it is as dumb as a computer with only a bare bone operating
    system. Modules and actions provide means for
    <application>Narval</application> to interact with the Outside World Where
    It's Cold. Actions can be very simple or very complex, they can do the
    processing themselves or act as interfaces to outside programs, enabling
    different proprietary pieces of software to talk to each other.</para>

    <para>Why should you write modules? Well, basically, there are two families
    of reasons. The first one is that you need to do something with
    <application>Narval</application>, and nothing exists yet to do it. This
    situation is likely to become less frequent as time passes, but since
    <application>Narval</application> is still young it may be the case. So you
    have checked the repositories, asked a few questions on the mailing lists,
    and it looks like you have found something missing. Well in that case, it is
    time to sharpen your Swiss army coding knife and confront yourself with
    writing the missing action. This book is here to help you, and so are the
    various mailing lists about <application>Narval</application>. </para>

    <para>The second one is that you may have written a new program, and you
    think that the <application>Narval</application> system is
    <emphasis>soooooo</emphasis> cool that you just <emphasis>have</emphasis> to
    write a module to interface your program with
    <application>Narval</application>, so that other users may pilot your
    program through recipes. This book is here to help you in that case
    too.</para>

    <para>Modules are used to extend narval with new interfaces, new
    elements and new actions. Interfaces and elements are used to
    describe physical or logical objects that can be manipulated by
    the narval interpreter. Actions are used to get a specific
    behaviour, optionaly using input elements and creating some others
    as output.</para>

    <para>In both cases, we thank you for contributing your efforts to the
    <application>Narval</application> user community.</para>

  </preface>

<!-- *********************************************************************** -->

  <chapter id="mdh-iae.en"> <title>Interfaces, adapters and elements</title>

      <para>Everything in narval is based on what we call <emphasis>elements</emphasis>.
      Those elements are used to hold any information but also to describe a 
      desired behaviour of the agent, using elements such as actions and
      recipes, or to control it, using elements such as start-plan, plans, etc... 
      All of them are manipulated by the interpreter and contained in it's
      <emphasis>memory</emphasis>.</para>

      <para>Elements are actually python class instances, and so defined in python 
      modules and (usually automatically) registered to the interpreter. Also, for a
      greater flexibility, the concepts of <emphasis>interface</emphasis> and
      <emphasis>adapter</emphasis> are used:
        <itemizedlist>
          <listitem><para>an interface is an abstract definition of available
	  attributes and methods that should define an element to provide a
	  logical set of data and services</para></listitem>
	  <listitem><para>an element implements some interfaces by having the required
	  attributes and methods. The implemented interfaces are explicitly
	  listed</para></listitem>
	  <listitem><para>an adapter is glue component providing a way adapt an interface to
          another</para></listitem>
	</itemizedlist>

      Interfaces and adapters, as elements, are defined by classes in python
      modules and registered to the interpreter.
      </para>

    <section id="mdh-iae-ia.en"><title>Writing a new interface and adapter</title>
      <para>Interfaces and adapter are usually defined in modules
      within the "interfaces" subpackage of narval. An interface must
      inherit (possibly indirectly) from the narval.public.Interface
      class, while an adapter must inherit (also possibly indirectly)
      from the narval.public.Adapter class.</para>
      
      <example id="ex-interface"><title>Defining an interface for URL object</title>
	<para>Suppose we want to describe a URL element. This element
	should contains the URL address and an optional attribute
	giving the encoding of the file locating at the address. We
	also want some methods to access to the different part of the
	url address or to do some actions on it such as normalizing
	it.</para>
        <para>This example is taken from the narval standard
	library. Notice that, by convention, all classes which are
	actually interfaces have a name beginning with a capitalized
	I.</para>

	<programlisting>
<![CDATA[
from narval.public import Interface

class IURL(Interface):
    """interface for url elements
        
    :ivar address: the URL string
    :ivar encoding: optional encoding of the file located at <address>
    """
    def normalize(self):
        """return the expanded normalized url string"""
    def protocol(self):
        """return the normalized url string"""
    def path(self):
        """return the expanded normalized path string"""
]]>
	</programlisting>

	<para>You can notice 3 main points: 
          <itemizedlist>
            <listitem><para>the class inherits from Interface</para></listitem>
            <listitem><para>attribute are defined in the class'docstring 
		(mainly because of a lack of the underlying interface
		implementation)</para></listitem>
            <listitem><para>methods are defined using regular
		python methods but with only a docstring documenting
		the method's aim as body</para></listitem>
	  </itemizedlist>
        </para>

      </example>
      
      <para>An adapter is used to "transform" an object implementing
      an interface into another one implementing another
      interface. They are usually defined in the same module as the
      source or/and target interface.</para>

      <example id="ex-adapter"><title>Defining an adapter from IURL to IOpen</title>
	<para>The standard library contains the IURL interface as
	defined above, but also a IOpen interface used to describe
	"openable" object such as file. We can easily provide a way to open a URL
	element using an adapter from IURL to IOpen.</para>
        <para>This example is taken from the narval standard
	library. Notice that, by convention, all classes which are
	actually adapters have a name using the pattern
	SourceInterfaceToTargetInterface.</para>

	<programlisting>
from narval.public import Adapter, Interface
import urllib2

class IOpen(Interface):
    """open anything and return a file-like object
    """
    def open(self):
        """return a readable file-like object"""

class IURLToIOpen(Adapter):
    """adapt IURL to IOpen"""
    __sources__ = (IURL,)
    __implements__ = (IOpen,)
        
    def open(self):
        """return a file-like object from an IURL object"""
        return urllib2.urlopen(self.original.normalize())
	</programlisting>

	<para>You can notice 4 main points: 
          <itemizedlist>
            <listitem><para>the class inherits from Adapter</para></listitem>
            <listitem><para>source interfaces (may be multiple) are
	    listed using the __sources__ attribute</para></listitem>
            <listitem><para>target interfaces (may be multiple) are
	    listed using the __implements__ attribute</para></listitem>
            <listitem><para>the adapted object is stored by the base
	    class in the <emphasis>original</emphasis> attribute</para></listitem>
	  </itemizedlist>
        </para>

      </example>

    </section>


    <section id="mdh-iae-e.en"><title>Writing a new element</title>
      <para>As you've seen in the earlier sections, interfaces are
      used to describe some data and services, while adapters are used
      to pass from an interface to another. But none of those classes
      defines concrete element living in the narval's memory. To do so
      you have to define another class, called
      <emphasis>element</emphasis> in this document.</para>

      <para>Elements are usually defined in modules within the
      "elements" subpackage of narval. Every narval elements should
      implement an internal interface (allowing for instance automatic
      xml marshalling). To ease things, narval provide a base class
      implementing this internal interface and other facilities to
      make development of new elements easier, the ALElement
      class.</para>
      
      <example id="ex-element"><title>Defining an element class implementing IURL</title>
	<para>The best way is probably to start with an example. You
	can see below a simple element definition implementing the
	IURL interface:</para>

	<programlisting>
from narval.public import NO_NS, normalize_url
from narval.element import NSAttribute, ALElement
from narval.interfaces.base

class URLElement(ALElement):
    """IURL implementation"""
    __implements__ = (IURL,)    
    __xml_element__ = (NO_NS, 'url')
    address = NSAttribute(NO_NS, None, str, str)
    encoding = NSAttribute(NO_NS, None, str, str)

    def raw_address(self):
        return self.address
    
    def normalize(self):
        """return the expanded normalized url string"""
        return normalize_url(self.raw_address())[0]

    def protocol(self):
        """return the normalized url string"""
        return normalize_url(self.raw_address())[1][0] or 'file'

    def path(self):
        """return the expanded normalized path string"""
        return normalize_url(self.raw_address())[1][2]
	</programlisting>

	<para>You can notice here, well, many points: 
          <itemizedlist>
            <listitem><para>the class inherits from ALElement</para></listitem>
            <listitem><para>implements interfaces (may be multiple) are
	    listed using the __implements__ attribute</para></listitem>
            <listitem><para>the __xml_element__ attribute is used to
	    defined that this element will be serialized into the xml
	    "url" element, within an empty namespace
	    (NO_NS)</para></listitem>
            <listitem><para>attributes are usually defined using the
	    NSAttribute property, which allow automatic marshalling of
	    the attribute to and from xml. In the example for
	    instance, we tell that the element have a "address"
	    attribute which is in an empty xml namespace, has None as
	    default value, and is deserializable and serializable
	    using the same "str" function (a python
	    builtin).</para></listitem>
	  </itemizedlist>
        </para>

        <para>For more information you should take a look at the
	ALElement implementation and at existing elements in the
	standard library.</para>
      </example>
     
    </section>


    <section id="mdh-iae-miae.en"><title>Manipulating interfaces, adapters and elements</title>
      <para>As interfaces and element classes are the base criteria to
      filter elements in prototypes, you need to know a few things
      about how to manipulate them. Notice that every interfaces and
      element classes name are available as identifier in the
      evaluation context of filter expressions.</para>

      <para>The expression <methodname>isinstance(elmt,
      AClass)</methodname> will be evaluated to True if the element
      is an instance of the AClass class (i.e. it's an instance of
      this class or of a children class).</para>

      <para>The expression <methodname>implements(elmt,
      IFace)</methodname> will be evaluated to True if the element
      is an instance of the a class implementing the IFace interface
      (i.e. it implements the IFace interface itself or a children
      interface). Notice that <emphasis>implements</emphasis> doesn't
      consider adaptation, that means that even if an adapter exists
      for an interface implemented by the element to the IFace
      interface, <emphasis>implements</emphasis> will be evaluated to
      False unless the element explicitly implements IFace.</para>

      <para>The expression <methodname>IFace(elmt)</methodname>
      will be evaluated to the element itself or to an adapted element
      if the element implements explicitly IFace or if an adapter for
      an interface implemented by the element to the IFace interface
      exists. In other cases, this will raise an error (and so in the
      context of filter expression evaluation, the element will be
      skipped, which is the desired behaviour).</para>
    </section>

  </chapter>

<!-- *********************************************************************** -->

  <chapter id="mdh-wa.en"> <title>Writing an action</title>
    
      <para>The action is the fundamental brick used by
      <application>Narval</application> to perform tasks. It is composed of two
      parts: an XML prototype and a python stub, both of which appear in the
      same file. An action is included in a python module.</para>

      <para>Since release 1.2, Narval use xml namespaces, and so
      action and prototype definition should belong to a narval
      specific namespace:
      <emphasis>'http://www.logilab.org/namespaces/Narval/1.2'</emphasis>. 
      Usualy this namespace is bound to the <emphasis>al</emphasis>
      prefix. This convention will be used in this document.</para>

    <section id="mdh-wa-wp.en"><title>Writing a prototype</title>

      <section id="sec-action-dtd"><title>The prototype DTD</title>

      <programlistingco>
	<areaspec>
	  <area id="dtd-action-element-action" coords='1'
		linkends="dtd-action-element-action-co"/>
	  <area id="dtd-action-attribute-name" coords='2'
		linkends="dtd-action-attribute-name-co"/>
	  <area id="dtd-action-attribute-func" coords='3'
		linkends="dtd-action-attribute-func-co"/>
	  <area id="dtd-action-element-description" coords='5'
		linkends="dtd-action-element-description-co"/>
	  <area id="dtd-action-element-input" coords='8'
		linkends="dtd-action-element-input-co"/>
	  <areaset id="dtd-action-attribute-optional" coords="" >
	    <area id="dtd-action-attribute-optional-1" coords='9'
		  linkends="dtd-action-attribute-optional-co"/>
	    <area id="dtd-action-attribute-optional-2" coords='18'
		  linkends="dtd-action-attribute-optional-co"/>
	  </areaset>
	  <area id="dtd-action-attribute-use" coords='10'
		linkends="dtd-action-attribute-use-co"/>
	  <area id="dtd-action-attribute-list" coords='11'
		linkends="dtd-action-attribute-list-co"/>
	  <area id="dtd-action-attribute-outdates" coords='12'
		linkends="dtd-action-attribute-outdates-co"/>
	  <areaset id="dtd-action-attribute-id" coords="" >
	    <area id="dtd-action-attribute-id-1" coords='15'
		  linkends="dtd-action-attribute-id-co"/>
	    <area id="dtd-action-attribute-id-2" coords='19'
		  linkends="dtd-action-attribute-id-co"/>
	  </areaset>
	  <area id="dtd-action-element-output" coords='17'
		linkends="dtd-action-element-output-co"/>
	  <area id="dtd-action-element-match" coords='21'
		linkends="dtd-action-element-match-co"/>
	</areaspec>

	  <programlisting><![CDATA[<!ELEMENT action (description*|input*|output*)>
<!ATTLIST action name CDATA #REQUIRED>
<!ATTLIST action func CDATA #REQUIRED>

<!ELEMENT description (#PCDATA)>
<!ATTLIST description lang CDATA #REQUIRED>

<!ELEMENT input (match*)>
<!ATTLIST input optional  (yes|no) "no">
<!ATTLIST input use       (yes|no) "no">
<!ATTLIST input list      (yes|no) "no">
<!ATTLIST input outdates  (yes|no) "no">
<!ATTLIST input from_context  (step|plan|parent_plan|memory) "step">
<!ATTLIST input to_context    (step|plan|parent_plan|memory) "memory">
<!ATTLIST input id        CDATA   #IMPLIED> 

<!ELEMENT output (match*)>
<!ATTLIST output optional (yes|no) "no">
<!ATTLIST output id        CDATA   #IMPLIED> 

<!ELEMENT match (#PCDATA)>
]]></programlisting>

	<calloutlist>

	  <callout id="dtd-action-element-action-co" arearefs="dtd-action-element-action">

	    <para>the <sgmltag class='element'>action</sgmltag> element is used as
	    a container for the action prototype. It holds a number of <sgmltag
	    class='element'>input</sgmltag> and <sgmltag
	    class='element'>output</sgmltag> child nodes. </para>

	  </callout>

	  <callout id="dtd-action-attribute-name-co" arearefs="dtd-action-attribute-name">

	    <para><sgmltag class='attribute'>name</sgmltag> is the name of the
	    action. It is used to identify an action in a recipe. To avoid name
	    clashes, the name of the module is prepended and used as a
	    namespace, so the name of an action needs only to be unique within a
	    given module.</para>

	  </callout>

	  <callout id="dtd-action-attribute-func-co" arearefs="dtd-action-attribute-func">

	    <para><sgmltag class='attribute'>func</sgmltag> is the name of the
	    python function that implements the action stub. This function
	    <emphasis>must</emphasis> be in the same module. The name is
	    often the name of the action prefixed with
	    <literal>act_</literal>.</para>

	  </callout>

	  <callout id="dtd-action-element-description-co" arearefs="dtd-action-element-description">

	    <para><sgmltag class='element'>description</sgmltag> elements are used
	    to provide useful information about the action. The <sgmltag
	    class='attribute'>lang</sgmltag> attribute is used to specify the
	    language of the description. It is useful for graphical interfaces and 
	    documentation generation which can use this for localization.</para>

	  </callout>

	  <callout id="dtd-action-element-input-co" arearefs="dtd-action-element-input">

	    <para>the <sgmltag class='element'>input</sgmltag> element describes
	    one input of the action. This description is a list of <sgmltag
	    class='element'>match</sgmltag> elements each of which contain an
	    python expression. In order to be accepted for the input,
	    an element must match all these expressions.</para>

	  </callout>

	  <callout id="dtd-action-attribute-optional-co" arearefs="dtd-action-attribute-optional">

	    <para>if <sgmltag class='attribute'>optional</sgmltag> is set to
	    <literal>yes</literal>, then the lack of element matching the
	    <sgmltag class='element'>input</sgmltag> or the <sgmltag
	    class='element'>output</sgmltag> will not cause an error.</para>

	  </callout>

	  <callout id="dtd-action-attribute-use-co" arearefs="dtd-action-attribute-use">

	    <para>if <sgmltag class='attribute'>use</sgmltag> is set to
	    <literal>yes</literal> this means that once an element has been
	    passed as an input to the action, it will be flagged as used and not
	    be reused by the action in subsequent evaluations of a plan
	    instantiated from the same recipe</para>

	  </callout>

	  <callout id="dtd-action-attribute-list-co" arearefs="dtd-action-attribute-list">

	    <para>if <sgmltag class='attribute'>list</sgmltag> is set to
	    <literal>yes</literal>, several arguments matching the input can be
	    passed to the action. This implies that the function stub is coded
	    accordingly.</para>

	  </callout>

	  <callout id="dtd-action-attribute-from_context-co" arearefs="dtd-action-attribute-list">

	    <para>the <sgmltag class='attribute'>from_context</sgmltag> maybe used
	    to control the memory area to use for the matching
	    elements lookup. This attribute defines the starting
	    context: from the step, the plan, the parent_plan or the memory.</para>

	  </callout>

	  <callout id="dtd-action-attribute-to_context-co" arearefs="dtd-action-attribute-list">

	    <para>the <sgmltag class='attribute'>to_context</sgmltag> maybe used
	    to control the memory area to use for the matching
	    elements lookup. This attribute defines the stoping
	    context: to the step, the plan, the parent_plan or the memory.</para>

	  </callout>

	  <callout id="dtd-action-element-output-co" arearefs="dtd-action-element-output">

	    <para>the <sgmltag class='element'>output</sgmltag> element describes
	    one output of the action. This description is a list of <sgmltag
	    class='element'>match</sgmltag> elements each of which contain an
	    XPath. <application>Narval</application> considers that an action
	    has failed if each output is not matched exactly once by one
	    different element output by the action (with the possible exception
	    of <sgmltag class='attribute'>optional</sgmltag> outputs.  all these
	    XPaths.</para>

	  </callout>

	  <callout id="dtd-action-attribute-outdates-co" arearefs="dtd-action-attribute-outdates">

	    <para>if <sgmltag class='attribute'>outdates</sgmltag> is set to
	    <literal>yes</literal>, once an element has been passed as an input
	    to the action, it will be flagged as outdated, and not be reused by
	    any action or transition in Narval. You can think of it as a super
	    <sgmltag class='attribute'>use</sgmltag> attribute, that affects all
	    the recipes, and not just the recipe that instanciated the current
	    plan.</para>

	  </callout>

	  <callout id="dtd-action-attribute-id-co" arearefs="dtd-action-attribute-id">

	    <para>The <sgmltag class='attribute'>id</sgmltag> attribute is used to
	    provide a way for steps to alter the prototype of the action, by
	    modifying the attributes of the input, or by adding <sgmltag
	    class='element'>match</sgmltag> nodes to the
	    prototype. It's also used in the action's stub to retreive
	    arguments corresponding to a given input.</para>

	  </callout>

	  <callout id="dtd-action-element-match-co" arearefs="dtd-action-element-match">

	    <para>A <sgmltag class='element'>match</sgmltag> contains an python
	    expression that describes an aspect of the expected
	    element. In this expression, the <emphasis>elmt</emphasis>
	    identifier is used to represent the element matched
	    against the expression. Notice that as inputs are
	    evaluated in the order of their definition in the xml, the
	    identifier of previously matched inputs will be bind to
	    the matched element(s) and so can be used to match
	    interdependant inputs.</para> 

	  </callout>

        </calloutlist>

	</programlistingco>

      </section>

<!-- *********************************************************************** -->

      <section><title>Examples</title>

	<para><xref linkend="ex-action-NOP"/> illustrates a minimal action: its
	takes no inputs, outputs nothing either. It could however, depending on
	what is in the function stub, have an effect. For example, it could be
	used to increment a hit counter on a web page. If no description is
	provided, it is not possible to tell what an action does, especially if
	the action name is not explicit. </para>

	<example id="ex-action-NOP"><title>the NOP action prototype</title>

	  <programlisting>
<![CDATA[
	  <al:action name='NOP' func='act_NOP'/>
]]>
</programlisting>

	</example>

	<para><xref linkend="ex-action-makemail"/> presents a typical action. A
	description is provided in English and in French. We notice that both
	inputs have a 'use' attribute: this is because we do not want to reuse
	the same header over and over again to produce an endless suite of
	identical mails. The <sgmltag class='element'>match</sgmltag> elements
	used in the prototype are self explanatory.</para>


	<example id="ex-action-makemail"><title>the make_mail action
	prototype</title>

	  <programlisting>
<![CDATA[
<al:action name='make_mail' func='act_make_mail'>
    <al:description lang='en'>Builds an email element given an element
    implementing (or adaptable to) IEmailAddress and another one
    implementing (without considering adaptation) IData.
and an email body</al:description>
    <al:description lang='fr'>Construit un élément email à partir d'un élément
implémentant (ou adaptable) l'interface IEmailAdress (adresse
électronique) et d'un élément implémentant IData (corps du message)</al:description>
    <al:input use='yes'>
        <al:match>IEmailAdress(elmt)</al:match>
    </al:input>
    <al:input use='yes'>
        <al:match>implements(elmt, IData)</al:match>
    </al:input>
    <al:output>
        <al:match>IEmail(elmt)</al:match>
    </al:output>
</al:action>

]]>
</programlisting>

	</example>

	<para><xref linkend="ex-action-httpgetext"/> shows a complex action: two
	out of the three input arguments are optional and the <sgmltag
	class='element'>match</sgmltag> for the first argument is much more
	elaborated than those we have seen so far. If you are not yet familiar
	with python, here is what it means: we are looking for a
	element implementing <sgmltag
	class='element'>IHTTPRequest</sgmltag> with at least a non
	null <sgmltag class='element'>url</sgmltag> attribute and a,
	also non null, <sgmltag class='element'>header</sgmltag>
	attribute.</para> 

	<example id="ex-action-httpgetext"><title>the http_get_ext action
	prototype</title>

	  <programlisting>
<![CDATA[
<al:action name='http_get_ext' func='http_get_ext_f'>
    <al:description lang='en'>Fetches a page on the web using an optional proxy,
and optionally filtering spam out</al:description>
    <al:description lang='fr'>Ramène une page depuis le Web, en passant par un
proxy (optionnel), et en supprimant le spam (optionnel)</al:description>
    <al:input>
        <al:match>IHTTPRequest(elmt).url and IHTTPRequest(elmt).header</al:match>
    </al:input>
    <al:input optional="yes">
        <al:match>IProxy(elmt).type == 'http'</al:match>
    </al:input>
    <al:input optional="yes">
        <al:match>ISpamPolicy(elmt)</al:match>
    </al:input>
    <al:output>
        <al:match>IHTTPResponse(elmt)</al:match>
    </al:output>
</al:action>'''
]]>
</programlisting>

	</example>
	
      </section>

    </section>

<!-- *********************************************************************** -->

    <section id="mdh-wa-cas.en"><title>Coding the action stub</title>

      <section><title>Prototype of a stub</title>
	
	<para>An action stub is a python function. Methods will not work,
	because there is no way to pass the object along with the
	call<footnote><para>Well, at least, this is what we believe. If there is
	a python hack that would do the trick, it's fine, but unless it is a
	very clean hack, we do not intend to support it.</para></footnote>. This
	function will be called passing one and only one argument, which will be
	a dictionary with input identifiers as key and matched
	elements for the input as value.</para>

      </section>

<!-- *********************************************************************** -->

      <section><title>Retrieving inputs</title>

	<para>If an input has the list attribute set to "yes", the
	value associated to the input identifier in the dictionary
	will be a list of matched elements. If not, it will be the
	matched element or None if the input is optional and has no
	matched element..</para> 
	  
	<para>It's so very easy to get elements associated to each
	input, as shown in the example below :</para>

	  <example id="ex-retrieve-single"><title>Retrieving inputs</title>

	    <para>Given the following action prototype:</para>

<programlisting>
<![CDATA[
<al:action name='dance-boogie-woogie' function='act_dance_boogie_woogie'>
    <al:input id='tempo'>
        <al:match>elmt.tempo</al:match>
    </al:input>
    <al:input id='dancers' list='yes'>
        <al:match>IDancer(elmt)</al:match>
    </al:input>
    <al:input id='song' optional='yes'>
        <al:match>ISong(elmt)</al:match>
    </al:input>
</al:action>
]]>
</programlisting>

	    <para>We would write the following code in the stub to bind the
	    various input elements to python identifier:</para>

<programlisting>
def act_dance_boogie_woogie(inputs):
    # access to the tempo element and get the value of its tempo
    # attribute 
    tempo = inputs['tempo'].tempo 
    # get dancer elements
    # as it's a non optional list, the dancers identifier will be
    # bound to a list with at least one dancer element
    dancers = inputs['dancers']
    # get the song element
    # as it's a optional element, the song identifier will be bound to
    # the song element or to None if no such element was found
    song = inputs['song']
    # do some stuff now

	</programlisting>

	  </example>

      </section>

      <section><title>Processing</title>

	<para>Well basically, you can do anything you want here: create arbitrary
	elements, modify elements you got as input, read and write
	files on hard disk, get a web page...</para> 

	<section><title>Writing to disk</title>
	  
	  <para>Nothing prevents you from writing anywhere on the disk, apart
	  from Operating System restrictions.</para>
	  
	</section>
<!--	
	<section><title>Sockets and <application>Narval</application>'s Socket
	Manager</title>
	  
	  <para>There is a issue with interprocess communication using sockets
	  within actions. As far as an action can <function>open</function> a
	  connection, do some processing with it and <function>close</function>
	  it by itself, everything goes fine. Things get a bit more difficult
	  when a connection is to be shared between generic actions. Such
	  actions are expected to take a <sgmltag class='element'>socket</sgmltag>
	  input. A <sgmltag class='element'>socket</sgmltag> has a <sgmltag
	  class='attribute'>port</sgmltag> attribute giving the port on which the
	  communication is taking place. The action should connect to this port
	  on <varname>localhost</varname> to communicate with the outside. Here
	  is some sample code taken from
	  <function>Http.Write_back_to_socket</function>:</para>

	  <programlistingco>

	    <areaspec>

	      <area id="sf-get-socket" coords='6' linkends="sf-get-socket-co"/>

	      <area id="sf-create-socket" coords='9'
	      linkends="sf-create-socket-co"/>

	      <area id="sf-connect-socket" coords='10'
	      linkends="sf-connect-socket-co"/>

 	      <area id="sf-use-socket" coords='11' linkends="sf-use-socket-co"/>

	      <area id="sf-close-socket" coords='24'
	      linkends="sf-close-socket-co"/>

	      <area id="sf-return" coords='25' linkends="sf-return-co"/>

	    </areaspec>

	    <programlisting>
##
## Write back to socket
##

def Write_back_to_socket_f(args) :
    sock = select_one(args,'socket')
    hi = select_one(args,'http-response')
    import socket
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect('localhost',int(sock.getAttributeNS(AL_NS,'port')))
    s.send('HTTP/1.0 '+hi.getAttributeNS(AL_NS,'errcode')+' '+\
                       hi.getAttributeNS(AL_NS,'errmsg')+'\r\n')
    data = select_one(hi,'headers/text()').data
    s.send(data)
    if data[-1] != '\n' :
        s.send('\r\n\r\n')
    else:
        s.send('\r\n')
    try:
        s.send(select_one(hi,'content/text()').data)
    except SelectException:
        pass
    s.send('\r\n')
    s.close()
    return args.ownerDocument.createElement('outputs')

<![CDATA[
MOD_XML=MOD_XML+'''
<al:action name='Write_back_to_socket' func='Write_back_to_socket_f'>
  <al:description lang="en">Send back response to client</al:description>
  <al:description lang="fr">Renvoie la réponse au client</al:description>
  <al:input use="yes"><al:match>http-response</al:match></al:input>
  <al:input><al:match>socket</al:match></al:input>
</al:action>'''
]]>
</programlisting>
	  <calloutlist>

	    <callout id="sf-get-socket-co" arearefs="sf-get-socket">

	      <para>We start by retrieving the socket element as described
	      previously. This element has a <sgmltag
	      class='attribute'>port</sgmltag> attribute which tells us on which
	      port we can communicate with the socket forwarder</para>

	    </callout>

	    <callout id="sf-create-socket-co" arearefs="sf-create-socket">

	      <para>Then we create a python socket object, using the standard
	      method&hellip;</para>

	    </callout>

	    <callout id="sf-connect-socket-co" arearefs="sf-connect-socket">

	      <para>&hellip;and we connect it to
	      <varname>localhost:port</varname>, which is the address of the
	      socket forwarder.</para>

	    </callout>

	    <callout id="sf-use-socket-co" arearefs="sf-use-socket">

	      <para>We can then use the socket normally. Depending on the
	      action, we shell <function>send</function> or
	      <function>recv</function> data. </para>

	    </callout>

	    <callout id="sf-close-socket-co" arearefs="sf-close-socket">

	      <para>One we have finished, it's time to close the
	      connection. <emphasis>Do not forget this step</emphasis>.</para>

	    </callout>

	    <callout id="sf-return-co" arearefs="sf-return">

	      <para>As for any action, we return an <sgmltag
	      class='element'>outputs</sgmltag> element. In this case, the action
	      has no declared outputs, so the element is empty. Refer to <xref
	      linkend="sec-return-results"/> for more information.</para>

	    </callout>

	  </calloutlist>

</programlistingco>
	  
	</section>
	
	<section><title>Memory management consideration</title>
	  
	  <para> You can create new elements within a stub. New
	  nodes are created using the <varname>ownerDocument</varname> member
	  variable of the argument passed to the stub as a node factory. In
	  order to avoid leaking memory, it is a good thing to attach the new
	  node to the document, or to release it using the
	  <function>ReleaseNode</function> function in
	  <classname>xml.dom.ext</classname> when you no longer need it. Even
	  better, do both (see <xref linkend="ex-create-and-release-node"/>), so
	  that if the stubs generates an exception before the element is
	  released, <application>Narval</application> will be able to cope with
	  the element.</para>

	  <note><title>Exception</title>
	    
	    <para>The <sgmltag class='element'>outputs</sgmltag> element that is
	    returned by the stub (See <xref linkend="sec-return-results"/>)
	    should not have a parent node.</para>

	  </note>

	  <para>More examples of node creation can be found in <xref
	  linkend="sec-return-results"/>.</para>

	  <example id="ex-create-and-release-node"><title>Avoiding memory leaks
	  with DOM</title>

<programlisting>
def myFunction_f(argument):
    # argument extraction code skipped

    # creating a new element
    doc = argument.ownerDocument 
    new_element = doc.createElementNS(None,'element')
    # attach the element to the document
    doc.documentElement.appendChild(new_element)

    # some processing skipped

    # release the element
    from xml.dom.ext import ReleaseNode
    ReleaseNode(new_element)
</programlisting>

	    </example>
	  
	</section>
 -->
	
      </section>

      <section id="sec-return-results"><title>Returning the results</title>

	<para>Obviously, once the processing is done, we want to return a
	result. <application>Narval</application> expects to get
	returned a dictionary containing output elements. As for
	inputs, the output dictionary has output identifiers for keys
	and associated elements for values. The same rules as for
	inputs apply (i.e. list or None value according to the value
	of the list and optional attributes). Moreover you can omit
	entries for optional output without element associated.</para>

	<example id="ex-build-outputs-by-block"><title>building the outputs</title>

<programlisting>
<![CDATA[
<al:action name='pastry' function='act_pastry'>
    <al:output id='muffin'>
        <al:match>isinstance(elmt, muffin)</al:match>
    </al:output>
</al:action>
]]>
</programlisting>

	    <para>We could write the following code:</para>

<programlisting>
def act_pastry(arg)
    # no inputs to read
    
    # we choose the flavour of the muffin
    from random import choice
    flavour = choice(['pumpkin','raisin', 'blueberry'])

    # build the output
    muffin_elmt = muffin(flavour=flavour)

    # return the result
    return {'muffin': muffin_elmt}
	</programlisting>
	</example>

      </section>

    </section>    

  </chapter>

<!-- *********************************************************************** -->

  <chapter id="mdh-bm.en"> <title>Building a module</title>


      <para> A naive way to define a module is saying that it is an action
      container. While this is true, there is also much more to modules than
      that. For one, functions in a module should have something in common, so a
      module is more something like an action library or a tool box.
      More generally, a module is a python file. Within narval,
      modules are used (appart for the core itself) to defines
      actions, but also interfaces, adapters and elements (<xref linkend="mdh-iae.en"/>).
      Furthermore, <xref linkend="mdh-gf-mi.en"/> will present yet another
      aspect of (more specifically actions') modules.</para>

    <section id="mdh-bm-mc.en"><title>The Module concept</title>

      <section><title>Packing actions</title>

      <para>Building a new module is quite easy. In the python module where all
      the actions are declared, you must have a global variable called
      <varname>MOD_XML</varname> that contains all the XML declaration for the
      module and the actions. The usual way to do this is by initializing the
      variable at the beginning of the file and building incrementally as
      function stubs are declared:</para>

	<programlisting>
<![CDATA[
from narval.public import AL_NS
MOD_XML = "<module xmlns:al='%s'>" % AL_NS

##
## Http Get Ext
##

def act_http_get_ext(args):
    pass
    # function code intentionally skipped

MOD_XML = MOD_XML+'''
<al:action name='http_get_ext' func='act_http_get_ext'>
    <al:input>
        <al:match>IHTTPRequest(elmt).url and IHTTPRequest(elmt).header</al:match>
    </al:input>
    <al:input optional="yes">
        <al:match>IProxy(elmt).type == 'http'</al:match>
    </al:input>
    <al:input optional="yes">
        <al:match>ISpamPolicy(elmt)</al:match>
    </al:input>
    <al:output>
        <al:match>IHTTPResponse(elmt)</al:match>
    </al:output>
</al:action>'''


##
## Write back to socket
##

def Write_back_to_socket_f(args) :
    pass
    # function code intentionally skipped

MOD_XML=MOD_XML+'''
<al:action name='Write_back_to_socket' func='Write_back_to_socket_f'>
  <al:description lang="en">Send back response to client</al:description>
  <al:description lang="fr">Renvoie la réponse au client</al:description>
  <al:input use="yes">
    <al:match>IHTTPResponse(elmt)</al:match>
  </al:input>
  <al:input>
    <al:match>isinstance(elmt, socket)</al:match>
  </al:input>
</al:action>'''


MOD_XML=MOD_XML+'</module>'

]]>
</programlisting>

	<para>This makes it very easy to add new functions to a module, since
	you only have to add the code in the file and add the action prototype
	to <varname>MOD_XML</varname> (MOD_XML is a special identifier
	that is used at actions'modules load time to extract available
	actions in the module, so you can't use it for another purpose
	or use a different identifier to hold actions'prototypes
	definitions).</para> 

      </section>

      <section><title>What makes a good candidate for an action?</title>

	<para>Choosing what to put in an action is difficult. This is really the
	same challenge as designing a software library. With
	<application>Narval</application> however, a new factor comes in
	play. Actions can be used in recipes. A typical recipe should use from
	three to a dozen actions to perform its task, so actions should not
	have a too small granularity. Actions are supposed to perform elementary
	tasks at the scale of the recipe, which itself is very high level, so
	actions are already quite high level. A typical recipe will for instance
	manage an address book, so the required actions for such a recipe would
	be adding or removing an address. Proposing an action to read the
	address book from hard disk is too fine grained. </para>

	<note><title>Remark</title>

	  <para>As always, everything is a matter of context, there may be
	  recipes which require reading files from disk, and an action that does
	  just that is provided in the standard distribution of
	  <application>Narval</application>. </para>

	</note>

	<para>This means that something that can be a good candidate for an
	action in a module can be a bad one in another module. Writing test
	recipes is a good way to tell if the actions in a module are too
	low-level: if you get the impression that you are writing a program in a
	programing language like Python, Java or Younameit, then you probably
	got it wrong. <application>Narval</application>'s ultimate goal is to
	bring the Power of computers in the hand of the average person in the
	street (well, to be honest we are still far away from that, so
	the "ultimate"...)<footnote>

	    <para>and when we say <emphasis>Power</emphasis>, we mean
	    <emphasis>Real</emphasis> Power, and not just surfing the web and
	    downloading WaReZ and thinking we are now 31337 hackers. Computers
	    are good because they can save time by doing boring stuff for
	    you. Using a word processor is not a progress over using a type
	    machine if you have to open each of the 456 files in your directory
	    to <emphasis>manually</emphasis> change the logo of your company on
	    the first page. Most people who use computers nowadays will have to
	    do it that way, though. <application>Narval</application> should
	    enable them to quickly write a recipe that will do this
	    automatically.</para>

	  </footnote>, so using your actions to write a recipe should not become
	  something like programming. When you add a new action to a module,
	  always ask yourself whether you would really like to have it in a
	  recipe. It is much better to share code between the implementation of
	  action stubs than to add an action that will have to be inserted in
	  every recipe that uses actions from the module.</para>

      </section>

      <section><title>Considering a actions'module as a unit</title>

	<para>Deciding to pack actions in modules is one thing, deciding
	<emphasis>how</emphasis> to pack them is another one. The logical
	decision is to group them around a common theme. Since all actions in a
	module share a common file <footnote>

	    <para>This is the case right now. It might be possible to build
	    modules that would be stored in directories, with the
	    <varname>MOD_XML</varname> variable initialized in the
	    <filename>__init__.py</filename> file. It has not been tested yet,
	    but if it is possible, we shall provide support for this in a future
	    release</para>

	  </footnote>, this encourages sharing utility code between action
	  stubs. In other words, all function within a module file need not be
	  action stubs. There can be any number of helper functions provided in a
	  module to avoid code duplication in action stubs, and ease the coding
	  of new actions.</para>

	  <para>Each module may want to deal with it's own set of
	  elements and / or interface. Before introducing a new
	  element or interface, you should carefully consider existing
	  ones and see if one of them would be fine for what you
	  need. If not so, you should write a python class for the
	  element / interface (or event an adapter)<!-- : see <xref
	  linkend="sec-create-elmt"/> -->.</para>

      </section>

    </section>

    <section id="mdh-bm-ts.en"><title>Testing strategies</title>

      <para>Before trying to test your module with
      <application>Narval</application>, you should perform some unitary
      testing, that will enable you to check that your module will behave as
      expected, or at least that it will not crash in a stupid way when
      loaded. This section introduces some testing techniques that isolates the
      module from <application>Narval</application> and thus make it easier to
      find some bugs. They especially enable the use of a python debugger and
      other standard debugging methods, which are rather difficult to set up
      when <application>Narval</application> is running.</para>

      <section><title>Checking the XML syntax</title>

      <para>One of the first thing you want to check in a module is that the
      prototypes of the actions are syntactically correct, since this will
      prevent <application>Narval</application> from being able to load the
      module. This is done by adding a python <function>main</function> function
      that will parse the string held by <varname>MOD_XML</varname>: </para>

	<programlisting>
if __name__ == "__main__" :
    print MOD_XML
    from xml.dom.ext.reader import Sax2
    doc=Sax2.FromXml(MOD_XML)
    print doc.documentElement
</programlisting>

	<para>Once this is done, you can run your module from the command line as
	you would for any other program. This can bring up two kinds of
	errors:</para>

	<itemizedlist>

	    <listitem>
	      
	      <para>Python syntax errors, which you are presumably familiar
	      with. These are beyond the scope of this document </para>

	    </listitem>
	    
	    <listitem>
	      
	      <para>XML syntax errors. The exception you will get will tell you
	      where the error occurred. Common errors include:</para>

	    <itemizedlist>
	      
	      <listitem>
		
		<para>missing '/' in the closing tag of an element;</para>
		
	      </listitem>
	      
	      <listitem>
		
		<para>missing '/' before the '>' of an empty element;</para>
		
	      </listitem>
	      
	      <listitem>
		
		<para>quoting mismatch in attributes;</para>
		
	      </listitem>
	      
	      <listitem>
		
		<para>typos that cause an opened element not to be
		closed.</para>
		
	      </listitem>
	      
	    </itemizedlist>
	    
	  </listitem>
	  
	</itemizedlist>
	
      </section>

      <section><title>Testing the actions individually</title>

	<para>Unfortunately, it is not possible to test all actions outside of
	<application>Narval</application>. If an action uses the socket
	forwarder, for instance, it will not be possible to test it if nothing
	is listening on the socket forwarder port, for instance. Similarly, if a
	group of actions are very tightly coupled, for instance if they share a
	common object and behave differently according to some internal state of
	the object, testing will be difficult.</para>

	<para>The proposed method for testing is to call the stub of the action
	by passing it elements as <application>Narval</application> would do
	it. The outputs of the method can be retrieved and compared to what is
	expected, and thus the action is validated.</para>
	
      </section>

      <section><title>The test framework</title>

	<para>For actions which can't be easily unittested, or to
	test how actions go together, or to test recipes, narval comes
	with a test framework. The principe is simple: launch narval
	with a memory file describing a recipe and starting it, make
	it stop when all plans are terminated, and then check the
	narval's memory after execution. You've so to write the
	initial memory file and a memory validation file. You can get
	more information about this in the "narval testing how-to"
	document.</para>
	
      </section>

      <section><title>The Big Game</title>

	<para>Well, obviously, when you created your actions, you had some
	precise idea about how they should be used in recipes. The time has now
	come to write test recipes, and run them. The Recipe Coding Manual is
	here to help you, and so is the Horn User Manual.</para>

      </section>

    </section>


    <section id="mdh-bm-re.en"><title>Releasing</title>

      <para>Writing modules is a Good Thing. Making them available for everyone
      is a Better Thing. So now that your module is coded and tested, now that
      you have sample recipes illustrating how to use your actions, it's release
      time!</para>

      <section><title>Documentation</title>
	
	<para>Maybe you have so far coded for your eyes only. You know your
	code, how it works, and that's good. However, maybe a bit of tidying
	would be welcome. </para>

	<section><title>Documenting action prototypes</title>

	  <para>We have seen in <xref linkend="sec-action-dtd"/> that the
	  <sgmltag class='element'>description</sgmltag> is optional. It is
	  <emphasis>strongly</emphasis> recommended that you should use it for
	  every action you write. This is one of the three indications a recipe
	  programmer will have about what your action does, the other two being
	  the module name and the action name. As the module and the action name
	  are one or two words, this leaves only the description for something a
	  bit more consistent. Keep in mind that due to screen space
	  limitations, it is best to keep the description string as a
	  one-liner. Avoid if possible repeating what can be guessed by looking
	  at the XML prototype, and rather elaborate about the
	  <emphasis>action</emphasis> that takes place to transform the inputs
	  into outputs.</para>

	</section>
	
	<section><title>Documenting stubs</title>

	  <para>The standard python documentation advices apply here. Use doc
	  strings wherever applies, use comments where needed. We believe at
	  Logilab that it is worth spending a <emphasis>lot</emphasis> of time
	  on the code so that it can be understood without using too many
	  comments. This includes using good variable names and good function
	  names, rewriting shaggy code again and again until it becomes clean,
	  using standard Design Patterns and naming the objects accordingly. We
	  encourage you to do the same, for the benefit of everyone.</para>

	</section>
	
      </section>

      <section><title>Licensing</title>

	<para>We have chosen to release <application>Narval</application> under
	LGPL. We do not wish to impose anything on the developer community about
	the modules they contribute, so you are free to distribute your modules
	under the license you wish. However, we encourage you to use a well
	known LGPL-compatible license. This will enable us to redistribute your
	modules in future <application>Narval</application> releases, and to
	make it available on our web site without worrying about possible legal
	problems. Please include a license statement with the modules you
	release.</para>
	
      </section>

    </section>

  </chapter>

  <!-- ************************************************************* -->

  <chapter id="mdh-gf.en"> <title>Going further</title>

    <!-- This chapter really needs to be worked on -->

    <section id="mdh-gf-ma.en"><title>Multiple actions for a single function</title>

      <para>There is no obligation of having a one to one correspondence between
      actions and stubs. It is perfectly acceptable to have a single stub that
      would behave differently according to the inputs is received. A typical
      example would be an action that acts as a proxy to some outer program
      using the same interface provided by the program. One could argue that it
      could be possible to have an single action with <sgmltag
      class='attribute'>optional</sgmltag> inputs that would be associated with
      the stub. This is true, but would nevertheless not be a good idea, because
      it would make recipes much less easy to read, whereas providing several
      actions with distinct and clear names can make things much easier to
      understand.</para>

    </section>

    <section id="mdh-gf-mi.en"> <title>Modules considered as interfaces</title>

      <para>Something great about modules is that they can behave as interfaces
      to programs, and as any OO programmer will tell you, Interfaces are Good
      Things. For instance, it is possible to identify the basic requirements
      for a mail module. However, the implementation of the stubs depend on the
      underlying operating system: under Unix, mail is often read in
      <filename>/var/spool/mail</filename> or another system mailbox, whereas
      Windows users generally use a POP3 or IMAP server. Yet, from an action
      point of view, they all receive and send mails, sometimes with attached
      documents, and that's about it. Once the module interface has been
      defined, it is possible to choose which implementation of the module
      should be installed on a given system, and all the recipes will keep on
      working <emphasis>regardless of the implementation</emphasis>.</para>

    </section>

  </chapter>
  

<!--   <appendix id="mdh-nlr.en"> -->

<!--     <title>Narval.public reference</title> -->

<!--      This appendix will provide a complete and precise documentation of -->
<!--     <classname>narval.public</classname>  -->

<!--     <section id="mdh-nlr-pr.en"> -->

<!--       <title>Presentation of the narval.public python module</title> -->

<!--       <para>WRITE ME! (herr, even GENERATE ME !)</para> -->

<!--     </section> -->

<!--   </appendix> -->

</book>