2012.03 WP8: iMarine Data e-Infrastructure Enabling Technology Development Monthly Activity By Task and Beneficiary

From IMarine Wiki

Jump to: navigation, search

Contents

This WP8 Activity Report described the activities performed in March 2012 by Beneficiary and Task.

It is part of the March 2012 Activity Report.

T8.1 iMarine Data e-Infrastructure Enabling-technology Development

CNR Activities

Following the solution shaped in MS33 for the new evolution of the Resource Management area, T8.1 prepared a workplan for the T8.1 implementation activities ahead. CNR led the organization of the workplan. The plan has been presented to the consortium at the 1st TCOM in Rome and reflected in a set of TRAC tickets.

The implementation on the new Software Gateway is almost completed. Stubs and service have been deployed in the development infrastructure and the Deployer and Resource Manager services are being adapted to interact with them. The role of the service and its implementation has been introduced at the TCOM.

In collaboration with WP5, new policies for distributing the encryption key have been settled and the common-utils-encryption library changed accordingly.

The implementation of the common-resources-publisher is ongoing but has been slowed but a non trivial issue: since it interacts with the Information System, based on on the current runtime stack, we need to understand how to avoid dependencies between this and the new stack. This is a common problem for any client of gCore services (such as the iMarine Gateway). Even the forthcoming common-resources-discovery will face it.

As part of cross-task activities, CNR chaired a very fruitful working session on T8.1/T8.2 at the TCOM. The discussion (joined also by WP5) led to some milestone agreements on T8.2 future directions according to the zero-dependency approach followed in T8.1.

At the end of the month, tickets (400, 399, 235) moved the effort on revising some parts of the Resource Manager, Deployer and Virtual Platform components.


A close collaboration with the CL working group of WP11 has been established to solve the so-called gCore client issue: how to avoid that clients of gCube service bring a whole gCore runtime stack as dependency.


  • workplan (reported also in TRAC tickets) for the implementation of the new resource management functions described in MS33
  • Software Gateway moved towards end of implementation phase
  • common-resources-publisher's final interface defined and implementation started

FAO Activities

In the reporting period, FAO has prepared the presentation for the TCom, which has reported the state of the previous four months of activities, including vision, approach, methodology, timeline, and some highlight on prototype implementations. In doing so, it has liaised with CNR.


None to report


  • Report on performed activities.

T8.2 iMarine Data e-Infrastructure Policy-oriented Security Facilities

E-IIS Activities

The activities performed on February 2012 have been continued and completed. In particular an analysis has been performed collecting some security requirements from other work packages and proposing other security requirements basing on ENG expertise on the field. A basic proposal of Policy Based Security Facilities has been presented during the first TCOM in Rome and has been refined basing on the observations and the comments of the partners. The proposal has been completed and reported on Milestone MS34. It is a complete security framework, called SOA3, providing authentication, authorization, accounting, user management services and identity federation. Standing on the current requirements of the project SOA3 provides:

  • service oriented authentication based on LDAP and transport of user attributes inside the infrastructure based on SAML assertions
  • service oriented authentication based on XACML with multiple attributes
  • CRUD operation on user lists stored on LDAP directory or Liferay database

SAML standard also provides Identity Federation which, standing on ENG experience, could became a future requirement.


None to report


T8.3 Workflow Management Facilities

NKUA Activities

During this period NKUA focused on specifying a set of tasks to be performed in order to achieve the goal of enhancing PE2ng workflow capabilities in a long term basis. This set of tasks was compiled and described in a presentation given by NKUA during the 1st TCOM meeting. A detailed description of the latter follows:

  • Parallel Execution Model Abstractions
    • Significant progress has been made in the design related to this task, which is described in more detail below.
  • Dynamic plans
    • Nested workflows within execution plans, which will be capable of receiving inputs from other plan elements. The instantiation of the workflows will take place "just-in-time", exploiting all the new data for parallelization or data handling.
  • Output status reporting and failure recovery
    • Refers to the definition of a pattern for handling output status of workflow elements. This will allow processing data that failed at the first execution attempt, as well as restarting failed paths of a parallelized execution.
  • Provenance Tracking
    • The incorporation of metadata inside the workflow. Will be achieved by defining a channel to pass metadata to the node and attaching provenence metadata to the data sets, if applicable.
  • Improved Matchmaking and Deployment
    • Will be achieved by the injection of deployment elements into execution plan. The deployers executed by deployment elements will be selected via technology and resource availability criteria.
  • Enhancement of error handling capabilities
    • This goal will be achieved by assigning unique identifiers for every entity referred to in the execution plan, and logging errors in association with these identifiers. Furthermore, a mechanism for declaring the identifiers to executable elements will be defined, in order for the latter to propagate them to their internal error logging.
  • Persistent Workflows
    • Workflows will be seen as invokable resources, capable for parameterized execution, parameter value assignment and on-launch data staging. Additional features such as notifications upon execution completion, execution cancellation and per-user aggregate monitoring are also under investigation.
  • User Interface and control
    • The basic management UI will be improved in order to allow launching, monitoring and introspecting tasks in a user-friendly manner. Furthermore, a GUI will be offered for the graphical composition of workflows.

Regarding parallel execution model abstractions, the design has advanced to a more concrete phase and the implementation of this feature, along with fault tolerance mechanisms will be given high priority. The objective of the abstractions is to provide a minimal set of concepts for running code over several infrastructures that include even the "local machine" and to simplify the way users access complex infrastructures by freeing them from having to adapt to platform models, APIs and their evolution. The two execution models which will be provided are the following:

  • The Master-Worker model
  • The Map-Reduce model
    • A specialization of Master-Worker itself, but with extra requirements

The Master-Worker abstraction consists of the following elements:

  • The Partitioner
  • The Processor
  • The Sink

Partitioners read input from a data source and are provided with information enabling them to partition the data set which they will forward to processors. Processors perform the actual processing and forward their data to the Sink, which combines them performing any post-processing operations necessary. Both partitioners and processors are executed to several internal nodes which are not known a priori but can be selected based on a set of policies or conditions, by means of the Node Selection library available now or other facilities implemented in the future. A set of generic Partitioners (gRS, url, filesystem, database,...) will be implemented in order to simplify the task of developers.

The described combination of Partitioner/Processor/Sink will be encapsulated in special constructs provided by the Workflow Engine. These constructs will enable the marking areas in the workflow as parallelizable, define indicators for parameter passing and provide the means to select the method of passing parallelization information (e.g. through parameters or environment variables).

An example of abstracted parallelization is shown below:

WorkflowReportMarch1.png

The above parallelized task will be instantiated as an Execution Plan as follows:

WorkflowReportMarch2.png

Briefly, the Map-Reduce abstraction consists of two Partitioner/Sink layers, the first corresponding to the map phase and the second to the shuffle/reduce phase. A more detailed description as well as the first implementation outcomes will be provided during the next periods.


None.


None.

FAO Activities

None


None to report


None

T8.4 Resource Model

The latest evolutions Facet-based Resource Model and their implementation in common-resources been presented at the 1st TCOM in order to gather feedback and further requirements.

After the event, CNR and FAO had a conference call to map the points discussed at the TCOM onto the Model and the implementation. Another call will follow in April to complete the analysis and shape solutions.

FAO Activities

Presentation at the TCOM and discussions on the received feedback


None to report


  • Feedback from the partners and new requirements collected on resources' classification

CNR Activities

Presentation at the TCOM and discussions on the received feedback. CNR also started to act as the first common-resources client with the new common-resources-publisher, giving in such a way early feedback and validations on the Resource Model implementation.


None to report


  • Feedback from the partners and new requirements collected on resources' classification

NKUA Activities

None


None to report


None

Personal tools