2012.07 WP8: iMarine Data e-Infrastructure Enabling Technology Development Monthly Activity By Task and Beneficiary

From IMarine Wiki

Jump to: navigation, search

Contents

This WP8 Activity Report described the activities performed in July 2012 by Beneficiary and Task.

It is part of July Monthly Activity Report.

T8.1 iMarine Data e-Infrastructure Enabling-technology Development

CNR Activities

CNR has continued the work on the new Resource Manager service started in the previous month. Several new features have been added to the service and its capabilities greatly expanded. In detail:

  • the internal state is centered around the concept of package instead of service
  • each package is bounded to its dependencies and the target node
  • the state maintains its consistency even if a target node is cleaned or redeployed
  • the operations of ResourceBinder allow the deployment of the same package in different nodes
  • a new Administration portType has been added to allow managers the partial cleanup of the service state and/or recovery

Along with these novelties, the service is now able to create, manage and destroy multiple scopes within the same instance.

JUnit tests integrated with Maven are under implementation allowing a comprehensive and layered testing reproducing several runtime conditions of the RM's service.

CNR also started the preparation of D8.2 due by the end of August.


The work on the new technological foundations has not been resumed yet.


  • Resource Manager is able to create, manage and destroy multiple scopes within the same instance.

FAO Activities

None to report.


None to report.


None to report.

T8.2 iMarine Data e-Infrastructure Policy-oriented Security Facilities

E-IIS Activities

ENG has finalized the implementation of SOA3 components. In particular:

  • the LDAP based Authentication
  • the Authentication REST interfaces
  • the User Management Service REST interface

These modules are completed and available on Etics in the HEAD configuration.

The activity of integration of Shibboleth Identity Provider and Service Provider with the rest of SOA3 architecture is ongoing.

The contribution to D8.2 with the technical information on the implemented modules has been started.


None


  • LDAP based Authentication module
  • Authentication REST interfaces module
  • User Management Service REST interface module

T8.3 Workflow Management Facilities

NKUA Activities

NKUA has been working in four directions:

In evolving the node selection library, where the following enhancements and fixes have been performed:

  • Colocation policies are now more aptly named Node Assignment policies. Minimum and maximum colocation remain unchanged, as special cases of node assignment policies.
  • The hosting node entity used by the library now enable the latter to distinguish the local node from the remote ones. The information of whether the entity is the local node or not is provided by the information provider used to harvest hosting node statistics, through the corresponding adaptor. Examples of such information providers are the Resource Registry or the direct usage of the gCube Information System.
  • The notion of tie breaking was introduced to all node selectors. In particular, all node selectors now support tie breaking in the form of an additional node selector. Tie breaking selectors are used if some of the scores produced by the node assessment procedure are equal. In that case, the subset of the candidate nodes which are involved in the ties is reassessed by the tie breaking selector and the new scores are interpolated into the final scores that are produced. As tie breakers are standard node selectors tie breaking can continue in a arbitrary number of levels, if so desired.
  • The SingleRemoteNode assignment policy which previously could somewhat controversially include the local node, if present in the candidate set, was specialized creating two new policies: The SingleNodePolicy and the SingleRemoteNodePolicy. The latter will never select the local node.
  • Node distance calculation support was added to the library. The latter can be performed by utilizing an XML describing node topology, or in the absence of such information using heuristics. Node distance was also added as a possible factor in the node selection policy based on a cost function.
  • MRU node selection was added as a counterpart to LRU node selection.
  • Various bugs and omissions in all selection and assignment policies were corrected.
  • Inconsistencies in hosting node property names were corrected.

In enhancing the distributed execution capabilities of the Workflow Adaptor used by the gCube Search System.

  • The adaptor is integrated with the node selection library; search plans can now be executed in multiple nodes, depending on their complexity. In particular, complex search plans are executed based on a configurable node assignment policy. For example, the maximum colocation policy can be used to control the number of nodes to which the execution of the plan is spread, with the aim of avoiding to overload nodes while at the same time keeping response times as low as possible. Briefly speaking, this policy will attempt to distribute execution to as low a number of nodes as possible, taking into account node computational capabilities when determining the size of subtrees which will be assigned to the latter.
  • A plan analyzer was developed and incorporated into the adaptor. The analyzer is used to determine the plan for execution is simple or complex. This characterization is based on a configurable cost threshold. In order to achieve low response times for simple queries, simple plans can be executed on the local node, i.e. the node on which the Search System is deployed.
  • A rewriter was developed and incorporated into the adaptor. The rewriter's task is to break complex operations like merge over a large number of sources into equivalent subtrees of simpler operations, in order to ease the management of the latter.
  • If the resources of the infrastructures are not enough to achieve optimal assignment and the final assignment results in nodes being over-utilized, the invocations of the search operators are instructed to limit the buffer capacity used by gRS2, trading off some fraction of the performance for robustness and stability.
  • The adaptor supports a number of new configuration parameters, in the form of hints. These parameters are used to support the functionality described above, have sensible default values and are the following:
    • The data source node selector to be used
    • The tie breaker selector of the data source selector
    • The node selector used to select nodes for seach operator invocations
    • The tie breaker selector of the operator node selector
    • The node assignment policy to be used
    • The maximum cost of search plan subtrees which can be executed in a single node, termed maximum colocation cost
    • A threshold value which is used by the node assignment policy. All nodes whose score fail to reach the threshold are excluded from selection.
    • Whether or not the local node should be used for the execution of simple plans.

In solving observed issues in the PE2ng Execution Engine. In particular, the following fixes have been made:

  • Array evaluation was not performed correctly in the case of remote execution. In order to solve this issue, a new parameter filter named ParameterArrayEvaluationFilter was implemented. This filter accept a set of execution plan data types and evaluates an array data type using their values. In this way, array evaluation depends on already evaluated variables and is performed correctly in all cases.
  • When transmitting an execution plan for remote execution, the Execution Engine avoids sending all variables present in the plan, in order conserve time and network resources. Instead, each element is expected to declare which variables should be included. A number of parameter filters did not declare all variables of importance. These omissions were corrected.

In designing and implementing a queue based scheduling facility for PE2ng. The design for the facility is complete and includes:

  • Execution node monitoring through an additional property which will be included in the entities representing Hosting Nodes in the Resource Registry. This property represents the computational cost of all PE2ng tasks which are currently running on each execution node.
  • The enhancement of Resource Registry in order to communicate the information described above among others using messaging publish-subscribe services. This functionality will be exploited by the Execution Engine, enabling it to obtain an as much up-to-date view of the infrastructure in terms of running tasks. The information will be updated upon the initiation and completion of execution tasks at the hosting node level.
  • The implementation of a queue-based scheduling component which will act as a front-end to PE2ng. This facility will also exploit the information describing the state of the infrastructure, deciding to forward or queue up incoming tasks.

Implementation of all three aspects of the functionality has started.


None to report.


  • The design of PE2ng queue based scheduling facility has been completed.
  • Search distributed execution has been implemented and tested.

FAO Activities

None to report.


None to report.


None to report.

T8.4 Resource Model

FAO Activities

None to report.


None to report.


None to report.

CNR Activities

None to report.


None to report.


None to report.

NKUA Activities

None to report.


None to report.


None to report.

Personal tools