2012.05 Activity Report
From IMarine Wiki
This Activity Report captures the activities performed in May 2012.
This report is organized in two parts:
- Progress and Achievements at the WP level reports a concise overview of the activities, issues and corrective actions, and major achievements produced by each WP Leader;
- Progress and Achievements by WP and Beneficiary reports a detailed description of the activities, issues and corrective actions, and major achievements produced by each beneficiary at the level of Project task.
Progress and Achievements at WP level
WP1: Administrative and Financial Management
WP Leader: ERCIM - Participants: CNR
- Finalising and submission Management report
- Monitoring and submission of the due deliverables and milestone achievement statement
- Notification of the new Project Officer
- Prepare reimbursement guidelines in terms of administration and finance for the iMarine Board Meeting and other events
- Deliverables submission: D8.1, D6.2 and D2.1
- 21 Milestones achieved up to date
WP2: Project Management
WP Leader: CNR - Participants: NKUA, FAO
The project was monitored and supervised from the scientific and technical point of view. The monthly SB (May 14 and May 18) and PEB teleconferences were organised. During these calls several strategic and management issues were raised that required further investigations, additional communications and bilateral phone calls between partners. A number of corrective actions were also identified and put in place. The D1.2 "Activity Management Report" was finalised by integrating and harmonising contributions received from WP leaders. As part of the production of this deliverable an analysis of the effort claimed by each partners with respect to the reported activities was performed. A number of telephone calls and e-mail exchange between CNR and FAO were performed to discuss how to address the collaboration with DG-MARE and plan the forthcoming meeting with DG-MARE representatives on June 22.
Delivery of D1.2 "Activity Management Report" was delayed. This was partially due to the late reception of contributes, especially concerning effort claims.
- Monitoring and supervision of project activities from the scientific, technical and strategic point of view
- Identification of a number of corrective actions
- Finalization of D1.2 "Activity Management Report"
- Organization of the monthly SB and PEB meetings
WP3: Governance and Policy Development
WP Leader: FAO - Participants: ERCIM, CNR, NKUA, FORTH, TERRADUE, FIN, UNESCO, CRIA, IRD
During May, effort in WP3 mainly focussed on
- 3.1: finalizing the cluster definitions and imarine Board report, and several meetings with project members to discuss their expectations and role in the project.
- 3.2: Continue the development of the iMarine Data access and Sharing policies, and present the opportunity to potential users. This will require that a governance mechanism for external partners be developed, an initiative was started to define such items.
- 3.3: deliver a new FLOD interface, experiment with WPS / Hadoop for intersection generation, review ICIS, and discuss the potential for a shared Code List Manager. The use cases in several clusters were reviewed.
D3.1 was reviewed, and is now ready to be produced. This deliverable blocks a range of other Deliverables and Milestones, and has highest priority.
The WP is accumulating delays, related to the work plan of the Board that requires completion.
In OBIS, a new staff was brought into the project. The arrangements with the iMarine project will require further clarifications over the coming weeks.
- The Board report and cluster descriptions have been circulated, the Work Plan must now be finalized.
- A potential collaboration has been negotiated, this now awaits review and comments from the SB.
- FLOD was enriched with a new and user friendlier UI.
- The experiment with WPS Hadoop resulted in a first intersection algorithm for testing purposes.
- The review of ICIS stressed again the need for work-flow support, Code List management, a generic approach to TimeSeries harmonization, and analytical capabilities.
WP4: Communication, Dissemination and Training
WP Leader: TRUST-IT - Participants: CNR, NKUA, FAO
The WP leader should report here a summary of the activities performed in the reporting period
The major issues faced in the reporting period and the identified corrective actions should be reported here
The WP leader should report here a bullet list highlighting the main achievements of the reporting period
WP5: iMarine Data e-Infrastructure Deployment and Operation
WP Leader: CERN - Participants: CNR, NKUA, FAO, FIN
CERN coordinated and performed together with CNR the CNR-PROD-PISA EGI site upgrade to UMD 1. In order to upgrade to the new middleware the installation from scratch of the old services running glite 3.2 and 3.1 has been performed by CNR while CERN was involved in the configuration of the installed services.
In addition in order to evaluate its possible usage in iMarine, an instance of a regional GOCDB has been installed at https://imarine2.cern.ch/portal/.
NKUA and FAO continued to maintain the provided infras operational. Changed the ips of provided hosts in NKUA was performed without major issues.
Regarding the infrastructure monitoring CERN installed and configured a new node running Nagios connected with LDAP to give access to Infrastructure and Site Managers. The new instance of the service is running at http://d4science-dt.cern.ch:8443/nagios/
In addition the latest version of the ActiveMQ broker (5.6.0) has been installed in the development infra for testing.
Lastly CERN participated at the 6th Spanish e-Science Network meeting in Madrid, with a presentation focused on iMarine and the D4Science infrastructure.
No news regarding the new server to be provided by FIN. ( the previous one is down due to HW problems)
The planned activity for infrastructure upgrade and VRE deployment is delayed, due to delay in the integration process
CNR-PROD-PISA site upgrade to UMD 1
New Nagios server for the D4science infrastructure installed.
WP6: Virtual Research Environments Deployment and Operation
WP Leader: CNR - Participants: ERCIM, NKUA, FAO, FIN, UNESCO, CRIA, IRD
No activity was performed in this Task
- UNESCO interacted with CNR and FAO for the development of services that will be made available through the VRE
- CNR finished with the implementation of the Geo Explorer web application
- CNR completed the implementation and testing of the new Workspace Portlet
- CNR synchronized the Workspace Light Tree Widget with the Home Library
- CNR enhanced the Resource Management Portlet
- CNR updated the Report Generation and Template Creation Portlets to support the new Workspace Portlet
- CNR started the development of a new Software Management Widget allowing to load any software artifact into gCube maven repositories and to register it correctly within the Software Gateway
- CNR completed and released the Species Discovery Portlet
- CNR started the development of a new Tabular Data Widget allowing the visualization, editing and processing of tabular data by a hosting application
- CNR released and updated the War Management Widget and Portlet in order to support the new Software Gateway and the Maven Repository
- CNR fixed and updated the Time Series gCube application
- CNR fixed and updated the Ecological Modeling gCube application
- CNR started the implementation of the Species View portlet
- NKUA updated the Search Manager Portlet to support the semantic search integration with the gCube Search System
No activity was performed in this Task
Some delays were due to the late gCube 2.9.0 release
- Collaboration between UNESCO, CNR and FAO for designing new VREs
- The Geo Explorer Portlet was released
- The Workspace feature was enhanced
- The Resource Management Portlet, Report Generation and Template Creation were enhanced
- The Search Manager Portlet was enhanced to support the semantic search
- Bug fixed on current applications running on the portal
- The Species Discovery Portlet was released
- New applications are being designed and implemented: Tabular Data Widget, Software Management Widget
WP7: Enabling-technology Integration and Distribution
WP Leader: E-IIS - Participants: CNR, NKUA
In the reporting period, integration of release gCube 2.9.0 started. E-IIS monitored the integration of the release identifying integration issues and supporting developers in its solution.
Mavenization process has been extended to all gCube components.
E-IIS performed the upgrade of testing infrastructure, installing new gHN v. 3.3.0 and new service Software Gateway. E-IIS performed deployment testing of gCube services released in gCube 2.9.0. A tight collaboration with CNR (responsible for software for dynamic deployment) was required to identify and solve several issues in service's deployment phase. As responsible of the software providing access to Maven Repositories and of the entire deployment phase, CNR was called to fix numerous integration issues, naturally expected in a so complex and new distribution procedure
As far as distribution and documentation activities are concerned:
- CNR updated the Software Gateway service in order to produce a registration report
- E-IIS implemented a new component build-distribution to aggregate registration reports and enrich them with svnpath and wikidoc information
- NKUA implemented a new version of the distribution site in order to be compliant with new format of distribution report
- NKUA implemented a crawler to download component's javadoc and make them available in the distribution site
- deployment testing phase of gCube 2.9.0 rlease took more than expecting and is still ongoing
- new distribution site ready
- scripts to produce build's distribution reports and new javadoc crawler ready
WP8: iMarine Data e-Infrastructure Enabling Technology Development
WP Leader: CNR - Participants: NKUA, E-IIS, FAO
As expected, the integration and release of gCube 2.9.0 (the first ever made according to the new distribution procedure based on Maven Repositories) called for tuning activities on the enabling components released last month by T8.1. Following some integration issues raised by T7.1, the Deployer service has been internally refactored to provide a better compliance with the new distribution procedure. The Software Gateway service has been tuned according to the directions raised by the integration phase. The report returned following the software registration has been enriched in order to provide the information required by the Mavenizer and the gCube distribution site. The interface of the service has been slightly changed following a different approach proposed by T7.1.
T8.1 has also upgraded to gCube 2.9.0 a significant part of the development infrastructure granting to all developers the possibility to exploit the latest features of the enabling technology as early as possible.
T8.2 has produced a set of tickets reporting a detailed workplan with effort and timing. The plan is actually still under discussion within the WP8 members because of the impact it will have on the system. The major point of the discussion is about the possibility to change the plan in order to have AuthN implemented firstly and then AuthZ will follow. In the meantime, T8.2 has started the implementation work according to this first version of workplan, i.e. the extended Authorization module supporting policies based on multiple attributes and context based attributes. The work is currently in progress. In Parallel, T8.2 is also working on a new UserManagement module based on LDAP, which will be used for the Authentication process. The release is planned for the first days of June.
In the workflow management area, T8.3 has completed the integration of PE2ng with the new Storage Manager facility started in the previous reporting period. Further, investigation of the issues reported in one ticket revealed that the matchmaking strategy followed by PE2ng was not optimal; to overcome the issues, a matchmaking preparation phase has been added to the plan generation procedure before JDL DAG jobs submission. This activity resulted in a new versions for (i) the Workflow Engine library; and (ii) for the Environment Provider library, in order to enable retrieval of the set of all matching nodes without node selection. Finally, other minor enhancements and fixes have been made to Execution Engine library and the Workflow Engine Service
T8.4 had a conference call late in May to resume the work on the 2nd generation of the gCube Resource Model. The analysis was made taking into consideration the deliverable D8.4 due to M8. Starting from this version of the Resource Model, gCube will take a more modular and abstract view over its management functions. Functions will be described solely by their APIs, and different implementations of the same functions (provided by T8.1) may exist within the system, over time or even concurrently.
At WP level, D8.1 (delayed from last month) was produced, reviewed, adjusted and officially released.
- implementation activities on VREModeler and Resource Manager services in order to offer simplified VRE creation (requiring only one instance of RM deployed per each VOVirtual Organization; )
- the formalization of the new Resource Model, the most common resource's types and their related interfaces
- the implementation of the new technological foundations will parallel the formal definition of the Resource Model
- finalization of the workplan for T8.2's implementation and integration activities
- release of deliverable D8.4
None to report.
- release of deliverable D8.1
- tuning of the distribution and deployment components
- proven effectiveness of the enabling components in the first mavenized release of gCube
- Software Gateway developer's guide
- integration of PE2ng with the new Storage Manager facility (in collaboration with WP9)
- new matchmaking preparation phase added to Workflow Engine and Environment Provider libraries
- release of the following components in the Workflow Management area:
WP9: Data Management Facilities Development
WP Leader: CNR - Participants: NKUA, CERN, TERRADUE, FAO
- FAO started the refactorisation of the tree-manager-library towards compliance with the design model defined in WP11
- FAO supported the evolution of the transfer services that make use of the tree-manager-library
- CNR finished with the development of the ITIS plugin
- CNR reproduced the ITIS datasource on a local MySQL database
- CERN extended the Data Transfer Agent service in order to integrate the URLResolution Library implemented by NKUA
- CERN integrated the Data Transfer Agent with the Storage Manager Library
- CERN updated the DTA development guide on the gCube wiki
- NKUA improved the TCP transfer method performance of gRS2
- Terradue documented the WPS-hadoop software on the gCube wiki pages
- Terradue implemented SPREAD intersection algorithms on WPS
- Terradue helped CNR in WPS integration with gCube
- No activity was performed in the direction of T9.3 during the month of May
The MS39 was written but is still in phase of internal validation.
- Refactorisation of the tree-manager-library following the indications from WP11
- Integration of the ITIS datasource
- URLResolution library included in the Data Transfer Agent
- TCP transfer performances enhancement
- Intersection algorithm added on WPS
- WPS integration started
WP10: Data Consumption Facilities Development
WP Leader: NKUA - Participants: CNR, FORTH, TERRADUE, FAO
During this period, activity has taken place both within the work package, with cross-task activities involving Data Retrieval and Semantic Data Analysis, and between work packages with conference discussions being held between WP10 and WP5.
As far as the latter is concerned, the decision of postponing the release of the already available new version of Resource Registry has been made, in order to decide upon and implement a dynamic configuration scheme, in order to make the facilitate the deployment of the increasing number of services which use the Resource Registry. The solution which was agreed upon is to create a default configuration ETICS artifact, as well as multiple specialized configuration artifacts for each service and declare these artifacts as dependencies of the services.
In the direction of integrating Semantic Data Analysis Facilities with Data Retrieval Facilities, a conference has been held in order to discuss the integration of Semantic Search components (X-Search) with the Search System of gCube. In these discussions, the naming of the subsystem under which the new components will be committed and the usage of Maven for the build process were agreed upon. Furthermore, having efficiency in mind it has been decided that Semantic Search functionality will be closely integrated with the gCube Search System, i.e. it will make direct exploitation of the programmatic interfaces available by the software layers comprising the latter. SPARQL endpoints needed by components in the area of Semantics are going to be modeled in gCube as Runtime Resources. Further discussion on the requirements of Semantic Search components (#363) led to the decision that the Search System should provide a means of annotating search fields with certain keywords, by implementing functionality at the layer of Resource Registry and by providing a new GUI to manage annotations. Development activity in this direction is in progress, while development activity of X-Search is progressing in parallel.
In the area of Data Manipulation, the new version of components comprising the Data Transformation Service have been released in gCube 2.9.0. In addition, a number of improvements have been made and many minor bugs in the new version of the Data Transformation Service have been fixed, as a result of further testing. Testing included transformations of images of various source and target type combinations with different data source and data sink configurations, thumbnail creation and feeding of indexes such as the Full Text and Forward indexes. The most important improvements made has been the handling of identity transformations in heterogeneous data sources and the improvement of the service in terms of modularity and portability.
In the area of Data Mining and Visualization, work has been performed in the area of testing Generators and Modelers. Furthermore, the interface of the Statistical Manager is under investigation and a number of existing systems is being analyzed in order to take the best ideas from each. The Thredds software was also used with the aim of adding Environmental Data Management capabilities to the Statistical Manager. A library (Environment Explorer Library) was implemented for retrieving physical or chemical features associated to some points by asking to a Geo Network installation. Finally, the Geo Explorer facility which is part of the Geospatial Data Visualization domain has been implemented and released in gCube 2.9.0.
Work on authoring and revising deliverables has continued. In particular, D10.1: gCube Query Language has undergone a lengthy reviewing and revising phase which aided in the improvement of its quality, but also postponed further its final delivery date. The document is almost finalized and its delivery is expected to take place early within the next period. The authoring of D10.3: iMarine Data Consumption Software is also complete, while a revised version has been made available taking into account the comments of the reviewer. However, the document has not been released yet as final approval is still expected.
- Release of D10.1 is delayed due to a more extended reviewing period than was initially anticipated, in order to meet the expected quality standards. Delivery is expected to take place during the first days of the next period.
- Release of D10.3 is delayed. Approval of the final version is pending.
- There have been delays within Data Mining activities due to the recent gCube 2.9.0 release.
- Statistical Manager was tested with local computation
- The Environmental Explorer library has been implemented
- GeoExplorer was released in gCube 2.9.0
- Release of Data Transformation components in gCube 2.9.0
WP11: Data e-Infrastructures Integration and Interoperability Facilities Development
WP Leader: NKUA - Participants: CNR, CERN, E-IIS, FORTH, TERRADUE, FAO
During this month, work in WP11 focused in many aspects of the Integration and Interoperability Framework covering mostly the topics for HTTP Front end layer, Client Libraries Design Model and Client Libraries Framework implementation. The restructuring of ASL HTTP Front End layer architecture has been completed and based on the design analysis performed during the previous month. The new applications derived are compliant with the model principles and they: - are extended to support both JSON and XML as standard data interchange formats - build upon the comon utilities implemented for context management and HTTP error handling - provide grouped access to logically related functionality. The design model for Client libraries has been extended with the cient security API and has been completed, insofar as the first iteration is concerned. Its analysis is presented in the dedicated space in gCube wiki. Based on the model, an implementation of a framework for Client Libraries has also been completed and is available in SVN as two libraries: common-clients and common-gcore-clients. The WIKI documentation of the framework is also ready and available here. Moreover, the verification of both the model and the framework has also started through implementations, including a sample library available in SVN, the re-factorisation of the tree-manager-library and a sample library for the gCube Search Service. The verification has led to the evolvement of the model and framework according to the comments and the observations made during implementation and aim at providing more feedback.
Finally D11.2 has been produced. It describes the novelties and evolution of the Application Programming Interface Software up to M6 and reports on the software released by WP11 as part of gCube release 2.9.0.
D11.2 has not been released yet. It is under review at FAO.
- the completion of the design model, which marks its official release
- release of two components as part of the HTTP Front End Framework:
- completion of restructuring and extension of the existing ASLHTTP application, via implementation of components compliant to predefined model principles:
- Preparation of D11.2 deliverable