2012.01 Activity Report
From IMarine Wiki
This Activity Report captures the activities performed in January 2012.
This report is organized in two parts:
- Progress and Achievements at the WP level reports a concise overview of the activities, issues and corrective actions, and major achievements produced by each WP Leader;
- Progress and Achievements by WP and Beneficiary reports a detailed description of the activities, issues and corrective actions, and major achievements produced by each beneficiary at the level of Project task.
Progress and Achievements at WP level
WP1: Administrative and Financial Management
WP Leader: ERCIM - Participants: CNR
The work performed in Month 3 primarily originated from the Quality Assurance Task Force (QATF), and included:
- Production and review of D1.1 Quality Plan;
- Monitoring and submission of the due deliverables and milestone achievement statements;
- Establishing effective communication techniques between the QATF and the consortium.
7 partners have not yet completed the bi-monthly effort report for November-December 2011 (M2). The next effort report will be due on 2 March and must include effort reported for M1-M2 and M3-M4 (in TWO reports, not one consolidated report!) from all partners. 3 deliverables are now overdue (D7.1, D3.1, D6.2).
- All CA signature pages have now been received by the Coordinator, allowing the completion of the advance payments.
- 11 milestone achievement statements have been documented and the project is now on track: http://bscw.d4science-ii.research-infrastructures.eu/bscw/bscw.cgi/219911
WP2: Project Management
WP Leader: CNR - Participants: NKUA, FAO
Activities of the period, deliverables, and milestones were monitored and supervised. Monthly PEB and SB teleconferences were organized in January 2011. A one day coordination meeting between FAO-CNR was held to discuss strategic and organizational aspects and to set priorities. Cooperation with external projects were established and a number of MoUs were finalised. Decisions about the activity to be assigned to J. Miur regarding the costing framework were taken. The agenda of the next f2f TCom (14-16 March 2012) was prepared and discussed with the project partners.
No major deviation to declare.
- Monthly SB and PEB meetings were held;
- A meeting between FAO-CNR was held to discuss strategic and organizational aspects and priority settings;
- The MoUs with EMI and TELDAP were finalized;
- Decisions about the activity to be assigned to J. Miur were taken;
- The agenda of the next f2f TCom (14-16 March 2012) was prepared.
WP3: Governance and Policy Development
WP Leader: FAO - Participants: ERCIM, CNR, NKUA, FORTH, TERRADUE, FIN, UNESCO, CRIA, IRD
In T3.1 FAO established the Board memberships, structured the documentation channels (wiki, bulletin board, documentation area), and added relevant content. Preparations for March Board meeting in Rome started.
In T3.2 FAO investigated on the data access and sharing policies in use by FAO. A similar effort on community software policies was initiated.
In T3.3 FAO worked with IRD on the establishment of SDMX functionality, with CNR on the identification of opportunities of advanced data analysis in BC2, and with iMarine Board Members on issues pertaining to statistical data collection and validation frameworks. This also saw the involvement of external parties from the CoP of BC1.
No corrective actions or deviations were reported.
- Establishment of the Board (MS5)
- Establishment of communication channels (MS6)
- Continue with documentation of Board activity and requirements collection (MS21)
- Initiated data access and sharing policy activity
- Identified focal point and define structure for software policy activity
- Analyzed SDMX options with IRD
- Collaborated with WP9 on providing access to the KB
- Prepared wiki and other materials to analyze CoP Business Cases
WP4: Communication, Dissemination and Training
WP Leader: TRUST-IT - Participants: CNR, NKUA, FAO
- Completion of D4.1 Communication, Dissemination & Training Plan;
- Generation & delivery of Flash Presentation Application for 3rd Marine Board Forum, Brussels;
- Content upload on iMarine Channel
- Preparation of synergies mutual benefits page (on-going);
- Preparation of interview list and questions (on-going);
- Generation of revised iMarine flyer;
- Validation & generation of contacts for iMarine dissemination database;
- Coordination & Liaison with iMarine Board for channel content.
January 2012 newsletter postponed to beginning of Feb 2012
- Completion and submission of the WP4 1st deliverable -  D4.1 Communication, Dissemination and Training Plan
- iMarine Channel - 90% completion of initial content in all sections, including fully functional area of iMarine Board Area
- 3 News & articles on iMarine published in January 2012 - CORDIS: Features Stories - Saving ecosystems with open data and e-infrastructure ecosystems, Article: Data e-Infrastructure Initiative for Fisheries Management and Conservation of Marine Living Resources (IMARINE) - Project description
WP5: iMarine Data e-Infrastructure Deployment and Operation
WP Leader: CERN - Participants: CNR, NKUA, FAO, FIN
The activities performed by WP5 during the project M3 can be summarized as follows:
- The deliverable D5.1 has been completed and after the SB approval has been sent to the EC.
- Achievement of the milestones M25 and MS27
- The D4Science Ecosystem has been continuously monitored and several tickets have been opened and monitored
- NKUA has been keeping up and running the provided nodes hosting GHNs and glite. Additionally provided gcube wiki maintenance and in this time have taken immediate corrective actions for any issues that occurred.
- In Adddition NKUA planned the upgrade of glite3.2 middleware services to UMD, which will start at the end of January.
- CNR has been keeping up and running the provided nodes hosting GHNs.
- CNR fixed two important incident tickets occurred in the reporting period. In particular, one of them required the migration of 14 nodes on a different hardware.
The D5.1 deliverable is of type Other ( wiki ) and it will be extended till March 2012 to complete the definition of the D4Science Infrastructure policies.
- D5.1 completed and sent to EC
- MS25 and MS27 achieved.
- NKUA Fixed incident regarding hardware problem
- CNR recovered from a hardware fault involving 14 nodes at CNR site (ticket #372)
- CNR restored a backup of a Content Manager instance (ticket #373)
WP6: Virtual Research Environments Deployment and Operation
WP Leader: CNR - Participants: ERCIM, NKUA, FAO, FIN, UNESCO, CRIA, IRD
The activities performed in this reporting period focused on (a) the maintenance of the existing VREs, (b) resources and tools provision, (c) the enhancement of the portal technology, and (d) supporting both project members and third party users while using the gCube technology.
For what concerns "VREs operation", the 13 VREs defined and deployed in the previous period (inherited by the D4Science-II project) have been maintained. There are no pending requests for additional VREs stemming from the EA-CoP through the WP3.
For what concerns "resources and tools provision", the study on identified data sources continued as well as early implementations of mediators facilitating access to species data have been released. In particular, the activity focuses on two data sources: OBIS and GBIF. Re OBIS, an early implementation of a mediator has been produced. Re GBIF, a prototype user interface allowing to discover and access occurrence points has been developed.
For what concerns "common interfaces and tools", a widget supporting the creation of "quick tour" per portlet has been designed and released. This will make it possible for portlet developers to create "quick tour" of their user interfaces. This facility is expected to be integrated in the next version of the portlets. Moreover, portlets have been upgraded to GWT 2.4. Besides this, a development portal has been deployed to support portlets development activity starting with forthcoming gCube 2.8.0.
For what concerns "development support" activity, the project team (mainly CERN) continued to support ISOFT/NAS (Ukraine) for the development and operation of a gCube based infrastructure. In particular, the enabling technology of the Ukraine infrastructure has been upgraded to gCube 2.7.2. In addition to that, the FORTH team has been helped to deploy a development portal and has been provided with guidelines for the exploitation of the HTTP ASL component.
- D6.2 "Applications and Tools Development Plan" (Lead beneficiary FAO, due date M3 a.k.a. January 2012) has not been produced. FAO is preparing a draft. CNR will support the finalization of this report;
- D6.1 "Virtual Research Environments Plan" has been finalized;
- A prototype mediator and a user interface for accessing GBIF data has been developed;
- A "quick tour" widget has been developed and released (including an how to guide);
- A deployment portal has been deployed;
- The enabling technology of the Ukraine infrastructure has been migrated to gCube 2.7.2;
WP7: Enabling-technology Integration and Distribution
WP Leader: E-IIS - Participants: CNR, NKUA
In the reporting period, E-IIS continued the activity aimed to provide the support in ETICS for the integration of maven components in org.gcube builds. CNR provided two updates for ghn-distribution v.3.2.5 and portal-bundle v.2.0.0 bundles for simplifying the management operations of the software resources deployed in the data infrastructure. They will be released in gCube 2.8 release. NKUA solved some problems with the Distribution Site that was not properly showing release's packages.
Deliverable D7.1 has been finalized in the reporting period.
Milestones MS30 and MS31 have been reached and documented.
The testing infrastructure setup has been delayed due to internal faults in E-IIS infrastructure. A new deadline for providing a fully functional testing infrastructure has been set by the end of February 2012.
- setup of org.gcube.HEAD to integrate maven components
- gCube 2.7.3 integration
- Deliverable D7.1
- Milestone MS30
- Milestone MS31
- gHN-distribution 3.2.5
- portal-bundle 2.0.0
WP8: iMarine Data e-Infrastructure Enabling Technology Development
WP Leader: CNR - Participants: NKUA, E-IIS, FAO
In the context of T8.1, there have been three directions of work:
- development of a new component for encrypting/decrypting textual information based on the AES standard algorithm for cryptography. This library has been then integrated in the current Resources Library in order to encrypt information published in the Information System;
- brainstorming activities on the new foundation technologies led to prototype some libraries that build on top of specifications instead of hosting environments;
- support to gCube Mavenization moved from the developer perspective to the infrastructure needs. The Software Repository underwent an analysis that will back the reshaping of the service around a cluster of Maven Repositories planned for the next month.
T8.2 is still in a course of study in which it is cooperating with T8.1 to understand the new directions of the foundations and implications on the security area.
T8.3 has been working on the implementation of a node selection and collocation policy library, by exploiting the existing limited functionality previously embedded in the PE2ng adapter employed by the search service and implementing additional methods and policies. Different Node Selectors and Collocation Policies have been implemented and tested. The primary goals of this work are (i) to support the upcoming execution abstraction layer which will be integrated into the PE2ng engine and (ii) to enhance the execution planner of the Search System.
T8.4 has produced a first draft of the classification of software resources in the new Resource Model, along with a high-level strategy to manage them under the new foundations. Some technical pointers on the implementation of the strategy are also included. Overall, the draft illustrates the new vision of a transparent and standards-based management for a broader class of software resources within and outside the system.
No major issues to report
- design and implementation of the
- developer's guide for the
- new version of
common-resourceslibrary integrated with
common-utils-encryptionfor encrypting sensible data inside gCube Resources
- definition of integration paths for introducing the new Resource Model in the existing enabling services have been identified
- analysis and experimentation for service's transparent resource management at run time
- prototype library for
- prototype library for
- sketched design for evolving the Security Subsystem as a pluggable and extensible module integrated with the Scope Management
- partial implementation of a node selection and collocation policy library
- a first draft of the classification of software resources in the new Resource Model,
- definition of a high-level strategy to manage software resources under the new foundations
WP9: Data Management Facilities Development
WP Leader: CNR - Participants: NKUA, CERN, TERRADUE, FAO
The activity on WP9 can be divided according to the tasks:
In T9.1 FAO mainly worked on the documentation of the plugin framework of the new Tree Manager Service and on a Tree Manager plugin for remote SPARQL endpoints. CNR created a prototype for the new Species Manager service.
In T9.2 CERN investigated on the data transfer protocols and on the enrichment of a document presenting the overall results about data transfer. Further tests on the protocols selected during December 2011 have been carried on. A first scheleton of the gCube Data Transfer Agent have been developed. NKUA activity regarded the extension of the gCube Result Set structure to the novel protocols to be supported. TERRADUE is continuing to work on THREDDS and WPS service. The activity has returned a first feedback about the use of a Thredds server, either using WCS or OpenDAP protocol. Then the focus moved on to the WPS block.
In T9.3 CNR worked on the tickets about Time Series components left on D4Science II project in order to produce requirement tickets on the iMarine project. The new tickets will be soon open on the iMarine issue tracker.
No corrective actions or deviations were necessary.
- FAO: documentation of the plugin framework of the Tree Manager service
- FAO: SPARQL plugin prototype
- CNR: Species Service prototype
- CNR: User Interface for Species Service prototype
- CNR: GBif Network plugin prototype
- CNR: iObis Information System plugin prototype
- CERN: investigation on the data transfer protocols and on the enrichment of a document presenting the overall results in this direction
- CERN: first scheleton of the gCube Data Transfer Agent has been prepared ( following new Maven structure),
- NKUA: implementing support to multiple transfer protocols
- NKUA: implementing of HTTP as gRS2 transfer method
- TERRADUE: integration of Thredds and WPS software. A prototype running with Bathymetry data should become available by mid of February.
- CNR: digest of tickets about Time Series Environment components coming from D4Science II project
WP10: Data Consumption Facilities Development
WP Leader: NKUA - Participants: CNR, FORTH, TERRADUE, FAO
During the period of January 2012 and regarding Data Retrieval, the design of the snippet mechanism for the gCube Search System was finalized by the involved partners, all development work has been completed and the mechanism is tested in development infrastructure in order to be ready to be included in the next gCube minor release. The snippet mechanism has been integrated into the search planner and in the CQL language as a projected field. Among the features of the implemented functionality are an identification mechanism for fields and terms that drive the creation of a snippet and the ability to configure the maximum size of snippets. The Resource Registry was also improved by fixing observed issues, while enhancements targeting performance improvements are under investigation. In the direction of enriching Data Retrieval facilities with semantic functionalities, the first actions in addition to the integration of snippets was the exploitation of the HTTP method of Application Support Layer for result retrieval from Search Service.
In the direction of providing Data Manipulation facilities, the new design of the gCube Data Transformation service is finalized and implementation has started.The new service is designed with scalability, bottleneck reduction, preservation of all useful features and new additions and improvements in mind. The most significant part of the design is the integration with the PE2ng Execution Engine, as DTS will now produce execution plans instead of performing the transformations locally. Each element of the execution plan instructs that a transformation be performed in a specific computational node having the required software. This software is independent of DTS, therefore the full set of computational resources present in the infrastructure can be used, as long as they support the required transformers. The transformation graph is consulted by a planner in order for the latter to create one execution plan per source data type which in turn are submitted to PE2ng. Result merging and final output into a data sink is also incorporated into a separate execution plan. Communication between computational nodes is performed via gRS2. Finally, the design allows the easy incorporation of future extensions regarding execution parallelization models which are to be provided by PE2ng and the further parallelization of the planning phase.
In the area of Data Mining and Visualization facilities, there has been further work on the development and testing of the Statistical Service which will provide data mining facilities on user data sources. The features of the service can be categorized into Generators, Modeling, Clustering, Transducers and Evaluators. Generators include probability distributions, classifications and matching or distance measurements, Modeling includes the models to be trained, such as neural networks or support vector machines, Clustering involves clustering procedures for grouping phenomena or multidimensional points, Transducers involve dataset transformation algorithms and Evaluators include a set of procedures for measuring the quality of a model. Currently several Aquamaps Generators have been implemented, two models (Aquamaps HSPEN and Feed Forward Neural Network) and several evaluators in the areas of Discrepancy and Quality Analysis. Furthermore, an experiment was performed to assess service perfomance, in which a feed forward neural network was used to simulate Aquamaps distributions for a specific species.
In the direction of providing Semantic Data Analysis Facilities, the use of search standards such as OpenSearch is being investigated for publishing search results using semantic web formats such as RDF. The design of a generic meta-search service is currently in progress. The service will employ an underlying search service via OpenSearch, which in this case will be the gCube Search Service and will provide advanced services for recall oriented needs, such as snippet-based result clustering, entity mining, etc. The output of these services will be made available also in OpenSearch formats, while the format of search responses and the selection of suitable ontologies and metadata schemas is under investigation. The development of a prototype has started.
- Design, implementation and testing of the snippet mechanism in the gCube Search System
- New version of Resource Registry included in gCube 2.7.3
- Finalization of the new design for gCube Data Transformation Service
- The Statistical Service has been tested and enhanced with new features and its performance has been assessed by performing experiments
- Quality analysis for the Statistical Service has been developed and improved and Statistical analysis has been introduced with a state-of-the-art system
WP11: Data e-Infrastructures Integration and Interoperability Facilities Development
During the third month of the project, work in WP 11 is mostly around the completion of the definition of the methodology and fine-tuning of the activities plan.
To better organize the WP tasks a further classification of the functional categories was considered necessary. The categories are now classified into functional areas. An analysis following the main principles of the WP (those driven from D4S, D4SII) will be performed on each functional area by the respective working group. This internal detailed organization of activities, is generally expected that will safeguard the overall deliveries of the workpackage, yet some fine tuning of timeline is to be decided within the next month.
The process of development has been drafted and established in wiki pages and a concrete plan has been presented and evaluated by WP partners. Two approaches have been launched in parallel to determine the upcoming tasks in further detail. In the context of this methodology, distinct framework layers and functional areas have been identified and working groups have been described. The members of one of the groups have been already assigned.
After an analysis of the needs and the options within the objectives of the WP, it was decided that
- there is little value and large cost in merging the existing Application Services Layer components with client libraries. Instead of undertaking such an approach, a framework of three-layer API is established.
- interaction with other "technology development" workpackages shall be strengthened, so that technological advances in these WPs are taken well into account in the Integration and Interoperability concepts and development.
In continuation of the activities analyzing the specifications of gCube services interface principles, the value of development or evolution of HTTP interfaces in some domains has been recognized. The designation of an HTTP API of the Explanatory Search Services has been analysed and client-side solutions to support the framework for local Java APIs have been prototyped. Another area of analysis has been the domain of OAI-ORE specification for handling the aggregations of resources in the data consumption layer of the infrastructure. Specifications defined by OGC are further analyzed along with the possibility to interact and contribute to their evolution along the line of Ecosystem enlargement.
In the area of development
- In the context of prototyping client-side solutions to support the framework for local Java APIs, a solution that deals with transparent scope propagation within the JVM and flexible scope map configuration has been developed and is under source revision
- Another solution that addresses issues of transparent interception of client calls over HTTP, particularly with regard to scope injection has been developed.
- New features have been added to the public HTTP Search interface of gCube
- There is a slight postponing of the activities of the WP, however along the new conceptualization of the WP work, which is compliant with initial objectives this better suits the work of the whole project, as alignment with other WPs activities is improved.
- Extension of the gCube exposed Search interface through HTTP
- Definition of API Framework principles and work areas
- Definition and implementation of two prototypes the common-scope prototype and the common-ghn-client prototype