2012.07 Activity Report
From IMarine Wiki
This Activity Report captures the activities performed in July 2012.
This report is organized in two parts:
- Progress and Achievements at the WP level reports a concise overview of the activities, issues and corrective actions, and major achievements produced by each WP Leader;
- Progress and Achievements by WP and Beneficiary reports a detailed description of the activities, issues and corrective actions, and major achievements produced by each beneficiary at the level of Project task.
Progress and Achievements at WP level
WP1: Administrative and Financial Management
WP Leader: ERCIM - Participants: CNR
- Consolidate effort report from M1-M8
- Prepare the 2nd Advance Payment ( will be proceeded after 15 Aug)
- Prepare 1st Periodic Report template and ask for contribution from all partners by 17 Sept 2012
- Follow and submit deliverables and milestones
- Prepare iMarine subcontract template
- Completion of 1st Periodic Report template
- Completion of subcontract template
- 15 deliverables submitted and 28 milestones achieved
WP2: Project Management
WP Leader: CNR - Participants: NKUA, FAO
- Day-by-day monitoring of the project scientific and technical activities;
- Organization of the July PEB (16 July 2012) and SB (17 July 2012). During the SB the release of the second part of the advanced payment has been discussed and compared with the effort claimed by each partner;
- Series of actions aimed at increasing the number of contributors in the project activities (e.g. Town Peterson in the validation of the biodiversity cluster activities);
- Meeting with different representatives of the LifeWatch research infrastructure to discuss possible collaborations (f2f meting 19 July 2012, Rome and series of telcos);
- Preparation of a high level draft agenda of the joint TCom+GA+iMarieneBoard October meeting;
- Organization of iMarine+VLIZ+OBIS meeting to be held in Ostende on 28 August 2012 dedicated to discuss (i) pending issues related to the OBIS exploitation, (ii) the involvment of VLIZ as iMarine subcontractor and (iii) possible future collaborations between iMarine and VLIZ beyond the project already scheduled activities.
- Finalization of the June activity report and preparation of the template for the July one.
- July SB and PEB;
- First draft of the joint TCOM+GA+iMarine Board meetings;
- Meeting with representatives of the LifeWatch research infrastructure;
- Organization of iMarine+VLIZ+OBIS meeting to be held in Ostende on 28 August 2012;
- Finalization of the June activity report and preparation of the template for the July one.
WP3: Governance and Policy Development
- Scheduled and discussed the iMarine Board meeting in October in Rome
- Practical organization of the Board Meeting
- Planned for a meeting with VLIZ / IOC / UNESCO
- Produced and dispatched the CFRAME position paper regarding BC1 / FLUX & Fishframe use cases
T3.2; Governance mechanisms
- A next draft on "Data access and sharing policies" for discussion with the imarine board was produced
- The Governance pages were further developed towards D3.2
- Technology discussion on geo-spatial for recommendations to project and communities
- BC2 discussion started on reconciliation governance
- Discussed the collaboration with other FAO departments on a Vocabulary / code list manager
- ICIS review report was finalized
- Discussed new VRE proposals for VME-DB and Species Fact Sheets
- Validated FCPPS with FAO data managers for next round of FCP's
- Continued development of SPREAD functionality with WPS
- FLOD contributed to the development of the top level ontology
- ICIS review report
- reporting tools comments
- FCPPS validation
- VME-DB feedback
- Fishfinder proposal
- COTRIX code list manager and vocabulary mangement draft discussed
WP4: Communication, Dissemination and Training
WP Leader: TRUST-IT - Participants: CNR, NKUA, FAO
- Development of iMarine Scientific Poster;
- Development of iMarine FishMApp on Species Fact Sheets specifications & design based on conference calls with CNR & FAO together with outline and preliminary draft SQLite DB with dummy content delivered to TRUST from FAO;
- Editing & circulation of iMarine July 2012 Newsletter;
- Liaison with OBIS for interview with Mark Fornwall, current Chair of the IODE Steering Group for OBIS, on Marine Biodiversity Data Management;
- Generation & circulation to 15 press contacts of iMarine press article on achievements to-date;
- Definition of iMarine Calendar for circulation to partners for feedback in August and printing in September 2012;
- Liaison with partners for outstanding video interviews and iMarine App demos;
- Participation at the GBIC Conference (2-4 July 2012, Copenhagen) CNR represented the iMarine project;
- maintenance of the iMarine channel & web conferencing system;
- Training of FAO staff on FCPPS, and collection their feed-back, including usability;
- Training / demo video on ICIS Curation;
- Discussed ICIS validation results to be re-used in training material / manuals;
- Conference call between Trust & FAO for training activities to discuss on-line training tools and demos.
Internal video interviews and iMarine App demos delayed
- July 2012 iMarine eNewsletter sent to 360+ contacts;
- Article entitled Introducing iMarine - Science driven policy information management published in ICES Inside Out July Newsletter ;
- Interview with Dr. Mark Fornwall, current Chair of the IODE Steering Group for OBIS, on Marine Biodiversity Data Management published on web channel and promoted in July 2012 eNewsletter ;
- Design of a first version of the iMarine mobile app sepcifications - iMarine FishMApp;
- Project dissemination at the GBIC Conference (2-4 July 2012, Copenhagen);
- ICIS video on curation;
- FAO Staff trained on FCPPS;
- Dummy database for discussion on contents and formats, structure and constraints, synchronization and versioning.
WP5: iMarine Data e-Infrastructure Deployment and Operation
WP Leader: CERN - Participants: CNR, NKUA, FAO, FIN
The Infrastructure upgrade to gCube 2.9.0 has been completed on 13 July. In addition the plan for the 2.9.1 upgrade to gCube 2.9.1 has been prepared ( upgrade scheduled on the 1st aug).
It has been discussed together with member of WP7 and Wp8 how to enhance the Integration and infrastructure updrade activities, in details:
- Editing of a guide for the generation of Maven based portlets in gCube ( together with CNR and ENG)
- discussions on the next version of RM and Deployer to support package upgrade ( together with CNR)
The Deliverable D5.2 ( as type report ) has been edited together with CNR. The Deliverable is under official review.
CNR has been keeping up and running the provided nodes hosting GHNs and UMD, In addition 12 new VMs have been created for Hadoop.
NKUA has been keeping up and running the provided nodes hosting GHNs, UMD and the gcube wiki.
The deliverable D5.2 has been extended with a section related to Monitoring and accounting Part. Statistics of the first 9 Months of the project have been prepared and included in the deliverable.
All the servers hosted in CNR have been moved to an ad-hoc subnet, and this has caused some issues ( reported in TRAC).
- Infrastructure upgrade to gCube 2.9.0 completed
- Deliverable D5.2 edited
WP6: Virtual Research Environments Deployment and Operation
WP Leader: CNR - Participants: ERCIM, NKUA, FAO, FIN, UNESCO, CRIA, IRD
Two VREs were created and put in operation: Biodiversity gCube Apps VRE and VME-DB VRE. A tailored FCPPS training was performed by FAO. The Tuna-Atlas use case based on the ICIS VRE produced an extensive validation report. Actions to fulfill the Tuna-Atlas requirements were discussed and a plan for the implementation of the new features was defined.
Several improvements were implemented to enhance the current gateways toward external Biodiversity providers. In particular the NCBI plugin was implemented and the OBIS plugin was extended by including validation of the data (e.g. the number of invalid and unrecognized dates was reduced from 7039 to 248). All biodiversity gateway can now deliver the classification capability. By implementing this capability, each gateway can return taxonomic information about the species, e.g. common names, phylogeny, and synonyms. Finally, all gateways can now profit from the transparent caching mechanism offered by the underlying framework.
Two major developments are progressing according the agreed plan, Software Management Widget and Social Networking facilities. The former is dedicated to the infrastructure maintainers and the communities manager. It allows to manage several kind of software - e.g. web service, web application, software patches, etc - through a simple and guided wizard. The release of this new component is planned for the next minor release. The latter is a completely new way to exploit the portal. It allows to exchange information, results, and links as in the most common and modern social networks. A prototype of the new social networking facility is available in the development infrastructure and it is planned to be released by the end of September.
The Search portlet was extended to support the semantic search while the xSearch portlet is under development by Forth.
- D6.3 is in delay. Actions to recover from delay and to deliver the deliverable by the end of August were agreed and now they are progressing as expected.
- the update of the User Management portlet is stuck due to a limitation of the user management core library, #398 ticket. Action by ENG is requested.
- FIN, Unesco, CRIA, and IRD did not report and perform any activity in July.
- a new VRE, VME-DB (Vulnerable Marine Ecosystem Database), has been created
- 2 FAO stuff were trained on the use of FCPPS
- validation of the first version of the Biodiversity VRE was performed
- NCBI gateway is ready for release
- all biodiversity gateways were improved and enhanced with the classification capability
WP7: Enabling-technology Integration and Distribution
WP Leader: E-IIS - Participants: CNR, NKUA
In the reporting period, E-IIS executed the integration of a new maintenance release of gCube (2.9.1) and completed the closure of gCube 2.9.0.
E-IIS, CNR, FAO and CERN started a discussion aimed to reduce the duration of release procedure by eliminating inefficiencies in the procedure and optimizing building and tests tasks. E-IIS carried out several tests on per-subsystem builds of the project and on incremental builds. Both this aspects will reduce time to have results of integration builds.
E-IIS re-factored partially BTRT tool and release scripts in order to support the per-subsystem builds.
NKUA started a reorganization of Developer's Guide proposing a draft for a new Table of Content. After a validation with other partners, the new TOC will replace the current one.
E-IIS and NKUA started the production of deliverable D7.2 which will due on next month.
Nothing to report
- gCube 2.9.1 integrated and released
WP8: iMarine Data e-Infrastructure Enabling Technology Development
WP Leader: CNR - Participants: NKUA, E-IIS, FAO
T8.1 has continued the work on the new Resource Manager service started in the previous month. Several new features have been added to the service and its capabilities greatly expanded. Along with them, the service is now able to create, manage and destroy multiple scopes within the same instance. JUnit tests integrated with Maven are under implementation allowing a comprehensive and layered testing reproducing several runtime conditions of the RM's service.
T8.2 has finalized the implementation of SOA3 components. In particular:
- the LDAP based Authentication
- the Authentication REST interfaces
- the User Management Service REST interface
These modules are completed and available on Etics in the HEAD configuration. The activity of integration of Shibboleth Identity Provider and Service Provider with the rest of SOA3 architecture is ongoing.
T8.2 and T8.1 have also started the preparation of deliverable D8.2.
T8.3 has been working in four directions:
- evolution of the node selection library, with several enhancements and fixes
- enhancement of the distributed execution capabilities of the Workflow Adaptor used by the gCube Search System
- solving observed issues in the PE2ng Execution Engine
- design and implementation of a queue based scheduling facility for PE2ng
T8.4 has no activity to report this month.
- T8.1's priority is to release Resource Manager 2.0, then resume the work on the next foundations and Resource Management Library
- T8.2 to stick to the implementation's roadmap of SOA3 defined in TRAC
- T8.3 to continue the evolution of PEng and integration with the Search System
- Deliverable D8.2
- T8.1 has not been resumed yet the work on the new technological foundations
- Resource Manager is able to create, manage and destroy multiple scopes within the same instance.
- Design and implementation of LDAP based Authentication module
- Design and implementation of Authentication REST interfaces module
- Design and implementation of User Management Service REST interface module
- Design of PE2ng queue based scheduling facility has been completed.
- Implementation and testing of Search distributed execution
WP9: Data Management Facilities Development
WP Leader: CNR - Participants: NKUA, CERN, TERRADUE, FAO In T9.1:
- FAO worked on a new version of the tree-based access subsystem meeting the requirements of the last version of the gRS2 library
- CNR enhanced the Species Product Discovery service by introducing the concept of caching
- CNR enhanced the Species Product Discovery service by building a query language for external interrogations
- CERN designed the Data Transfer Scheduler Service along with the Data Transfer Scheduler DB Library
- CERN enhanced the Data Transfer Agent and Library by means of asynchronous transfers handling
- NKUA completed the the fault transportation management between consumers and producer in the gRS2 library
- Terradue did not perform activities in July.
- CNR planned a complete refactoring of the Time Series Environment in order to address the requirements for supporting SPREAD, the Atlas-Tuna use case and the old requirements contained D4Science-II wish list
- CNR planned a modular design of the Time Series Service in order for this to be reused in other infrastructure services
No deviations to report
- The Species Product Discovery service was enhanced and released in the development infrastructure
- A SPQL language parser was produced for the Species Discovery service
- The Data Transfer Scheduler Service and the Data Transfer Scheduler DB Library were designed
- The fault management was introduced in the gRS2 library
- A Time Series Environment refactoring plan was designed
WP10: Data Consumption Facilities Development
WP Leader: NKUA - Participants: CNR, FORTH, TERRADUE, FAO
During this period, there has been progress in the activity of integrating XSearch with the gCube Search System. There has been close collaboration by the two involved partners to ensure that the activity remains within the target time frame. In the context of this activity, two components have been developed: the XSearch Portlet and the XSearch Service. The XSearch portlet acts as a consumer of search results and performs an analysis of the results. In the current version of the portlet, the analysis is perfomed over the textual metadata of the results and their snippet, as given from gCube Search. The portlets provides the necessary information to the XSearch service by means of gRS2. The results from the XSearch service, in particular the entities that have been mined from the search results and the search results clustering, are then returned back to the XSearch portlet and are presented to the user.
Remaining activities in the Data Retrieval area were (i) enhancements of the gCube Search Service to support the latest version of the Workflow adaptor for search developed in the context of WP8, which features enhanced distributed execution features and (ii) enhancements and fixes in the Resource Registry, with the introduction of new plugins and the resolution of a number of issues and omissions.
In the context of Data Manipulation, the integration of the new node selection library which is part of org.gcube.execution.MadgikCommons into Data Transformation Service Workflow Adaptor has been performed. In that way, during the construction of a data transformation execution plan, the node selection library is responsible for finding the appropriate execution nodes that will be used. During the exploration of the available nodes, Merger node must take into account the high demands for i/o, instructing high resource needs. In this case, a special purpose cost function based selector named "best node selector" is invoked. The rest of the nodes must distribute evenly to all available nodes, considering the distance between subsequent execution nodes. For this case, distance based selector with MRU policy is being used.
Finally, in the area of Data Mining and Visualization, a distributed network of Executor instances along with an ActiveMQ installation was used for performing data mining experiments in parallel fashion. Tests were made in order to evaluate performances and a first version is currently ready to be integrated with the Statistical Service. The DBSCAN algorithm was investigated about performances and limitations for its application to species occurrence points clustering. In addition, a GDAL Java bindings-based library called gtuploader was developed to retrieve all layers from a NetCDF file and their metadata, in the context of the WPS-Hadoop activity.
The major issues faced in the reporting period and the identified corrective actions should be reported here
- Development of first versions of XSearch Portlet and Service
- A beta version of a parallel processing system for niche modeling experiments was released
- The DBSCAN algorithm was implemented and evaluated in the context of the Statistical Manager Service.
WP11: Data e-Infrastructures Integration and Interoperability Facilities Development
WP Leader: NKUA - Participants: CNR, CERN, E-IIS, FORTH, TERRADUE, FAO
During this month, work in WP11 has progressed in all tasks. In task T11.1, the Access Management utility of the HTTP framework was extended to support anonymous access based on the server configuration, at deployment time, which binds the installed web application to a specific scope. In those cases, there is no requirement for scope awareness clint side. This mechanism could be used withing the HTTP URIs resolution mechanism for accessing gCube content across infrastucture scopes. An early discussion has started on this subject, targetting the adoption of web friendly HTTP URIs to disseminate all content from within the infrastructure. The main facets under investigation are related to the adoption of HTTP Front End framework, the attachment of a contextual prefix to URI fragments produced by service-specific components and the deployment of the dedicated websevers as part of the infrastructure. Moreover, the implementation of the CredentialManager which is responsible to manage host certificates has started. A number of Open Source APIs have been tested as possible candidates to be the client API for ID-Federation Service Provider. The goal of this activity is the integration of the Enhanced Client or Proxy Profile and the Attribute Authority of Shibboleth in gCube.
In T11.2, the tree-manager-library, and the spd-client-libary and the Data Transfer Agent Library have been refactored towards compliance with the CL framework, as per TCOM plans. The refactoring of the tree-manager-library will incorporate recent changes to the underlying gRS2 resultset, which serve as an indirect dependency of the library. Moreover, the Data Transfer Agent Library has been enhanced with Async delegates.
In T11.3 the OAI-ORE provider for gCube has been implemented, as part of the HTTP Front End Layer of the Integration and Interoperability Framework. It offers an OAI-ORE interface for disseminating gCube documents and collections as OAI-ORE Resource Maps. The details of the implementation are described here: https://gcube.wiki.gcube-system.org/gcube/index.php/HTTP_Front_End:_OAI-ORE_Implementation
None to report
- Extension of Access Management utility for configuring web applications bound to specific infrastructure scope, for Open Access.
- Refactoring of tree-manager-library towards CL FWK adoption
- Refactoring of spd-client-libary towards CL FWK adoption
- gCube OAI-ORE Provider implementation