2012.06 Activity Report
From IMarine Wiki
This Activity Report captures the activities performed in June 2012.
This report is organized in two parts:
- Progress and Achievements at the WP level reports a concise overview of the activities, issues and corrective actions, and major achievements produced by each WP Leader;
- Progress and Achievements by WP and Beneficiary reports a detailed description of the activities, issues and corrective actions, and major achievements produced by each beneficiary at the level of Project task.
Progress and Achievements at WP level
WP1: Administrative and Financial Management
WP Leader: ERCIM - Participants: CNR
- Monitoring and submission of the due deliverables and milestone achievement statement
- Consolidating effort contribution from partners for M7 and M8
- Preparing the first periodic report (M10)
- Preparing the first project review (agenda requested by PO)
- Implementing the project sub-contracts with NEAFC and VLIZ
- 6/13 partners have uploaded bi-monthly effort report. Reminders and follow up should be carried out next month as it is the due time for 10 month Periodic Report
- 15 Deliverables submitted up to now
- 26 Milestones achieved
WP2: Project Management
WP Leader: CNR - Participants: NKUA, FAO
- Day-by-day monitoring of the project scientific and technical activities;
- Organization and chairing of the June SB meeting(5 June 2012);
- Organization and chairing of the June PEB meeting(4 June 2012);
- Organization and chairing of the June TCom (18-21 June 2012);
- Preparation of a number of versions of the first review agenda and discussion of them with the iMarine appointed PO;
- Organization of a phone meeting with the project coordinator and the iMarine Board Director to take decisions about subcontracting (8 June 2012).
- Preparation of the presentation of the iMarine project to DG-MARE, delivery of this presentation and subsequent meeting with DG-Mare representatives;
- Initial steps performed in the organization of a workshop between iMarine and other Marine Knowledge related projects (GeoSeas, EuroMarine, Assemble, EMBRC, JPI Oceans, EMODNET)to be held on 11-12 October 2012;
- Monthly SB and PEB meetings;
- Quaterly TCom;
- Draft version first review agenda;
- Meeting with DG-MARE;
- Preliminary programme for a workshop between iMarine and other Marine Knowledge related projects.
WP3: Governance and Policy Development
WP Leader: FAO - Participants: ERCIM, CNR, NKUA, FORTH, TERRADUE, FIN, UNESCO, CRIA, IRD
- The iMarine Board closed its first meeting, and the Board report and work plan were finalized.
- Continued activities to raise interest in the use of the e-Infrastructure. FAO reported progress in e.g. the start of a Cotrix collaboration (sharing components and concepts), FIN prometes the use of VREs to WFC Atlasses and Coral MPA's.
- The e-Infrastructure was discussed with DG-MARE for support to BC1.
- FAO worked on a release of a next version of iMarine Data Access and Sharing Policies released.
- Work on COTRIX started; a tool to manage external codelists. It aims to add versioning and life-cycle support to entire code-lists and individual codes to the existing Vocbench tool (which is being rdf-ized). Aim is to have in November a working prototype for AFSIS species lists.
- FORTH continued on the Top-level ontology, and on X-Search to access FAO Species Profiles.
- SPREAD continued testing the JSON based JAVA re-allocation and the use of WPS.
- ICIS and the the gCubeApp for Biodiversity data management were reviewed.
None reported. The transition of activities of OBIS to Ostend is progressing. It requires that Ostend continues to familiarize with the tools, and prepares for the use and development of the tools.
- iMarine Board meeting 1 report closed.
- iMarine Board work plan finalized
- MS8 - Definition of policy and governance development model;
- MS11 - Definition of Guidelines and Best Practices.
- FLOD released a new UI
- Cotrix; code list manager activity started in FAO, with a tentative agreement of FAO/OEK to use Vocbench technology.
WP4: Communication, Dissemination and Training
WP Leader: TRUST-IT - Participants: CNR, NKUA, FAO
T4.1: Communication and Dissemination - Task leader: Trust-IT; Participants: CNR, NKUA, FAO; Generation of 5 interview texts with members of iMarine Consortium:
- Donatella Castelli – iMarine Project Director – outstanding
- Pasquale Pagano - iMarine Technical Director – outstanding
- Andrea Manzi - iMarine Deputy Technical Director - completed
- Ward Appeltans – OBIS Project Manager – outstanding
- Julien Barde – IRD – outstanding
Generation of 1 interview with Mark Fornwall, Chair of the IODE Steering Group for OBIS to be released in July 2012
Preparation of 1 interview with Francky Callaewart, EC DG MARE – to be revised according to DG MARE meeting notes & followed up in July
1 project overview generated & published in ICES newsletter
Drafting iMarine 16 month calendar – proposal to be circulated to consortium by end of July for production in early September
Involvement in organisation of All Marine Knowledge Meeting for October 2012 Brussels
Submission of a contribution to EGI Technical Forum 2012 titled "Managing Virtual Research Environments in Hybrid Data Infrastructures" by CNR
NKUA representatives participated a consensus building meeting among three projects: iMarine, EarthServer and ESPAS, held in Greece on 29 June 2012. In this meeting the gCube system has been briefly introduced and the discussion has touched the potential of integrating gCube technology with array databases under various perspectives.
37 new entries in Dissemination & Contacts database
T4.2: Knowledge Management Channel - Task leader: Trust-IT; Participants: CNR, NKUA, FAO;
Generation of iMarine Application Texts in collaboration with FAO & CNR
- Ecological Niche Modelling – new area on channel with link from home page. Extended (downloadable pdf) & edited version of texts available on channel * Integrated Capture Information System – new area on channel with link from home page. Extended (downloadable pdf) & edited version of texts available on channel
- Vessel Transmitted Information– new area on channel with link from home page. Extended (downloadable pdf) & edited version of texts available on channel
Editing & upload of 1 interview with Andrea Manzi - iMarine Deputy Technical Director
T4.3 User Community Capacity and Competence Building - Task leader: FAO; Participants: CNR, NKUA, US, Trust-IT; NKUA has contributed to User Community Capacity and Competence Building via presentations or contribution made during the 2nd TCom sessions, most notably the ones referring to Process Execution and Search/xSearch integration which are gCube facilities within direct reach of user communities.
The planned interviews and dissemination releases were not achieved in June and will be targetted for completion in July. The planned demos to support the iMarine Apps (FAO & CNR) will be completed in July 2012.
- 1 June 2012 newsletter released
- 13 new iMarine Channel Members
- 1 video interview recorded, edited and released
- Contribution to EGI Technical Forum 2012.
- Revision of homepage & inclusion of iMarine Apps on channel
- 37 new entries in Dissemination & Contacts database
WP5: iMarine Data e-Infrastructure Deployment and Operation
WP Leader: CERN - Participants: CNR, NKUA, FAO, FIN
CERN has organized and started the infrastructure upgrade to gCube 2.9.0. During the period, both iMarine and D4science.org portals have been upgraded together with the gCube Apps VOVirtual Organization; services. The preproduction infrastructure ( Ecosystem VOVirtual Organization; ) has also started to be upgraded, activity which is still blocked because of some issues on the remote undeployment facilities.
The new VRE BiodiversityResearchEnvironement has been also deployed and made accessible trough the D4science.org portal.
The status of the infra, the current issues and the plans have been presented during the iMarine TCOM in Rhodes
CNR has been keeping up and running the provided nodes hosting GHNs. The configuration issue on UMD1 reported in M7 has been solved.
NKUA has been keeping up and running the provided nodes hosting GHNs, UMD and the gcube wiki. During this month, among others, we attended 2nd TCom Meeting, encountered and solved computer-room's air-conditioning issues, tested upgrading gcube wiki, upgraded dl19.di.uoa.gr memory from 2 to 3 gbs because containers were running out of memory and solved UMD Apel publishing issues.
Documentation related to gCube infrastructure monitoring using Nagios has been edited on the gCube WIKI.Some tests have been performed on the new Nagios server to integrate NRPE and new servicegroups/hostgroups layout.
The new ActiveMQ v5.6 with Kaha DB as backend has been deployed in the infrastructure during the upgrade to gCube 2.9.0.
- The infrastructure upgrade is still not completed ( FARM VOVirtual Organization; and part of the iMarine portal components still running gCube 2.8.1).
- 70% of VMs hosting GHNs at CNR went down for a power cut, all VMs have been restarted and GHNs are up and running
- The BiodiversityResearchEnvironement VRE has been deployed.
- Upgrade to gCube 2.9.0 started
WP6: Virtual Research Environments Deployment and Operation
WP Leader: CNR - Participants: ERCIM, NKUA, FAO, FIN, UNESCO, CRIA, IRD
Re VRE Operation, the previosly create VREs have been maintained, e.g. they have been monitored, requests for subscriptions have been managed. A new VRE "BiodiversityResearchEnvironment" has been deployed to provide the community with the Species Products Discovery facilities. In particular, this VRE supports the discovery of Occurrence Points from data providers including GBIF and OBIS.
Re Resources and Tools provision, a new Species Products Discovery plugin has been developed to interface with the NCBI data source. Morever, minor bugs affecting the OBIS plugin has been fixed.
Re common interfaces and tools, a lot of new or enhanced facilities have been developed during the period:
- the workspace has been revised with a new GUI supporting WebDAV and a new backend aiming at enhancing the discovery performance;
- the widget allowing portlets to access workspace items has been enhanced to improbe the visualization performance;
- a new version of the Species Products Discovery portlet supporting the specification of filters (eg data source, date, geographic boundaries) and providing users with detailed Information on discovered products;
- a new set of facilities for declino with tabular data have been produced. This includes a widget for visualising tabular data and a JDBC-based data access module;
- an enhanced version of the Software Management Widget allowing to register a rich array of software;
- a revised version of the SearchManager GUI to Support the needs of the semantic search;
- a new version of the UserManagement GUI to cope with tickets #398 and #400
No deviation to report.
- A new VRE - BiodiversityResearchEnvironment - has been deployed;
- A plugin for interfacing with the NCBI data source has been developed;
- A new GUI of Species Discovery Portlet has been released in gCube 2.9.0;
- The GUI of Workspace Portlet has been updated and released in gCube 2.9.0;
- Workspace light tree widget has been optimized and released in gCube 2.9.0;
- First alpha of TabularDataWidget as been released and integrated in Statistical Manager Portlet;
WP7: Enabling-technology Integration and Distribution
WP Leader: E-IIS - Participants: CNR, NKUA
Integration of gCube 2.9.0 release continued in the reporting period. E-IIS performed integration builds and, mainly, deployment tests. E-IIS, CNR and CERN collaborated in discovery and resolving some dynamic deployment issues.
gCube 2.9.0 release have been made available in Maven Repositories making possible the upgrade of production infrastructure.
Regarding documentation and distribution acitvities:
- NKUA put in production the new version of the Distribution Site including the updated web pages and the javadoc crawler
- E-IIS adapted the ReleaseNotes Generator tool to new distribution reports
- NKUA produced a news in gCube website concerning gCube 2.9.0 release. Available here
The major issues faced in the reporting period and the identified corrective actions should be reported here
The WP leader should report here a bullet list highlighting the main achievements of the reporting period
- gCube 2.9.0 artifacts made available for deployment in production infrastructure
- new distribution site in production
WP8: iMarine Data e-Infrastructure Enabling Technology Development
WP Leader: CNR - Participants: NKUA, E-IIS, FAO
The biggest area of work for T8.1 in June was the Virtual Research Environment management. The task was almost completely devoted to design and implement a new approach for creating VREs driven by functionality instead of services. VREModeler and Resource Manager have been ported on the Maven platform and implementation on both of them is progressing. Specifically, The work on Resource Manager was along three main directions:
- capability of managing multiple scopes in the same instance (ticket #348)
- capability of deploying/undeploying single gCube software packaging, web applications, plugins (ticket #410)
- future possibility of package upgrading (ticket #554)
The SoftwareGateway service has been modified about policies for dependencies resolution.
T8.1 also worked to solve several minor integration issues allowing to finalize gCube 2.9.0.
T8.2 completed the work on the Policy Management library and the component has been published on Etics. A version of the UserManagement library, providing CRUD operations on users records stored in an LDAP directory, has been developed and it is also available on Etics. The development activities on the REST interfaces of the Authentication module and the User Management Module have been started as well as the activities of study possible improvement and integration of Shibboleth Identity Provider.
T8.3 has been working in enriching the hosting node information supported by the corresponding entity and adaptors within the node selection library in order to prepare the ground for the integration of PE2ng with Resource Registry. Following some requirements collected at the TCOM, the incorporation of a queueing mechanism as a layer operating on top of the Execution Engine is being designed while its interrelation with the computational models which will be introduced in the near future is being investigated.
The Workflow adaptor implementing the execution layer of the gCube Search System has been made more flexible by modularizing the data source wrappers which implement the logic of web service invocations.
T8.4 has worked on D8.4 and completed the work on the new resource model to the degree required for D8.4. Moreover, T8.4 held a session at the TCOM to introduce the Interface and Function Layers.
- complete the work on VRE creation and package deployment/undeployment
- resume the work on the new foundations
- continue the implementation of the security components according to tickets
- introduce the package upgrade features within nodes and VREs
- adapt the Resource Registry repository provider for gCube to enrich the hosting node information
- integrate PE2ng with Resource Registry
- D8.2 (due in August)
Resource Manager able to manage multiple scopes has delayed in favor of a better integration with future capabilities of the service. Globally, these features won't be in delay, but just released all together.
- Formalization of the Resource Model (2nd generation)
- Release of deliverable D8.4
- Resource Manager service able to handle multiple scopes
- New Resource Manager public interface
- Design of deployment/undeployment features for Plugins, WebApplications, Libraries and single gCube packages
- Policy Management library is ready to be released
- LDAP User Management library is ready to be released
WP9: Data Management Facilities Development
WP Leader: CNR - Participants: NKUA, CERN, TERRADUE, FAO
- FAO begun the re-factorisation of the tree-manager-library according to the specifications of WP11, with related documentation
- CNR team finished with the development of the Synonyms and Layers Capabilities for the Species Discovery Service
- CERN worked on extending the Data Transfer Agent in order to implement asynchronous operations
- NKUA worked on improving the TCP transfer method performance of gRS2
- NKUA worked on providing the capability of transporting errors between consumers and producer in the gRS2 library
- NKUA worked on the gRS2Broker in order to present the data in XML format instead of HTML format
- Terradue continued with supporting WPSHadoop CNR and FAO developers and fixed related bugs
- CNR worked on a reusable component for tabular data visualization and processing
- CNR released a Tabular Data widget alpha version
- CNR created an interface for data sources definition and access through JDBC
- CNR performed a first integration of the Tabular Data Widget with the Statistical Manager Portlet
No deviations to report
- Synonyms and Layers Capability were implemented in the Species Discovery Service
- The Data Transfer facility was enhanced
- The gRS2 was enhanced
- WPS-hadoop adoption is growing due to CNR and FAO developers evaluating the framework
- A first alpha of Tabular Data Widget was developed
WP10: Data Consumption Facilities Development
WP Leader: NKUA - Participants: CNR, FORTH, TERRADUE, FAO
During this period, deliverables D10.1: gCube Query Language Specification and D10.3: iMarine Data Consumption Software were finalized and submitted. In addition, final decisions on the exact means of communication and integration between the gCube Search System and Semantic Search have been made. In particular, it has been decided that (i) Semantic Search will comprise a separate Semantic Search Service in the infrastructure and a Semantic Search Portlet; (ii) the existing Search Portlet will include a means to enable Semantic Search facilities and will forward the outcome of the search query to the Semantic Search portlet, along with the search terms in free text format. The latter will use the provided software abstractions to read the returned results and will use its own presentation mechanisms in order to present the produced semantic information; finally, (iii) the gCube ResultSet will be used as a means of transfer of the needed data, such as result snippets and the payloads of annotated fields, from the portlet to the Semantic Search Service. Development work both in the area of the Service and the Portlet has started and involved partners are in contact in order to provide support where needed. Activity in the context of Semantic Data Analysis also included the refinement of the MS45-Semantic Data Analysis Specification.
In the area of Data Manipulation, the distributed computing capabilities of the Data Transformation Service have been enhanced by exploiting former knowledge about the transformation, in order to construct efficient workflows. Regarding geo-spatial data manipulation, focus was on the integration of a Thredds Service with GeoNetwork and the management of NetCDF-CF files. The Environmental Explorer library was developed in order to perform intersections of geo-spatial layers with coordinate points. In addition there has been activity in WPS integration based on the WPS-hadoop framework developed by Terradue in WP9.
Finally, in the context of Data Mining, the efficiency of the Statistical Manager was enhanced. A connection to the Executor engine was implemented in order to parallelize algorithms on several GHN. The performance and robustness of the system have been evaluated and reported. On the other side, the Statistical Manager was redesigned in order to rely on a external message queue for user requests. In the area of Data Visualization, the GeoExplorer portlet was improved by designing the management of multiple GIS workspaces and performing optimization and bug fixing of the current portlet and a WPSHadoop connector to a WMS hosted by GeoServer is under design, according to the outline from the 1st TCom meeting.
The major issues faced in the reporting period and the identified corrective actions should be reported here
- Finalization and submission of deliverable D10.1: gCube Query Language Specification
- Finalization and submission of deviverable D10.3: iMarine Data Consumption Software
- Finalization of design of gCube Search and Semantic Search integration
- NetCDF-CF files management and publication in D4Science
- Development and testing of Environmental Explorer library
- Release of Thredds in the infrastructure
- Thredds\NetCDF-CF files are indexed on GeoNetwork
- Design of Map Reduce procedures in D4Science
- Performance evaluation and usage of the Executor for distributed computations
- Test of failure and robustness for the distributed computation
- Re-design of the Statistical Manager for managing requests by means of an external queue
- Enhancements in GeoExplorer Portlet
WP11: Data e-Infrastructures Integration and Interoperability Facilities Development
WP Leader: NKUA - Participants: CNR, CERN, E-IIS, FORTH, TERRADUE, FAO
During this period, work in WP11 focused on the refinement of the Integration and Interoperability framework and the launching of implementation towards its practices. The CL Application Framework and the CL Design model have been revisited, both in code, associated documentation and examples. Extensive presentations have been prepared on both model and framework for the CL and the HTTP Front End layers of the system and have been delivered at the 2nd TCOM. A concrete plan has been decided on the next implementation steps towards the adoption of the CL framework practices from existing or new client libraries. The timeplan prioratized the integration of Client Libraries for the Tree Manager, Species Manager, Occurence Service, Transfer Agent and Statistical Manager and foresees with lower priority the integration for Resource Manager, Search, DTS, Personalization and Workflow Engine.
The management model for Client Libraries has also been completed and presented here: https://gcube.wiki.gcube-system.org/gcube/index.php/Integration_and_Interoperability_Facilities_Framework:_Client_Libraries_Management_Model. The model describes best practices and/or tools in the areas of:
- building of client libraries, including interactive builds and continuous integration builds;
- profiling of client libraries as system components
- packaging of client libraries for distribution purposes
- testing of client libraries, including unit testing an integration testing
Moreover, the discussion on the architectural details of the common-security library has lead to the decision for structuring the client security library in a similar way to the common scope library. It is planned to be developed as a part of the connector between gCube and SOA3 and to be released by the end of July.
In the context of Data Management APIs activity, the Data Transfer agent CL has been modified to be compliant with the Integration and Interoperability facilities framework for CL.
In the context of Data Consumption APIs, the implementation of the gCube-OaiORE provider has been launched that will support the OAI-ORE standard to expose gCube compound digital objects as aggregations to external applications and the Search CL has been implemented in compliance with the CL framework of T11.1.
Finally, the milestone MS47:'Data Management APIs Secification' has been achieved, with the inclusion of all the APIs of the software subsystems respective to the functional category and the analysis of their key features, framework compliance and adoption of protocols and specifications.
None to report.
- Management Model for CL Libraries completed
- Data Transfer agent CL integrated with Integration and Interoperability FWK
- MS47: Data Management APIs Specification achieved
- Prioritization List for API implementation