2012.08 Activity Report
From IMarine Wiki
This Activity Report captures the activities performed in August 2012.
This report is organized in two parts:
- Progress and Achievements at the WP level reports a concise overview of the activities, issues and corrective actions, and major achievements produced by each WP Leader;
- Progress and Achievements by WP and Beneficiary reports a detailed description of the activities, issues and corrective actions, and major achievements produced by each beneficiary at the level of Project task.
Progress and Achievements at WP level
WP1: Administrative and Financial Management
WP Leader: ERCIM - Participants: CNR
During August 2012, tasks implemented by ERCIM:
- Proceed 2nd Advanced Payment for 5 beneficiaries: ERCIM, CNR,NKUA,TRUST-IT, CERN (completed 24 Aug)
- Beneficiaries who were approved by SB from SB Meeting (4 September 2012) include: FORTH, E-IIS, UNESCO, IRD, FAO (to be launched by 11 Sept)
- Beneficiaries who are waiting for validation: Terradue, FIN
- Beneficiary has not uploaded effort report: CRIA
- Produce and deliver Periodic Report 1 Template to the consortium - All partners must send their contributions by 17 Sept 2012
- Follow and submit deliverables and milestones
- 2nd advance payment completed for 5 partners
- Periodic Report template completed and start the collection phase by 17 Sept 2012
WP2: Project Management
- Day-by-day scientific and technical monitoring of the project activities;
- Chairing of the August PEB (6 August 2012);
- Interaction with some project partner to clarify and better focus their activity in the project;
- Discussion with LifeWatch representatives for the organization of a dedicated gCube+D4Science training event;
- Finalization of the Agenda for the “Marine Knowledge All Projects Meeting” to be held on 11-12 October in Brussels;
- Organisation of a telco with Townsend Peterson to discuss about a possible collaboration between iMarine and his research team;
- Discussion about possible liaisons with EMODNET Biology during the iMarine+VLIZ+OBIS meeting held on 28 August 2012 in Ostende;
- Finalization of the July activity report and preparation of the template for the August one.
None to report.
- Finalization of the July activity report and preparation of the template for the August one;
- August PEB;
- Agenda for the “Marine Knowledge All Projects Meeting” to be held on 11-12 October in Brussels;
- Establishment of new collaboration opportunities with other projects and initiatives;
- Improved activity plan.
WP3: Governance and Policy Development
- Practical organization of the Board Meeting
- Meeting with VLIZ / IOC / UNESCO
T3.2; Governance mechanisms
- The Governance pages were further developed towards D3.2
- BC2 discussion on reconciliation governance for biodiversity data continued
- IRD setting up Opensearch protocol to enable data discovery on Ecoscope Knowledge Base (X-Search, CSW3 if possible).
- COTRIX code list manager identified 2 candidate developers
- COTRIX first internal UI proposal circulated and commented
- Discussed new Android App proposal Species Fact Sheet data extraction from FAO XML sources
- Continued development of SPREAD functionality with WPS
- FLOD contributed to the development of the top level ontology
- COTRIX code list manager and vocabulary mangement draft discussed
WP4: Communication, Dissemination and Training
WP Leader: TRUST-IT - Participants: CNR, NKUA, FAO
The WP leader should report here a summary of the activities performed in the reporting period
The major issues faced in the reporting period and the identified corrective actions should be reported here
The WP leader should report here a bullet list highlighting the main achievements of the reporting period
WP5: iMarine Data e-Infrastructure Deployment and Operation
WP Leader: CERN - Participants: CNR, NKUA, FAO, FIN
CERN planned and coordinated the infrastructure upgrade to gCube 2.9.1 which fixed some production incidents. (https://portal.i-marine.d4science.org/web/guest/gcube-version)
NKUA has been keeping up and running the provided nodes hosting GHNs, UMD and the gcube wiki. Nagios NRPE has been installed to all NKUA's nodes that are hosting GHNs, for central monitoring from CERN hosted Nagios.
With CERN, FAO changed the resource manager for ICIS VOVirtual Organization; configuration files. Services were up-and-running the entire reporting period.
Prepared the documentation for the installation of Nagios NRPE and Plugins on the D4science infra nodes (https://gcube.wiki.gcube-system.org/gcube/index.php/Monitoring_a_gCube_infrastructure_With_Nagios) . The installation and configuration have started, completed by NKUA on all production nodes. CERN hosted Nagios has been configured accordingly.
All messaging and accounting components ( except for portlets) have been moved to build with Maven.
The CNR infrastructure nodes have been moved to new IPs. Some issues have been found and properly fixed by CNR
The re-installation of gCube node in Kiel is still awaiting pending the acquisition of a new server. The previous server having crashed has been repaired, unfortunately, IT Department in Kiel has taken it back for other operations.
The acquisition of a new server was planned but then cancelled. So currently, there is no solution to re-install a gCube node in Kiel and Operations for the AquaMaps VRE are still not available from Kiel.
- Infrastructure upgrade to gCube 2.9.1
WP6: Virtual Research Environments Deployment and Operation
WP Leader: CNR - Participants: ERCIM, NKUA, FAO, FIN, UNESCO, CRIA, IRD
- FAO continued with the evaluation of the Tuna Atlas in ICIS;
- FAO discussed on the suitability of geospatial services to support the SPREAD use case;
- FAO continued validating the analysis with R;
- FAO identified new services to be developed on top of Biodiversity data services.
- CNR fixed bugs in the existing plugins;
- CNR created a new IRMNG Species Discovery Plugin;
- FIN analyzed the algorithm of Taxamatch (Tony Rees, CSIRO) and produced documentation;
- FIN implemented the basic functions for the Taxamatch algorithm in PHP;
- FIN produced a new climate change scenario HCAF for the year 2100.
- CNR released an enhanced version of the Species Discovery Portlet;
- CNR released a new Species View Portlet;
- CNR released a new Software Upload Wizard;
- NKUA enhanced the Search Portlet.
- NKUA discussed with FORTH for supporting the development of the xSearch portlet and the integration with the gCube components.
D6.3 Virtual Research Environments Activity Report production has accumulated some delay. However, a draft has been produced during the August period and there is a plan to finalise it by the end of September.
- Enhanced analysis of the features supplied by the current ICIS and geospatial environment
- IRMNG Species Discovery Plugin was implemented
- The Taxamatch algorithm was analyzed and partially implemented
- The Species Discovery Portlet was enhanced
- The Species View Portlet was enhanced
- the Software Upload Wizard was released
- The Search Portlet was enhanced
WP7: Enabling-technology Integration and Distribution
WP Leader: E-IIS - Participants: CNR, NKUA
- Deliverable D7.2 "Software release report" has been finalized and circulated for internal review.
- Updated release procedures have been circulated to the tcom mailing list, aiming at reducing integration times and the release cycles.
- Investigation on how to perform incremental builds (i.e. avoid re-building unchanged components) hasn't yet led to a positive outcome.
- An improvement of success ratio for HEAD configurations has been achieved through more strict monitoring and problem notification.
- Testing infrastructure is being enlarged with eight more nodes to host a testing VOVirtual Organization; and a portal instance planned to be used in the next release cycle.
- NKUA has been working on the deliverable D7.2 filling the sections about Documentation, Javadoc Status and gCube guide status.
- The news for the minor release of gCube 2.9.1 release have been published on the gcube website.
- NKUA updated the gCube Distribution site for 2.9.1 & 2.9.2 releases by inserting in the database the corresponding entry.
- A slight delay in the delivery of D7.2.
The WP leader should report here a bullet list highlighting the main achievements of the reporting period
WP8: iMarine Data e-Infrastructure Enabling Technology Development
WP Leader: CNR - Participants: NKUA, E-IIS, FAO
T8.1, T8.2 and T8.3 worked together towards the production of deliverable D8.2 describing the novelties and evolution of the software produced by WP8 from M7 (May ’12) to M10 (Aug.'12). Specifically, the deliverable reported on the software released as part of gCube 2.9.1 and the forthcoming gCube 2.10.0.
T8.1 has worked towards the release of common-scope 1.0.0, core API to handle scope propagation in component's runtime as well as scope map handling, and gCF 1.5.0 to add facilities for error management and propagation over the network. The release of maven-dependency-plugins 2.5 caused an unexpected major revision of the approach to software dependency resolution taken in the Software Gateway service. Finally, the work on the new Resource Manager 2.0 continued and part of the testing phase was successfully performed and several fixes applied, while the enhanced Deployer service (2.4.0) was completed and released.
T8.2 has continued the development of SOA3 Id-Federation module is in progress. In particular:
- Shibboleth Identity Provider has been integrated as the Identity Provider of the security framework
- a SAML Consumer is currently under development and be integrated with Shibboleth SP
T8.3 has implemented a queue-based scheduling facility for PE2ng which will act as a front-end to Execution Engine. Execution plans are analyzed, then added up to the queue, examined for requirements satisfaction, and if so, assigned to the execution engine. This new facility also exploits the information describing the state of the infrastructure, deciding to forward or queue up incoming tasks, and it requires an Execution node monitoring mechanism actually based on JMS. On this node, a monitor/consumer is subscribed on a load topic while each execution engine is subscribed as a producer on the same topic reporting the utilization of that execution engine. T8.3 also worked on the transition of the corresponding released components from ant to maven. This procedure includes many library components and two services. Also, all released components have been refactored to use the slf4j logging facade in order to facilitate the configuration of loggers at deployment time.
T8.4 has nothing to report.
- In T8.1, the unexpected work on the new integration of Maven 3 with the SoftwareGateway has delayed the testing phase of Resource Manager 2.0. Tests will be completed in the first 2 weeks of September.
- In T8.3, in order to communicate the load information among execution components, a JMS notification mechanism was used instead of the enhancement of Resource Registry previously designed. The originally designed mechanism will be definitely used when it gets finally implemented.
- deliverable D8.2
- design and implementation of a new communication layer between Software Gateway and Maven 3
- Software Gateway's developer and administration guides updated
- Resource Manager's developer guide updated
- several PE2ng components have been mavenized
- Software released:
- software-gateway 1.1.0
- common-scope 1.0.0
- gcf 1.5.0
- deployer-service 2.4.0
- resource-manager-stubs 2.0.0 release
WP9: Data Management Facilities Development
- NKUA started the development of a file storage component abstracting over storage systems clients;
(NKUA erroneously reported activity on task T9.2)
- CERN started the implementation of a first version of the Data Transfer scheduler, with architecture details fully documented;
- NKUA worked on the bug fixing of the gRS2 and MadgikCommons components;
- NKUA wrote documentation for the gRS2Broker;
- TERRADUE performed tests and bug fixing on the WPS-Hadoop framework.
- CNR worked on the design of the occurrence reconciliation service;
- CNR designed and implemented a common library for remote executions handling based on JMX and ActiveMQ.;
- CNR applied the designed architecture to the Statistical Manager described in T10.3.
No deviations to report.
- start of the implementation of the Data Transfer Scheduler
- start of the development of another file storage system by NKUA
- implementation of a common library for queued requests handling
WP10: Data Consumption Facilities Development
WP Leader: NKUA - Participants: CNR, FORTH, TERRADUE, FAO
During this month, there has been further progress in the integration of XSearch with the gCube Search System which is now in the final stage. In particular, the development of the XSearch portlet is complete in terms of functionality, while some minor issues have been addressed. Further activities specifically in the Data Retrieval area include (i) dynamic configuration support for the Resource Registry with the creation of a number of ETICS configurations for this purpose and (ii) modifications in the gCube Search Service and library to support the latest changes performed to the corresponding Workflow Engine adaptor in the context of WP8 in order to support distributed execution.
In the area of Data Manipulation, there has been activity in the context of the WPS-hadoop framework developed under WP9. A Hadoop 2.1 cluster was attached and the resampling algorithm was run on it. Further effort is needed to run algorithms on the Hadoop cluster, as the developed algorithms were designed to be run on a single node. The publishing of a WPS single node server in production environment was achieved and released in the upcoming gCube release, while effort is being put for extending the currently available algorithms to the cluster.
In the context of Data Mining, clustering features (DBSCAN, KMEANS) and a message queue for storing the requests coming from the users have been added to the Statistical Manager, while its web interface has been designed and its development was started. In Data Visualization, the development of a new WPSHadoop algorithm called tiff_uploader is complete. This software represents the missing "connector" to a WMS hosted by GeoServer, providing a "visualization" tool to the gCube WPSHadoop framework.
Finally, there has been work for the preparation for the upcoming minor release in both the Data Retrieval and Manipulation tasks. In particular, all components under release achieved transition from ant to maven and were refactored to use the slf4j logging facade instead of the logger provided by Java, in order to facilitate the configuration of logging at deployment time and overcome significant configuration obstacles presented by the standard Java logging environment. The number of components affected by these modifications are 4 libraries and 2 services for Data Retrieval and 4 libraries and 1 service for Data Manipulation, totaling 8 libraries and 3 services.
- Further effort will be needed for running the currently supported WPS-hadoop algorithms on an Hadoop 2.1 cluster. The activity required in order to achieve this goal has already started.
- Transition to maven for a number of components
- Transition to slf4j for a number of components in order to streamline logging configuration
- Dynamic configuration support for Resource Registry
- Hadoop cluster 2.1 has been attached to the WPS-hadoop framework
- Clustering features have been added to the Statistical Manager
- The design of the web interface of the Statistical Manager has been finished
WP11: Data e-Infrastructures Integration and Interoperability Facilities Development
WP Leader: NKUA - Participants: CNR, CERN, E-IIS, FORTH, TERRADUE, FAO
During this month, work in WP11 focused on the integration of system components with the interoperability framework, and the procedure towards their release. In HTTP Front End layer of the framework and in the context of Data Consumption APIs, the new HTTP components that comply to the framework principles and support json have been tested and configured for the release. The documentation with the new descriptions has also been added in the wiki:
- asl HTTP Information Retrieval APIs: https://gcube.wiki.gcube-system.org/gcube/index.php/ASL_HTTP_InformationRetrieval
- asl HTTP Content Access APIs: https://gcube.wiki.gcube-system.org/gcube/index.php/ASL_HTTP_ContentAccess
- asl HTTP Infrastructure Login APIs: https://gcube.wiki.gcube-system.org/gcube/index.php/ASL_HTTP_InfrastructureLogin
In client libraries layer, the framework components that iplement the CL model into a framework, named common-clients and common-gcore-clients have been configured to be relesed. Moreover, the Data Transfer Scheduler Library is under implemantation based on the framework.
Finally, MS48 for 'Data Consumption APIs Specification' has been achieved, with the inclusion of all the APIs of the software subsystems respective to the functional category and the analysis of their key features, framework compliance and adoption of protocols and specifications.
None to report.
- The first release of the CL framework through version 2.0.0.
- First release of the new components complying to the HTTP Front End framework.
- MS48: Data Consumption APIs Specification achieved.