Software Gateway

From IMarine Wiki

Jump to: navigation, search


Contents

Motivations

Motivations for refactoring:

  • as final step of the gCube mavenization, we want to have all our software stored in a Maven Repository for distribution and deployment purposes
  • Service Archive is an arranged format, not comply with the standard and/or widely accepted packaging format
  • SR has an inconsistent and proprietary interface that prevents any sort of interoperability or openness

Role

Primary Roles:

  • the new service will be a gateway over N Maven Repositories for the enabling layer
  • the new service will certify which software is deployable withing the infrastructure

More specifically:

  • it will represent a single point of configuration of a cluster of Maven Repositories in the whole infrastructure (hence, will avoid to configure each gHN for accessing to the recognized Maven repositories)
  • enabling services aiming at communicating with the registered Maven Repositories must talk with the Software Gateway
  • external services may communicate directly with the Maven Repositories
  • it must minimize the impact of the new physical storage (Maven) on the other enabling layer services
  • it will serve as mediator from the gCube coordinates (service name, class, version, package name, package version) to the Maven coordinates (groudID,artifactID,version>
  • the new service is responsible for publishing service profiles in the Information System
  • the new service must not deal with the packages storage
  • the new service must be scope-agnostic (all the software available in the maven repository must be visible to all the clients)

What it will not be:

  • a gateway over N Maven Repositories for the rest of the world (only for the enabling layer)
  • a wrapper around the Maven API

Packaging Model

As a result of the gCube mavenization, the following packaging model is foreseen in the near future.

About gCube components, a build of component generates 2 artifacts:

  1. the main artifact (JAR/WAR/GAR);
  2. a second artifact for deployment and distribution purposes (Software Archive) with the descriptors of the artifact (README, LICENSE, INSTALL, MAINTAINERS), the pom.xml, profile.xml and all the other files belonging the package (such as installation script, reboot scripts and so on), including the main artifact. It will be a tar.gz archive.

2) may contain more than one artifact, depending on the nature of the gCube component (Maven-based or Ant-based).

About external components:

  • usually there is 1 artifact (the main one) per component (hosted in a non-gCube Maven repository)
  • optionally, there exists a Software Archive artifact per component (hosted in a gCube Maven repository) in case that custom deployment actions or special needs are needed.

How SG deals with them? Regarding the main artifacts, the SG "simply" interfaces the configured Maven repositories to look for them and deals only with their main binary artifact.

Regarding the SA artifacts, the SG needs to do some extra-management actions when they are multi-packages.

New Scenarios

gCube Software Registration

gCube Software Registration

Removal

Software Removal

Support to Deployment

Software Deployment Interactions

Service Interface

Registration portType:

  • String register(profile): accepts a service profile with a list of packages inside, return the ID assigned. The profile must include the Maven coordinates for each package
  • void unregister(ServiceName, ServiceClass, ServiceVersion, PackageName, PackageVersion)

Access portType:

  • URL getLocation(ServiceName, ServiceClass, ServiceVersion, PackageName, PackageVersion): returns the HTTP location from which the package can be downloaded
  • URL getSALocation(ServiceName, ServiceClass, ServiceVersion, PackageName, PackageVersion): returns the HTTP location of the Service Archive connected to the package, if any. If a SA is returned, the Deployer executes the instructions inside as pre/post deployment and undeployment operations
  • List<package> getDependencies(ServiceName, ServiceClass, ServiceVersion, PackageName, PackageVersion): returns an XML document reporting the actual resolved dependencies and the missed dependencies (unresolved). PackageName and PackageVersion are optional parameters: if they are not set, the operation must look for the package declared as "service" and for the last version
  • List<package> getPackages(ServiceName, ServiceClass, ServiceVersion): return the gCube packages belonging that service.
  • List<package> getPlugins(ServiceName, ServiceClass, ServiceVersion, PackageName, PackageVersion): returns the list of plugins targeting the package or an empty list

All the above operations throw a ServiceNotAvailableFault if the requested package is not available in any repository.

Implementation

Conversions

Software Gateway deals with two types of coordinates: gCube coordinates and Maven coordinates.

GCube coordinates are expressed as quintuples: <ServiceClass, ServiceName, ServiceVersion, PackageName, PackageVersion>.

Maven coordinates are expressed as triples: <groupId, artifactId, version>.

The Software Gateway often needs to convert gCube coordinates in Maven coordinates and vice versa. In such cases, the following conversions are applied:

  • Conversion gCubeToMaven
  • groupId = service class
  • artifactid = package name
  • version = package version
  • Conversion MavenTogCube
  • service class = groupId
  • service name = artifactId
  • service version = 1.0.0
  • package name = artifactId
  • package version = version

Registration PortType

Registering a New Software

The service receives in input a service profile and registers it in all the scopes it is configured to run. The service returns a report that describes the outcome of the operation. The report contains an entry for every package. For every package the report contains:

  • The maven coordinates of the package: groupID, artifactID and version tags
  • the ID of the newly registered resource (assigned at load time by the resource library)
  • the URL where is possible download the jar of the package
  • the javadoc tag that contains the url where is possible download the javadoc related to the package
  • the Status tag that contains the outcome of the operation
  • the operation tag that can be
  • NEW if the resource is not present in the Information System service
  • UPDATE if the resource is already present in the Information System service
  • the timestamp tag that contains the timestamp of the operation

Here's an example of the report returned:


<Packages>
	<Package>
		<groupID>org.gcube.common</groupID>
		<artifactID>rapidminer-custom</artifactID>
		<version>1.0.0</version>
		<ID>d3217750-9b6a-11e1-be35-f7d8862056ea</ID>
		<URL>http://maven.research-infrastructures.eu/nexus/service/local/artifact/maven/redirect?r=gcube-staging&g=org.gcube.common&a=rapidminer-custom&v=1.0.0&e=jar</URL>
		<javadoc>http://maven.research-infrastructures.eu/nexus/service/local/artifact/maven/redirect?r=gcube-staging&g=org.gcube.common&a=rapidminer-custom&v=1.0.0&e=javadoc</javadoc>
		<Status>SUCCESS</Status>
		<Operation>NEW</Operation>
		<Timestamp>1336741892744</Timestamp>
	</Package>
</Packages>

Unregistering a Software

The service receives in input the gCube coordinates that identify a package. Then, it performs the following steps:

  • query the IS in order to harvest the profile matching such gCube coordinates
  • if the profile reports only that package, the whole profile is deleted from the IS (all scopes)
  • if the profile reports other packages, the package is withdrawn from the list of packages and the modified profile is updated in the IS (all scopes)

Access PortType

Getting a Software Location

The service receives in input the gCube coordinates that identify a package. Then, it performs the following steps:

  • query the IS in order to harvest the profile matching such gCube coordinates
  • if a profile is returned, then extract the Maven coordinates from the profile
  • if not, the conversion gCubeToMaven is applied
  • having the Maven coordinates, query the configured Maven repositories until a main artifact with such coordinates is found
  • find the URL of the artifact (with the REST interface exposed by Maven)
  • return the URL from which the package can be downloaded

Getting Dependencies

The service receives in input the gCube coordinates that identify a package. Then, it performs the following steps:

  • query the IS in order to harvest the profile matching such gCube coordinates
  • if a profile is returned, then extract the Maven coordinates from the profile
  • if not, the conversion gCubeToMaven is applied
  • having the Maven coordinates, query the configured Maven repositories and prepare the following 2 lists:
  • resolved dependencies
  • missing dependencies
  • elements in the list are expressed in Maven coordinates that must be converted in gCube coordinates
  • for each element in the lists, query the IS in order to find the service profile with such Maven coordinates
  • if a profile is returned, the gCube package's coordinates are extracted from the profile
  • if not, the conversion MavenTogCube is applied
  • finally, prepare the XML report to return to the caller

Here's an example of the report returned:

<DependencyResolutionReport>
	<ResolvedDependencies>
	 	<Dependency>
	 		<Service>
	 			<Class>Portal</Class>
				<Name>ThumbnailService</Name>
				<Version>1.00.00</Version>
 			</Service>
 			<Package>Thumbnailer-service</Package>
 			<Version>1.00.00</Version>
 			<Scope level="GHN"/>
 		</Dependency>
 		<Dependency>
 			<Service>
				<Class>Portal</Class>
 				<Name>ThumbnailService</Name>
				<Version>1.00.00</Version>
 			</Service>
 			<Package>Thumbnailer-stubs</Package>
			<Version>1.00.00</Version>
			<Scope level="GHN"/>
 		</Dependency>
 	</ResolvedDependencies>
 	<MissingDependencies>
 		<MissingDependency>
	 		 	<Service>
	 		 		<Class>ExternalSoftware</Class>
	 		 		<Name>imageJ</Name>
	 		 		<Version>1.40.00</Version>
	 		 	</Service>
	 		 <Package>imageJ</Package>
	 		 <Version>1.40.00</Version>
	 		 <Scope level="GHN"/>
 		</MissingDependency>
	</MissingDependencies>
</DependencyResolutionReport>

Getting a Software Archive

The service receives in input the gCube coordinates that identify a package. Then, it performs the following steps:

  • query the IS in order to harvest the profile matching such gCube coordinates
  • if a profile is returned, then extract the Maven coordinates from the profile
  • if not, the conversion gCubeToMaven is applied
  • having the Maven coordinates, query the configured Maven repositories to find the secondary artifact (we will see later on how it is characterized) related to the package
  • find the URL of the artifact (with the REST interface exposed by Maven)
  • download locally the tarball and uncompress it
  • if it contains more than one package, create a local SA archive with only the package requested
  • re-tar the SA and put in a local (cachable folder)
  • return the URL from which the local SA can be downloaded

Open issues

  • How to manage the inverted dependency of the plugins?
  • Re: by querying the IS to find plugins targeting the service
  • Is still necessary the local Database?
  • Re: no
  • What to do when a new VOVirtual Organization; becomes available? How to register old packages in the IS?
  • How to download artifacts from the SG? Is it possible to have an URI pointing to a Maven location or is a local download (hence, with a local cache) needed?
  • the cache might be obtained with a local Maven Repository too
  • Can we allow deployments of artifacts that are not registered in the IS?
  • Re: no, we will lose the control over the sw deployed in the infrastructure
  • SA will not exist anymore: where the auxiliary and distribution files such as installation scripts, README, MAINTAINERS, etc. will be placed?
  • do we need a secondary artifact for a package for them? to investigate

Support from other components

Modifications required to other components for compliance:

  • Service XML schema and parsers: must be extended to include the Maven coordinates in the profile per package
  • Maven-service-plugin: must fill the service profile with the coordinates extracted from the POM
  • Deployer: must ask for both the main artifact and the related SA with the new operation. Then, if the SA exists, it performs the standard deployment operations, otherwise it will just deploy the main artifact as it is in the local platform
Personal tools