io.cloudex.framework.cloud.api

Interface CloudService

public interface CloudService

An interface for cloud API operations. This is generic and is not cloud provider specific. Implementations are needed for the various cloud providers

Author:

Omer Dawelbeit (omerio)

Method Summary

Modifier and Type	Method and Description
void	blockOnComputeOperations(List<String> references, String zoneId) Block and wait the operations with the provided references to complete
void	createCloudStorageBucket(String bucket, String location) Create a bucket on the mass cloud storage
void	downloadObjectFromCloudStorage(String object, String outFile, String bucket) Download an object from cloud storage to a file, this method will block until the file is downloaded
void	downloadObjectFromCloudStorage(String object, String outFile, String bucket, Callback callback) Download an object from cloud storage to a file
int	getApiRecheckDelay() Get the sleep period in seconds when waiting for async API operations to complete
String	getInstanceId() Get the id of the current instance
int	getMaximumMetaDataSize() Return the maximum metadata size for the underlying cloud provider implementation
VmMetaData	getMetaData(boolean waitForChange) Get the meta data of the current instance, Wait for change will block until there is a change
VmMetaData	getMetaData(String instanceId, String zoneId) Get the meta data for the provided instance id
VmMetaData	init() Perform any required initialization before this cloud service is used
List<StorageObject>	listCloudStorageObjects(String bucket) List all the objects in the provided cloud storage bucket
String	loadCloudStorageFilesIntoBigData(List<String> files, BigDataTable table, boolean createTable)
List<String>	loadLocalFilesIntoBigData(List<String> files, BigDataTable table, boolean createTable) Load a list of local files into a big data table
QueryStats	saveBigQueryResultsToCloudStorage(String jobId, String bucket, String filename) TODO rename to BigData Polls a big data job and once done exports the results to a cloud storage file then download it
QueryStats	saveBigQueryResultsToFile(String jobId, String filename) TODO rename to BigData Polls a big data job and once done save the results to a file
QueryStats	saveBigQueryResultsToFile(String jobId, String filename, String bucket, Integer minFiles, int directDownloadRowLimit) TODO rename to BigData Polls a big data job and once done save the results to a file
<T> void	setAuthenticationProvider(AuthenticationProvider<T> provider) The authentication provider used by this instance
void	setRemote(boolean remote) Usually this cloud service is used in a cloud VM environment.
void	shutdownInstance() Delete the currently running vm, i.e.
void	shutdownInstance(List<VmConfig> configs) Delete the VMs provided in this config
String	startBigDataQuery(String querySql) Creates an asynchronous Query Job for a particular query on a dataset
String	startBigDataQuery(String querySql, BigDataTable table) Creates an asynchronous Query Job for a particular query on a dataset.
boolean	startInstance(List<VmConfig> configs, boolean block) Create VM instances, optionally block until all are created.
void	updateMetadata(VmMetaData metaData) Update the meta data of the current instance.
String	updateMetadata(VmMetaData metaData, String zoneId, String instanceId, boolean block) Update the meta data of the the provided instance
StorageObject	uploadFileToCloudStorage(String filename, String bucket) Upload a file to cloud storage and block until it's uploaded
StorageObject	uploadFileToCloudStorage(String filename, String bucket, Callback callback) Upload the provided file into cloud storage

Method Detail

init

VmMetaData init()
         throws IOException

Perform any required initialization before this cloud service is used

Throws:

IOException - if any of the cloud api calls fail

getMetaData

VmMetaData getMetaData(boolean waitForChange)
                throws IOException

Get the meta data of the current instance, Wait for change will block until there is a change

Parameters:

waitForChange - - true to block until there is a change, false to return immediately

Returns:

VmMetaData

Throws:

IOException - if any of the cloud api calls fail

getMetaData

VmMetaData getMetaData(String instanceId,
                       String zoneId)
                throws IOException

Get the meta data for the provided instance id

Parameters:

instanceId - - the id of the vm instance

zoneId - - the zone or region id

Returns:

VmMetaData

Throws:

IOException - if any of the cloud api calls fail

updateMetadata

void updateMetadata(VmMetaData metaData)
             throws IOException

Update the meta data of the current instance.

Parameters:

metaData - VmMetaData

Throws:

IOException - if any of the cloud api calls fail

updateMetadata

String updateMetadata(VmMetaData metaData,
                      String zoneId,
                      String instanceId,
                      boolean block)
               throws IOException

Update the meta data of the the provided instance

Parameters:

metaData - VmMetaData

zoneId - - the zone or region id

instanceId - - the id of the vm instance

block - - true to wait for the api call to complete, false to return immediately

Throws:

IOException - if any of the cloud api calls fail

blockOnComputeOperations

void blockOnComputeOperations(List<String> references,
                              String zoneId)
                       throws IOException

Block and wait the operations with the provided references to complete

Parameters:

references - - the operations references

zoneId - - the zone or region id

Throws:

IOException - if any of the cloud api calls fail

startInstance

boolean startInstance(List<VmConfig> configs,
                      boolean block)
               throws IOException

Create VM instances, optionally block until all are created. If any fails then the returned flag is false

Parameters:

configs - - the VmConfig of the instances to start

block - - true to wait for all the instance to start up

Returns:

true if the operation is successful

Throws:

IOException - if any of the cloud api calls fail

shutdownInstance

void shutdownInstance(List<VmConfig> configs)
               throws IOException

Delete the VMs provided in this config

Parameters:

configs - - the VmConfig of the instances to shutdown

Throws:

IOException - if any of the cloud api calls fail

shutdownInstance

void shutdownInstance()
               throws IOException

Delete the currently running vm, i.e. self terminate

Throws:

IOException - if any of the cloud api calls fail

getInstanceId

String getInstanceId()

Get the id of the current instance

Returns:

the id of the current instance

createCloudStorageBucket

void createCloudStorageBucket(String bucket,
                              String location)
                       throws IOException

Create a bucket on the mass cloud storage

Parameters:

bucket - - the cloud storage bucket

location - - the location where the bucket should be created

Throws:

IOException - if any of the cloud api calls fail

uploadFileToCloudStorage

StorageObject uploadFileToCloudStorage(String filename,
                                       String bucket,
                                       Callback callback)
                                throws IOException

Upload the provided file into cloud storage

Parameters:

filename - - the name of the file to upload

bucket - - the cloud bucket

callback - - an optional callback when the upload is done

Returns:

a StorageObject

Throws:

IOException - if any of the cloud api calls fail

uploadFileToCloudStorage

StorageObject uploadFileToCloudStorage(String filename,
                                       String bucket)
                                throws IOException

Upload a file to cloud storage and block until it's uploaded

Parameters:

filename - - the name of the file to upload

bucket - - the cloud storage bucket

Returns:

a StorageObject of the uploaded file

Throws:

IOException - if any of the cloud api calls fail

downloadObjectFromCloudStorage

void downloadObjectFromCloudStorage(String object,
                                    String outFile,
                                    String bucket,
                                    Callback callback)
                             throws IOException

Download an object from cloud storage to a file

Parameters:

object - - the name of the object to download

outFile - - the local file to save the object

bucket - - the cloud storage bucket

callback - - call when the operation completes

Throws:

IOException - if any of the cloud api calls fail

downloadObjectFromCloudStorage

void downloadObjectFromCloudStorage(String object,
                                    String outFile,
                                    String bucket)
                             throws IOException

Download an object from cloud storage to a file, this method will block until the file is downloaded

Parameters:

object - - the name of the object to download

outFile - - the local file to save the object

bucket - - the cloud storage bucket

Throws:

IOException - if any of the cloud api calls fail

listCloudStorageObjects

List<StorageObject> listCloudStorageObjects(String bucket)
                                     throws IOException

List all the objects in the provided cloud storage bucket

Parameters:

bucket - - the cloud storage bucket

Returns:

a list of StorageObject

Throws:

IOException - if any of the cloud api calls fail

loadCloudStorageFilesIntoBigData

String loadCloudStorageFilesIntoBigData(List<String> files,
                                        BigDataTable table,
                                        boolean createTable)
                                 throws IOException

Throws:

IOException

loadLocalFilesIntoBigData

List<String> loadLocalFilesIntoBigData(List<String> files,
                                       BigDataTable table,
                                       boolean createTable)
                                throws IOException

Load a list of local files into a big data table

Parameters:

files - - the names of the local files to upload to big data

table - - the table definition

createTable - - true if a new table should be created if it doesn't exist

Returns:

a List of all the job ids

Throws:

IOException - if any of the cloud api calls fail

startBigDataQuery

String startBigDataQuery(String querySql)
                  throws IOException

Creates an asynchronous Query Job for a particular query on a dataset

Parameters:

querySql - the actual query string

Returns:

a reference to the inserted query job

Throws:

IOException - if any of the cloud api calls fail

startBigDataQuery

String startBigDataQuery(String querySql,
                         BigDataTable table)
                  throws IOException

Creates an asynchronous Query Job for a particular query on a dataset. If a table is supplied then large results are stored in a new temporary table with the table name driven from the supplied table

Parameters:

querySql - the actual query string

table - the table being queried

Returns:

a reference to the inserted query job

Throws:

IOException - if any of the cloud api calls fail

saveBigQueryResultsToFile

QueryStats saveBigQueryResultsToFile(String jobId,
                                     String filename)
                              throws IOException

TODO rename to BigData Polls a big data job and once done save the results to a file

Parameters:

jobId - - the id of a previous big data job

filename - - the destination filename

Throws:

IOException - if any of the cloud api calls fail

saveBigQueryResultsToFile

QueryStats saveBigQueryResultsToFile(String jobId,
                                     String filename,
                                     String bucket,
                                     Integer minFiles,
                                     int directDownloadRowLimit)
                              throws IOException

TODO rename to BigData Polls a big data job and once done save the results to a file

Parameters:

jobId - - the id of a previous big data job

filename - - the destination filename

bucket - - the cloud storage bucket

minFiles - - the minimum number of files to create

directDownloadRowLimit - - the limit to use for direct download, large than this limit will use export to cloud storage

Throws:

IOException - if any of the cloud api calls fail

saveBigQueryResultsToCloudStorage

QueryStats saveBigQueryResultsToCloudStorage(String jobId,
                                             String bucket,
                                             String filename)
                                      throws IOException

TODO rename to BigData Polls a big data job and once done exports the results to a cloud storage file then download it

Parameters:

jobId - - the id of a previous big data job

filename - - the destination filename

bucket - - the cloud storage bucket

Throws:

IOException - if any of the cloud api calls fail

getMaximumMetaDataSize

int getMaximumMetaDataSize()

Return the maximum metadata size for the underlying cloud provider implementation

Returns:

maximum metadata size

getApiRecheckDelay

int getApiRecheckDelay()

Get the sleep period in seconds when waiting for async API operations to complete

Returns:

API recheck delay

setRemote

void setRemote(boolean remote)

Usually this cloud service is used in a cloud VM environment. If remote is set to true, then it's assumed that this service is not run on a VM environment. Useful for local testing of cloud services

Parameters:

remote - - true if this instance is not running on a VM

setAuthenticationProvider

<T> void setAuthenticationProvider(AuthenticationProvider<T> provider)

The authentication provider used by this instance

Type Parameters:

T - the class of the authentication object returned by the provider

Parameters:

provider - the authentication provider to use