Following last year success, the event will cover recent developments and progress in cloud technologies. Presenting with top-of-the-line companies active in this field, including (partial list): Amazon, Google, eBay, IBM, HP, Sun, RedHat and more.

Additional “hands-on” labs and workshops are offered during the event for participants that would like to learn more about cloud technologies and integration possibilities.

We are also presenting Hoopoe at the summit, for GPU Cloud Computing, and providing a workshop on GPU Computing in general and Hoopoe as well.

This event ends 2009 and symbolically the last decade, marking cloud computing as a major development that we are about to see more and more in the next years.

You are invited to join us during the event.
Agenda
Registration

We added support for Amazon S3 storage services recently to Hoopoe. Following the previous article with our general account details, we wanted to share with you a regular expression we use for validating S3 URL as sources of data and files.

You may find more information about S3 naming conventions and requirements in the manuals available from http://aws.amazon.com/s3.

When submitting a task to Hoopoe with input/output sources from Amazon S3, one must specify the S3 URL of the resource. A simple format for a resource can be:
https://test-bucket.s3.amazonaws.com/dir1/input.bin.
With this example, the bucket of the user storing the object is called “test-bucket“, and the file for input is “dir1/input.bin”, called the key of the object (in the bucket).

This is a general form for S3 URL to make them accessible over the internet.

Regular Expression

We are using a regular expression to validate all Amazon S3 URLs with submitted tasks to Hoopoe.

In .NET (and general) manners, the RegEx is:
https://[a-z0-9][a-z0-9\-\.]*\.s3\.amazonaws\.com/[\w][\w\W]*

As you may see, the following limitations exist:

1. For DNS compatibility, bucket names must be lower case and start with a letter or number
2. In S3, and following DNS limitations, bucket names should not exceed 63 characters in length
3. Object keys can be of variable length, must start with a valid character but can follow with other possible characters, also to denote paths (a file named: “dir/input.bin” is located under “dir” directory)
4. In addition to the above, Hoopoe restricts S3 URL to be up to 256 characters in length

In case you find a mistake in the regular expression definition, whether possible URLs do not fit or it is permissive, please send us an email.
We also hope you may find this information useful for your own purposes.

This post will present the File System interface to Hoopoe distributing engine. The File System (FS) interface can be used by users to transfer data files to be processed by Hoopoe with CUDA computing kernels. After processing completes, the same interface can be used to read back computed results.

Topics

• Features
• General terms
• API description
• API examples
  • Creating new instance
  • Authenticating
  • Creating a file
  • Creating a directory
  • Creating a file under a sub-directory
  • Deleting a file
  • Writing data into files
  • Reading data from files

1. Features

HoopoeFS exposes a simple interface for data and file management. In general, most features available by general OS file systems are given by HoopoeFS service, providing high flexibility for users.

Taking security into consideration, every users is provided with a completely isolated environment, so no special security functions should be used or exposed, as every users sees, and able to access only the files he generated or uploaded.

As a simplified manner, the API provided by HoopoeFS is generic, but there are few limitations to user operations and capabilities. For example, a user is allowed to place files in the root directory or under sub directories. A user is also allowed to create only one level of sub directories being able to contain additional files.

2. General terms

As previously mentioned, HoopoeFS provides all general constructs for working with files and directories.

A file, is simply a container for data, either raw or compressed and can be named using every supported character.

A directory, is a container for files, and provided is the root directory, and further sub directories that can be created by the user.

3. API description

For data constructs (File, Directory), are provided management functions as follows:

• CreateFile/CreateDirectory – allows to create a new file or directory, respectively. Calling these functions is required as a first step prior to accessing them.
• DeleteFile/DeleteDirectory – deletes a previously created file or directory.
• RenameFile/RenameDirectory – given existing files or directories, allows to modify their name.
• WriteData/ReadData – modifies the content of a file (write) or reads content from a specific file.
• GetFileSize – returns the number of bytes in a file.

For general operation, few more functions are given:

• Authenticate – returns a value indicating whether the user is registered and recognized by HoopoeFS
• IsUserOverQuota – returns a value indicating whether the user has exceeded the allowed storage space. In such case the user cannot create new files or directories, but can delete and read contents of files.

4. API examples

4.1 Creating new instance

In order to work with HoopoeFS, it is necessary to create a new instance of HoopoeFS class:
HoopoeFS hps = new HoopoeFS();

4.2 Authenticating

It is a good practice to check with HoopoeFS if we are authenticated, before performing futher operations. Every operation to be performed must use this authentication level:

Authentication a = new Authentication();
a.User = test@company_alias;
a.Password = "my_password"
hfs.AuthenticationValue = a;


4.3 Creating a file 

Creating a file is a simple task with HoopoeFS API:

hfs.CreateFile("test.dat");

4.4 Creating a directory

Following the previous example, a similar API can be used to create a new sub-directory (all directories are created under the root):

hfs.CreateDirectory("test_data");

4.5 Creating a file under a sub-directory

Once created a sub directory, any number of files can be created under it.

To do that, the following operations are necessary:

Directory d = new Directory();
d.Name = "test_data";
hfs.DirectoryValue = d;
hfs.CreateFile("child_temp.dat");

You may note, that once hfs.DirectoryValue is set, all file related operations correspond to the directory (creating new files, deleting, modifying etc.), so when working with the directory ends, hfs.DirectoryValue should be set to null.

4.6 Deleting a file

A very straight forward operation:

hfs.DeleteFile("test.dat");

4.7 Writing data into files

The concept of writing data to files within Hoopoe maps to the real world, with a simplified API.

byte[] data = new byte[512*1024];
// Load/generate data
....

// If willing to write more data, then consider a
// new offset
offset += data.Length;


4.8 Reading data from files

The same rules for writing data apply to reading it from files.

// Read the data, starting at offset 0 of the file
long offset = 0;
// Determines the amount of bytes to read
int length = 512*1024;
byte[] data = hfs.ReadFile("temp.dat", offset, length);

// If willing to read more data, then consider a
// new offset
offset += data.Length;


Security models in Hoopoe

Hoopoe provides several features to overcome this problem.

Isolated user environment

Hoopoe provides each user with a unique, isolated, environment. This way, only the user can access its files and computations, using the specific mechanism provided by for file management and related operations.

Hiding the “metal”

Hoopoe hides the “metal” from the user, providing access only through a web service interface to communicate with the system.
Thus, the user is limited with the flexibility of the code it can run.
There is no direct access to machines, so the user is able to submit his task to Hoopoe for further processing of the system. After the submission point, the user waits for the task to finish, and copy the results back.

Independent data management

User data is managed by Hoopoe as files, either raw or compressed (using GZip).
A buffer is then read in a fully managed (.NET) environment, thus reducing the risk for malformed or “bad” files.

Running computations

Hoopoe is meant to run computations, and not serve as an operating system. By such, user tasks are compiled on demand for the platform it should be processed on (if 32/64 bit, or specific hardware support).

Computations are running on the GPU itself, and this is where the interaction with the GPU ends. Copying the relevant data, performing the computations and placing the results back in the appropriate buffer.