Monday, September 19, 2011

The Role of Linux in Grid Computing

It is this objective that grid computing wants to meet. The objective of grid-based computing is to virtualize, manage and allocate distributed physical resources (processing power, memory, storage, networking) to applications and users on an as-needed (on-demand) basis--regardless of the resources' location. Grid networks transcend physical components, organizational units, enterprise infrastructure and geographic boundaries. Naturally, software plays a vital role in determining the success of grid computing. In this article, we focus on the role of Linux in grid computing.


Four arguments can be made for Linux becoming the basis of grid computing:
1. Open Grid Services Architecture (OGSA) is a service architecture built on the open-source paradigm of community participation and sharing code. According to the father of the grid, Ian Foster, a chief scientist at Argonne National Laboratory, the long-term success of grid computing depends on four issues: open standards, open software, open infrastructure and commercializing grid services to speed enterprise adoption. The development of Linux has progressed along similar lines.

The Globus Toolkit, which formed the basis of OGSA, is a community-based, open-architecture, open-source set of services and software libraries. Globus addresses issues of security, information discovery, resource management, data management, communication, fault detection and portability. Thus, it mirrors the community processes used for the development and evolution of the Linux kernel. Any grid network must accommodate a heterogeneous mix of existing resources. However, future generations of grid networks likely will center around operating system and development environments that support an open and collaborative community process whose infrastructure evolves through an open-source process. Because Linux has evolved from the same open-source process, there is a high degree of affinity between Linux and grid-computing projects. Open standards and protocols lead to the building of services, and services are at the heart of the grid.

2. The grid concept is based on the management and allocation of distributed resources rather than on a vertically integrated, monolithic resource tightly coupled to the underlying operating-system architecture of the platform.
The adoption of grid computing from single platform architectures will not happen all of a sudden. A few computational units will have to be deployed in small, inexpensive increments. The performance of these units will be measured and compared to the expected results. If the gain is significant, only then would there be a next round of deployments. This is in contrast to the major investments needed for large-scale monolithic systems, which typically are obsolete within four or five years and thus are a drain on capital and operating budgets.
Linux has gained a reputation for being a highly efficient operating system in simpler application environments running on smaller hardware configurations, the type that will be enabled by the grid architecture. In such experimentation-based systems, the free nature of Linux will play a crucial important role due to lower investments.

3. Computing grids are virtual, extensible and horizontally scalable and use open network protocols. Many of the early instances of grid networks were developed in the scientific and technical computing environments of universities, technical laboratories, health and bio-informatics consortia. Most of them have relied on native operating system processes for the hosting environment, typically UNIX and Linux. Their experience suggests that Linux is the best platform available for grid computing. There has hardly been any evidence of grid computing projects being deployed on other operating systems, such as Windows 98 or Windows XP.

For example, the TeraGrid is designed to help the National Science Foundation address complex scientific research, including molecular modeling for disease detection, cures and drug discovery, automobile crash simulation and investigations of alternative energy sources. TeraGrid will use more than 3,000 Intel processors running Linux. The Grid Forum, a research consortium, is aiming its research at applications in the oil industry, physical disaster prediction and simulation, biological and ocean modeling, industrial simulations, agriculture applications, health service applications and e-utilities. Many of these applications currently run on UNIX or Linux.

4. Vendor-specific initiatives are promoting Linux. Although IBM's grid architectural block diagrams show the OGSA framework supporting operating system heterogeneity, they also clearly point to the centrality of Linux in IBM's grid strategy. Sun Microsystems has written an edition of its Grid Engine 5.3 software for the version of Linux available through SuSE Linux AG. Other vendors are investing in grid computing as well. Hewlett-Packard has incorporated software specs for massive grids into the Utility Data Center, a computing power-on-demand product that supports Linux. An Oracle spokesman recently said Linux is "the smart option for grid computing". In addition, Oracle recently announced that the Oracle 10g package is grid-enabled and runs smoothly on Linux.

On the whole, Linux is the buzzword as far as the platform for grid computing is concerned. But, arguments against pervasive adoption of Linux exist. A few them are listed below:

1. Grid computing is based on the principle of heterogeneity, where virtual organizations are formed with no discrimination between resources and systems, as long as the standard toolkit services are implemented.
The OGSA model does not specify an operating system. On the contrary, it has been developed so as to invite all computing architectures into the grid-computing family. Given grid computing's emphasis on resource virtualization and usage and the heterogeneous nature of most enterprises' IT infrastructures, enterprises are under no pressure to change their hardware and software to participate in grids or establish internal grids.

2. The grid philosophy does not specify implementations: its fundamental principle is to adapt to the operating system environment of specific hosts and exploit their native capabilities.
The grid architecture does not suggest or inhibit ways or solutions for implementation of grid architecture. Similarly, it does not specify anything about the platform to be used.

3. Grid computing addresses only a small part of the IT infrastructure. Grid computing is exactly what it implies--a mesh of distributed resources whose members share each other's resources through a specifically enforced set of protocols. The heterogeneity that OGSA attempts to address is in itself recognition that IT infrastructures will consist of a mix of computing architectures.The role Linux will play in such heterogeneous environments will depend largely on its performance, reliability and economics when running on such hosts.
Both grid computing and Linux are too immature for us to forecast that Linux will dominate in commercial grid applications. Only the future will tell whether the realm of grid computing is ruled by Linux. One thing is for sure, however; Linux definitely will form a large chunk in the Grid Computing Platform market.




What OS does Google run on its servers?


Google uses a customized Linux operating system that is optimized for use in massive commodity-based hardware clusters.


They are designed to be fault tolerant and fail over with many multiple redundancies both inside their server rooms and regionally. This way when one of those commodity-based systems crap out they simply ignore it until somebody has time to replace it. This way they can have very large datacenters and manage it without large amounts of effort.

They don't use SQL databases or RAId or anything like that for their search engine stuff. Simply because those sort of things don't scale very well. Instead they have a specialized google-specific file system with super-secret algorythms and whatnot to manage all of it.

Enterprise FileSystem (EFS)

Enterprise File System (EFS)

EFS is an open-source, Apache-licensed Linux/Unix-based software product that provides software distribution and change control for heterogeneous clients using distributed filesystems (currently NFSv3, with plans for supporting both NFSv4 and OpenAFS). EFS was designed to support extremely large, global enterprises, but can be scaled down for smaller environments as well.

- A special-purpose Unix global file system
- A collaborative mechanism for developers to quickly and easily share and deploy software
- A system for tracking releases of software
Overall Reduction in: Time, Resources and Budget

EFS Benefits
EFS is a client/server software product which implements a versionized namespace for application deployment in a distributed filesystem (currently NFSv3, with plans to support NFSv4 and OpenAFS). The EFS namespace makes it feasible to deploy multiple versions of the same application side by side, each with its own distinct set of dependencies.
- Less time spent deploying software to internal clients
- If the software is in EFS any client attached to EFS will have access to all core applications available
- Recovery is quick and relatively painless
- Roll-back of software is done in near-seconds instead of hours or days

EFS or Enterprise File System is a suite of Perl-based commands to help deploy, distributed and maintain versioning of software releases in the UNIX environment. These commands structure your file system to allow ease of deployment and, as a byproduct, standardize your environment. It even abstracts the OS from the end user so that file locations are common across different UNIX flavors

In addition to the Perl layer, EFS infrastructure also consits of an NFS namespace and a database system (Oracle or MySQL are the supported flavors as of the time of this writing)

EFS is not:
- a software repository like Source Forge or Download.com
- a place to store source code for revision control purposes, like CVS, Subversion, or the like.
- just another NFS variant. EFS leverages NFS under the hood, but it's a suite of functionality layered on top of NFS

Introduction to the Basic Structure of EFS
An “EFS Cell” is the building block of the global EFS environment. It is simply the NAS filer (or filers) and the NFS clients that mount the /efs file systems from them, as a single administrative subdomain of the global environment. This cell can have the celltype of either DEV (for development and QA/UAT testing) or PROD (production). There is at most one cell per distinct data center per celltype. A single cell, though, can support multiple data centers.

EFS has 2 main namespaces: “/efs/dev” (read/write) for development, and “/efs/dist” (read only) for distributed code. The DEV celltype supports both /efs/dev and /efs/dist (for UAT and QA testing). The PROD celltype supports only /efs/dist.

A NAS filer containing the EFS data is called an EFS “Cell.” This cell consists of every part of EFS’s infrastructure in a single location, including storage, EFS management servers and caches. The collection of cells within a region is referred to as a “Regional Cell.” When any UNIX server is connected to EFS, they access the EFS Namespace.

Each regional cell may contain the EFS DEV or EFS PROD environments or both.
- A NAS filer containing the EFS data is called an EFS “CELL”
- A cell consists of every part of EFS’s infrastructure in a single location
- The collection of cells within a region is referred to as a “CELL SET” or “REGIONAL CELL”

There are 2 environments in EFS
dev
The Dev environment supports /efs/dev (read-write) and /efs/dist( read-only , simulating PROD used for testing debugging QA )

prod
The Prod environment contains a “locally” form of /efs/dist only

Connecting to EFS
To connect a client to EFS, a simple boot script is installed on the client machine. This script makes the static NFS mount points necessary to make the /efs/dist, and optionally /efs/dev, file systems available on the client.

EFS Components
EFS is built using NAS Filers and the NFS Protocol. The original Morgan Stanley project was based on AFS, but Merrill Lynch was uninterested in this technology. The NAS Filers are abstracted from clients originally by using Acopia file switches, but eventually by using Linux automounter technology for the DEV environment and NAS’ FlexCache technology for PROD.

There are two components to EFS: (1) the File System and (2) the Management Utility. The File System is the EFS namespace - NFS mounted on EFS clients. The Management Utility is the EFS Command Line Utility (CLI) for creating and managing projects in EFS.

EFS Commands
EFS is managed from a core Perl Library that contains most application logic. The library is accessed via a simple command line interface (CLI), via a client/server architecture which centralizes all privileged EFS commands to limited set of administrative servers. The basic structure of an EFS command is as follows: “efs action object [arguments]”action is an operation, (e.g. create, define, destroy, etc.) object is what the operation acts on, (e.g. business, metaproj, release, cell, etc.) arguments are the additional strings of information needed to complete the command
Developers and EFS Operations each run different commands to complete tasks in the EFS environment.

Example:-
efs create release wlm xcat 2.6.6-01
efs create install wlm xcat 2.6.6-01 common

Reference:-
http://openefs.org/
Google Search