# WebServer Server
- A Web server serves pages for viewing in a Web browser, while an application server provides methods that client
applications can call.
- A Web server exclusively handles HTTP requests
- A Web server handles the HTTP protocol. When the Web server receives an HTTP request, it responds with an HTTP response, such as sending back an HTML page. To process a request, a Web server may respond with a static HTML page or image, send a redirect, or delegate the dynamic response generation to some other program such as CGI scipts , JSPs (JavaSever Pages), servlets, ASPs(Active Server Pages), server-side JavaScripts, or some other server side technology.
- The server side program usually provides for itself such functions as transaction processing, database connectivity and messaging
- Webserver may not itself support transactions or database connection pooling, may employ various strategies for fault tolerance and scalability such as load balancing, caching and clustering
- Eg: Apache HTTP Server, Sun ONE Web Server, iPlanet Web Server, Microsoft IIS's (Internet Information server)
- Webserver - any company's website, it responds with a static or dynamic html pages for each click providing the information you requested
- Protocol dependent (Handle HTTP only)
# Application Server
- Application server serves business logic to application programs through any number of protocols. \
- An application server exposes business logic to client applications through various protocols, possibly including HTTP.
- An application server provides access to business logic to use by client applicatin programs. The application program can use this logic just as it would call a method on an object
- Application server manages its own resources. Such gate-keeping duties include security, transaction process, resource pooling, and messaging.
- Application Server is a super set of which web server is a sub-set
- Both application/webserver offers caches, clustering and load balancing
- Application server means we can store application (DATA) in that Server
- Application server - any business application such as icicibank, hdfc(These applications are called as enterprise applications) or any bank website as in these sort of application, you not only click the login button, affer that u login giving credentials, check your balance, transfer money online... These things you cannot do in web application.
- Webserver handles only http protocols where as application server can handle any type of protocols.
- Application server maintains the application logic and serves the webpages in response to user request
- Application server can do both application logic maintanence and webapage serving
- Application server is used for processing/applying logic
- E.g: weblogic server is an example of Application server
- protocol independent
- Application server = webserver + EJB Container
- Application server is a server computer on a computer network dedicated to running certain 'Software application'
- An Application server is a software engine that delivers applications to client computers
- Benefits of application server technology are ease of application development and centralization
- Application server is more capable of dynamic behavior than webserver. We can also configure application server to work
as a webserver. Simply application server is a superset of webserver
- Eg: IBM WebSphere, Redhat JBOSS, Weblogic server, tomcat
- An application server has a 'built-in' webserver, in addition to that it supports other modules or features like
e-business integration, independent management and security module etc
### Portal Server ###
Portal server is just an application server running a portal software or a portal application
Different networking types
Bridged Networking
Bridged networking connects a virtual machine to a network by using the host computer’s network adapter. If your host computer is on a network, this is often the easiest way to give your virtual machine access to that network. The virtual network adapter in the virtual machine connects to the physical network adapter in your host computer, allowing it to connect to the LAN the host computer uses.
Bridged networking configures your virtual machine as a unique identity on the network, separate from and unrelated to its host. It makes the virtual machine visible to other computers on the network, and they can communicate directly with the virtual machine. Bridged networking works with both wired and wireless physical host network cards.
Note: Bridge network uses the host Computer network adapter, the VM Can access the host computer Network
Network Address Translation (NAT)
NAT configures a virtual machine to share the IP and MAC addresses of the host. The virtual machine and the host share a single network identity that is not visible outside the network. NAT can be useful when your network administrator lets you use a single IP address or MAC address. If cannot give your virtual machine an IP address on the external network, you can use NAT to give your virtual machine access to the Internet or another TCP/IP network. NAT uses the host computer’s network connection. NAT works with Ethernet, DSL, and phone modems.
Setup Requirements for IP Addresses
If you use NAT, your virtual machine does not have its own IP address on the external network. Instead, a separate private network is set up on the host computer.
Note: NAT will share the IP and MAC address of the host machine with the VM, it can access the Internet and host network as well
Host-Only Networking
Host-only networking creates a network that is completely contained within the host computer. Host-only networking provides a network connection between the virtual machine and the host computer, using a virtual network adapter that is visible to the host operating system. This approach can be useful if you need to set up an isolated virtual network. In this configuration, the virtual machine cannot connect to the Internet.
Note: Isolated Network
### Virtualization ###
virtualization has been around in one form or another since it was first introduced on IBM mainframe operating systems in the 1960s. The reason for the sudden popularity of virtualization can be attributed to a number of largely unconnected
trends
- Green computing- Recent trend to reduce the power consuption of computer systems
- Increased computing power - Virtualization allows companines to maximize utilization of hardware by running multiple operating systems concurrently on single physical systems
- financial constraints - A key technique for reducing IT overheads is to use virtualization to gain maximum return on investments in computer hardware
- Web 2.0 & Cloud Computing - The term web 2.0 has primarily come to represent the gradual shift away form hosting applications and data on local computer systems ot a web based approach. Example many users and companies now use Google apps for spreadsheet and word processing instead of installing office sutie software on local computers
- Operating system market fragmentation - In recent years the operating system market has increasingly fragmented with Microsoft ceding territory to offerings such as linux and Mac
Guest Operating System Virtualization
Guest OS Virtualization, also referred to as application based virtualization. Running on Operating system is a virtualization application which executes in much the same way as any other application such as word Processor. The virtualization application is responsible for starting hardware resources on behalf of the individual virtual machines.
The virtualization application also engages in a process known as 'binary rewriting' which involves scanning the instruction stream of the executing guest system and replacing any privileged instructions with safe emulations. This has the effect of making the guest system think it is running directly on the system hardware.
Guest Operating system <-> Virtualization Application <-> Host Opearating System <-> Host Hardware
Shared Kernel Virtualization
shared kernel virtualization (also known as system level or operating system virtualization) takes advantage of the architectural design of linux and Unix based Operating systems.
The kernel, in simple terms, handles all the interactions between the operating system and the physical hardware. The second key component is the 'root file system' which contains all the libraries,files and utilities necessary for the operating system to function.Under shared kernel virtualization the virtual guest systems each have their own root file system but share the kernel of the host operating system. This type of virtualization is made possible by the ability of the kernel to dynamically change the current root file system(a concept known as chroot) to a different root file system without having to reboot the entire system.
Drawback The guest operating systems must be compatible with the version of the kernel which is being shared. It is not, for example, possible to run Microsoft windows, Nor possible to share a linux guest system designed for the 2.6 to 2.4
Eg. Solaris Zones and Containers
Host Operating system Guest - Operating sytem & root filesystem -Shared Kernel <-> Host Hardware
Kernel Level Virtulization
Under Kernel level virtualization the host operating system runs on a specially modified kernel which contains extensions designed to manage and control multiple virtual machines each containing a guest Operating system.
Examples of Kernel level Virtualization technologies include User Mode Linux (UML) and Kernel-based Virtual Machine(KVM)
Guest Operating system - Linux Kernel <-> Host Hardware
### Hypervisor Virtualization ###
The x86 family of CPU's provide a range of protection levels also known as rings in which code can execute. Ring 0 has the highest level privilege and it is in this ring that the operating system kernel noramally runs. Code executing in ring0 is said to be running in system space, kernel mode or supervisor mode. All other code such as applications running on the operating system operates in less privileged rings, typically ring 3
Under hypervisor virtualization a program known as a hypervisor runs directly on the hardware of the host system in ring 0. The task of this hypervisor is to handle resource and memory allocation for the virtual machines in addition to providing interfaces for higher level administration and monitoring tools
clearly, with the hypervisor occupying ring0 of the CPU, the kernels for any guest operating systems running on the system must run in less privileged CPU rings. Unfortunately, most operating system kernels are written explicitly to run in ring 0 for the simple reason that they need to perform tasks that are only available in that ring, such as the ability to execute privileged CPU instructions and directly manipulate memory. A number of different solutions to this problem have been devised in recent years
Paravirtualization
Under paravirtualization the kernel of the guest operating system is modified specifically to run on the hyperisor.This typically involves replacing any privileged operations that will only run in ring 0 of the CPU with calls to the hypervisor(known as hypercalls). The hypervisor in turn performs the task on behalf of the guest kernel.
Note: The guest kernel to communicate directly with the hypervisor results in greater performance levels compared to other virtualization approaches
E.g Citrix XEN
Full Virtualization
Full Virtualization provides support for unmodified guest operating systems(Unmodified Kernel ). The term unmodified refers to operating system kernels which have not been altered to run on a hypervisor and therefore still execute privileged operations as though running in ring 0 of the CPU. In this scenario, the hypervisor provides CPU emulation to handle and modify privileged and protected CPU operations made by unmodified guest operating system kernels
Note: Unfortunately this emulation process requires both time and system resources to operate resulting in inferior performance levels
Hardware Virtualization
Hardware Virtualization leverages virtualization features built into the latest generations of CPU's from both intel and AMD. These technologies, known as Intel VT and AMD-V respectively, provide extensions necessary to run unmodified guest virtual machines.
Note: In very simplistic terms these new processors provide an additional privilege mode above ring 0 in which the hypervisor can operate essentially leaving ring 0 available for unmodified guest operating systems
Administrative Env - Virtual Machine <-> Hypervisor <-> Host Hardware
Hypervisor based virtualizatin solutions include Xen, VMware Esx Server and Microsoft's Hyper-V technology
XEN
Managing VMs
- starting VMs(xm create)
- stopping VMs(xm shutdown)
- Really stopping VMs(xm destroy)
- Monitoring (xm list, xm top)
xen config file - It will be having all the details including memory, disk,kernel, swap,name, mac address
Xen is a feature rich, opensource, hypervisor-based virtualization solution
Xen supports a variety of operating systems, which increases its applicability for running many popular applications on whichever OS you desire. Xen currently supports linux kernels through an approach called paravirtualization in which the operating system is modified to become aware of Xen
Xen system has a single privileged OS, called Doamin-0, that is responsible for starting and managing the other unprivileged OS instances. Domain-0 is the OS that boots when you start your computer, and it has the tools necessary to manage other domains.
Xen itself is called a hypervisor because it operated at a higher privilege level than the supervisor code of the guest operating systems that it hosts
At boot time, Xen is loaded into memery in ring 0. It starts a patched kernel in ring1; this is called domain0. From this domain you can create other domains, destroy them, perform migrations of domains, set parameters to a domain and more. the domains you create also run their kernels in ring 1. User applications run in ring 3.
Xend Deamon: Xen controller daemon, meaning it handles creating new domains, destroying extant domains, migration and many other domain management tasks.
The idea behind Xen is to run guest operating systems not in ring 0, but in a higher and less privileged ring. Running guest OSes in a ring higher than 0 is called "ring deprivileging". The default Xen installation n x86 runs guest OSes in ring 1, termed Current Privilege Level 1 of the processor. It runs a virtual machine monitor(VMM), the "hypervisor", in CPL 0. The applications run in ring 4 without any modification
Xen is a hypervisor virtual machine that runs multiple open-source operating systems
Clustering
Clustering essentially is the unification of multiple independent computers into a single system through the use of software and networking. By definition, a cluster must have a least two computers(called nodes), one master and one slave. The master node typically has a job scheduler that schedules the work to be done by the slave nodes. As the name implies, the slave nodes perform whatever work has been scheduled for them to do. The nodes of the Cluster communicate using a message passing interface. Two of the most commonly used message passing interfaces are PVM and MPI.
What files makes up a virtual machine
Vmware standard
A virtual machine typically is stored on the host computer in a set of files, usually in a directory created by workstation for the specific virtual machine
- vmware.log This is the file that keeps a log of key activity, can be useful for troubleshooting
-.nvram This is the file that stores the state of the virtual machine's BIOS
-.vmdk This is a virtual disk file, which stores the contents of the virtual machine's hard disk drive
--<###>.vmdk This is a redo-log file, created automatically when a virtual machine has one or more snapshots
-.vmem The virtual machine's paging file, which backs up the guest main memory on the host file system.
This file exists only when the virtual machine is running
-.vmss This is the suspeneded state file, which stores the state of a suspended virtual machine
-.vmtm This is the configuration file containing team data
-.vmx This is the primary configuration file , which stores settings chosen in the New virtual machine
-.vmxf This is a supplemental configuration file for virtual machines that are in a team
- delta.vmdk Snapshot differences file
Snapshot
A snapshot preserves the state and data of a virtual machine at a specific point in time
- state includes the virtual machine's power state(powered-on,off,suspended etc)
- Data includes all the files that make-up the virtual machine, including disks,memory and other devices
KVM (Kernel-based Virtual Machine)
It is a full virtualization solution for Linux on x86 hardware containing virtualization extensions intel VT or AMD-V
Why KVM?
Xen allows several guest Operating systems to execute on the same computer hardware and it is also included with RHEL
KVM is part of the official linux kernel and fully supported by NOVELL and REDHAT. Xen boots from GRUM and loads a modified host operating system such as RHEL into dom0(host domain). KVM don not have concept of dom0 and domU. It uses
/dev/kvm interface to setup the guest operating systems
Packages:
kmod-kvm: kvm kernel module, kvm: kernel-based virtual machine, kvm-qemu-img: Qemu disk image utility, kvm-tools, python-virtinst: Python modules and utilites for installing virtual machines, virt-manager: GUI app, virt-viewer: virtual machine viewer, bridge-utils: Utilities for configuring the Linux Ethernet bridge
A Note About libvirt
libvirt is an open source API and management tool for managing platform virtualzation. It is used to manage linux KVM and Xen virtual machines through graphical interfaces such as virtual Machine Manager
A Note About QEMU
QEMU is a processor emulator that relies on dynamic binary translation to achieve a reasonable speed while being easy to port on new host CPU architecutes. When used as a virtualizer, QEMU achieves near native performances by executing the guest code directly on the host CPU. QEMU supports virtualization when executing under the XEN hypervisor or using the KVM kernel module in Linux
A Note About Virtio Drivers
Virtio is paravirtualized drivers of kvm/Linux. With this you can run multiple virtual machines running unmodified Linux or Windows VMs. Each Virtual Machine has private virtualized hardware a network card, disk, graphics adapter etc
Para-virtualized drivers enchance the performance of fully virtualized guests. With the para-virtualized drivers guest I/O latency decreases and throughput increases to near bare-metal levels. It is recommended to use the para-virtualized drivers for fully virtualized guest running I/O heavy tasks applications
Important Configuration and Log files (Dirctories ) Location
- /etc/libvirt/ - Main Configuration directory
- /etc/libvirt/qemu/ - Virtual Machine configuratin directory. All xml files regarding VMs are stored here
- /etc/libvirt/qemu/networks/ - Networking for your KVM including default NAT. I strongly suggest you use bridge based networking for performance
- /etc/libvirt/qemu/networks/default.xml - The default NAT Configuration used by NAT device virbr0
- /var/log/libvirt/ - The default log file directory. All VM specifig logs files are stored here
- /etc/libvirt/libvirtd.conf - Master libvirtd configuration file
- /etc/libvirt/qemu.conf - master Configuration file for QEMU driver
KVM in-kernel device emulation
- Certain devices emulated in-kernel for performance reasons(APIC,IO-APIC,PIT)
- Memslots are in-kernel device ranges protected by a read-write semaphore
- Cacheline bouncing
- Wait for grace period
Virtualization progress has almost eliminated the barriers between operating systems and dramatically increased utilization of powerful servers, bringing immediate benefit to companies.
Software-emulated Virtulization
- full virtualization: a layer, commonly called the hypervisor or the virtual machine monitor, exist between the virtualized operating systems and the hardware.
- paravirtualization: is different in that the hypervisor operates in a more cooperative fashion, bacause each guest operating system is aware that it is running in a virtualized environment, so each cooperatives with the hypervisor to virtualize the underlying hardware
Both approaches have advantages and disadvantages. The primary advantage of the paravirtulization approach is that it allows the possible software-based virtualization, at the cost of not supporting proprietary operating systems. Full virtualization approaches, of course, do not have this limitation: however, full virtualization hypervisors are very complex pieces of software.
E.g: vmware full virtualization
E.g: paravirtualization is provided by Xen, User-mode Linux(UML)
Hardware-based virtualization: With the advent of Intel's VT and ADM's SVM, is possible to enjoy the benefits of full virtulization while keeping the hypervisor's complexity at a minimum
Xen, the classic paravirtualization engine, now supports fully virtualized MS Windows, with the help of hardware-based virtualization.
KVM is a relatively MS Windows, with the help of hardware-based virtualization. KVM is a relatively new and simple, yet powerful, virtualization engine, which has found its way into the Linux Kernel, giving the linux Kernel native
LAN --> Swith --> eth0 --> -++ 192.168.155.32
|--> br0 -+
| |
| |==> RHEL SERVER <--> | All the VM's
| | KVM
Wan --> Router --> eth1 --> -++ -+ br1 - +
Kernel
Kernel is heart of linux OS. It manages resource of linux O/S.
- I/O Management
- Process Management
- Device Management
- File Management
- Memory Management
Shell accepts your instruction or commands in English and translate it into computers native binary language
Your Command ==> Linux SHELL ==> Converted to Binary ==> Now linux kernel understand your
or Shell script Language by shell Request
ls ==> bash ==> 10010101 ==> Linux Kernel
A Look at IPv6
With IPv6, IP addresses go from 32-bit to 128-bit. Here's why the change is being made
We Achieved 32-bit addressing and a hierarchical arry of classes of networks: A,B,C,D and E. Class E addresses have never been available for general use.
CIDR: Classles Inter-Domain Routing,The Internet has grown with the number of intranets
IPv6 provides for 128-bit addressing. This is a gigantic number, larger than the extimated total number of molecules in the moon.
Over the next few years, conventional computers on the internet will be joined by a variety of new devices, including palmtop personal data assistants(PDA), hybrid mobile-phone technology with data processing capabilites, smart set-top boxes with integrated web browsers, and embedded network components in equipment ranging from Office copy machines to kitchen appliances. Many devices requiring IP addresses and connectivity will be consumer-oriented, such as your coffee machine, dishwasher etc
Extract rpm file to a particular Directory
rpm2cpio myrpmfile.rpm | cpio -idmv
Incremental backup with tar
tar pczvf - --listed-incremental /mnt/backups/ispconfig-log.snar /root/ispconfig | split -b 20m - /mnt/backups/ispconfig-`date '+%d-%B-%Y'`.tar.gz.
tar pczvf - --listed-incremental /mnt/backups/admispconfig-log.snar /home/admispconfig | split -b 20m - /mnt/backups/admispconfig-`date '+%d-%B-%Y'`.tar.gz.
### Multi-Tenancy and Load Simulation ###
Benchmarking of public IAAS clouds involves a certain amount of ambiguity due to the scheduling and allocation of resources in multi-tenant virtualized environments. One of the fundamental jobs of a hypervisors such as VMware and Xen is to allocate shared resources in a fair and consistent manner. In order to maximize performance and utilization, they are designed to allocate resources such as CPU and Disk IO using a combination of fixed and burstable methods. For example, when a VM requests CPU resources, the hypervisor will generally provide more resources when neighboring VMs are idle versus when they are also requesting CPU resources. In very busy environments, this often results in variable and inconsistent VM performance.
In the first run, the benchmarks were run individually to provide maximum performance. In the second run, we attempted to simulate a loaded environment by filling the cloud to about 70% capacity with VMs instructed to perform a random sample of load simulating benchmarks (using mostly non-synthetic benchmarks like tpcc, blogbench and pgbench). The benchmarks for the second run were conducted concurrently with the load simulation. The tables and graphs below provide the unloaded benchmark results. Differences between those and the loaded results are noted above the results.
- CPU Performance
- Disk I/O Performance
- Programming Language Performance
- Memory IO Performance
- Encoding & Encryption Performance
### Zen Load-Balancer ###
In computer networking, load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload. Using multiple components with load balancing, instead of a single component, may increase reliability through redundancy. The load balancing service is usually provided by a dedicated program or hardware device
It is commonly used to mediate internal communications in computer clusters, especially high-availability clusters. If the load is more on a server, then the secondary server takes some load while the other is still processing requests.
Zen is a collection of software which provide a full-featured load Balancer Server. TCP Simple Load Balancer that could work with services as SMTP, LDAP, FTP, IMAP, POP, HTTP, or other services over the TCP Protocol
It is a logical map about Zen Load Balancer Architecture (LDAP FARM, MAIL FARM, WEB FARM)
# Building a Multisourced Infrastructure Using OpenVPN
To satisfy redundancy and fault-tolerance requirements, we needed an active-active pair of OpenVPN servers to share a common address space - all hosts must be able to access each other by static IP addresses at all times, no matter which OpenVPN server provides connectivity at either end of the communication. Then, if we lose one OpenVPN server, the other will provide all connectivity. And, if they are both up, both will accepting connections from clinets to share the load. This feature was not available as a part of OpenVPN source distribution, so we developed a standalone dynamic routing daemin to facilitate active-active load balancing.
find -type f -mmin +3 -delete ( Delete files)
Creating Custom live CD for Compute nodes from Chroot Env
The diagram below is an attempt to show how some of the virtualisation components in linux relate. The lower part of the diagram
SOAP, originally defined as Simple Object Access Protocol, is a protocol specification for exchanging structured information in the implementation of Web Services in computer networks.
http://www.tuxradar.com/content/howto-linux-and-windows-virtualization-kvm-and-qemu
http://code.google.com/p/appscale/wiki/Introduction_to_Xen
http://www.ibm.com/developerworks/linux/library/l-xen/
http://lazysystemadmin.blogspot.com/2010/05/xen-virtualization-installation.html
http://en.wikipedia.org/wiki/Xen
http://www.logicsupply.com/blog/2009/11/25/virtualization-with-xen/
http://linuxdevcenter.com/pub/a/linux/2006/01/26/xen.html
http://www.linuxjournal.com/content/storage-cluster-challenge-lj-staff-and-readers
http://www.linuxjournal.com/article/9764
- A Web server serves pages for viewing in a Web browser, while an application server provides methods that client
applications can call.
- A Web server exclusively handles HTTP requests
- A Web server handles the HTTP protocol. When the Web server receives an HTTP request, it responds with an HTTP response, such as sending back an HTML page. To process a request, a Web server may respond with a static HTML page or image, send a redirect, or delegate the dynamic response generation to some other program such as CGI scipts , JSPs (JavaSever Pages), servlets, ASPs(Active Server Pages), server-side JavaScripts, or some other server side technology.
- The server side program usually provides for itself such functions as transaction processing, database connectivity and messaging
- Webserver may not itself support transactions or database connection pooling, may employ various strategies for fault tolerance and scalability such as load balancing, caching and clustering
- Eg: Apache HTTP Server, Sun ONE Web Server, iPlanet Web Server, Microsoft IIS's (Internet Information server)
- Webserver - any company's website, it responds with a static or dynamic html pages for each click providing the information you requested
- Protocol dependent (Handle HTTP only)
# Application Server
- Application server serves business logic to application programs through any number of protocols. \
- An application server exposes business logic to client applications through various protocols, possibly including HTTP.
- An application server provides access to business logic to use by client applicatin programs. The application program can use this logic just as it would call a method on an object
- Application server manages its own resources. Such gate-keeping duties include security, transaction process, resource pooling, and messaging.
- Application Server is a super set of which web server is a sub-set
- Both application/webserver offers caches, clustering and load balancing
- Application server means we can store application (DATA) in that Server
- Application server - any business application such as icicibank, hdfc(These applications are called as enterprise applications) or any bank website as in these sort of application, you not only click the login button, affer that u login giving credentials, check your balance, transfer money online... These things you cannot do in web application.
- Webserver handles only http protocols where as application server can handle any type of protocols.
- Application server maintains the application logic and serves the webpages in response to user request
- Application server can do both application logic maintanence and webapage serving
- Application server is used for processing/applying logic
- E.g: weblogic server is an example of Application server
- protocol independent
- Application server = webserver + EJB Container
- Application server is a server computer on a computer network dedicated to running certain 'Software application'
- An Application server is a software engine that delivers applications to client computers
- Benefits of application server technology are ease of application development and centralization
- Application server is more capable of dynamic behavior than webserver. We can also configure application server to work
as a webserver. Simply application server is a superset of webserver
- Eg: IBM WebSphere, Redhat JBOSS, Weblogic server, tomcat
- An application server has a 'built-in' webserver, in addition to that it supports other modules or features like
e-business integration, independent management and security module etc
### Portal Server ###
Portal server is just an application server running a portal software or a portal application
Different networking types
Bridged Networking
Bridged networking connects a virtual machine to a network by using the host computer’s network adapter. If your host computer is on a network, this is often the easiest way to give your virtual machine access to that network. The virtual network adapter in the virtual machine connects to the physical network adapter in your host computer, allowing it to connect to the LAN the host computer uses.
Bridged networking configures your virtual machine as a unique identity on the network, separate from and unrelated to its host. It makes the virtual machine visible to other computers on the network, and they can communicate directly with the virtual machine. Bridged networking works with both wired and wireless physical host network cards.
Note: Bridge network uses the host Computer network adapter, the VM Can access the host computer Network
Network Address Translation (NAT)
NAT configures a virtual machine to share the IP and MAC addresses of the host. The virtual machine and the host share a single network identity that is not visible outside the network. NAT can be useful when your network administrator lets you use a single IP address or MAC address. If cannot give your virtual machine an IP address on the external network, you can use NAT to give your virtual machine access to the Internet or another TCP/IP network. NAT uses the host computer’s network connection. NAT works with Ethernet, DSL, and phone modems.
Setup Requirements for IP Addresses
If you use NAT, your virtual machine does not have its own IP address on the external network. Instead, a separate private network is set up on the host computer.
Note: NAT will share the IP and MAC address of the host machine with the VM, it can access the Internet and host network as well
Host-Only Networking
Host-only networking creates a network that is completely contained within the host computer. Host-only networking provides a network connection between the virtual machine and the host computer, using a virtual network adapter that is visible to the host operating system. This approach can be useful if you need to set up an isolated virtual network. In this configuration, the virtual machine cannot connect to the Internet.
Note: Isolated Network
### Virtualization ###
virtualization has been around in one form or another since it was first introduced on IBM mainframe operating systems in the 1960s. The reason for the sudden popularity of virtualization can be attributed to a number of largely unconnected
trends
- Green computing- Recent trend to reduce the power consuption of computer systems
- Increased computing power - Virtualization allows companines to maximize utilization of hardware by running multiple operating systems concurrently on single physical systems
- financial constraints - A key technique for reducing IT overheads is to use virtualization to gain maximum return on investments in computer hardware
- Web 2.0 & Cloud Computing - The term web 2.0 has primarily come to represent the gradual shift away form hosting applications and data on local computer systems ot a web based approach. Example many users and companies now use Google apps for spreadsheet and word processing instead of installing office sutie software on local computers
- Operating system market fragmentation - In recent years the operating system market has increasingly fragmented with Microsoft ceding territory to offerings such as linux and Mac
Guest Operating System Virtualization
Guest OS Virtualization, also referred to as application based virtualization. Running on Operating system is a virtualization application which executes in much the same way as any other application such as word Processor. The virtualization application is responsible for starting hardware resources on behalf of the individual virtual machines.
The virtualization application also engages in a process known as 'binary rewriting' which involves scanning the instruction stream of the executing guest system and replacing any privileged instructions with safe emulations. This has the effect of making the guest system think it is running directly on the system hardware.
Guest Operating system <-> Virtualization Application <-> Host Opearating System <-> Host Hardware
Shared Kernel Virtualization
shared kernel virtualization (also known as system level or operating system virtualization) takes advantage of the architectural design of linux and Unix based Operating systems.
The kernel, in simple terms, handles all the interactions between the operating system and the physical hardware. The second key component is the 'root file system' which contains all the libraries,files and utilities necessary for the operating system to function.Under shared kernel virtualization the virtual guest systems each have their own root file system but share the kernel of the host operating system. This type of virtualization is made possible by the ability of the kernel to dynamically change the current root file system(a concept known as chroot) to a different root file system without having to reboot the entire system.
Drawback The guest operating systems must be compatible with the version of the kernel which is being shared. It is not, for example, possible to run Microsoft windows, Nor possible to share a linux guest system designed for the 2.6 to 2.4
Eg. Solaris Zones and Containers
Host Operating system Guest - Operating sytem & root filesystem -Shared Kernel <-> Host Hardware
Kernel Level Virtulization
Under Kernel level virtualization the host operating system runs on a specially modified kernel which contains extensions designed to manage and control multiple virtual machines each containing a guest Operating system.
Examples of Kernel level Virtualization technologies include User Mode Linux (UML) and Kernel-based Virtual Machine(KVM)
Guest Operating system - Linux Kernel <-> Host Hardware
### Hypervisor Virtualization ###
The x86 family of CPU's provide a range of protection levels also known as rings in which code can execute. Ring 0 has the highest level privilege and it is in this ring that the operating system kernel noramally runs. Code executing in ring0 is said to be running in system space, kernel mode or supervisor mode. All other code such as applications running on the operating system operates in less privileged rings, typically ring 3
Under hypervisor virtualization a program known as a hypervisor runs directly on the hardware of the host system in ring 0. The task of this hypervisor is to handle resource and memory allocation for the virtual machines in addition to providing interfaces for higher level administration and monitoring tools
clearly, with the hypervisor occupying ring0 of the CPU, the kernels for any guest operating systems running on the system must run in less privileged CPU rings. Unfortunately, most operating system kernels are written explicitly to run in ring 0 for the simple reason that they need to perform tasks that are only available in that ring, such as the ability to execute privileged CPU instructions and directly manipulate memory. A number of different solutions to this problem have been devised in recent years
Paravirtualization
Under paravirtualization the kernel of the guest operating system is modified specifically to run on the hyperisor.This typically involves replacing any privileged operations that will only run in ring 0 of the CPU with calls to the hypervisor(known as hypercalls). The hypervisor in turn performs the task on behalf of the guest kernel.
Note: The guest kernel to communicate directly with the hypervisor results in greater performance levels compared to other virtualization approaches
E.g Citrix XEN
Full Virtualization
Full Virtualization provides support for unmodified guest operating systems(Unmodified Kernel ). The term unmodified refers to operating system kernels which have not been altered to run on a hypervisor and therefore still execute privileged operations as though running in ring 0 of the CPU. In this scenario, the hypervisor provides CPU emulation to handle and modify privileged and protected CPU operations made by unmodified guest operating system kernels
Note: Unfortunately this emulation process requires both time and system resources to operate resulting in inferior performance levels
Hardware Virtualization
Hardware Virtualization leverages virtualization features built into the latest generations of CPU's from both intel and AMD. These technologies, known as Intel VT and AMD-V respectively, provide extensions necessary to run unmodified guest virtual machines.
Note: In very simplistic terms these new processors provide an additional privilege mode above ring 0 in which the hypervisor can operate essentially leaving ring 0 available for unmodified guest operating systems
Administrative Env - Virtual Machine <-> Hypervisor <-> Host Hardware
Hypervisor based virtualizatin solutions include Xen, VMware Esx Server and Microsoft's Hyper-V technology
XEN
Managing VMs
- starting VMs(xm create)
- stopping VMs(xm shutdown)
- Really stopping VMs(xm destroy)
- Monitoring (xm list, xm top)
xen config file - It will be having all the details including memory, disk,kernel, swap,name, mac address
Xen is a feature rich, opensource, hypervisor-based virtualization solution
Xen supports a variety of operating systems, which increases its applicability for running many popular applications on whichever OS you desire. Xen currently supports linux kernels through an approach called paravirtualization in which the operating system is modified to become aware of Xen
Xen system has a single privileged OS, called Doamin-0, that is responsible for starting and managing the other unprivileged OS instances. Domain-0 is the OS that boots when you start your computer, and it has the tools necessary to manage other domains.
Xen itself is called a hypervisor because it operated at a higher privilege level than the supervisor code of the guest operating systems that it hosts
At boot time, Xen is loaded into memery in ring 0. It starts a patched kernel in ring1; this is called domain0. From this domain you can create other domains, destroy them, perform migrations of domains, set parameters to a domain and more. the domains you create also run their kernels in ring 1. User applications run in ring 3.
Xend Deamon: Xen controller daemon, meaning it handles creating new domains, destroying extant domains, migration and many other domain management tasks.
The idea behind Xen is to run guest operating systems not in ring 0, but in a higher and less privileged ring. Running guest OSes in a ring higher than 0 is called "ring deprivileging". The default Xen installation n x86 runs guest OSes in ring 1, termed Current Privilege Level 1 of the processor. It runs a virtual machine monitor(VMM), the "hypervisor", in CPL 0. The applications run in ring 4 without any modification
Xen is a hypervisor virtual machine that runs multiple open-source operating systems
Clustering
Clustering essentially is the unification of multiple independent computers into a single system through the use of software and networking. By definition, a cluster must have a least two computers(called nodes), one master and one slave. The master node typically has a job scheduler that schedules the work to be done by the slave nodes. As the name implies, the slave nodes perform whatever work has been scheduled for them to do. The nodes of the Cluster communicate using a message passing interface. Two of the most commonly used message passing interfaces are PVM and MPI.
What files makes up a virtual machine
Vmware standard
A virtual machine typically is stored on the host computer in a set of files, usually in a directory created by workstation for the specific virtual machine
- vmware.log This is the file that keeps a log of key activity, can be useful for troubleshooting
-
-
-
-
This file exists only when the virtual machine is running
-
-
-
-
- delta.vmdk Snapshot differences file
Snapshot
A snapshot preserves the state and data of a virtual machine at a specific point in time
- state includes the virtual machine's power state(powered-on,off,suspended etc)
- Data includes all the files that make-up the virtual machine, including disks,memory and other devices
KVM (Kernel-based Virtual Machine)
It is a full virtualization solution for Linux on x86 hardware containing virtualization extensions intel VT or AMD-V
Why KVM?
Xen allows several guest Operating systems to execute on the same computer hardware and it is also included with RHEL
KVM is part of the official linux kernel and fully supported by NOVELL and REDHAT. Xen boots from GRUM and loads a modified host operating system such as RHEL into dom0(host domain). KVM don not have concept of dom0 and domU. It uses
/dev/kvm interface to setup the guest operating systems
Packages:
kmod-kvm: kvm kernel module, kvm: kernel-based virtual machine, kvm-qemu-img: Qemu disk image utility, kvm-tools, python-virtinst: Python modules and utilites for installing virtual machines, virt-manager: GUI app, virt-viewer: virtual machine viewer, bridge-utils: Utilities for configuring the Linux Ethernet bridge
A Note About libvirt
libvirt is an open source API and management tool for managing platform virtualzation. It is used to manage linux KVM and Xen virtual machines through graphical interfaces such as virtual Machine Manager
A Note About QEMU
QEMU is a processor emulator that relies on dynamic binary translation to achieve a reasonable speed while being easy to port on new host CPU architecutes. When used as a virtualizer, QEMU achieves near native performances by executing the guest code directly on the host CPU. QEMU supports virtualization when executing under the XEN hypervisor or using the KVM kernel module in Linux
A Note About Virtio Drivers
Virtio is paravirtualized drivers of kvm/Linux. With this you can run multiple virtual machines running unmodified Linux or Windows VMs. Each Virtual Machine has private virtualized hardware a network card, disk, graphics adapter etc
Para-virtualized drivers enchance the performance of fully virtualized guests. With the para-virtualized drivers guest I/O latency decreases and throughput increases to near bare-metal levels. It is recommended to use the para-virtualized drivers for fully virtualized guest running I/O heavy tasks applications
Important Configuration and Log files (Dirctories ) Location
- /etc/libvirt/ - Main Configuration directory
- /etc/libvirt/qemu/ - Virtual Machine configuratin directory. All xml files regarding VMs are stored here
- /etc/libvirt/qemu/networks/ - Networking for your KVM including default NAT. I strongly suggest you use bridge based networking for performance
- /etc/libvirt/qemu/networks/default.xml - The default NAT Configuration used by NAT device virbr0
- /var/log/libvirt/ - The default log file directory. All VM specifig logs files are stored here
- /etc/libvirt/libvirtd.conf - Master libvirtd configuration file
- /etc/libvirt/qemu.conf - master Configuration file for QEMU driver
KVM in-kernel device emulation
- Certain devices emulated in-kernel for performance reasons(APIC,IO-APIC,PIT)
- Memslots are in-kernel device ranges protected by a read-write semaphore
- Cacheline bouncing
- Wait for grace period
Virtualization progress has almost eliminated the barriers between operating systems and dramatically increased utilization of powerful servers, bringing immediate benefit to companies.
Software-emulated Virtulization
- full virtualization: a layer, commonly called the hypervisor or the virtual machine monitor, exist between the virtualized operating systems and the hardware.
- paravirtualization: is different in that the hypervisor operates in a more cooperative fashion, bacause each guest operating system is aware that it is running in a virtualized environment, so each cooperatives with the hypervisor to virtualize the underlying hardware
Both approaches have advantages and disadvantages. The primary advantage of the paravirtulization approach is that it allows the possible software-based virtualization, at the cost of not supporting proprietary operating systems. Full virtualization approaches, of course, do not have this limitation: however, full virtualization hypervisors are very complex pieces of software.
E.g: vmware full virtualization
E.g: paravirtualization is provided by Xen, User-mode Linux(UML)
Hardware-based virtualization: With the advent of Intel's VT and ADM's SVM, is possible to enjoy the benefits of full virtulization while keeping the hypervisor's complexity at a minimum
Xen, the classic paravirtualization engine, now supports fully virtualized MS Windows, with the help of hardware-based virtualization.
KVM is a relatively MS Windows, with the help of hardware-based virtualization. KVM is a relatively new and simple, yet powerful, virtualization engine, which has found its way into the Linux Kernel, giving the linux Kernel native
LAN --> Swith --> eth0 --> -++ 192.168.155.32
|--> br0 -+
| |
| |==> RHEL SERVER <--> | All the VM's
| | KVM
Wan --> Router --> eth1 --> -++ -+ br1 - +
Kernel
Kernel is heart of linux OS. It manages resource of linux O/S.
- I/O Management
- Process Management
- Device Management
- File Management
- Memory Management
Shell accepts your instruction or commands in English and translate it into computers native binary language
Your Command ==> Linux SHELL ==> Converted to Binary ==> Now linux kernel understand your
or Shell script Language by shell Request
ls ==> bash ==> 10010101 ==> Linux Kernel
A Look at IPv6
With IPv6, IP addresses go from 32-bit to 128-bit. Here's why the change is being made
We Achieved 32-bit addressing and a hierarchical arry of classes of networks: A,B,C,D and E. Class E addresses have never been available for general use.
CIDR: Classles Inter-Domain Routing,The Internet has grown with the number of intranets
IPv6 provides for 128-bit addressing. This is a gigantic number, larger than the extimated total number of molecules in the moon.
Over the next few years, conventional computers on the internet will be joined by a variety of new devices, including palmtop personal data assistants(PDA), hybrid mobile-phone technology with data processing capabilites, smart set-top boxes with integrated web browsers, and embedded network components in equipment ranging from Office copy machines to kitchen appliances. Many devices requiring IP addresses and connectivity will be consumer-oriented, such as your coffee machine, dishwasher etc
Extract rpm file to a particular Directory
rpm2cpio myrpmfile.rpm | cpio -idmv
Incremental backup with tar
tar pczvf - --listed-incremental /mnt/backups/ispconfig-log.snar /root/ispconfig | split -b 20m - /mnt/backups/ispconfig-`date '+%d-%B-%Y'`.tar.gz.
tar pczvf - --listed-incremental /mnt/backups/admispconfig-log.snar /home/admispconfig | split -b 20m - /mnt/backups/admispconfig-`date '+%d-%B-%Y'`.tar.gz.
### Multi-Tenancy and Load Simulation ###
Benchmarking of public IAAS clouds involves a certain amount of ambiguity due to the scheduling and allocation of resources in multi-tenant virtualized environments. One of the fundamental jobs of a hypervisors such as VMware and Xen is to allocate shared resources in a fair and consistent manner. In order to maximize performance and utilization, they are designed to allocate resources such as CPU and Disk IO using a combination of fixed and burstable methods. For example, when a VM requests CPU resources, the hypervisor will generally provide more resources when neighboring VMs are idle versus when they are also requesting CPU resources. In very busy environments, this often results in variable and inconsistent VM performance.
In the first run, the benchmarks were run individually to provide maximum performance. In the second run, we attempted to simulate a loaded environment by filling the cloud to about 70% capacity with VMs instructed to perform a random sample of load simulating benchmarks (using mostly non-synthetic benchmarks like tpcc, blogbench and pgbench). The benchmarks for the second run were conducted concurrently with the load simulation. The tables and graphs below provide the unloaded benchmark results. Differences between those and the loaded results are noted above the results.
- CPU Performance
- Disk I/O Performance
- Programming Language Performance
- Memory IO Performance
- Encoding & Encryption Performance
### Zen Load-Balancer ###
In computer networking, load balancing is a technique to distribute workload evenly across two or more computers, network links, CPUs, hard drives, or other resources, in order to get optimal resource utilization, maximize throughput, minimize response time, and avoid overload. Using multiple components with load balancing, instead of a single component, may increase reliability through redundancy. The load balancing service is usually provided by a dedicated program or hardware device
It is commonly used to mediate internal communications in computer clusters, especially high-availability clusters. If the load is more on a server, then the secondary server takes some load while the other is still processing requests.
Zen is a collection of software which provide a full-featured load Balancer Server. TCP Simple Load Balancer that could work with services as SMTP, LDAP, FTP, IMAP, POP, HTTP, or other services over the TCP Protocol
It is a logical map about Zen Load Balancer Architecture (LDAP FARM, MAIL FARM, WEB FARM)
# Building a Multisourced Infrastructure Using OpenVPN
To satisfy redundancy and fault-tolerance requirements, we needed an active-active pair of OpenVPN servers to share a common address space - all hosts must be able to access each other by static IP addresses at all times, no matter which OpenVPN server provides connectivity at either end of the communication. Then, if we lose one OpenVPN server, the other will provide all connectivity. And, if they are both up, both will accepting connections from clinets to share the load. This feature was not available as a part of OpenVPN source distribution, so we developed a standalone dynamic routing daemin to facilitate active-active load balancing.
find -type f -mmin +3 -delete ( Delete files)
Creating Custom live CD for Compute nodes from Chroot Env
The diagram below is an attempt to show how some of the virtualisation components in linux relate. The lower part of the diagram
SOAP, originally defined as Simple Object Access Protocol, is a protocol specification for exchanging structured information in the implementation of Web Services in computer networks.
http://www.tuxradar.com/content/howto-linux-and-windows-virtualization-kvm-and-qemu
http://code.google.com/p/appscale/wiki/Introduction_to_Xen
http://www.ibm.com/developerworks/linux/library/l-xen/
http://lazysystemadmin.blogspot.com/2010/05/xen-virtualization-installation.html
http://en.wikipedia.org/wiki/Xen
http://www.logicsupply.com/blog/2009/11/25/virtualization-with-xen/
http://linuxdevcenter.com/pub/a/linux/2006/01/26/xen.html
http://www.linuxjournal.com/content/storage-cluster-challenge-lj-staff-and-readers
http://www.linuxjournal.com/article/9764
No comments:
Post a Comment