The user to own Oracle Grid Infrastructure is grid.
The user to own Oracle Database software is oracle.
The installation is done on a Windows 7 (64 bit) host with Core i5 Processor and 8G RAM. Each node will have 2.5G RAM. Installation is not possible on low spec machine.
Below are the software used:
Oracle Enterprise Linux 5.3 (32-bit)
VMware Server 2.0
Oracle 11gR2 Grid Infrastructure and database (11.2.0.3 - 32-bit)
Install the VMware Server.
Install the VMware Server software the same as any other application installation on your Windows 7 host.
Access the web-based VMware Infrastructure Web Access Console by pointing your browser to any of the following URLs:
http://windowshostname:8222
https://windowshostname:8333
Login with your windows admin account
Now that the VMware server is setup, we will create the virtual machines. To save time, we will only configure one (1) virtual machine and clone it later.
> Open VMware Intrastructure Web Access Console
> Click the "Virtual Machine > Create Virtual Machine" menu option, or click the "Create Virtual Machine" link on the bottom right of the console.
> Enter the name "RAC1" and accept the standard datastore by clicking the "Next" button.
Install Guest Operating System
> Start the virtual machine by clicking the play button on the toolbar.
> Skip CD media testing
Throughout the lesson, the following IP will be used. You can choose your own IP to suit your network.
-- Machine 1:
hostname: rac1.localdomain
IP Address eth0: 192.168.130.101 -- This is your Public IP Address
Subnet mask eth0: 255.255.255.0
Default Gateway eth0: 192.168.130.1
IP Address eth1: 192.168.131.111 (rac1-priv) -- This is your Private IP Address (Should be on different subnet with Public)
Subnet mask eth1: 255.255.255.0
Default Gateway eth1: NONE
192.168.130.121 (rac1-vip) -- This is your Virtual IP Address (The same subnet with Public)
---------------------------------------
-- Machine 1:
hostname: rac2.localdomain
IP Address eth0: 192.168.130.102
Subnet mask eth0: 255.255.255.0
Default Gateway eth0: 192.168.130.1
IP Address eth1: 192.168.131.112 (rac2-priv)
Subnet mask eth1: 255.255.255.0
Default Gateway eth1: NONE
192.168.130.122 (rac2-vip)
SCAN IPs:
(Should be the same subnet with Public and defined in DNS server)
rac-scan 192.168.130.201
rac-scan 192.168.130.202
rac-scan 192.168.130.203
> Click "Continue" and ignore no "Primary DNS" specified. We will set this later.
GNOME Desktop Environment
Editors
Graphical Internet
Text-based Internet
Development Libraries
Development Tools
X Software Development
Administration Tools
Base
Java
System Tools
X Window System
> Click "Next". Installer will automatically check dependencies in packages.
> Change directory to your DVD installer.
cd /media/Enterprise\ Linux\ dvd\ 20090127/Server/
rpm -Uvh binutils-2*
rpm -Uvh binutils-devel-2*
rpm -Uvh compat-libstdc++-33*
rpm -Uvh elfutils-libelf-0*
rpm -Uvh elfutils-libelf-devel-*
rpm -Uvh gcc-4*
rpm -Uvh gcc-c++-4*
rpm -Uvh glibc-2*
rpm -Uvh glibc-common-2*
rpm -Uvh glibc-devel-2*
rpm -Uvh glibc-headers-2*
rpm -Uvh ksh-*
rpm -Uvh libaio-0*
rpm -Uvh libaio-devel-0*
rpm -Uvh libgcc-4*
rpm -Uvh libstdc++-4*
rpm -Uvh libstdc++-devel-4*
rpm -Uvh make-3*
rpm -Uvh sysstat-7*
rpm -Uvh unixODBC-2*
rpm -Uvh unixODBC-devel-2*
rpm -Uvh oracleasm-support-2*
rpm -Uvh oracleasm-2.6.18-128.el5-2*
Oracle Installation Prerequisites
> Perform the following steps whilst logged into the RAC1 virtual machine as the root user.
> Make sure the shared memory filesystem is big enough for Automatic Memory Manager to work.
[root@rac1 ~]# df -h |grep tmpfs
[root@rac1 ~]# umount tmpfstmpfs 1.3G 0 1.3G 0% /dev/shm
[root@rac1 ~]# mount -t tmpfs shmfs -o size=1500m /dev/shm
> Make the setting permanent by amending the "tmpfs" setting of the "/etc/fstab" file to look like this.
> Edit "/etc/hosts" file. It must contain the following information./dev/VolGroup00/LogVol00 / ext3 defaults 1 1
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs size=1500m 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0
127.0.0.1 localhost.localdomain localhost
# Public
192.168.130.101 rac1.localdomain rac1
192.168.130.102 rac2.localdomain rac2
# Private (Should be on different subnet with Public)
192.168.131.111 rac1-priv.localdomain rac1-priv
192.168.131.112 rac2-priv.localdomain rac2-priv
# Virtual (The same subnet with Public)
192.168.130.121 rac1-vip.localdomain rac1-vip
192.168.130.122 rac2-vip.localdomain rac2-vip
# SCAN (Should be the same subnet with Public and defined in DNS server)
#192.168.130.201 rac-scan.localdomain rac-scan
#192.168.130.202 rac-scan.localdomain rac-scan
#192.168.130.203 rac-scan.localdomain rac-scan
The SCAN address should not really be defined in the hosts file. Instead is should be defined on the DNS to round-robin between 3 addresses on the same subnet as the public IPs.
> Modify "/etc/sysctl.conf" file to include the following:
fs.aio-max-nr = 1048576
fs.file-max = 6815744
kernel.shmall = 2097152
kernel.shmmax = 1329047552
kernel.shmmni = 4096
# semaphores: semmsl, semmns, semopm, semmni
kernel.sem = 250 32000 100 128
net.ipv4.ip_local_port_range = 9000 65500
net.core.rmem_default=262144
net.core.rmem_max=4194304
net.core.wmem_default=262144
net.core.wmem_max=1048586
> Reload the current kernel parameters.
/sbin/sysctl -p
> Add the following lines to the "/etc/security/limits.conf" file.
oracle soft nproc 2047
oracle hard nproc 16384
oracle soft nofile 1024
oracle hard nofile 65536
grid soft nproc 2047
grid hard nproc 16384
grid soft nofile 1024
grid hard nofile 65536
> Add the following lines to the "/etc/pam.d/login" file, if it does not already exist.
session required pam_limits.so
> Ensure secure linux is disabled in "/etc/selinux/config" file
SELINUX=disabled
> Deconfigure NTP. We will use Oracle Cluster Time Synchronization Service (ctssd) to synchronize the times of the RAC nodes.
[root@rac1 ~]# service ntpd stop
Shutting down ntpd: [OK]
[root@rac1 ~]# chkconfig ntpd off
[root@rac1 ~]# mv /etc/ntp.conf /etc/ntp.conf.org
[root@rac1 ~]# rm /var/run/ntpd.pid
> Create the new groups and users.
[root@rac1 ~]# groupadd dba
[root@rac1 ~]# groupadd oper
[root@rac1 ~]# groupadd oinstall
[root@rac1 ~]# groupadd asmdba
[root@rac1 ~]# groupadd asmadmin
[root@rac1 ~]# groupadd asmoper
[root@rac1 ~]# useradd -g oinstall -G dba,asmdba,asmadmin,oper oracle
[root@rac1 ~]# useradd -g oinstall -G dba,asmdba,asmadmin,asmoper grid
[root@rac1 ~]# passwd oracle
[root@rac1 ~]# passwd grid
> Create the directories in which the Oracle software will be installed.
[root@rac1 ~]# mkdir -p /u01/app/grid
[root@rac1 ~]# mkdir -p /u01/app/11.2.0/grid
[root@rac1 ~]# mkdir -p /u01/app/oracle/product/11.2.0/db_1
[root@rac1 ~]# chown -Rv grid:oinstall /u01/app/grid
[root@rac1 ~]# chown -Rv grid:oinstall /u01/app/11.2.0
[root@rac1 ~]# chown -Rv oracle.oinstall /u01/app/oracle
[root@rac1 ~]# chmod -R 775 /u01/
Additional Setup
> Perform the following steps whilst logged into the "rac1" virtual machine as the root user.
Install the following package from the Oracle grid media after you've defined groups.
[root@rac1 ~]# cd /home/grid/staging/grid/rpm
[root@rac1 rpm]# rpm -ivh cvuqdisk-1.0.9-1.rpm
> Login as the oracle user and add the following lines at the end of the ".bash_profile" file.
[root@rac1 ~]# su - oracle
[oracle@rac1 ~]$ vi .bash_profile
# Oracle Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac1.localdomain; export ORACLE_HOSTNAME
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=/u01/app/oracle/product/11.2.0/db_1; export ORACLE_HOME
ORACLE_SID=RAC1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "oracle" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
> Login as the grid user and add the following lines at the end of the ".bash_profile" file.
[root@rac1 ~]# su - grid
[grid@rac1 ~]$ vi .bash_profile
# Grid Settings
TMP=/tmp; export TMP
TMPDIR=$TMP; export TMPDIR
ORACLE_HOSTNAME=rac1.localdomain; export ORACLE_HOSTNAME
ORACLE_BASE=/u01/app/grid; export ORACLE_BASE
ORACLE_HOME=/u01/app/11.2.0/grid; export ORACLE_HOME
ORACLE_SID=+ASM1; export ORACLE_SID
ORACLE_TERM=xterm; export ORACLE_TERM
BASE_PATH=/usr/sbin:$PATH; export BASE_PATH
PATH=$ORACLE_HOME/bin:$BASE_PATH; export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib:/lib:/usr/lib; export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE:$ORACLE_HOME/jlib:$ORACLE_HOME/rdbms/jlib; export CLASSPATH
if [ $USER = "grid" ]; then
if [ $SHELL = "/bin/ksh" ]; then
ulimit -p 16384
ulimit -n 65536
else
ulimit -u 16384 -n 65536
fi
fi
Install VMware Client Tools
> On the web console, highlight the "RAC1" VM and click the "Install VMware Tools" link and click the subsequent "Install" button.
> Accept all the default settings and pick the screen resolution of your choice. Ignore any warnings or errors. The VMware client tools are now installed.
> Issue the "vmware-toolbox" command as the root user. On the subsequent dialog, check the "Time synchronization between the virtual machine and the host operating system" option and click the "Close" button.
Create Shared Disks
> Shut down the RAC1 virtual machine using the following command.
[root@rac1 ~]# shutdown -h now
> Create a directory on the host system to hold the shared virtual disks.[root@rac1 ~]# shutdown -h now
Broadcast message from root (pts/1) (Thu Oct 11 10:03:47 2012):
The system is going down for system halt NOW!
C:\Users\gellee>mkdir "E:\DATA\Gellee\Renz\VMWare\RACShared"
> On the VMware Intrastructure Web Access Console, click the "Add" link from the Commands.
- Add Command.jpg (18.78 KiB) Viewed 2634 times
File Name: [R3nZ] RACShared/asm2.vmdk
Virtual Device Node: SCSI 1:2
Mode: Independent and Persistent
File Name: [R3nZ] RACShared/asm3.vmdk
Virtual Device Node: SCSI 1:3
Mode: Independent and Persistent
File Name: [R3nZ] RACShared/asm4.vmdk
Virtual Device Node: SCSI 1:4
Mode: Independent and Persistent
File Name: [R3nZ] RACShared/asm5.vmdk
Virtual Device Node: SCSI 1:5
Mode: Independent and Persistent
> At the end of this process, the virtual machine should look something like the screenshot below.
> Start the RAC1 virtual machine by clicking the "Play" button on the toolbar, then start the console as before. When the server has started, log in as the root user so you can partition the disks. The current disks can be seen by issuing the following commands.disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
scsi1.present = "TRUE"
scsi1.sharedBus = "VIRTUAL"
scsi1.virtualDev = "lsilogic"
scsi1:1.present = "TRUE"
scsi1:1.fileName = "E:\DATA\Gellee\Renz\VMWare\RACShared\asm1.vmdk"
scsi1:1.writeThrough = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.deviceType = "plainDisk"
scsi1:1.redo = ""
scsi1:2.present = "TRUE"
scsi1:2.fileName = "E:\DATA\Gellee\Renz\VMWare\RACShared\asm2.vmdk"
scsi1:2.writeThrough = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.deviceType = "plainDisk"
scsi1:2.redo = ""
scsi1:3.present = "TRUE"
scsi1:3.fileName = "E:\DATA\Gellee\Renz\VMWare\RACShared\asm3.vmdk"
scsi1:3.writeThrough = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.deviceType = "plainDisk"
scsi1:3.redo = ""
scsi1:4.present = "TRUE"
scsi1:4.fileName = "E:\DATA\Gellee\Renz\VMWare\RACShared\asm4.vmdk"
scsi1:4.writeThrough = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.deviceType = "plainDisk"
scsi1:4.redo = ""
scsi1:5.present = "TRUE"
scsi1:5.fileName = "E:\DATA\Gellee\Renz\VMWare\RACShared\asm5.vmdk"
scsi1:5.writeThrough = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.deviceType = "plainDisk"
scsi1:5.redo = ""
[root@rac1 ~]# fdisk -l
> Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the expected fdisk output for the sdb disk.[root@rac1 ~]# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 3916 31350847+ 8e Linux LVM
Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdb doesn't contain a valid partition table
Disk /dev/sdc: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdc doesn't contain a valid partition table
Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdd doesn't contain a valid partition table
Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sde doesn't contain a valid partition table
Disk /dev/sdf: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Disk /dev/sdf doesn't contain a valid partition table
[root@rac1 ~]# fdisk /dev/sdb
> In each case, the sequence of answers is "n", "p", "1", "Return", "Return", "p" and "w".[root@rac1 ~]# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> Continue partitioning the other disks.
[root@rac1 ~]# fdisk /dev/sdc
[root@rac1 ~]# fdisk /dev/sdd[root@rac1 ~]# fdisk /dev/sdc
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@rac1 ~]# fdisk /dev/sde[root@rac1 ~]# fdisk /dev/sdd
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
[root@rac1 ~]# fdisk /dev/sdf[root@rac1 ~]# fdisk /dev/sde
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> Once all the disks are partitioned, the results can be seen by "fdisk -l".[root@rac1 ~]# fdisk /dev/sdf
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
Warning: invalid flag 0x0000 of partition table 4 will be corrected by w(rite)
Command (m for help): n
Command action
e extended
p primary partition (1-4)
p
Partition number (1-4): 1
First cylinder (1-652, default 1):
Using default value 1
Last cylinder or +size or +sizeM or +sizeK (1-652, default 652):
Using default value 652
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
Syncing disks.
> Download and install the following packages if you have not done so:[root@rac1 ~]# fdisk -l
Disk /dev/sda: 32.2 GB, 32212254720 bytes
255 heads, 63 sectors/track, 3916 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 3916 31350847+ 8e Linux LVM
Disk /dev/sdb: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdb1 1 652 5237158+ 83 Linux
Disk /dev/sdc: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdc1 1 652 5237158+ 83 Linux
Disk /dev/sdd: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdd1 1 652 5237158+ 83 Linux
Disk /dev/sde: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sde1 1 652 5237158+ 83 Linux
Disk /dev/sdf: 5368 MB, 5368709120 bytes
255 heads, 63 sectors/track, 652 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Device Boot Start End Blocks Id System
/dev/sdf1 1 652 5237158+ 83 Linux
oracleasm-support-2*
oracleasmlib-2*
oracleasm-2*
> Install the packages using the following command.
rpm -Uvh oracleasm*.rpm
> Configure ASMLib using the following command.
[root@rac1 ~]# oracleasm configure -i
> Load the kernel module using the following command.[root@rac1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []:
[root@rac1 ~]# oracleasm configure -i
Configuring the Oracle ASM library driver.
This will configure the on-boot properties of the Oracle ASM library
driver. The following questions will determine whether the driver is
loaded on boot and what permissions it will have. The current values
will be shown in brackets ('[]'). Hitting <ENTER> without typing an
answer will keep that current value. Ctrl-C will abort.
Default user to own the driver interface []: grid
Default group to own the driver interface []: asmadmin
Start Oracle ASM library driver on boot (y/n) [n]: y
Scan for Oracle ASM disks on boot (y/n) [y]: y
Writing Oracle ASM library driver configuration: done
[root@rac1 ~]# /usr/sbin/oracleasm init
> Mark the five shared disks as follows. Disks sdb1, sdc1 and sdd1 is for Diskgroup DATA and sde1 and sdf1 is for FLASH diskgroup.[root@rac1 ~]# /usr/sbin/oracleasm init
Creating /dev/oracleasm mount point: /dev/oracleasm
Loading module "oracleasm": oracleasm
Mounting ASMlib driver filesystem: /dev/oracleasm
> Refresh the ASM disk configuration.[root@rac1 ~]# /usr/sbin/oracleasm createdisk DATA01 /dev/sdb1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# /usr/sbin/oracleasm createdisk DATA02 /dev/sdc1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# /usr/sbin/oracleasm createdisk DATA03 /dev/sdd1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# /usr/sbin/oracleasm createdisk FRA01 /dev/sde1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# /usr/sbin/oracleasm createdisk FRA02 /dev/sdf1
Writing disk header: done
Instantiating disk: done
[root@rac1 ~]# /usr/sbin/oracleasm scandisks
> Check if disks are now visible to ASM using the "listdisks" command.[root@rac1 ~]# /usr/sbin/oracleasm scandisks
Reloading disk partitions: done
Cleaning any stale ASM disks...
Scanning system for ASM disks...
> The shared disks are now configured for the grid infrastructure.[root@rac1 ~]# /usr/sbin/oracleasm listdisks
DATA01
DATA02
DATA03
FRA01
FRA02
Clone the Virtual Machine
> We can clone the Virtual Machine manually and rename it to RAC2
> Shut down the RAC1 virtual machine using the following command.
[root@rac1 ~]# shutdown -h now
> Copy the RAC1 virtual machine using the following command. Or you can just copy and paste from your Windows 7 Host machine.
C:\Users\gellee>xcopy "E:\DATA\Gellee\Renz\VMWare\RAC1" "E:\DATA\Gellee\Renz\VMWare\RAC2" /c /e /f /y
> Edit the contents of the "E:\DATA\Gellee\Renz\VMWare\RAC2\RAC1.vmx" file, making the following change.C:\Users\gellee>xcopy "E:\DATA\Gellee\Renz\VMWare\RAC1" "E:\DATA\Gellee\Renz\VMW
are\RAC2" /c /e /f /y
Does E:\DATA\Gellee\Renz\VMWare\RAC2 specify a file name
or directory name on the target
(F = file, D = directory)? D
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1.nvram -> E:\DATA\Gellee\Renz\VMWare\RAC2\RA
C1.nvram
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1.vmdk -> E:\DATA\Gellee\Renz\VMWare\RAC2\RAC
1.vmdk
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1.vmsd -> E:\DATA\Gellee\Renz\VMWare\RAC2\RAC
1.vmsd
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1.vmx -> E:\DATA\Gellee\Renz\VMWare\RAC2\RAC1
.vmx
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1.vmxf -> E:\DATA\Gellee\Renz\VMWare\RAC2\RAC
1.vmxf
E:\DATA\Gellee\Renz\VMWare\RAC1\RAC1_2.vmdk -> E:\DATA\Gellee\Renz\VMWare\RAC2\R
AC1_2.vmdk
E:\DATA\Gellee\Renz\VMWare\RAC1\vmware-0.log -> E:\DATA\Gellee\Renz\VMWare\RAC2\
vmware-0.log
E:\DATA\Gellee\Renz\VMWare\RAC1\vmware.log -> E:\DATA\Gellee\Renz\VMWare\RAC2\vm
ware.log
8 File(s) copied
From displayName = "RAC1" to displayName = "RAC2"
> Ignore discrepancies with the file names in the "E:\DATA\Gellee\Renz\VMWare\RAC2" directory. This does not affect the action of the virtual machine.
> In the VMware Infrastructure Web Access Console, select the "Virtual Machine > Add Virtual Machine to Inventory" menu options and browse for the"E:\DATA\Gellee\Renz\VMWare\RAC2\RAC1.vmx" file. Once opened, the RAC2 virtual machine is visible on the console.
> Log in to the RAC2 virtual machine as the root user and start the "Network Configuration" tool (System > Administration > Network).
> Edit the "/home/grid/.bash_profile" file on the RAC2 node to correct the ORACLE_SID
> Reboot the RAC2 virtual machine. Start the RAC1 virtual machine. When both nodes have started, check they can both ping all the public and private IP addresses using the following commands.
ping -c 3 rac1
ping -c 3 rac1-priv
ping -c 3 rac2
ping -c 3 rac2-priv
> At this point the virtual IP addresses defined in the "/etc/hosts" file will not work, so don't bother testing them.[root@rac1 ~]# ping -c 3 rac1
PING rac1.localdomain (192.168.130.101) 56(84) bytes of data.
64 bytes from rac1.localdomain (192.168.130.101): icmp_seq=1 ttl=64 time=1.10 ms
64 bytes from rac1.localdomain (192.168.130.101): icmp_seq=2 ttl=64 time=0.127 ms
64 bytes from rac1.localdomain (192.168.130.101): icmp_seq=3 ttl=64 time=0.067 ms
--- rac1.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.067/0.433/1.106/0.476 ms
[root@rac1 ~]# ping -c 3 rac1-priv
PING rac1-priv.localdomain (192.168.131.111) 56(84) bytes of data.
64 bytes from rac1-priv.localdomain (192.168.131.111): icmp_seq=1 ttl=64 time=0.091 ms
64 bytes from rac1-priv.localdomain (192.168.131.111): icmp_seq=2 ttl=64 time=0.082 ms
64 bytes from rac1-priv.localdomain (192.168.131.111): icmp_seq=3 ttl=64 time=0.082 ms
--- rac1-priv.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 0.082/0.085/0.091/0.004 ms
[root@rac1 ~]# ping -c 3 rac2
PING rac2.localdomain (192.168.130.102) 56(84) bytes of data.
64 bytes from rac2.localdomain (192.168.130.102): icmp_seq=1 ttl=64 time=2.87 ms
64 bytes from rac2.localdomain (192.168.130.102): icmp_seq=2 ttl=64 time=0.369 ms
64 bytes from rac2.localdomain (192.168.130.102): icmp_seq=3 ttl=64 time=0.153 ms
--- rac2.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2003ms
rtt min/avg/max/mdev = 0.153/1.132/2.874/1.234 ms
[root@rac1 ~]# ping -c 3 rac2-priv
PING rac2-priv.localdomain (192.168.131.112) 56(84) bytes of data.
64 bytes from rac2-priv.localdomain (192.168.131.112): icmp_seq=1 ttl=64 time=1.50 ms
64 bytes from rac2-priv.localdomain (192.168.131.112): icmp_seq=2 ttl=64 time=0.343 ms
64 bytes from rac2-priv.localdomain (192.168.131.112): icmp_seq=3 ttl=64 time=0.789 ms
--- rac2-priv.localdomain ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 0.343/0.878/1.502/0.477 ms
> It's a good idea to take a snapshot of the virtual machines, so you can repeat the following stages if you run into any problems. To do this, shutdown both virtual machines and issue the following commands.
[root@rac1 ~]# shutdown -h now
[root@rac2 ~]# shutdown -h now
Configure DNS on RAC1 for the SCAN used with Oracle RAC Database 11g Release 2
> This step is not necessary if you have an actual DNS Server. But here since it's a test environment, we will create our own DNS server.
> Install DNS server (BIND) packages. Mount your OS Media Installer.
[root@rac1 ~]# cd /media/Enterprise\ Linux\ dvd\ 20090127/Server/
[root@rac1 Server]# rpm -ivh bind-9*
[root@rac1 Server]# rpm -ivh bind-libs*
[root@rac1 Server]# rpm -ivh system-config-bind-4*
> Run GUI DNS editor to load default configuration files for you to amend. Then close the editor.
[root@rac1 Server]# system-config-bind
> Next we must edit the contents of the "/etc/named.conf" file.
[root@rac1 Server]# vi /etc/named.conf
options {
directory "/var/named";
dump-file "/var/named/data/cache_dump.db";
statistics-file "/var/named/data/named_stats.txt";
/*
* If there is a firewall between you and nameservers you want
* to talk to, you might need to uncomment the query-source
* directive below. Previous versions of BIND always asked
* questions using port 53, but BIND 8.1 uses an unprivileged
* port by default.
*/
// query-source address * port 53;
// Set IP address correctly.
listen-on port 53 { 127.0.0.1; 192.168.130.101; };
};
// The "file" parameter specifies the name of the file in the "/var/named/" directory that contains the configuration for this zone.
zone "localdomain." IN {
type master;
file "localdomain.zone";
allow-update { none; };
};
zone "130.168.192.in-addr.arpa." IN {
type master;
file "130.168.192.in-addr.arpa";
allow-update { none; };
};
zone "." in {
type hint;
file "/dev/null";
};
> [root@rac1 named]# vi /var/named/localdomain.zone
> Create /var/named/130.168.192.in-addr.arpa for the public network reverse lookups.$TTL 86400
@ IN SOA localhost root.localdomain. (
44 ; serial (d. adams)
3H ; refresh
15M ; retry
1W ; expiry
1D ) ; minimum
IN NS localhost
localhost IN A 127.0.0.1
rac1 IN A 192.168.130.101
rac2 IN A 192.168.130.102
rac1-vip IN A 192.168.130.121
rac2-vip IN A 192.168.130.122
rac-scan IN A 192.168.130.201
rac-scan IN A 192.168.130.202
rac-scan IN A 192.168.130.203
[root@rac1 named]# vi /var/named/130.168.192.in-addr.arpa
> Start the DNS Service$ORIGIN 130.168.192.in-addr.arpa.
$TTL 1H
@ IN SOA rac1.localdomain. root.rac1.localdomain. ( 2
3H
1H
1W
1H )
130.168.192.in-addr.arpa. IN NS rac1.localdomain.
101 IN PTR rac1.localdomain.
102 IN PTR rac2.localdomain.
121 IN PTR rac1-vip.localdomain.
122 IN PTR rac2-vip.localdomain.
201 IN PTR rac-scan.localdomain.
202 IN PTR rac-scan.localdomain.
203 IN PTR rac-scan.localdomain.
[root@rac1 named]# service named start
> Issue the following command to make sure the "named" service starts automatically after reboots.
[root@rac1 named]# chkconfig named on
> The DNS server is now running, but each server must be told to use it. This is done by editing the "/etc/resolv.conf" file on each server, including the RAC nodes. Make sure the file contains the following entries, where the IP address and domain match those of your DNS server and the domain you have configured.
> On RAC1,
[root@rac1 named]# vi /etc/resolv.conf
search localdomain
nameserver 192.168.130.101
> On RAC2,
[root@rac2 ]# vi /etc/resolv.conf
search localdomain
nameserver 192.168.130.101
> Test it. You should now be able to test the forward and reverse lookups using the "nslookup" command. The output below shows the forward and reverse lookups of the SCAN address.
[root@rac1 ~]# nslookup rac-scan.localdomain
Server: 192.168.130.101
Address: 192.168.130.101#53
Name: rac-scan.localdomain
Address: 192.168.130.203
Name: rac-scan.localdomain
Address: 192.168.130.201
Name: rac-scan.localdomain
Address: 192.168.130.202
[root@rac1 ~]# nslookup 192.168.130.201
Server: 192.168.130.101
Address: 192.168.130.101#53
201.130.168.192.in-addr.arpa name = rac-scan.localdomain.
[root@rac1 ~]# nslookup 192.168.130.202
Server: 192.168.130.101
Address: 192.168.130.101#53
202.130.168.192.in-addr.arpa name = rac-scan.localdomain.
Install the Grid Infrastructure[root@rac1 ~]# nslookup 192.168.130.203
Server: 192.168.130.101
Address: 192.168.130.101#53
203.130.168.192.in-addr.arpa name = rac-scan.localdomain.
> Start the RAC1 and RAC2 virtual machines.
> Login to RAC1 as the grid user, upload the Grid Infrastructure Installer and unzip.
[grid@rac1 ~]$ cd ~/staging/grid/
[grid@rac1 grid]$ ./runInstaller
> Skip software updates.
[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
[root@rac1 ~]# /u01/app/11.2.0/grid/root.sh[root@rac1 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
> On RAC2,[root@rac1 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
root wallet
root wallet cert
root cert export
peer wallet
profile reader wallet
pa wallet
peer wallet keys
pa wallet keys
peer cert request
pa cert request
peer cert
pa cert
peer root cert TP
profile reader root cert TP
pa root cert TP
peer pa cert TP
pa peer cert TP
profile reader pa cert TP
profile reader peer cert TP
peer user cert
pa user cert
Adding Clusterware entries to inittab
CRS-2672: Attempting to start 'ora.mdnsd' on 'rac1'
CRS-2676: Start of 'ora.mdnsd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'rac1'
CRS-2676: Start of 'ora.gpnpd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'rac1'
CRS-2672: Attempting to start 'ora.gipcd' on 'rac1'
CRS-2676: Start of 'ora.cssdmonitor' on 'rac1' succeeded
CRS-2676: Start of 'ora.gipcd' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'rac1'
CRS-2672: Attempting to start 'ora.diskmon' on 'rac1'
CRS-2676: Start of 'ora.diskmon' on 'rac1' succeeded
CRS-2676: Start of 'ora.cssd' on 'rac1' succeeded
ASM created and started successfully.
Disk Group DATA created successfully.
clscfg: -install mode specified
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
CRS-4256: Updating the profile
Successful addition of voting disk 418fae60abd74fecbf15c709fa614ab0.
Successfully replaced voting disk group with +DATA.
CRS-4256: Updating the profile
CRS-4266: Voting file(s) successfully replaced
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 418fae60abd74fecbf15c709fa614ab0 (/dev/oracleasm/disks/DATA01) [DATA]
Located 1 voting disk(s).
CRS-2672: Attempting to start 'ora.asm' on 'rac1'
CRS-2676: Start of 'ora.asm' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.DATA.dg' on 'rac1'
CRS-2676: Start of 'ora.DATA.dg' on 'rac1' succeeded
CRS-2672: Attempting to start 'ora.registry.acfs' on 'rac1'
CRS-2676: Start of 'ora.registry.acfs' on 'rac1' succeeded
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh[root@rac2 ~]# /u01/app/oraInventory/orainstRoot.sh
Changing permissions of /u01/app/oraInventory.
Adding read,write permissions for group.
Removing read,write,execute permissions for world.
Changing groupname of /u01/app/oraInventory to oinstall.
The execution of the script is complete.
> Once the scripts have completed, return to the "Execute Configuration Scripts" screen on RAC1 and click the "OK" button.[root@rac2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= grid
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node rac1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Configure Oracle Grid Infrastructure for a Cluster ... succeeded
> Wait for the configuration assistants to complete.
> Create the other ASM Disk Group (FLASH) for Flash Recovery Area.
[grid@rac1 ~]# asmca
> Click on "Create"
[root@rac1 ~]# shutdown -h now
[root@rac2 ~]# shutdown -h now
Install the Database
> Start the RAC1 and RAC2 virtual machines
> Upload and unzip the Database installer
> Login to RAC1 as the oracle user and start the Oracle installer.
[oracle@rac1 ~]$ cd ~/staging/database/
[oracle@rac1 database]$ ./runInstaller
> Uncheck the security updates checkbox and click the "Next" button.
> Click the "Close" button to exit the installer.
Check the Status of the RAC
There are several ways to check the status of the RAC. The srvctl utility shows the current configuration and status of the RAC database.
[oracle@rac1 ~]$ srvctl config database -d RAC
[oracle@rac1 ~]$ srvctl status database -d RAC[oracle@rac1 ~]$ srvctl config database -d RAC
Database unique name: RAC
Database name: RAC
Oracle home: /u01/app/oracle/product/11.2.0/db_1
Oracle user: oracle
Spfile: +DATA/RAC/spfileRAC.ora
Domain: localdomain
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: RAC
Database instances: RAC1,RAC2
Disk Groups: DATA
Mount point paths:
Services:
Type: RAC
Database is administrator managed
[oracle@rac1 ~]$
> The V$ACTIVE_INSTANCES view can also display the current status of the instances.[oracle@rac1 ~]$ srvctl status database -d RAC
Instance RAC1 is running on node rac1
Instance RAC2 is running on node rac2
[oracle@rac1 ~]$ sqlplus '/as sysdba'
[oracle@rac1 ~]$ sqlplus '/as sysdba'
SQL*Plus: Release 11.2.0.3.0 Production on Mon Oct 15 15:11:03 2012
Copyright (c) 1982, 2011, Oracle. All rights reserved.
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Data Mining and Real Application Testing options
SQL> select inst_name from v$active_instances;
INST_NAME
--------------------------------------------------------------------------------
rac1.localdomain:RAC1
rac2.localdomain:RAC2
