Why and How to install RAC DB 19c on Solaris 11.4 ?

 Why ?

RAC DB is the most powerful RDBMS DB for high performance and high availability. It is an expected task for any Oracle DBA to do RAC installation and here I am sharing my quick reference while do this task along with some details related to disks and OS configuration to achieve this.

How ?

 

Below are steps that I follow to do this installation where I have the below:

Download GI+DB binary for Solaris Sparc64

Download the latest GI patch that will include both GI and DB and make sure you have opatch latest one as well :

1- OS pre configuration:

## check NTP

svcs -xv ntp

svcs /network/ntp:default

svcs -l /network/ntp:default

ntpq -p

https://docs.oracle.com/cd/E26502_01/html/E28996/time-20.html

 

#TO setup the slew

svccfg -s svc:/network/ntp:default setprop config/slew_always = true

svcadm refresh svc:/network/ntp:default

#To check it :

svcprop -p config/slew_always svc:/network/ntp:default

svccfg -s svc:/network/ntp:default listprop | /bin/grep slew

## check DNS

nslookup

## install packages

root@node1:~# pkg list entire

NAME (PUBLISHER)                                  VERSION                    IFO

entire                                            11.4-11.4.52.0.1.132.2     i--

root@node1:~# pkg install oracle-database-preinstall-19c

2- Users/group setup:

## create user/groups

  groupadd -g 54321 oinstall

  groupadd -g 54329 asmadmin

  groupadd -g 54327 asmdba

  groupadd -g 54328 asmoper

  groupadd -g 54322 dba

  groupadd -g 54323 oper

  groupadd -g 54324 backupdba

  groupadd -g 54325 dgdba

  groupadd -g 54326 kmdba

  groupadd -g 54330 racdba

 

useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,racdba -d "/export/home/oracle" -m oracle

useradd -u 54331 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,asmadmin,asmoper -d "/export/home/grid" -m grid

3-User project in OS:

# to check current configured project

cat /etc/project

 

## using project to create users:

projadd -p 100 -G dba -c "Oracle Project" -K "project.max-shm-memory=(privileged,10G,deny)" group.dba

projmod -sK "project.max-sem-ids=(privileged,100,deny)" group.dba

 projmod -sK "project.max-shm-ids=(privileged,100,deny)" group.dba

  projmod -sK "project.max-sem-nsems=(privileged,256,deny)" group.dba

projmod -sK"process.max-stack-size=(basic,10485760,deny)" group.dba

 

useradd -u 54321 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,racdba -d "/export/home/oracle" -m -K project=group.dba oracle

useradd -u 54331 -g oinstall -G dba,asmdba,backupdba,dgdba,kmdba,racdba,asmadmin,asmoper -d "/export/home/grid" -m -K project=group.dba grid

 

## to create a project and then add the user use the below :

 

projadd -G dba -c "Oracle dba Group for CRS and RDBMS" group.dba

projmod -sK "project.max-shm-memory=(priv,32G,deny)" group.dba

projmod -sK "project.max-sem-ids=(priv,256,deny)" group.dba

projmod -sK "process.max-sem-nsems=(priv,256,deny)" group.dba

projmod -sK "process.max-file-descriptor=(priv,65536,deny)" group.dba

projmod -sK "process.max-file-descriptor=(basic,65536,deny)" group.dba

projmod -sK "process.max-stack-size=(basic,10485760,deny)" group.dba

usermod -K project=group.dba grid

usermod -K project=group.dba oracle

4- Create directories:

## create directories

mkdir -p /u01/app/19.0.0/grid

mkdir -p /u01/app/grid

chown -R grid:oinstall /u01

chmod -R 775 /u01/

 

5- Create directories:

## create directories

mkdir -p /u01/app/19.0.0/grid

mkdir -p /u01/app/grid

chown -R grid:oinstall /u01

chmod -R 775 /u01/

 

6- prepare ASM disks:

## Some references for the disk setup :

 

AFD Storage Disks Not Appearing Under "CREATE ASM DISK GROUP" In 19c Grid Installation (Doc ID 2878933.1)

Configuring Disk Devices for Oracle ASM on Oracle Solaris

https://support.oracle.com/epmos/faces/CommunityDisplay?resultUrl=https%3A%2F%2Fcommunity.oracle.com%2Fmosc%2Fdiscussion%2Fcomment%2F14658232&_afrLoop=72302341112795&resultTitle=Re%3A+ASMFD+%3A+before+installing+GRID+use+of+asmfd_label+seems+to+do+nothing.+AFD_STATE+showing+nothing.&commId=&displayIndex=1&_afrWindowMode=0&_adf.ctrl-state=wiy4fphbq_634#Comment_14658232

 

You have to get the list of shared disk from OS Admin include the cylinder to be used, below I will show samples only :

 

you may use command “format” to see all disks details, below is a sample of initiate and label the disks.

 

After unzip the GI home by following the below link, start the labeling.

https://docs.oracle.com/en/database/oracle/oracle-database/19/cwsol/installing-oracle-standalone-cluster.html#GUID-7BAE8D62-4D26-424C-BBB4-B993F248C163

 

chown grid:asmadmin /dev/rdsk/c0t60000970000297801140533030303344d0s0

chown -h grid:asmadmin /dev/rdsk/c0t60000970000297801140533030303344d0s0

chmod g+w /devices/scsi_vhci/disk@g60000970000297801140533030303344:a,raw

ls -ltrh /dev/rdsk/c0t60000970000297801140533030303344d0s0

ls -ltrh /devices/scsi_vhci/disk@g60000970000297801140533030303344:a,raw

 

DATA Disk Group

As root user :

export ORACLE_HOME=/u01/app/19.0.0/grid

export ORACLE_BASE=/u01/app/grid

cd /u01/app/19.0.0/grid/bin

 

./asmcmd afd_label MGMT1 /dev/rdsk/c0t60000970000297801140533030303344d0s0 –init

./asmcmd afd_lslbl /dev/rdsk/c0t60000970000297801140533030303344d0s0

 

https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=368998006909019&parent=EXTERNAL_SEARCH&sourceId=PROBLEM&id=2355039.1&_afrWindowMode=0&_adf.ctrl-state=168weusfr1_4

 

 

In case you do a deinstall, below will help you to wipe and clean up the disks :

 

dd if=/dev/zero of=/dev/rdsk/c0t60000970000297801140533030303344d0s0 bs=1024 count=100

./asmcmd afd_unlabel /dev/rdsk/c0t60000970000297801140533030303344d0s0 –init

 

 

7- start GI instllation:

## To avoid any issue and bugs ,I do prefer to patch the GI during the installation with the latest RU .

## to avoid GI installation issue , chwon to grid after setup the disks using root ,some logs will be owned by root and this will affect the GI installation:

 chown -R grid:oinstall /u01/app/19.0.0/grid/log

 

You may run cluvy as well prior to installation or you may depned on GUI cluvy during the installation.

## Cluvy command , you have to install clufy separately

cluvfy stage -pre crsinst -n node1,node2 -fixup -verbose

 

 

## patch GRID_HOME with latest page during the installation ** make sure to update OPatch version

 

## run installer from Grid user home

/u01/app/19.0.0/grid/gridSetup.sh -applyRU /u01/files/34762026

 

Follow the wizard till the scripts step by step.

 

8- DB instllation:

mkdir -p /u02/app/oracle

mkdir -p /u02/app/oracle/product/19.0.0/dbhome_1

cd /u02/app/oracle/product/19.0.0/dbhome_1

unzip -q /export/home/sal3570/oracle_files/DB_19c_instllation.zip

#update opatch and prepare DB, OCW patches to be applied during installation

./runInstaller -applyRU /u02/patches/34762026/34765931 -applyOneOffs /u02/patches/34762026/34768559

 

Follow the wizard till the scripts step by step.

 

9- GI wizard sample :

 

10- DB wizard sample :

## Then DBCA 

## this should be confirmed with the Application requirement 

Why and How to use sqlcmd in bash script ?

 Why ?

SQLCMD is a command line utility to be used to run T-SQL for MS SQL server, below I will show how to use it. I have create this to redirect and append the output to the file by date at every execute.

How ?

1- create sql file( myscript.sql) contain whatever script you need to execute.

Use master

Go

Select @@version

go

2- Create bat file contains the script to execute the above sql query:

@echo off

:: to get the date

for /f "delims=" %%a in ('powershell get-date((get-date^).addDays(-1^)^) -uformat "%%Y%%m%%d"') do set date=%%a

:: to run the sql file

sqlcmd -S <server/instane> -U <sqllogin> -P "password" -i C:\path_to_file\myscript.sql >> C:\path_to_file\output%date%.txt

 

:: to print the output file on screen

type C:\path_to_file\output%date%.txt

 

:: pause the screen

set /p delExit=Press the ENTER key to exit...:

 

note : you can't use os authentication for remote sqlcmd in -U , -U used for sql login user only.

Why and How to create a SAN SSL certificate request?

Why ?

The purpose of SAN is to have multiple CN and this will be used instead of wild card (*.domain.com) CN if same certificate will be used for multiple subdomain.

How ?

1- create a configuration file as below.

vi req.conf

 

[req]

distinguished_name = req_distinguished_name

req_extensions = v3_req

prompt = no

[req_distinguished_name]

C = BH

ST = MANAMA

L = MANAMA

O = <companyName>

OU = IT

CN = CN.Domain

[v3_req]

keyUsage = keyEncipherment, dataEncipherment

extendedKeyUsage = serverAuth

subjectAltName = @alt_names

[alt_names]

DNS.1 = CN1.Domain

DNS.2 = CN2.Domain

2- Create CSR file along with the key and keep a copy of this private key that you may use it if needed:

openssl req -new -out certificate.csr -newkey rsa:2048 -nodes -sha256 -keyout certificate.key -config req.conf

Submit the certificate to CA to be singed , you may want to verify the CSR file information, use the below :

openssl req -text -noout -verify -in certificate.csr

Below are pem file certificate sequence for a reference in case needed:

-----BEGIN CERTIFICATE-----

(Your Primary SSL certificate: your_domain_name.crt)

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

(Your Intermediate certificate: DigiCertCA.crt)

-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----

(Your Root certificate: TrustedRoot.crt)

-----END CERTIFICATE-----

Why and How to generate self-singed or public singed certificate ?

Why ?

Certificate can be self-singed or publicly singed , self-singed is singed by your server only and no public trusted authority know this certificate, it is good to be used for testing only, production servers/website services that may need a certificate , you have to generate a request file to be signed by public known authority and load the singed in your public website or service that may need this certificate.

How ?

1- generate a private key that will be used to generate a certificate request, this key is important to be kept with you for future needed in case .

 ## with password:

openssl genrsa -des3 -out private.key 2048

## without password:

openssl genrsa -out VISA_ACS1_PROD_Signing.key 2048

2- Create the Certificate Singed Request:

openssl req -new -key privkey.key -sha256  -out server.csr

If this certificate will be singed by public authority , you have to share “server.csr” and they will provide you a singed certiticate (mainly .csr or .crt ) file

3- For Selef-Singed Certificate do the below, First generate a server certificate that will act like private authority to singe the csr:

openssl x509 -req -days 1825 -in server.csr -signkey server.key -out server.crt

4- In many cases you may need to convert crt to PEM format, use the below:

openssl x509 -in server.crt -out server.pem -outform PEM

Why and How to setup the xdisplay parameter after switch user in Linux ?

Why ?

Display on ssh session that is support X11-forwarding  (remote display forwarded through SSH) will appeared for the first login user only , if -X -Y used the display will be transferred as well however if just switch user only using ‘su’ this will lead to lose the display from server to client , use the below steps to move the display from first to the second switched user

How ?

1- After login with first user before switching take the details of display .

$ xuath list $DISPLAY

<output1>

$ echo $DISPLAY

<output2>

2- Switch to the user:

$ xauth add <output1>

$ export DISPLAY=<output2>

Why and How to implement TDE in 11g and 19c ?

Why ?

As clarified by Oracle ,“Transparent Data Encryption (TDE) enables you to encrypt sensitive data that you store in tables and tablespaces” . Below are a quick note using oracle document for both versions 19c or lower versions (11g or 12c).

How ?

1- Set the Software Keystore Location in the sqlnet.ora File . For 11g or 12.1 or 12.2c .

ENCRYPTION_WALLET_LOCATION=

  (SOURCE=

   (METHOD=FILE)

    (METHOD_DATA=

     (DIRECTORY=/etc/ORACLE/WALLETS/orcl)))

For  19c, it is recommended to use spfile parameters as below :

ALTER SYSTEM SET WALLET_ROOT = '$ORACLE_BASE/ADMIN/ORCL/WALLET' SCOPE = SPFILE SID = '*';

ALTER SYSTEM SET TDE_CONFIGURATION="KEYSTORE_CONFIGURATION=FILE" SCOPE = BOTH SID = '*';

2- Create the Software Keystore:

ADMINISTER KEY MANAGEMENT CREATE KEYSTORE '/etc/ORACLE/WALLETS/orcl' IDENTIFIED BY password;

You may create an auto login to avoid re-entering the wallet password every time:

ADMINISTER KEY MANAGEMENT CREATE AUTO_LOGIN KEYSTORE FROM KEYSTORE '/etc/ORACLE/WALLETS/orcl' IDENTIFIED BY password;

3- Open the Software Keystore ## In case autologin not configured:

ADMINISTER KEY MANAGEMENT SET KEYSTORE OPEN IDENTIFIED BY password;

4- Set the Software TDE Master Encryption Key:

ADMINISTER KEY MANAGEMENT SET KEY IDENTIFIED BY keystore_password WITH BACKUP USING 'key_backup';

5- Encrypt Your Data:

CREATE TABLESPACE encrypt_ts

  DATAFILE '$ORACLE_HOME/dbs/encrypt_df.dbf' SIZE 1M

  ENCRYPTION USING 'AES256'

  DEFAULT STORAGE (ENCRYPT);

6- Convert exist Tablespace, for prior 12.2c it is offline only , for 12.2 and later version , both offline and online can be used:

## offline

ALTER TABLESPACE users OFFLINE NORMAL;

ALTER TABLESPACE users ENCRYPTION OFFLINE USING 'AES256' ENCRYPT;

ALTER TABLESPACE users ONLINE;

## online

ALTER TABLESPACE users ENCRYPTION ONLINE USING 'AES256' ENCRYPT FILE_NAME_CONVERT = ('users.dbf', 'users_enc.dbf');

 

Why and How to fix Red hat Linux library issue?

Why ?

For any reason you may discover an issue with library like the below issue , where you will not be able even to access the server with emergency mode/target , use the below to fix it, for example , I got the below error.

/usr/sbin/sulogin: error while loading shared libraries: libcrypt.so.1: cannot open shared object file: permission denied

 

How ?

1- First it needs to boot the system into the rescue mode using DVD/CD ISO on the installation media:

How to boot Red Hat Enterprise Linux to Rescue Mode for Data Collection (sosreport, vmcore, etc.)

https://access.redhat.com/articles/3405661

2- Check the rpm package that is behind this library as below:

# rpm -qf /usr/lib64/libcrypt.so.1

libxcrypt-4.1.1-4.el8.x86_64

3- re-install the package using the below and reboot the system to disk boot again:

yum reinstall libxcrypt