RMAN backups are slow with TSM

[jimb]

Hi,
I encountered a problem using RMAN with TSM to backup a database. It seems that the RMAN channel doesn't use the entire bandwidth to the TSM (or whatever) and is idle most of the time. This makes the database backup really slow.
As a comparison, we backup also filesystem and SQL Server, they both use very few channels and each channel backup about X5 more data per second than Oracle. For example, SQL Server uses 5 channels to the TSM, and we get a total of around 100MB/s. When we configure Oracle to backup using 5 channels we get about 20MB/s and we see that the SQL Server is writing to the TSM all the time while Oracle is idle a large portion of the time.

Any ideas? Known configuration issues?
I hope I'm clear about the problem.

Thanks,
Liron

[jimb]

Hi Liron;

I talked with our storage team about your issue. They mention below:

1. Both sql and oracle are in same network?
2. It could be difference compare backup period between oracle and sql, that's why please compare backup period between 2 oracle db
3. Drivers are up to date?
4. Please see:

Poor RMAN restore performance from TSM backup [ID 1130593.1]

Regard
Helios

[jimb]

Hi Helios,
Thanks for you quick response. I'll further explain the issue according to your questions.

1. All SQL Server and filesystem backups are fast, all Oracle (10.2.0.4-11.2.0.3) are slow (same/different networks)
2. We did several tests, all tests took place when nothing else was writing to the TSM
3. I think the drivers are up to date, I'll check
4. I saw this note already, I couldn't make it work faster. Maybe I did it wrong...
5. Actually, I saw that happening at several sites (different companies), and still I can't explain/solve it.

Do you work with TSM?

[jimb]

Hi;

Yes we are working wiht tsm but storage&backup team makes all adjustment. Here is some other questions

1. We asume your oracle db are working on different server, so question is do you take file system backup? Its also slow like your oracle backup?
2. Your system CPU has enough source?

They suggest also below:
http://www-01.ibm.com/support/docview.w ... wg21142185
http://publib.boulder.ibm.com/tividd/td ... 101-01.htm

Regard
Helios

[jimb]

Hi

kindly inform large_pool_size in you database. is this configured properly.

Thanks
Krishna

[jimb]

Hi

kindly inform large_pool_size in you database. is this configured properly.

Thanks
Krishna

How does the sizing of the large pool impact rman performance?

[jimb]

Liron,

Just for fun...

ATM is supposed to give money, ACE is supposed to give answer... I wonder/fear; what will happen if ATM demand the money vs ACE ask the question ...

Only Bank/Oracle can give the Answer.

Regards
Girish Sharma

[jimb]

Hi,
Did you test an OS backup of a datafile (or a file of similar size on same storage) with TSM from the same database server and check the speed?
Rgds,
Tycho

[jimb]

Hi,
Thanks for everybody for your responses, I'll try to sum the answers here:
1. Yes, we backup the filesystem, and everything works fine. The problem is only with RMAN backups
2. Resources are not an issue, it happens on strong and weak servers (and several O/S as well)
3. Database versions are not the issue, it happens on 10.2 as well as 11.2

My feeling is that this has something to do with the integration, either oracle setting or TSM setting, but I can't be sure until I solve this.
For those of you who backup with TSM, did you notice that filesystem / SQL Servers are being backed up faster? Did you check the Oracle wait events and TSM status while you run the backup?

Thanks

BTW Girish, great comment, I wish I had all the answers...

[jimb]

Hi Liron,
Try to find the highest wait event for you rman sessions and search on MOS for a direction for a solution.

Code: Select all

set line 150
set pages 100
break on sid
column e.event format a20

Code: Select all

select s.SID, e.EVENT, TOTAL_WAITS , TIME_WAITED, AVERAGE_WAIT, MAX_WAIT
from v$session_event e, v$session s
where e.sid = s.sid
and s.program like 'rman%'
order by 1
/

Regards,
Tycho