接到告警，发现extract进程ABENDED

查看进程日志如下：

***********************************************************************

**                     Run Time Messages                             **

***********************************************************************

2024-08-25 13:22:11  INFO    OGG-01517  Position of first record processed Log Number: 79079

Record Offset: 67072343, Aug 24, 2024 12:32:26 AM.

2024-08-25 13:22:11  INFO    OGG-06507  MAP (TABLE) resolved (entry pub_credit.ent_core_data_count): TABLE “pub_credit”.”ent_core_data_

count”.

2024-08-25 13:22:11  INFO    OGG-06509  Using the following key columns for source table pub_credit.ent_core_data_count: ogg_key_id.

2024-08-25 13:23:25  WARNING OGG-01266  cm_mf_write_lower: write: to_write: 98304  errno: 28 (No space left on device)  co_uid: 1  /ogg

/dirtmp/e_ph02_p48330_extr_0x7f968d981020_00016.cm.

2024-08-25 13:23:25  WARNING OGG-01857  cachemgr: filecaching: cm_mf_write_upper: /ogg/dirtmp.

2024-08-25 13:23:25  WARNING OGG-01262  The call to the cm_mf_write_lower() function from line 3164 in cm_mf_get() failed with reason ‘

no space on directories: error: 108: co: 00007F968D981020 obj_id: <30><30><30><30><30><30><30><30><30><30><30><30><30><30><30><30><37><

39><30><37><39><3A><30><30><30><30><30><30><30><36><37><30><37><32><33><34><33>’.

Source Context :

SourceModule            : [ggapp.cmgr_filecaching]

SourceID                : [/scratch/aime/adestore/views/aime_adc4150378/oggcore/OpenSys/src/gglib/ggapp/cachemgr/cmgr_filecache.c]

SourceFunction          : [cm_mf_write_upper]

SourceLine              : [854]

ThreadBacktrace         : [6] elements

: [/ogg/libgglog.so(CMessageContext::AddThreadContext()+0x26) [0x7f969a2d6336]]

: [/ogg/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, …)+0x6a8) [0x7f969a2c9358

]]

: [/ogg/libgglog.so(_MSG_ERR_CACHEMGR_FILECACHING_FILE(CSourceContext*, char const*, CMessageFactory::Message

Disposition)+0x41) [0x7f969a270a41]]

: [/ogg/extract(cm_filecaching_thread(void*)+0x2b49) [0x6ed689]]

: [/lib64/libpthread.so.0(+0x7dd5) [0x7f96971b2dd5]]

: [/lib64/libc.so.6(clone+0x6d) [0x7f969579fead]]

2024-08-25 13:23:25  ERROR   OGG-01853  cachemgr: filecaching: cm_mf_write_upper: /ogg/dirtmpcm_mf_get: no space on directories: error:

108: co: 00007F968D981020 obj_id: <30><30><30><30><30><30><30><30><30><30><30><30><30><30><30><30><37><39><30><37><39><3A><30><30><30>

<30><30><30><30><36><37><30><37><32><33><34><33>.

***********************************************************************

*                   ** Run Time Statistics **                         *

***********************************************************************

【分析】

从日志看到报错，是因为空间满了：

ERROR   OGG-01853  cachemgr: filecaching: cm_mf_write_upper: /ogg/dirtmpcm_mf_get: no space on directories: error:

发现dirtmp目录占了32G，这是罪魁祸首。

参考：

https://docs.oracle.com/en/middleware/goldengate/core/19.1/installing/temporary-disk-requirements.html

当总缓存事务数据超过 CACHEMGR CACHESIZE 参数时，Extract 将开始将缓存数据写入临时文件。

默认会使用整改目录的大小，建议设置一个单独专用的目录，并通过 CACHEMGR CACHEDIRECTORY 参数指定。

注意：

CACHEMGR 是一个内部自动配置和自动调节的参数，Oracle 建议不要更改 CACHESIZE，因为根据环境的不同，可能会对性能产生不利影响。

通常情况下，操作系统写 swap 的效率高于写临时文件的效率。默认的 CACHESIZE 参数就是这样设定的。

当事务提交或回滚之后，对应的临时文件也会被删除。所以这里dirtmp目录占用了32G导入整个/ogg目录满了，是因为这是一个大事务。

了解发现由于上游系统有个表做了历史数据清理，导致OGG同步这个delete语句的事务很大。

【解决】

临时扩容/ogg 目录，等这个大事务执行完成之后，目录空间就自动释放了。

OGG同步恢复正常

OGG报错dirtmp目录占满最先出现在dbAnt。

今天有同事反馈业务系统有个简单的查询SQL跑的很慢，要3秒左右，我的第一直觉就是索引问题。

下面我们就用这个例子，来看看怎么用 SQL Tuning Advisor 做SQL优化。

【分析】

使用SQL Tuning Advisor 来代替人工分析，包括：

• 为统计信息丢失或失效的对象收集统计信息
• 考虑优化器的任何数据偏差、复杂谓词或失效的统计信息
• 重新构建 SQL 以优化性能
• 提出新索引建议

原始SQL如下：

select decode(remamt1, '', decode(remamt2, '', t3.remamt, remamt2), remamt1) remamta,
 t3.remamt remamt3,
 tt.*
 from (select t1.remamt remamt1,
 t2.remamt as remamt2,
 t1.bizid,
 t1.serseqno
 from b_wf_voumng t1
 left join bedc_detail t2
 on t1.bizid = t2.bizid) tt
 left join bedc_hisdetail t3
 on tt.bizid = t3.bizid
 where serseqno = #{serseqno}

查找SQL ID

col SQL_FULLTEXT for a100
select SQL_ID, PLAN_HASH_VALUE, SQL_FULLTEXT from v$sql where SQL_TEXT like 'select decode%bedc_detail%';

查看SQL执行计划

select * from table(dbms_xplan.display_cursor('aqfdvtkcrptbx'));

这里可以看到 B_WF_VOUMNG 表确实是走了全表扫描，只要加上对应的索引就好了。

下面我们看看 SQL Tuning Advisor 的分析建议

根据SQL_ID创建优化分析任务，如果有多个执行计划，也可以指定执行计划的hash值

DECLARE
 MY_TASK_NAME VARCHAR2(1024);
BEGIN
 MY_TASK_NAME := DBMS_SQLTUNE.CREATE_TUNING_TASK(
 sql_id =>; 'aqfdvtkcrptbx',
 --plan_hash_value => '2744121178',
 scope => 'COMPREHENSIVE',
 time_limit => 60,
 task_name => 'my_tuning_task',
 description => 'TUNING TASK');
END;
/

查看创建的优化任务

SELECT task_name, status FROM dba_ADVISOR_TASKS WHERE task_name ='my_tuning_task';

执行优化任务，进行分析

EXEC DBMS_SQLTUNE.EXECUTE_TUNING_TASK('my_tuning_task');

查看分析结果

set LONG 999999
set SERVEROUTPUT ON SIZE 999999
set LINESIZE 200
set PAGES 9999
col DBMS_SQLTUNE.REPORT_TUNING_TASK('my_tuning_task') for a200

SELECT DBMS_SQLTUNE.REPORT_TUNING_TASK('my_tuning_task') FROM DUAL;

【分析结果如下】
分析结果比较长，分成3段看一下

第一段如下：

DBMS_SQLTUNE.REPORT_TUNING_TASK('MY_TUNING_TASK')
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
GENERAL INFORMATION SECTION
-------------------------------------------------------------------------------
Tuning Task Name : my_tuning_task
Tuning Task Owner : SYS
Workload Type : Single SQL Statement
Scope : COMPREHENSIVE
Time Limit(seconds): 60
Completion Status : COMPLETED
Started at : 07/31/2024 14:10:31
Completed at : 07/31/2024 14:10:41

-------------------------------------------------------------------------------
Schema Name: CMS
SQL ID : aqfdvtkcrptbx
SQL Text : select decode(remamt1, '', decode(remamt2, '', t3.remamt,
 remamt2), remamt1) remamta,
 t3.remamt remamt3,
 tt.*
 from (select t1.remamt remamt1,
 t2.remamt as remamt2,
 t1.bizid,
 t1.serseqno
 from b_wf_voumng t1
 left join bedc_detail t2
 on t1.bizid = t2.bizid) tt
 left join bedc_hisdetail t3
 on tt.bizid = t3.bizid
 where serseqno = :1
Bind Variables :
 1 - (VARCHAR2(128)):SKSY24072400000833

-------------------------------------------------------------------------------
FINDINGS SECTION (3 findings)
-------------------------------------------------------------------------------

1- SQL Profile Finding (see explain plans section below)
--------------------------------------------------------
 A potentially better execution plan was found for this statement.

 Recommendation (estimated benefit: 98.26%)
 ------------------------------------------
 - Consider accepting the recommended SQL profile to use parallel execution
 for this statement.
 execute dbms_sqltune.accept_sql_profile(task_name => 'my_tuning_task',
 task_owner => 'SYS', replace => TRUE, profile_type =>
 DBMS_SQLTUNE.PX_PROFILE);

 Executing this query parallel with DOP 64 will improve its response time
 98.27% over the original plan. However, there is some cost in enabling
 parallel execution. It will increase the statement's resource consumption by
 an estimated 11.03% which may result in a reduction of system throughput.
 Also, because these resources are consumed over a much smaller duration, the
 response time of concurrent statements might be negatively impacted if
 sufficient hardware capacity is not available.

 The following data shows some sampled statistics for this SQL from the past
 week and projected weekly values when parallel execution is enabled.

 Past week sampled statistics for this SQL
 -----------------------------------------
 Number of executions 0
 Percent of total activity 0
 Percent of samples with #Active Sessions > 2*CPU 0
 Weekly DB time (in sec) 0

 Projected statistics with Parallel Execution
 --------------------------------------------
 Weekly DB time (in sec) 0

SQL Tuning Advisor 给的第一个建议是开启并发执行，可以看到预计收益达到98.26%（estimated benefit: 98.26%），
效果是非常好的，但是可能会对数据库其他SQL造成影响，我们先不考虑。

第二段如下：

2- Index Finding (see explain plans section below)
--------------------------------------------------
 The execution plan of this statement can be improved by creating one or more
 indices.

 Recommendation (estimated benefit: 99.99%)
 ------------------------------------------
 - Consider running the Access Advisor to improve the physical schema design
 or creating the recommended index.
 create index CMS.IDX$$_516130001 on CMS.B_WF_VOUMNG("SERSEQNO","BIZID","REM
 AMT");

 Rationale
 ---------
 Creating the recommended indices significantly improves the execution plan
 of this statement. However, it might be preferable to run "Access Advisor"
 using a representative SQL workload as opposed to a single statement. This
 will allow to get comprehensive index recommendations which takes into
 account index maintenance overhead and additional space consumption.
</pre><pre>

第二个建议是添加索引，也就是我们刚开始说的，这个预估收益达到99.99%（estimated benefit: 99.99%），效果比开启并发执行还要好。

再来看看第三段

3- Alternative Plan Finding
---------------------------
 Some alternative execution plans for this statement were found by searching
 the system's real-time and historical performance data.

 The following table lists these plans ranked by their average elapsed time.
 See section "ALTERNATIVE PLANS SECTION" for detailed information on each
 plan.

 id plan hash last seen elapsed (s) origin note
 -- ---------- -------------------- ------------ --------------- ----------------
 1 3788302452 2024-07-26/17:00:04 0.684 AWR
 2 2290443376 2024-07-31/08:39:39 5.907 Cursor Cache original plan

 Recommendation
 --------------
 - Consider creating a SQL plan baseline for the plan with the best average
 elapsed time.
 execute dbms_sqltune.create_sql_plan_baseline(task_name =>
 'my_tuning_task', owner_name => 'SYS', plan_hash_value =>
 3788302452);

-------------------------------------------------------------------------------
EXPLAIN PLANS SECTION
-------------------------------------------------------------------------------

1- Original
-----------
Plan hash value: 2290443376

---------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 175 | 77663 (1)| 00:15:32 |
| 1 | NESTED LOOPS OUTER | | 1 | 175 | 77663 (1)| 00:15:32 |
| 2 | NESTED LOOPS OUTER | | 1 | 111 | 77661 (1)| 00:15:32 |
|* 3 | TABLE ACCESS FULL | B_WF_VOUMNG | 1 | 44 | 77659 (1)| 00:15:32 |
| 4 | TABLE ACCESS BY INDEX ROWID| BEDC_DETAIL | 1 | 67 | 2 (0)| 00:00:01 |
|* 5 | INDEX UNIQUE SCAN | BEDC_DETAIL_INDEX | 1 | | 1 (0)| 00:00:01 |
| 6 | TABLE ACCESS BY INDEX ROWID | BEDC_HISDETAIL | 1 | 64 | 2 (0)| 00:00:01 |
|* 7 | INDEX UNIQUE SCAN | SYS_C00139561 | 1 | | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 3 - filter("T1"."SERSEQNO"=:1)
 5 - access("T1"."BIZID"="T2"."BIZID"(+))
 7 - access("T1"."BIZID"="T3"."BIZID"(+))

2- Using New Indices
--------------------
Plan hash value: 2517111532

---------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 175 | 7 (0)| 00:00:01 |
| 1 | NESTED LOOPS OUTER | | 1 | 175 | 7 (0)| 00:00:01 |
| 2 | NESTED LOOPS OUTER | | 1 | 111 | 5 (0)| 00:00:01 |
|* 3 | INDEX RANGE SCAN | IDX$$_516130001 | 1 | 44 | 3 (0)| 00:00:01 |
| 4 | TABLE ACCESS BY INDEX ROWID| BEDC_DETAIL | 1 | 67 | 2 (0)| 00:00:01 |
|* 5 | INDEX UNIQUE SCAN | BEDC_DETAIL_INDEX | 1 | | 1 (0)| 00:00:01 |
| 6 | TABLE ACCESS BY INDEX ROWID | BEDC_HISDETAIL | 1 | 64 | 2 (0)| 00:00:01 |
|* 7 | INDEX UNIQUE SCAN | SYS_C00139561 | 1 | | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 3 - access("T1"."SERSEQNO"=:1)
 5 - access("T1"."BIZID"="T2"."BIZID"(+))
 7 - access("T1"."BIZID"="T3"."BIZID"(+))

3- Using Parallel Execution
---------------------------
Plan hash value: 3538277696

----------------------------------------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | TQ |IN-OUT| PQ Distrib |
----------------------------------------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 175 | 1347 (0)| 00:00:17 | | | |
| 1 | PX COORDINATOR | | | | | | | | |
| 2 | PX SEND QC (RANDOM) | :TQ10000 | 1 | 175 | 1347 (0)| 00:00:17 | Q1,00 | P->S | QC (RAND) |
| 3 | NESTED LOOPS OUTER | | 1 | 175 | 1347 (0)| 00:00:17 | Q1,00 | PCWP | |
| 4 | NESTED LOOPS OUTER | | 1 | 111 | 1347 (0)| 00:00:17 | Q1,00 | PCWP | |
| 5 | PX BLOCK ITERATOR | | | | | | Q1,00 | PCWC | |
|* 6 | TABLE ACCESS FULL | B_WF_VOUMNG | 1 | 44 | 1347 (0)| 00:00:17 | Q1,00 | PCWP | |
| 7 | TABLE ACCESS BY INDEX ROWID| BEDC_DETAIL | 1 | 67 | 0 (0)| 00:00:01 | Q1,00 | PCWP | |
|* 8 | INDEX UNIQUE SCAN | BEDC_DETAIL_INDEX | 1 | | 0 (0)| 00:00:01 | Q1,00 | PCWP | |
| 9 | TABLE ACCESS BY INDEX ROWID | BEDC_HISDETAIL | 1 | 64 | 0 (0)| 00:00:01 | Q1,00 | PCWP | |
|* 10 | INDEX UNIQUE SCAN | SYS_C00139561 | 1 | | 0 (0)| 00:00:01 | Q1,00 | PCWP | |
----------------------------------------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 6 - filter("T1"."SERSEQNO"=:1)
 8 - access("T1"."BIZID"="T2"."BIZID"(+))
 10 - access("T1"."BIZID"="T3"."BIZID"(+))

-------------------------------------------------------------------------------
ALTERNATIVE PLANS SECTION
-------------------------------------------------------------------------------

Plan 2
------

 Plan Origin :Cursor Cache
 Plan Hash Value :2290443376
 Executions :10777
 Elapsed Time :5.907 sec
 CPU Time :0.723 sec
 Buffer Gets :263726
 Disk Reads :263668
 Disk Writes :0

Notes:
 1. Statistics shown are averaged over multiple executions.
 2. The plan matches the original plan.

---------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 175 | 77663 (1)| 00:15:32 |
| 1 | NESTED LOOPS OUTER | | 1 | 175 | 77663 (1)| 00:15:32 |
| 2 | NESTED LOOPS OUTER | | 1 | 111 | 77661 (1)| 00:15:32 |
|* 3 | TABLE ACCESS FULL | B_WF_VOUMNG | 1 | 44 | 77659 (1)| 00:15:32 |
| 4 | TABLE ACCESS BY INDEX ROWID| BEDC_DETAIL | 1 | 67 | 2 (0)| 00:00:01 |
|* 5 | INDEX UNIQUE SCAN | BEDC_DETAIL_INDEX | 1 | | 1 (0)| 00:00:01 |
| 6 | TABLE ACCESS BY INDEX ROWID | BEDC_HISDETAIL | 1 | 64 | 2 (0)| 00:00:01 |
|* 7 | INDEX UNIQUE SCAN | SYS_C00139561 | 1 | | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 3 - filter("T1"."SERSEQNO"=:1)
 5 - access("T1"."BIZID"="T2"."BIZID"(+))
 7 - access("T1"."BIZID"="T3"."BIZID"(+))

Plan 1
------

 Plan Origin :AWR
 Plan Hash Value :3788302452
 Executions :5764
 Elapsed Time :0.684 sec
 CPU Time :0.678 sec
 Buffer Gets :71476
 Disk Reads :0
 Disk Writes :0

Notes:
 1. Statistics shown are averaged over multiple executions.

---------------------------------------------------------------------------------------------------
| Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time |
---------------------------------------------------------------------------------------------------
| 0 | SELECT STATEMENT | | 1 | 175 | 81836 (1)| 00:16:23 |
| 1 | NESTED LOOPS OUTER | | 1 | 175 | 81836 (1)| 00:16:23 |
| 2 | NESTED LOOPS OUTER | | 1 | 111 | 81834 (1)| 00:16:23 |
| 3 | TABLE ACCESS BY INDEX ROWID| B_WF_VOUMNG | 1 | 44 | 81832 (1)| 00:16:22 |
|* 4 | INDEX SKIP SCAN | B_WF_VOUMNG_IDX7 | 1 | | 81830 (1)| 00:16:22 |
| 5 | TABLE ACCESS BY INDEX ROWID| BEDC_DETAIL | 1 | 67 | 2 (0)| 00:00:01 |
|* 6 | INDEX UNIQUE SCAN | BEDC_DETAIL_INDEX | 1 | | 1 (0)| 00:00:01 |
| 7 | TABLE ACCESS BY INDEX ROWID | BEDC_HISDETAIL | 1 | 64 | 2 (0)| 00:00:01 |
|* 8 | INDEX UNIQUE SCAN | SYS_C00139561 | 1 | | 1 (0)| 00:00:01 |
---------------------------------------------------------------------------------------------------

Predicate Information (identified by operation id):
---------------------------------------------------

 4 - access("T1"."SERSEQNO"=:1)
 filter("T1"."SERSEQNO"=:1)
 6 - access("T1"."BIZID"="T2"."BIZID"(+))
 8 - access("T1"."BIZID"="T3"."BIZID"(+))

-------------------------------------------------------------------------------

这一段主要是列出各个建议的执行计划情况，包括原始执行计划、添加了索引的执行计划、并发执行的执行计划等等， 这里我们可以看到添加索引的执行计划（2- Using New Indices）的成本是最低的，也是最优的。

最后，删除刚刚创建的 SQL Tuning Advisor 任务

EXEC dbms_sqltune.drop_tuning_task('my_tuning_task');

【解决】

最后我们只要根据建议加上索引就可以了。

create index CMS.IDX$$_516130001 on CMS.B_WF_VOUMNG("SERSEQNO","BIZID","REM AMT");

查看索引

col COLUMN_NAME for a20
SELECT TABLE_OWNER,
 TABLE_NAME,
 INDEX_NAME,
 COLUMN_NAME,
 COLUMN_POSITION
FROM dba_ind_columns
WHERE table_name='B_WF_VOUMNG'
order by INDEX_NAME,COLUMN_POSITION;

【参考】

Using the DBMS_SQLTUNE Package to Run the SQL Tuning Advisor (Doc ID 262687.1)

基于SQL Tuning Advisor做SQL优化的方法最先出现在dbAnt。

【报错】

接到告警电话，查看alert 日志报错如下：

Thu Jun 20 22:17:11 2024

Errors in file /oracle/diag/rdbms/coredb/coredb/trace/coredb_j003_23800.trc  (incident=288071):

ORA-04030: out of process memory when trying to allocate 432 bytes (kxs-heap-c,kprbalo temp memory)

Use ADRCI or Support Workbench to package the incident.

See Note 411.1 at My Oracle Support for error and packaging details.

End automatic SQL Tuning Advisor run for special tuning task  “SYS_AUTO_SQL_TUNING_TASK”

Thu Jun 20 22:17:13 2024

Sweep [inc][288071]: completed

【分析】

当操作系统请求内存或其他资源，而系统资源不足时，就会报错 ORA-4030，这里我们可以看到 kxs-heap-c,kprbalo temp memory 是内存的问题，也就是PGA内存不足了。

1、如果 ORA-4030 报错没有对应的trace详细信息，可以手动开启

## 开启trace

ALTER SYSTEM SET EVENTS '4030 trace name heapdump level 536870917;name errorstack level 3';

## 等ORA-4030错误再次出现之后，关闭trace

ALTER SYSTEM SET EVENTS '4030 trace name context off; name context off';

2、查看trace文件：/oracle/diag/rdbms/coredb/coredb/trace/coredb_j003_23800.trc

*** 2024-06-20 22:00:04.116

*** SESSION ID:(406.2663) 2024-06-20 22:00:04.116

*** CLIENT ID:() 2024-06-20 22:00:04.116

*** SERVICE NAME:(SYS$USERS) 2024-06-20 22:00:04.116

*** MODULE NAME:(DBMS_SCHEDULER) 2024-06-20 22:00:04.116

*** ACTION NAME:(ORA$AT_SQ_SQL_SW_3355) 2024-06-20 22:00:04.116

 

status=(pst=ERROR)

status=(pst=ERROR)

status=(pst=ERROR)

…

Incident 273219 created, dump file: /oracle/diag/rdbms/coredb/coredb/incident/incdir_273219/coredb_j003_23800_i273219.trc

ORA-04030: out of process memory when trying to allocate 432 bytes (kxs-heap-c,kprbalo temp memory)

 

Incident 273220 created, dump file: /oracle/diag/rdbms/coredb/coredb/incident/incdir_273220/coredb_j003_23800_i273220.trc

ORA-04030: out of process memory when trying to allocate 169040 bytes (pga heap,kgh stack)

ORA-04030: out of process memory when trying to allocate 432 bytes (kxs-heap-c,kprbalo temp memory)

 

Incident 273221 created, dump file: /oracle/diag/rdbms/coredb/coredb/incident/incdir_273221/coredb_j003_23800_i273221.trc

ORA-04030: out of process memory when trying to allocate 82456 bytes (pga heap,control file i/o buffer)

ORA-04030: out of process memory when trying to allocate 432 bytes (kxs-heap-c,kprbalo temp memory)

从上面的日志可以看到是DBMS_SCHEDULER 定时任务调用的 ORA$AT_SQ_SQL_SW_3355 模块导致 4030 报错，这个是Oracle定时执行的Automatic SQL Tuning job。

## 查看trace文件：/oracle/diag/rdbms/coredb/coredb/incident/incdir_273219/coredb_j003_23800_i273219.trc

这里可以看到 pid 145 这个进程执行了一个存储过程占用了90%的内存。

查看pid 145的详细信息，确实是ORA$AT_SQ_SQL_SW_3355 模块。

这里可以看到32G的内存，剩下232M 可用，基本消耗完了。

到这里，我们已经确认是Oracle每天晚上定时执行的 Automatic SQL Tuning job 导致内存耗尽的，可以选择关闭定时任务解决问题。

## 关闭job
BEGIN
dbms_auto_task_admin.disable(
client_name => 'sql tuning advisor',
operation => NULL,
window_name => NULL);
END;
/

3、如果不想关闭这个定时任务，进一步分析日志

从上面的trace日志可以看到，还有另外两个错误提示：

(pga heap,kgh stack)

(pga heap,control file i/o buffer)

参考MOS文档：

PLSQL Procedure Causing ORA-04030: (pga heap,control file i/o buffer) And ORA-04030: (koh-kghu sessi,pmuccst: adt/record) or ORA-04030: (koh-kghucall ,pmucalm coll) Errors (Doc ID 1325100.1)

该文章提到即使PGA已经设置大于4GB，当一个进程使用内存达到4GB，就会报这个错误。

## 查看trace文件：/oracle/diag/rdbms/coredb/coredb/incident/incdir_273221/coredb_j003_23800_i273221.trc

可以看到trace日志中，确实内存使用达到4GB就被限制了。

为什么会有4GB这个限制呢？

## 查看trace文件的 Process Map Dump，内存dump信息

从日志看出，进程的内存map记录数是 70986-5453+1=65534 条。

查看系统最大内存记录数65530（默认应该是65536），上面那个进程的记录数已经超过系统限制了

more /proc/sys/vm/max_map_count

默认的内存分配器（realfree allocator）页大小是64KB（65536），所以64K 条记录 64KB * 64K 就是4GB，所以单进程使用达到4GB内存的限制，就报错了。

【解决】

我们可以从系统层面 或 数据库层面修改这个限制。

• 操作系统修改内存最大记录数为256K（262144），即 64KB * 256K = 16GB

more /proc/sys/vm/max_map_count

## 临时生效

sysctl -w vm.max_map_count=262144

## 永久生效

echo "vm.max_map_count=262144" >> /etc/sysctl.conf

sysctl -p

• 数据库修改内存分配器的页大小256KB（262144），即256KB * 64K = 16GB

## 11.2.0.4 及以下版本:

_use_realfree_heap=TRUE

_realfree_heap_pagesize_hint = 262144

## 12.1 及以上版本:

_use_realfree_heap=TRUE

_realfree_heap_pagesize = 262144

对于多租户数据库，只要修改CDB的参数即可，PDB会继承该参数。

注意：

请修改OS 或 DB 其中一个即可，因为一个修改的是记录数，一个修改的是页大小，如果同时修改，那么限制就是 256KB * 256K = 64GB了。

【其他】

参考文档：FAQ: ORA-4030 (Doc ID 399497.1)

该案例只是其中一种情况，导致  ORA-4030 的原因很多，典型的原因有：

• 系统内存或swap空间达到限制；
• 系统达到内核或用户shell等限制，限制用户级或进程级的内存使用；
• 系统内存地址限制（32位系统），SGA占用了所有地址，导致PGA无法分配地址；
• 触发内部未公开的Bug:3130972
• 应用设计问题导致限制；
• 内存空间溢出或内存堆溢出等bug

Diagnostic Tools Catalog (Doc ID 559339.1)

MOS上面有很多诊断工具

可以使用 ORA-4030-Troubleshooting Tool

## 新建一个issue

## 支持上次TFA、IPS、RDA 3种工具收集的日志，选择其一上传即可

## 上传之后，如果是一个有已知问题，会给出现象、原因以及解决方案。如果是一个未知的问题，会给出相关文档以及内存监控工具，便于继续分析或提交SR。

也可以使用RDA工具检查系统配置，内存使用情况等

./rda.sh T hcve

## 检查修复检查结果为 FAILED 的

参考：

Primary Note for Diagnosing OS Memory Problems and ORA-4030 (Doc ID 1088267.1)

FAQ: ORA-4030 (Doc ID 399497.1)

https://community.oracle.com/mosc/discussion/3900920/ora-04030-out-of-process-memory-when-trying-to-allocate-x-bytes-kxs-heap-c-kprbalo-temp-memory

ORA-04030: out of process memory when trying to allocate 432 bytes (kxs-heap-c,kprbalo temp memory)最先出现在dbAnt。

最后在中科院的网站找到了国家授时中心的NTP域名： ntp.ntsc.ac.cn

原公告地址：

http://www.cas.cn/tz/201809/t20180921_4664344.shtml

中科院国家授时中心NTP服务器最先出现在dbAnt。

官方文档的解释：

Unique Constraints

A unique key constraint requires that every value in a column or set of columns be unique. No rows of a table may have duplicate values in a column (theunique key) or set of columns (the composite unique key) with a unique key constraint.

Note:

The term key refers only to the columns defined in the integrity constraint. Because the database enforces a unique constraint by implicitly creating or reusing an index on the key columns, the term unique key is sometimes incorrectly used as a synonym for unique key constraint or unique index.

Unique and Nonunique Indexes

Indexes can be unique or nonunique. Unique indexes guarantee that no two rows of a table have duplicate values in the key column or column. For example, no two employees can have the same employee ID. Thus, in a unique index, one rowid exists for each data value. The data in the leaf blocks is sorted only by key.

Nonunique indexes permit duplicates values in the indexed column or columns. For example, the first_name column of the employees table may contain multiple Mike values. For a nonunique index, the rowid is included in the key in sorted order, so nonunique indexes are sorted by the index key and rowid (ascending).

Oracle Database does not index table rows in which all key columns are null, except for bitmap indexes or when the cluster key column value is null.

【实验】

创建表T1并添加唯一性约束：

create table t1 (
id varchar2(20),
name varchar2(30),
constraints t1_id_uk unique(id));

创建表T2并创建唯一索引：

create table t2 (
id varchar2(20),
name varchar2(30));

create unique index ind_t2_id on t2(id);

查看两个表的索引，发现T1表自动创建了唯一索引：

select OWNER,TABLE_NAME,INDEX_NAME,INDEX_TYPE,UNIQUENESS from dba_indexes where table_name in ('T1','T2');

TABLE_NAME     INDEX_NAME    INDEX_TYPE      UNIQUENES
-------------- ------------- --------------- ---------
T1             T1_ID_UK      NORMAL          UNIQUE
T2             IND_T1_ID     NORMAL          UNIQUE

索引列也是ID列

select TABLE_OWNER,TABLE_NAME,INDEX_NAME,COLUMN_NAME,COLUMN_POSITION from dba_ind_columns where table_name in ('T1','T2');

TABLE_NAME      INDEX_NAME      COLUMN_NAME     COLUMN_POSITION
--------------- --------------- --------------- ---------------
T1              T1_ID_UK        ID                            1
T2              IND_T1_ID       ID                            1

检查两个表的约束，只有T1表存在唯一性约束：

select OWNER, TABLE_NAME, CONSTRAINT_NAME, CONSTRAINT_TYPE, STATUS from dba_constraints where table_name in ('T1','T2');

TABLE_NAME      CONSTRAINT_NAME      C STATUS
--------------- -------------------- - --------
T1              T1_ID_UK             U ENABLED

T1表插入数据：

insert into t1 values('A0001','TOM');
insert into t1 values('A0001','JACK');
commit;

kamner@TESTDB> insert into t1 values('A0001','JACK');
insert into t1 values('A0001','JACK')
*
ERROR at line 1:
<strong>ORA-00001: unique constraint (KAMNER.T1_ID_UK) violated</strong>

T2表出入数据

insert into t2 values('A0001','TOM');
insert into t2 values('A0001','JACK');
commit;

kamner@TESTDB> insert into t2 values('A0001','JACK');
insert into t2 values('A0001','JACK')
*
ERROR at line 1:
<strong>ORA-00001: unique constraint (KAMNER.IND_T1_ID) violated</strong>

可以看到，T1、T2表都报唯一性约束冲突的错误。

接着，我们插入NULL值。

kamner@TESTDB> insert into t1 values(null,'SCOTT');
1 row created.

kamner@TESTDB> commit;
Commit complete.

kamner@TESTDB> select * from t1;

ID                   NAME
-------------------- ------------------------------
A0001                TOM
                     SCOTT

kamner@TESTDB> insert into t1 values(null,'SCOTT');
1 row created.

kamner@TESTDB> insert into t1 values(null,'SCOTT');
1 row created.

kamner@TESTDB> commit;
Commit complete.

kamner@TESTDB> select * from t1;

ID                   NAME
-------------------- ------------------------------
A0001                TOM
                     SCOTT
                     SCOTT
                     SCOTT

我们看到，在Oracle中，每个NULL值都是不同的，因此他不受唯一索引的约束。这里给我们一点启发，在实际引用中，对于有唯一性要求的列，一般要同时添加非空约束。

综上，可以看出对一个表建唯一性约束和唯一索引的效果是一样的，而且可以看到，唯一性约束是通过唯一性索引来实现的。对于NULL值的情况，由于索引不会对NULL建索引页，因此更准确的说是在Oracle中，数据库没办法对NULL值进行比较，因此每个NULL值都看成是不一样的值。

Unique constraint & Unique index in Oracle 11gR2最先出现在dbAnt。

【问题现象】

exp system/123456 OWNER=rs2 file=20160823_rs2_17_8.dmp log=20160823_rs2_17_8.log

...

EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
. . exporting table ACCEPT_DETAILS 70239 rows exported
EXP-00091: Exporting questionable statistics.
"20160823_rs2_17_8.log" 5686L, 314896C 1,0-1 顶ç«EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
EXP-00091: Exporting questionable statistics.
. . exporting table ACCEPT_DETAILS0527 64422 rows exported

...

【问题处理】

由于exp工具会依赖与操作系统的字符串编码，所以需要将NLS_LANG环境变量设置与数据库一致。

SQL> select * from nls_database_parameters;

PARAMETER                      VALUE
------------------------------ ------------------------------
NLS_LANGUAGE                   AMERICAN
NLS_TERRITORY                  AMERICA
NLS_CURRENCY                   $
NLS_ISO_CURRENCY               AMERICA
NLS_NUMERIC_CHARACTERS         .,
NLS_CHARACTERSET               ZHS16GBK
NLS_CALENDAR                   GREGORIAN
NLS_DATE_FORMAT                DD-MON-RR
NLS_DATE_LANGUAGE              AMERICAN
NLS_SORT                       BINARY
NLS_TIME_FORMAT                HH.MI.SSXFF AM
NLS_TIMESTAMP_FORMAT           DD-MON-RR HH.MI.SSXFF AM
NLS_TIME_TZ_FORMAT             HH.MI.SSXFF AM TZR
NLS_TIMESTAMP_TZ_FORMAT        DD-MON-RR HH.MI.SSXFF AM TZR
NLS_DUAL_CURRENCY              $
NLS_COMP                       BINARY
NLS_LENGTH_SEMANTICS           BYTE
NLS_NCHAR_CONV_EXCP            FALSE
NLS_NCHAR_CHARACTERSET         AL16UTF16
NLS_RDBMS_VERSION              10.2.0.4.0

20 rows selected.

设置Linux的NLS_LANG环境变量：

[oracle@mfs ~]$ export NLS_LANG=AMERICAN_AMERICA.ZHS16GBK

设置windows的NLS_LANG环境变量：

C:\Users\Administrator>set NLS_LANG=AMERICAN_AMERICA.ZHS16GBK

设置了环境变量之后，重新导出一切正常。

不过还是建议使用数据泵来导数，因为数据泵不会收到操作系统的字符集编码的影响，就不会有这样的报错了。

EXP-00091: Exporting questionable statistics最先出现在dbAnt。

test@TESTDB> insert into t1 select * from dba_objects where rownum < 100;
insert into t1 select * from dba_objects where rownum < 100
*
ERROR at line 1:
ORA-00001: unique constraint (TEST.PK_T1_OBJECT_ID) violated

查询T1表主键及索引，发现主键索引仍然存在

test@TESTDB> select * from dba_cons_columns where table_name='T1';

no rows selected

test@TESTDB> select INDEX_OWNER,INDEX_NAME,TABLE_NAME,COLUMN_NAME from dba_ind_columns where table_name='T1';

INDEX_OWNER                    INDEX_NAME                     TABLE_NAME                     COLUMN_NAME
------------------------------ ------------------------------ ------------------------------ --------------------
TEST                           PK_T1_OBJECT_ID                T1                             OBJECT_ID

手动删除该索引，插入数据成功

test@TESTDB> drop index test.PK_T1_OBJECT_ID;

Index dropped.

test@TESTDB> insert into t1 select * from dba_objects where rownum < 100; 

99 rows created.

正常情况下删掉主键的时候会同时删除主键附带的索引，这里为什么会出现不删除的情况呢？

在MOS上面有一篇文章解答了这个疑问：

Indexes Associated With Primary Key Constraints Of Imported Tables Are Not Dropped When Constraints Are Disabled (文档 ID 887208.1)

由于T1表是imp导入的，而在import的时候，oracle会先创建唯一性索引，然后在添加主键约束，这就导致在创建主键的时候，唯一索引已经存在，并被主键使用，而对于并非主键创建的唯一性索引，在删除主键的时候，索引是会保留的。

如果想解决该问题，可以通过如下的方法进行修复：

test@TESTDB> alter table t1 enable constraint PK_T1_OBJECT_ID;

Table altered.

test@TESTDB>
test@TESTDB> select * from dba_cons_columns where table_name='T1';

OWNER CONSTRAINT_NAME TABLE_NAME COLUMN_NAME POSITION
------------------------------ ------------------------------ ------------------------------ -------------------- ----------
TEST PK_T1_OBJECT_ID T1 OBJECT_ID 1

test@TESTDB>
test@TESTDB> select INDEX_OWNER,INDEX_NAME,TABLE_NAME,COLUMN_NAME from dba_ind_columns where table_name='T1';

INDEX_OWNER INDEX_NAME TABLE_NAME COLUMN_NAME
------------------------------ ------------------------------ ------------------------------ --------------------
TEST PK_T1_OBJECT_ID T1 OBJECT_ID

Indexes Are Not Dropped When Primary Key Are Dropped最先出现在dbAnt。

DB：ORACLE 11.2.0.3

通过EXPDP导出全库，查看log发现如下报错：

ORA-31693: 表数据对象 “AUDIT_GD”.”PUB_LOBFILE” 无法加载/卸载并且被跳过, 错误如下:
ORA-02354: 导出/导入数据时出错
ORA-01555: 快照过旧: 回退段号 7 (名称为 “_SYSSMU7_93985497$”) 过小
已成功加载/卸载了主表 “EXP_DBA”.”JOB_EXP_FULL_2″

从报错信息ORA-31693，我们可以看到由于”AUDIT_GD”.”PUB_LOBFILE”对象的UNDO段过期导致。我们知道LOB对象的UNDO段保留时间通过PCTVERSION和RETENTION参数控制。

注意：

1、SecureFiles类型的LOB，只能通过RETENTION参数控制。

2、BasicFiles类型的LOB，可以通过PCTVERSION 或 RETENTION中的其中一个进行控制。

PCTVERSION：表示整个BasicFiles LOB对象的空间允许被该LOB对象的UNDO段占用的最大百分比。

RETENTION：表示保留的时间，单位秒。当数据库为自动UNDO段管理是，该参数为默认值，并参考undo_retention的参数值。

查看该LOB对象的UNDO段保留情况：

SQL> select COLUMN_NAME,SECUREFILE,PCTVERSION,RETENTION from dba_lobs where OWNER=upper('AUDIT_GD') and TABLE_NAME=upper('PUB_LOBFILE') ;

COLUMN_NAME                    SEC PCTVERSION  RETENTION
------------------------------ --- ---------- ----------
LOB                            NO          10
TEXTLOB                        NO                    900

查看数据库中使用UNDO段的最大保留时间：

SQL> select max(maxquerylen) from v$undostat;

MAX(MAXQUERYLEN)
----------------
1839

为了防止数据导出过程中UNDO段过期，增加undo_retention参数值：(设置该参数之前，必须保证UNDO表空间有足够的空间)

SQL> alter system set undo_retention=2000 scope=both sid='*';

设置LOB对象的UNDO段保留时间：

SQL> alter table AUDIT_GD.PUB_LOBFILE modify lob(LOB) (retention);
Table altered.

SQL> alter table AUDIT_GD.PUB_LOBFILE modify lob(TEXTLOB) (retention);
Table altered.

SQL> select COLUMN_NAME,SECUREFILE,PCTVERSION,RETENTION from dba_lobs where OWNER=upper('AUDIT_GD') and TABLE_NAME=upper('PUB_LOBFILE') ;

COLUMN_NAME                    SEC PCTVERSION  RETENTION
------------------------------ --- ---------- ----------
LOB                            NO                   2000
TEXTLOB                        NO                   2000

修改完参数之后，再进行EXPDP导出。

参考：ORA-31693, ORA-02354 and ORA-01555 with Export Datapump (文档 ID 1580798.1)

版权所有，转载请注明出处：

EXPDP导出全库报错ORA-01555

EXPDP导出全库报错ORA-01555最先出现在dbAnt。