Due to "No space left on device", Many Concurrent Manager are under 'No Manager' Status

Found Many Concurrent Managers are under 'No Manager' Status, Then I tried to restart Concurrent Manager like(http://blog.csdn.net/pan_tian/article/details/7765256)

Then saw following error:

--------------------------------------------------------------------------------------------------------------

[oracle@bej301441 scripts]$ adcmctl.sh  start apps/apps diag=Y

You are running adcmctl.sh version 120.17.12010000.5

Starting concurrent manager for mc3yd213 ...
Starting mc3yd213_0127@mc3yd213 Internal Concurrent Manager
Default printer is noprint
/u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/admin/scripts/adcmctl.sh: line 547: printf: write error: No space left on device
/u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/admin/scripts/adcmctl.sh: line 548: printf: write error: No space left on device

adcmctl.sh: exiting with status 0
--------------------------------------------------------------------------------------------------------------

Use Linux Command to check directory space
du -sh <Directory>

Apps Server Side Logs Directory
Forms dump files : $INST_TOP/logs/ora/10.1.2/forms
Reports Cache : $INST_TOP/logs/ora/10.1.2/reports/cache
Apache logs : $INST_TOP/logs/ora/10.1.3/Apache
OPMN Logs : $INST_TOP/logs/ora/10.1.3/opmn

[oracle@bej301441 forms]$ du -sh $INST_TOP/logs/ora/10.1.2/forms
207M    /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.2/forms
[oracle@bej301441 forms]$ du -sh $INST_TOP/logs/ora/10.1.2/reports/cache
145M    /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.2/reports/cache
[oracle@bej301441 forms]$ du -sh $INST_TOP/logs/ora/10.1.3/Apache
640M    /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.3/Apache
[oracle@bej301441 forms]$ du -sh $INST_TOP/logs/ora/10.1.3/opmn
6.8M    /u01/oracle/mc3yd213/inst/apps/mc3yd213_bej301441/logs/ora/10.1.3/opmn

DB Server Side Logs Directory
Check user_dump_dest and core_dump_dest
SELECT * FROM v$parameter WHERE name in ('user_dump_dest','core_dump_dest')
[oracle@bej301441 trace]$ du -sh /u01/oracle/mc3yd213/db/tech_st/11.1.0/admin/mc3yd213_bej301441/diag/rdbms/mc3yd213/mc3yd213/trace
22G     /u01/oracle/mc3yd213/db/tech_st/11.1.0/admin/mc3yd213_bej301441/diag/rdbms/mc3yd213/mc3yd213/trace
[oracle@bej301441 trace]$ du -sh /u01/oracle/mc3yd213/db/tech_st/11.1.0/admin/mc3yd213_bej301441/diag/rdbms/mc3yd213/mc3yd213/cdump
4.0K    /u01/oracle/mc3yd213/db/tech_st/11.1.0/admin/mc3yd213_bej301441/diag/rdbms/mc3yd213/mc3yd213/cdump
[oracle@bej301441 trace]$


It's clear in Oracle DB Side,sql trace are too big under user_dump_dest dir.

After remove all sql trace from user_dump_dest,concurrent manager works.

reference: Cleaning up the system – 11i and R12