Thursday, November 13, 2014

"CPU :In-memory query" in Oracle 12c

 In 12c EM, we have a  " CPU :in-memory query" displayed in the  graphical report of performance tab. Since Oracle Database In-Memory sessions access only memory, it makes perfect sense to track the CPU utilized for such queries.  I did not see any separate event or  info of how it is derived. 
However, there are new columns called "in_memory_query" and "in_memory_populate" which have values 'Y' and 'N'
I think  "CPU :In-memory query" is derived from the following logic.

 From the v$active_session_history columns session_state, session_type and in_memory_query or in_memory_populate values and its meaning.

      session_state + session_type       + in_inmemory_xxx                                = "Wait-class"
     "ON CPU"        + "FOREGROUND" + in_inmemory_query ='Y'                      =  CPU in-memory query
      "ON CPU"       + "FOREGROUND" + in_inmemory_query ='N'                      =  CPU
      "ON CPU"       + "BACKGROUND" + in_inmemory_populate ='Y'                  =  CPU: in-memory populate
     "ON CPU"        + "BACKGROUND" + in_memory_populate='N'                     =  BCPU /* background CPU */

else
      "WAITING"                                                                                                 = "wait_class value"


the query for the above logic will be something like
 
SELECT sample_time,
DECODE (session_state,'ON CPU',
   DECODE(session_type,'FOREGROUND',
        DECODE( in_inmemory_query,'Y','CPU in-memory query','CPU'),'BACKGROUND',DECODE(in_inmemory_populate,'Y','CPU: in-memory populate','BCPU')),wait_class)
        FROM v$active_session_history order by sample_time desc;

Tweaks for Linux VirtualBox (VBOX) to use SIMD for 12c InMemory.


  Oracle InMemory uses  SSE4.2 extensions to run SIMD.
  However, by default, Virtualbox(VBOX) may not configure this. You can verify if the extensions are enabled
by checking  the flag in cpuinfo as follows in your Linux guest OS

  cat /proc/cpuinfo | grep sse4

If it is not set,  then no rows will be returned.

 you have to do the following tweak to enable it.
1)  Shutdown the guest VM, and the GUI VM manager.
2) cd to VM home directory.
    eg.  cd  D:\software\virtualbox
3) run the following command  Allowing VirtualBox to pass the SSE 4.2 extensions
( supported in VirtualBox starting in Versions 4.3.8)

eg.
D:\software\virtualbox>VBoxManage setextradata "VM name" VBoxInternal/CPUM/SSE4.2 1

Where "VM Name" is the name of your guest Linux VM.

4) start the GUI VManager  and start the guest OS.

5) verify if the  flag is set.
cat /proc/cpuinfo | grep sse4

[oracle@bigdatalite ~]$ cat /proc/cpuinfo | grep sse4
flags        : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx rdtscp lm constant_tsc rep_good nopl pni ssse3 sse4_2 lahf_lm

To see if the library is linked dynamically , you can run the following 
ps -ef | grep ora_p | awk '{system ( "pmap "$2 " |grep libshpk " ) ; print "-->"$8 }'

[oracle@bigdatalite ~]$  ps -ef | grep ora_p | awk '{system ( "pmap "$2 " |grep libshpk " ) ; print "-->"$8 }'
00007f732897d000   1500K r-x--  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so
00007f7328af4000   2044K -----  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so
00007f7328cf3000     72K rw---  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so
-->ora_pmon_orcl
00007f8e01a71000   1500K r-x--  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so
00007f8e01be8000   2044K -----  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so
00007f8e01de7000     72K rw---  /u01/app/oracle/product/12.1.0/dbhome_1/lib/libshpksse4212.so

Tuesday, September 30, 2014

12c PDBs and SPFILE

In 12c multi-tenant database, the spfile is for CDB only. A newly created PDB will inherit the initialization values from the container database if they are created from the pdb$seed ( default empty PDB). 
 There are about 184 parameters that can be modified at the PDB level and are stored in the sys table of the PDB. If you altered the parameter in a PDB  and then plugged in to a different Container database, these modified parameters go with the PDB.

The v$system_parameter shows the different configured parameters of all plugged in database. To see all non default parameters, you can run the following query.
select  name,value, con_id from v$system_parameter where con_id!=0;


One interesting fact is, once a parameter has been set, later even if you reset the value equal to container default, the PDB will still maintain the parameter values.

Tuesday, January 21, 2014

Oracle In-database MapReduce in 12c (big data)

There is some interest from the field about what is In-database map-reduce option and why and how it is different than hadoop solution.
I though I will share my thoughts on it.

 In-database map-reduce is an umbrella term that includes two features.
  •             "SQL Map-reduce" or  "SQL pattern matching".
  •              In database container for Hadoop.  to be released in future release. 


  • "SQL MapReduce" : Oracle database 12c introduced a new feature called PATTERN MATCHING using "MATCH_RECOGNIZE" clause in SQL. This is one of the latest ANSI SQL standards proposed and implemented by Oracle. The new sql syntax helps to intuitively solve complex queries that are not easy to implement using 11g analytical functions alone. Some of the use cases are fraud detection, gene sequencing, time series calculation, stock ticker pattern matching . Etc.  I found most of the use case for Hadoop can be done using match_recognize in database on structured data. Since this is just a SQL enhancement , it is there in both Enterprise & Standard Edition database.


  • "In database container for Hadoop  (beta)" : if you have your development team more skilled at Hadoop and not SQL , or want to implement some complex pre-packaged Hadoop algorithms, you could use oracle container for Hadoop (beta). It is a Hadoop prototype APIs  which run within the java virtual machine in the database. It implements Hadoop Java APIs and interfaces with database using parallel table functions to read data in parallel. One interesting fact about parallel table functions is that it can run in parallel across RAC cluster and also can also route data to a specific parallel processes . This functionality is the key in making Hadoop scale across clusters and  this functionality exited in database for over 15 years now.  Advantage of in-database Hadoop  is 
  1.  No need to move data out of database for running Mapreduce functions and hence save time and resources.
  2.  More  real time data could be used.
  3.  Less redundant copies of data and hence better security & less disk space used.
  4.  The servers could be used for not just MapReduce work, but also used to run the database making better resource utilization,
  5. The output of the MapReduce is immediately available for analytic tools and can combine this functionality along with database features like "in-memory option (beta) to get near real time analysis of Big Data. 
  6. Combine db features for security. Backup, auditing, performance with MapReduce. API.
  7. The ability to stream the output of one parallel table function as input to the next parallel table function has an advantage of not needing to maintain any intermediate stages.
  8. Features like graphical, test, spacial and semantic within oracle database can be used for further analysts.
In addition to this, Oracle 12c will support schema less access using JSON protocol. That will help big data use cases of NOSQL to run on data within Oracle database as well.  

Conclusion.
Having these features will help to solve MapReduce challenges when the data is mostly within database and reduce data movement and make better use of available resources.. 
If Most of your data is outside the DB, then sql Connectors for hadoop and Oracle Loader for Hadoop could be used. 


Reference

1) presentation from Kuassi Mensah 

Friday, January 3, 2014

Sql* developer guage sample report

 I had tried to show something like guage in sqlplus using RPAD trick.
Howevere, if you are running the query in sql*developer , using GUAGE will be a good way to display the graph. 

below is the  output in sql*developer and sqlplus.


Sql*Developer OUTPUT
select name,con_id, bufferCache_MB, 'SQLDEV:GAUGE:0:100:0:100:' ||PCT as PCT from
(  SELECT NVL(pdb.name,'CDB') name,b.con_id,
  ROUND(b.subtotal *
  (SELECT value FROM V$PARAMETER WHERE name = 'db_block_size'
  )                        /(1024 *1024)) bufferCache_MB,
  ROUND(b.subtotal         /b.total * 100) PCT,
  rpad('*',ROUND(b.subtotal/b.total * 100),'*') PCT_BARCHART
FROM
  ( SELECT DISTINCT a.*
  FROM
    (SELECT con_id,
      COUNT(*) over (partition BY con_id ) subtotal ,
      COUNT(*) over () total
    FROM sys.gv$bh
    ) a
  )b, v$pdbs pdb
  where b.con_id=pdb.con_id(+)
ORDER BY con_id);
Sqlplus Output 
col PCT_BARCHART format a30
set linesize 200
 col name format a10
 col con_id format 99
 col PCT format 999
    SELECT NVL(pdb.name,'CDB') name,b.con_id,
  ROUND(b.subtotal *  
 (SELECT value FROM V$PARAMETER WHERE name = 'db_block_size'
  )                        /(1024 *1024)) bufferCache_MB,
  ROUND(b.subtotal         /b.total * 100) PCT,
  rpad('*',ROUND(b.subtotal/b.total * 100),'*') PCT_BARCHART
FROM
  ( SELECT DISTINCT a.*
  FROM
    (SELECT con_id,
      COUNT(*) over (partition BY con_id ) subtotal ,
      COUNT(*) over () total
    FROM sys.gv$bh
    ) a
  )b, v$pdbs pdb
  where b.con_id=pdb.con_id(+)
ORDER BY con_id;



Friday, December 27, 2013

12c partial index to speed OLTP






In 11g database, the more index you have in a table, the longer it takes to insert rows into that table. When a row is inserted into a table, rowid, and column values are inserted into the index. Further, if several sessions are concurrently inserting data, there will be contention to insert data into the same table and index block. This issue can be further magnified in RAC servers, where these user sessions will have block contention across different RAC servers. Some of the solutions to remove these contention is to try to distribute the data inserted across several blocks by using hash partitions or reverse key indexing or even in some cases use exchange partition strategy.
 While all these methods are still relevant in 12c as well, it is possible to reduce the index contention completely on some indexes by not having the indexes built during inserts.
This can be done using 12c new feature called partial index. Note that indexes are used mainly to speed queries or reports. There may be still some indexes like unique index, indexes on foreign key column and some additional indexes which might be required to speed  look-ups to do the insert into tables. Rest of the indexes which could wait to be asynchronously built but still report in real time could use PARTIAL indexing.


To create partial indexes, you have to set INDEXING parameter on the table and then alter or create index with the PARTIAL clause.

To configure indexes to have PARTIAL GLOBAL and  PARTIAL LOCAL index  on a partitioned table.  There are two places you can set this feature.
1) indexing ON/OFF at table level
2) indexing ON/OFF at the table partition level.
and then
Alter or create  index  with partial clause.
e.g. alter index global_idx indexing partial;
      create index local_idx on tabname(colname) local indexing PARTIAL;

The parameter set at table level to OFF will set newer default partitions created with indexing OFF at partition level. Indexing ON at table level will set INDEXING ON at partition level for newly created  partition. The key to note is "newly created".  We will make use of this feature to implement a  performance strategy.

  The global partial index will have only one index segment. The size of the global partial index will be smaller than a full index as it will not have data of partitions where indexing is OFF. 
The local partial index will have  index segments on partitions where indexing is ON and no index segments on partitions where it is set OFF. In 11g, it was possible to make some of the index segments unuseable. however, it was not easy to maintain. It would not work with rebuilds and the index segments were created even if they are unusable. 

The query optimizer will use global/local partial index where ever available and do partition scan where indexing is OFF to get the full picture. The execution plan will have a UNION-ALL to combine the data from index range scan and partition range scan. The cost of this operation will be higher than full index range scan but lesser than a full table scan.
Below explain plan shows the UNION-ALL to combine partial index data with partition scan.



The trade off to use partial index will depend on

  1.  the cost & performance of the queries  using partial index
  2. to the improved speed of OLTP transaction. 
This is really application dependent.  
It is more favorable to use partial index when 

  • The oltp transaction rate is critical.
  • The performance of queries with partial index is not tht different .
  • The resource used to run queries with partial index does not impact the overall performance of the system.

 When you ALTER the partition level indexing from OFF to ON, the local index will be created and global index rebuild to add the partition's data.

When you have INTERVAL PARTITION  table, Oracle will automatically create the next partition when the data inserted into the table is beyond the current high value of range partition. These partitions will pickup the information from the table level.
At the partition level,  INDEXING OFF will mean that the indexes will not be created for that partition.  

  In order to reduce index contention for application with high inserts, one way is to not have any non-essential indexes on the latest partition where concurrent sessions are inserting data. Having no index on the latest partition will speedup inserts. Reports that do use these indexes will continue to use PARTIAL indexes on older partitions and a Full Table Scan (FTS) on the latest partition.
Steps below describes  how to do this.

 1) Alter current partitioned table to have indexing OFF at table level. This will ensure that all new partitions added will not automatically enable indexes. ( global and local)
 2) If the partition scheme is range partition, you can alter table to have interval partition so that you need not manually add partitions. Although this is a optional step,it is better to have partitions created automatically.
3) create or alter all nonessential indexes with the partial index clause.
4) During inserts into these partitioned tables, newly created partitions will not have partial indexes created. With interval partition, the automatically created partition will not have any partial indexes created.  it is the last partition where data is inserted and has usually has the highest contention. Now with less indexes on this partition, the inserts will run a lot faster. At the same time , the reports will still run real time with the latest data inserted. It could be slightly more expensive due to the full scan on the last partition.
5)  Have a background job or manually  turn INDEXING ON for all partitions where the heavy inserting is not happening.  This will automatically rebuild global partial indexed on that table as well as create  local partial indexes on that table partition.

That is it. This is one of the strategy you can use to speed OLTP inserts and reduce contention in 12c using PARTIAL INDEXING.
Note: although the example below is using interval range partition, partial indexing will work on other partitions as well. 
 At the time of writing this blog, there is a documented Bug 14558315 that does a FTS  instead of union-all between index scan and partition scan. 

1)  Create partitioned table. set indexing off at table. You can then set indexing on at partition level.

drop table poc_data purge;
create table
poc_data (
   start_date        DATE,
   store_id          NUMBER,
   inventory_id      NUMBER,
   qty_sold          NUMBER
)
INDEXING  OFF
PARTITION BY RANGE (start_date)
INTERVAL(NUMTOYMINTERVAL(1, 'MONTH'))
(
   PARTITION poc_data_p2 VALUES LESS THAN (TO_DATE('1-7-2007', 'DD-MM-YYYY')) INDEXING on,
   PARTITION poc_data_p3 VALUES LESS THAN (TO_DATE('1-8-2007', 'DD-MM-YYYY')) INDEXING on
);


-- create global and local index with PARTIAL clause.

create index global_partial_index on poc_data(store_id) GLOBAL INDEXING PARTIAL;
create index local_partial_index on poc_data(inventory_id) LOCAL INDEXING PARTIAL;

2) Load some data to create newer interval partition.

begin
for i in 1..144 loop
  for j in 1..10000 loop
insert into poc_data values (TO_DATE('1-8-2007', 'DD-MM-YYYY') +i,i,J,i+j);
  end loop;
commit;
end loop;
commit;
end;
/


-- check the partition indexing parameter is off as it is set to off at table level.
SQL> col table_name format a15
SQL> col partition_name format a20
SQL> select table_name, partition_name, indexing from user_tab_partitions where table_name='POC_DATA';













-- check  the local indexes are created to only partitions where indexing ON is set. and obser the leaf_blocks count for PARTIAL global index.
SQL> col index_name format a30
SQL> select  index_name, partition_name,segment_created,status,leaf_blocks from
user_ind_partitions where index_name in (select index_name from user_indexes where table_name='POC_DATA')
union all
select Index_name, 'SINGLE GLOBAL INDEX' PARTITION_NAME,'YES' SEGMENT_CREATED,status, leaf_blocks from user_indexes where table_name='POC_DATA'
and partitioned='NO';





-- Manually enable the partial index on on one of the new partitions..
alter table poc_data modify  partition SYS_P1283 indexing on ;
-- this will automatically rebuild global index and create a partial local index on that partition.
-- update statistics to update leaf_block info
begin
dbms_stats.gather_table_stats(ownname=>user, tabname=>'POC_DATA', estimate_percent=>null, cascade=> true);
end;
/

-- check the index segments and observer leaf_blocks in global nonpartitioned index has increased.










Conclusion.

Combining 12c PARTIAL INDEXING feature and INTERVAL PARTITION , you can easily automate index maintenance that could speed up performance of OTLP applications. 

Monday, December 23, 2013

In-memory option in oracle 12c

I clench  when people say oracle is playing catchup with in-memory option. Any option oracle came up was integral part of database. It takes time to build on your own, rather than buy a company like SAP did and call it its own. 

I recall Teradata gave a anti Exadata compete paper saying "Exadata is still  Oracle" database. They totally missed the point that oracle worked hard to make it transparent to end users. It is what people want. 
 Now with in-memory option, will SAP dare to call it  "In-memory option is still a Oracle database "? 

Remember, Times-ten is a Row-store in memory database and it will stay that way. In-memory option is a columnar-store format and is created ground up. There is no change of code required for any of the existing application to run on oracle in-memory option. 
 The underlying code change to make this happen cannot be a after though after HANA announcement. It is foresight, hard work and great development team that can code.