shareplex oracle to jms 基础上再度集成flume测试

dongxujian

1.基于测试三十五的环境下测试（注意 post 设置为投递到队列  SHAREPLEX.Q2)

2.安装用户oracle环境变量设置：
[oracle@node2 ~]$ cat .bash_profile
# ---------------------------------------------------
# OS User:    oracle
# Application:  Oracle Database Software Owner
# Version:    Oracle 11g release 2
# ---------------------------------------------------

# Get the aliases and functions
if [ -f ~/.bashrc ]; then
   . ~/.bashrc
fi
export JAVA_HOME=/usr/java/jdk1.8.0_73
export JRE_HOME=$JAVA_HOME/jre
ORACLE_SID=orcl; export ORACLE_SID #another is dgh2
ORACLE_BASE=/u01/app/oracle; export ORACLE_BASE
ORACLE_HOME=$ORACLE_BASE/product/11.2.0/dbhome_1; export ORACLE_HOME
ORACLE_TERM=xterm; export ORACLE_TERM
NLS_DATE_FORMAT="DD-MON-YYYY HH24:MI:SS"; export NLS_DATE_FORMAT
TNS_ADMIN=$ORACLE_HOME/network/admin; export TNS_ADMIN
PATH=.

{JAVA_HOME}/bin

{PATH}

HOME/bin

ORACLE_HOME/bin
PATH=${PATH}:/usr/bin:/bin:/usr/bin/X11:/usr/local/bin
PATH=$JAVA_HOME/bin

JRE_HOME/bin

{PATH}:/u01/app/proddir/bin:/u01/app/common/oracle/bin
export PATH
LD_LIBRARY_PATH=$ORACLE_HOME/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}

ORACLE_HOME/oracm/lib
LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/lib:/usr/lib:/usr/local/lib
export LD_LIBRARY_PATH
CLASSPATH=$ORACLE_HOME/JRE
CLASSPATH=${CLASSPATH}

ORACLE_HOME/jlib
CLASSPATH=${CLASSPATH}

ORACLE_HOME/rdbms/jlib
CLASSPATH=${CLASSPATH}

ORACLE_HOME/network/jlib
export CLASSPATH
THREADS_FLAG=native; export THREADS_FLAG
export TEMP=/tmp
export TMPDIR=/tmp
# ---------------------------------------------------
# UMASK
# ---------------------------------------------------
# Set the default file mode creation mask
# (umask) to 022 to ensure that the user performing
# the Oracle software installation creates files
# with 644 permissions.
# ---------------------------------------------------
umask 022

[oracle@node2 ~]$

3.root 用户上传软件至/home/oracle目录下，然后oracle用户执行解压
tar -zxf apache-flume-1.8.0-bin.tar.gz
mv apache-flume-1.8.0-bin flume

[oracle@node2 ~]$ ls -l
total 874772
drwxr-xr-x. 11 oracle oinstall    4096 Mar 15 03:04 activemq
-rw-r--r--.  1 root root    58149484 Mar 13 02:37 apache-activemq-5.15.3-bin.tar.gz
-rw-r--r--.  1 root root    58688757 Mar 14 23:29 apache-flume-1.8.0-bin.tar.gz
drwxr-xr-x.  7 oracle oinstall    4096 Mar 15 01:09 flume
-r-xr-xr-x.  1 root root    106782720 Apr  6  2017 shareplex-8.6.6-b64-oracle110-rh-40-amd64-m64.tar
-rwxr-xr-x.  1 oracle oinstall 106779251 Jan 13  2017 SharePlex-8.6.6-b64-oracle110-rh-40-amd64-m64.tpm
-rw-r--r--.  1 oracle oinstall 132628480 Aug  1  2017 shareplex-9.0.0-b6-oracle110-rh-40-amd64-m64.tar
-rwxr-xr-x.  1 oracle oinstall 132622107 May 26  2017 SharePlex-9.0.0-b6-oracle110-rh-40-amd64-m64.tpm
-rw-r--r--.  1 oracle oinstall 150046720 Mar  7 07:11 shareplex-9.1.0-b11-oracle110-rh-40-amd64-m64.tar
-rwxr-xr-x.  1 oracle oinstall 150039141 Jan 30 12:57 SharePlex-9.1.0-b11-oracle110-rh-40-amd64-m64.tpm

4.调整/home/oracle/flume/bin目录下flume-ng文件配置：

################################
# main
################################

# set default params
FLUME_CLASSPATH=""
FLUME_JAVA_LIBRARY_PATH=""
JAVA_OPTS="-Xmx2048m"  ####加大此值的大小
LD_LIBRARY_PATH=""

opt_conf=""
opt_classpath=""
opt_plugins_dirs=""
arr_java_props=()
arr_java_props_ct=0
opt_dryrun=""

mode=$1

5.拷贝 /home/oracle/activemq/lib/目录下activemq-all-5.15.3.jar至/home/oracle/flume/lib目录下

oracle 用户拷贝

6./home/oracle/flume/conf 目录下编辑配置文件

[oracle@node2 conf]$ ls -l
total 24
-rw-r--r--. 1 oracle oinstall  642 Mar 15 08:44 a.conf
-rw-r--r--. 1 oracle oinstall 1661 Sep 15 08:26 flume-conf.properties.template
-rw-r--r--. 1 oracle oinstall 1661 Mar 15 01:10 flume-conf.properties.template.bak
-rw-r--r--. 1 oracle oinstall 1455 Sep 15 08:26 flume-env.ps1.template
-rw-r--r--. 1 oracle oinstall 1568 Mar 15 08:07 flume-env.sh
-rw-r--r--. 1 oracle oinstall 3107 Sep 15 08:26 log4j.properties
[oracle@node2 conf]$ pwd
/home/oracle/flume/conf
[oracle@node2 conf]$ cat a.conf
a.sources  = r
a.sinks =  k
a.channels  = c

a.sources.r.channels = c
a.sources.r.type  = jms
a.sources.r.initialContextFactory = org.apache.activemq.jndi.ActiveMQInitialContextFactory
a.sources.r.connectionFactory = ConnectionFactory
a.sources.r.destinationName = SHAREPLEX.Q2
a.sources.r.providerURL = tcp://localhost:61616
a.sources.r.username=admin
a.sources.r.password=admin
a.sources.r.destinationType = QUEUE

a.channels.c.type  = memory
a.channels.c.capacity  = 1000000
a.channels.c.transactionCapacity  = 100

a.sinks.k.type = file_roll
a.sinks.k.sink.directory = /tmp/flume
a.sinks.k.sink.rollInterval = 60
a.sinks.k.channel = c

[oracle@node2 conf]$

7.启动 flume
/home/oracle/flume目录下oracle用户启动
[oracle@node2 flume]$ bin/flume-ng agent --conf conf --conf-file conf/a.conf --name a  -Dflume.root.logger=INFO,console

8.确认文件是否生成

/tmp/flume 目录下（注意此目录的权限）

[oracle@node2 flume]$ pwd
/tmp/flume
[oracle@node2 flume]$ ls -ltr
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:13 1521117900455-89
-rw-r--r--. 1 oracle oinstall 7977626 Mar 15 08:54 1521117900455-9
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:14 1521117900455-90
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:15 1521117900455-91
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:16 1521117900455-92
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:17 1521117900455-93
-rw-r--r--. 1 oracle oinstall    0 Mar 15 10:18 1521117900455-94

9.查看生成的文件的内容

[oracle@node2 flume]$ tail -200 1521117900455-9
<col name="GENERATED">N</col>
<col name="SECONDARY">N</col>
<col name="NAMESPACE">1</col>
</row>
</cmd>
</tbl>
</opentarget>

<?xml version="1.0" encoding="UTF-8"?>
<?opentarget version="1.1"?>
<opentarget>
<txn id="1441765" msgIdx="29193" />
<tbl name="ACTIVEMQ.TTT">
<cmd ops="ins">
<row id="AAAVqJAAEAAAAggAAp">
<col name="OWNER">

UBLIC</col>
<col name="OBJECT_NAME">/5537b1b9_DOMAdapter</col>
<col name="OBJECT_ID">29471</col>
<col name="OBJECT_TYPE">SYNONYM</col>
<col name="CREATED">2013-08-24T11:44:17</col>
<col name="LAST_DDL_TIME">2013-08-24T11:44:17</col>
<col name="TIMESTAMP">2013-08-24:11:44:17</col>
<col name="STATUS">VALID</col>
<col name="TEMPORARY">N</col>
<col name="GENERATED">N</col>
<col name="SECONDARY">N</col>
<col name="NAMESPACE">1</col>
</row>
</cmd>
</tbl>
</opentarget>

<?xml version="1.0" encoding="UTF-8"?>
<?opentarget version="1.1"?>
<opentarget>
<txn id="1441765" msgIdx="29194" />
<tbl name="ACTIVEMQ.TTT">
<cmd ops="ins">
<row id="AAAVqJAAEAAAAggAAq">
<col name="OWNER">SYS</col>
<col name="OBJECT_NAME">/7965285_AdaptiveResultTreeImp</col>
<col name="OBJECT_ID">29472</col>
<col name="OBJECT_TYPE">JAVA CLASS</col>
<col name="CREATED">2013-08-24T11:44:17</col>
<col name="LAST_DDL_TIME">2013-08-24T11:44:17</col>
<col name="TIMESTAMP">2013-08-24:11:44:17</col>
<col name="STATUS">VALID</col>
<col name="TEMPORARY">N</col>
<col name="GENERATED">N</col>
<col name="SECONDARY">N</col>
<col name="NAMESPACE">1</col>
</row>
</cmd>
</tbl>
</opentarget>

测试完成

此时ActiveMQ 会出现入队出队消息统计信息，见截图

图文精华

shareplex oracle to jms 基础上再度集成flume测试