Written by Deepak Vohra Oracle Database is a relational database and Couchbase database is a NoSQL database. The data model of each is completely different. While Oracle Database is fixed schema based Couchbase database is schema-free. While Oracle Database stores data in a table with a fixed number of columns, Couchbase stores data in JSON documents, which do not have a fixed structure and different JSON documents may have different fields. Apache Flume does not provide a built-in source or sink for either of the databases. We shall use third-party Flume SQL Source for Oracle Database and third party Flume Couchbase Sink to stream Oracle Database table data to Couchbase server. Setting the Environment Installing Flume Sink for Couchbase Configuring Flume SQL Source Creating Oracle Database Table Configuring Flume Agent Running the Flume Agent Displaying Streamed Data in Couchbase server Admin Console Setting the Environment The following software is required for this tutorial. Oracle Database Couchbase Server Apache Flume Maven Java 7 Create a directory to install the software and set its permissions to global (777). mkdir /flume chmod -R 777 /flume cd /flume Download and extract Maven 3.3.x tar file. wget http://mirror.its.dal.ca/apache/maven/maven-3/3.3.3/binaries/apache-maven-3.3.3-bin.tar.gz tar xvf apache-maven-3.3.3-bin.tar.gz Download and extract Apache Flume 1.6 tar file. wget http://archive.apache.org/dist/flume/stable/apache-flume-1.6.0-bin.tar.gz tar -xvf apache-flume-1.6.0-bin.tar.gz Download Couchbase Server rpm from http://www.couchbase.com/nosql-databases/downloads. Install Couchbase Server using the rpm. The rpm version could be different. rpm -i couchbase-server-community_2.2.0_x86.rpm Set environment variables for Apache Flume, Oracle Database, Maven and Java. vi ~/.bashrc export ORACLE_HOME=/home/oracle/app/oracle/product/11.2.0/dbhome_1 export ORACLE_SID=ORCL export MAVEN_HOME=/flume/apache-maven-3.3.3-bin export FLUME_HOME=/flume/apache-flume-1.6.0-bin export FLUME_CONF=/flume/apache-flume-1.6.0-bin/conf export JAVA_HOME=/flume/jdk1.7.0_55 export PATH=$PATH:$FLUME_HOME/bin:$MAVEN_HOME/bin: $ORACLE_HOME/bin export CLASSPATH=$FLUME_HOME/lib/* Download the Couchbase Client jar and copy the jar to the Flume lib directory. wget http://central.maven.org/maven2/com/couchbase/client/couchbase-client/1.4.10/couchbase-client-1.4.10.jar cp couchbase-client-1.4.10.jar $FLUME_HOME/lib We also need to download the following jars for Couchbase server from Maven repository and copy the jars into Flume classpath (Flume lib directory). Jar File File Name Description The Apache Commons Codec package commons-codec-1.10.jar Encoder and decoder for various formats. Apache HttpComponents Core httpcore-4.4.2.jar Apache HttpComponents for the classic (blocking) I/O model. Apache HttpComponents Core (NIO) httpcore-nio-4.4.2.jar Apache HttpComponents for the non-blocking I/O model. Jettison jettison-1.3.7.jar A StAX implementation for JSON Netty/All In One netty-all-4.0.31.Final.jar A framework for network applications. Spymemcached spymemcached-2.12.0.jar A client library for memcached Commons Lang commons-lang-2.3.jar Java utility classes for classes in the java.lang package. Copy the jar files to the Flume lib directory, which adds the jars to the Flume classpath. cp commons-codec-1.10.jar $FLUME_HOME/lib cp httpcore-4.4.2.jar $FLUME_HOME/lib cp httpcore-nio-4.4.2.jar $FLUME_HOME/lib cp jettison-1.3.7.jar $FLUME_HOME/lib cp netty-all-4.0.31.Final.jar $FLUME_HOME/lib cp spymemcached-2.12.0.jar $FLUME_HOME/lib cp commons-lang-2.3.jar $FLUME_HOME/lib Create and configure a CouchbaseServer cluster from URL http://localhost:8091 . If a cluster has already been created login to Couchbase Console from the same url. The Cluster Overview tab displays the RAM Overview and the Disk Overview. The Data Buckets tab displays the data buckets. Initially the “default” bucket is the only bucket in the Couchbase server and the bucket is empty as indicated by the Item Count of 0. Click on the Documents button to list the documents in the default bucket. No document gets listed. Installing Flume Sink for Couchbase Download the source code for Flume Sink for Couchbase and compile and install the Flume sink jar file. git clone https://github.com/voidd/flume-couchbase-sink.git mvn compile mvn install Copy the jar file generated to the Flume lib directory. cp flume-couchbase-sink-1.0.0.jar $FLUME_HOME/lib Configuring Flume SQL Source The Flume source for Oracle Database is packaged with the Apache Flume distribution. Download the keedio/flume-ng-sql-source from the git hub. CD to the flume-ng-sql-source directory and compile and package the Flume SQL source to a jar file. git clone https://github.com/keedio/flume-ng-sql-source.git cd flume-ng-sql-source mvn package Create a plugins.d/sql-source/lib directory for the SQL Source plugin, set its permissions to global and copy the flume-ng-sql-source-1.3-SNAPSHOT.jar file to the lib directory. mkdir -p $FLUME_HOME/plugins.d/sql-source/lib chmod -R 777 $FLUME_HOME/plugins.d/sql-source/lib cp flume-ng-sql-source-1.3-SNAPSHOT.jar $FLUME_HOME/plugins.d/sql-source/lib Similarly, create a libext directory plugins.d/sql-source/libext for the Oracle Database JDBC jar file ojdbc6.jar , set its permissions to global (777) and copy the ojdbc6.jar to the libext directory. mkdir $FLUME_HOME/plugins.d/sql-source/libext chmod -R 777 $FLUME_HOME/plugins.d/sql-source/libext cp ojdbc6.jar $FLUME_HOME/plugins.d/sql-source/libext We also need to copy the ojdbc6.jar and flume-ng-sql-source-1.3-SNAPSHOT.jar to the Flume lib directory, which adds the jars to the runtime classpath of Flume. cp flume-ng-sql-source-1.3-SNAPSHOT.jar $FLUME_HOME/lib cp ojdbc6.jar $FLUME_HOME/lib Creating Oracle Database Table Drop the OE.WLSLOG table as it may have been created for another application and create the table with the following SQL script in SQL*Plus. DROP TABLE OE.WLSLOG; CREATE TABLE OE.WLSLOG (id INTEGER PRIMARY KEY, time_stamp VARCHAR2(4000), category VARCHAR2(4000), type VARCHAR2(4000), servername VARCHAR2(4000), code VARCHAR2(4000), msg VARCHAR2(4000)); Table OE.WLSLOG gets created. Add data to the table with the following SQL script. INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(1,'Apr-8-2014-7:06:16-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000365','Server state changed to STANDBY'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(2,'Apr-8-2014-7:06:17-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000365','Server state changed to STARTING'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(3,'Apr-8-2014-7:06:18-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000365','Server state changed to ADMIN'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(4,'Apr-8-2014-7:06:19-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000365','Server state changed to RESUMING'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(5,'Apr-8-2014-7:06:20-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000361','Started WebLogic AdminServer'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(6,'Apr-8-2014-7:06:21-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000365','Server state changed to RUNNING'); INSERT INTO OE.WLSLOG(id,time_stamp,category,type,servername,code,msg) VALUES(7,'Apr-8-2014-7:06:22-PM-PDT','Notice','WebLogicServer','AdminServer','BEA-000360','Server started in RUNNING mode'); 7 rows of data get added. An SQL query may be used to list the data added. Exit SQL*Plus after creating the table as Flume agent is not able to connect to Oracle Database if SQL*Plus is also connected. Configuring Flume Agent Create a Flume configuration file ( flume.conf ; name is arbitrary) in the FLUME_CONF directory and set the configuration for a Flume agent for a Flume source for SQL Source, Flume channel and Flume sink for Couchbase database. Configuration Property Description Value agent.sources Sets the Flume Source. sql-source agent.sinks Sets the Flume sink. couchbaseSink agent.channels Sets the Flume channel. ch1 agent.sources.sql-source.channels Sets the channel on the source. ch1 agent.channels.ch1.capacity Sets the channel capacity. 1000000 agent.channels.ch1.type Sets the channel type. memory agent.sources.sql-source.type Sets the SQL Source type class. org.keedio.flume.source.SQLSource agent.sources.sql-source.connection.url Sets the connection URL for Oracle Database. jdbc:oracle:thin:@127.0.0.1:1521:ORCL agent.sources.sql-source.user Sets the username for Oracle Database. OE agent.sources.sql-source.password Sets the password for Oracle Database. OE agent.sources.sql-source.table Sets the Oracle Database table. WLSLOG agent.sources.sql-source.columns.to.select Sets the columns to select. The * setting selects all columns. * agent.sources.sql-source.incremental.column.name Sets the incremental column name. id agent.sources.sql-source.incremental.value Sets the incremental column value to start streaming from. 0 agent.sources.sql-source.run.query.delay Sets the frequency in millisecond to poll sql source. 10000 agent.sources.sql-source.status.file.path Sets the directory path for the SQL source status file. /var/lib/flume agent.sources.sql-source.status.file.name Sets the status file. sql-source.status agent.sinks.couchbaseSink.channel Sets the channel on the Couchabse sink. ch1 agent.sinks.couchbaseSink.type Sets the sink type for Couchabse. org.apache.flume.sink.couchbase.CouchBaseSink agent.sinks.couchbaseSink.hostNames Sets the Host name/s for the Couchbase cluster http://127.0.0.1:8091/pools agent.sinks.couchbaseSink.bucketName Sets the Couchbase bucket default The flume.conf is listed: agent.channels = ch1 agent.sinks = couchbaseSink agent.sources = sql-source agent.channels.ch1.type = memory agent.channels.ch1.capacity = 1000000 agent.sources.sql-source.channels = ch1 agent.sources.sql-source.type = org.keedio.flume.source.SQLSource # URL to connect to database agent.sources.sql-source.connection.url = jdbc:oracle:thin:@127.0.0.1:1521:ORCL # Database connection properties agent.sources.sql-source.user = OE agent.sources.sql-source.password = OE agent.sources.sql-source.table = OE.WLSLOG agent.sources.sql-source.columns.to.select = * # Increment column properties agent.sources.sql-source.incremental.column.name = id # Increment value is from you want to start taking data from tables (0 will import entire table) agent.sources.sql-source.incremental.value = 0 # Query delay, each configured milisecond the query will be sent agent.sources.sql-source.run.query.delay=10000 # Status file is used to save last readed row agent.sources.sql-source.status.file.path = /var/lib/flume agent.sources.sql-source.status.file.name = sql-source.status agent.sinks.couchbaseSink.channel = ch1 agent.sinks.couchbaseSink.type =org.apache.flume.sink.couchbase.CouchBaseSink agent.sinks.couchbaseSink.hostNames =http://127.0.0.1:8091/pools agent.sinks.couchbaseSink.bucketName =default Create the directory for the status file, which contains the incremental column value that was last streamed. Any row with incremental value greater than status file value gets streamed. Set the directory permissions to global (777). Remove the status file from any previous run of Flume. sudo mkdir -p /var/lib/flume sudo chmod -R 777 /var/lib/flume cd /var/lib/flume rm sql-source.status We also need to copy the template flume-env.sh.template to flume-env.sh . cp $FLUME_HOME/conf/flume-env.sh.template $FLUME_HOME/conf/flume-env.sh Running the Flume Agent To stream data from Oracle Database to Couchbase server run the Flume agent with the following command. flume-ng agent --conf $FLUME_CONF -f $FLUME_CONF/flume.conf -n agent -Dflume.root.logger=INFO,console Flume agent gets started. Oracle Database data gets streamed to Couchbase server and the Flume agent continues to run. A more detailed output from the Flume agent is as follows. [root@localhost flume]# flume-ng agent --conf $FLUME_CONF -f $FLUME_CONF/flume.conf -n agent -Dflume.root.logger=INFO,console Info: Including Hive libraries found via () for Hive access + exec /flume/jdk1.7.0_55/bin/java -Xmx20m -Dflume.root.logger=INFO,console -cp '/flume/apache-flume-1.6.0-bin/conf:/flume/apache-flume-1.6.0-bin/lib/*:/flume/apache-flume-1.6.0-bin/plugins.d/sql-source/lib/*:/flume/apache-flume-1.6.0-bin/plugins.d/sql-source/libext/*:/lib/*' -Djava.library.path= org.apache.flume.node.Application -f /flume/apache-flume-1.6.0-bin/conf/flume.conf -n agent o.a.f.n.PollingPropertiesFileConfigurationProvider - Checking file:/flume/apache-flume-1.6.0-bin/conf/flume.conf for changes 13:38:23.976 [conf-file-poller-0] INFO o.a.f.n.PollingPropertiesFileConfigurationProvider - Reloading configuration file:/flume/apache-flume-1.6.0-bin/conf/flume.conf 13:38:24.022 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Processing:couchbaseSink 13:38:24.028 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Created context for couchbaseSink: channel 13:38:24.046 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Added sinks: couchbaseSink Agent: agent 13:38:24.054 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Processing:couchbaseSink 13:38:24.058 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Processing:couchbaseSink 13:38:24.062 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Processing:couchbaseSink 13:38:24.073 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Starting validation of configuration for agent: agent, initial-configuration: AgentConfiguration[agent] SOURCES: {sql-source={ parameters:{run.query.delay=10000, columns.to.select=*, connection.url=jdbc:oracle:thin:@127.0.0.1:1521:ORCL, incremental.value=0, channels=ch1, table=OE.WLSLOG, status.file.name=sql-source.status, type=org.keedio.flume.source.SQLSource, password=OE, user=OE, incremental.column.name=id, status.file.path=/var/lib/flume} }} CHANNELS: {ch1={ parameters:{capacity=1000000, type=memory} }} SINKS: {couchbaseSink={ parameters:{hostNames=http://127.0.0.1:8091/pools, type=org.apache.flume.sink.couchbase.CouchBaseSink, bucketName=default, channel=ch1} }} 13:38:24.145 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Created channel ch1 13:38:24.223 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Creating sink: couchbaseSink using OTHER 13:38:24.243 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Post validation configuration for agent AgentConfiguration created without Configuration stubs for which only basic syntactical validation was performed[agent] SOURCES: {sql-source={ parameters:{run.query.delay=10000, columns.to.select=*, connection.url=jdbc:oracle:thin:@127.0.0.1:1521:ORCL, incremental.value=0, channels=ch1, table=OE.WLSLOG, status.file.name=sql-source.status, type=org.keedio.flume.source.SQLSource, password=OE, user=OE, incremental.column.name=id, status.file.path=/var/lib/flume} }} CHANNELS: {ch1={ parameters:{capacity=1000000, type=memory} }} SINKS: {couchbaseSink={ parameters:{hostNames=http://127.0.0.1:8091/pools, type=org.apache.flume.sink.couchbase.CouchBaseSink, bucketName=default, channel=ch1} }} 13:38:24.265 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Channels:ch1 13:38:24.266 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Sinks couchbaseSink 13:38:24.268 [conf-file-poller-0] DEBUG o.a.flume.conf.FlumeConfiguration - Sources sql-source 13:38:24.271 [conf-file-poller-0] INFO o.a.flume.conf.FlumeConfiguration - Post-validation flume configuration contains configuration for agents: [agent] 13:38:24.277 [conf-file-poller-0] INFO o.a.f.n.AbstractConfigurationProvider - Creating channels 13:38:24.378 [conf-file-poller-0] INFO o.a.f.channel.DefaultChannelFactory - Creating instance of channel ch1 type memory 13:38:24.427 [conf-file-poller-0] INFO o.a.f.n.AbstractConfigurationProvider - Created channel ch1 13:38:24.431 [conf-file-poller-0] INFO o.a.f.source.DefaultSourceFactory - Creating instance of source sql-source, type org.keedio.flume.source.SQLSource 13:38:24.437 [conf-file-poller-0] DEBUG o.a.f.source.DefaultSourceFactory - Source type org.keedio.flume.source.SQLSource is a custom type 13:38:24.488 [conf-file-poller-0] INFO org.keedio.flume.source.SQLSource - Reading and processing configuration values for source sql-source 13:38:24.510 [conf-file-poller-0] INFO o.k.flume.source.SQLSourceHelper - Status file not created, using start value from config file 2015-11-12 13:38:26,038 (conf-file-poller-0) [INFO - org.hibernate.annotations.common.reflection.java.JavaReflectionManager. (JavaReflectionManager.java:66)] HCANN000001: Hibernate Commons Annotations {4.0.5.Final} 2015-11-12 13:38:26,148 (conf-file-poller-0) [INFO - org.hibernate.Version.logVersion(Version.java:54)] HHH000412: Hibernate Core {4.3.10.Final} 2015-11-12 13:38:26,174 (conf-file-poller-0) [INFO - org.hibernate.cfg.Environment. (Environment.java:239)] HHH000206: hibernate.properties not found 2015-11-12 13:38:26,202 (conf-file-poller-0) [INFO - org.hibernate.cfg.Environment.buildBytecodeProvider(Environment.java:346)] HHH000021: Bytecode provider name : javassist 13:38:26.383 [conf-file-poller-0] INFO o.k.flume.source.HibernateHelper - Opening hibernate session 13:38:26.961 [lifecycleSupervisor-1-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:{ file:/flume/apache-flume-1.6.0-bin/conf/flume.conf counterGroup:{ name:null counters:{file.checks=1, file.loads=1} } provider:org.apache.flume.node.PollingPropertiesFileConfigurationProvider agentName:agent } supervisoree:{ status:{ lastSeen:1447353503922 lastSeenState:IDLE desiredState:START firstSeen:1447353503922 failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@26a4ce } 13:38:26.961 [lifecycleSupervisor-1-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Status check complete 2015-11-12 13:38:27,103 (conf-file-poller-0) [WARN - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:93)] HHH000402: Using Hibernate built-in connection pool (not for production use!) 2015-11-12 13:38:27,113 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:166)] HHH000401: using driver [null] at URL [jdbc:oracle:thin:@127.0.0.1:1521:ORCL] 2015-11-12 13:38:27,131 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:175)] HHH000046: Connection properties: {user=OE, password=****} 2015-11-12 13:38:27,139 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildCreator(DriverManagerConnectionProviderImpl.java:180)] HHH000006: Autocommit mode: false 2015-11-12 13:38:27,153 (conf-file-poller-0) [INFO - org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:102)] HHH000115: Hibernate connection pool size: 20 (min=1) 13:38:29.962 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:{ file:/flume/apache-flume-1.6.0-bin/conf/flume.conf counterGroup:{ name:null counters:{file.checks=1, file.loads=1} } provider:org.apache.flume.node.PollingPropertiesFileConfigurationProvider agentName:agent } supervisoree:{ status:{ lastSeen:1447353506961 lastSeenState:START desiredState:START firstSeen:1447353503922 failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@26a4ce } 13:38:29.962 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Status check complete 2015-11-12 13:38:32,387 (conf-file-poller-0) [INFO - org.hibernate.dialect.Dialect. (Dialect.java:145)] HHH000400: Using dialect: org.hibernate.dialect.Oracle10gDialect 2015-11-12 13:38:32,798 (conf-file-poller-0) [INFO - org.hibernate.engine.transaction.internal.TransactionFactoryInitiator.initiateService(TransactionFactoryInitiator.java:62)] HHH000399: Using default transaction strategy (direct JDBC transactions) 2015-11-12 13:38:32,845 (conf-file-poller-0) [INFO - org.hibernate.hql.internal.ast.ASTQueryTranslatorFactory. (ASTQueryTranslatorFactory.java:47)] HHH000397: Using ASTQueryTranslatorFactory 13:38:32.963 [lifecycleSupervisor-1-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:{ file:/flume/apache-flume-1.6.0-bin/conf/flume.conf counterGroup:{ name:null counters:{file.checks=1, file.loads=1} } provider:org.apache.flume.node.PollingPropertiesFileConfigurationProvider agentName:agent } supervisoree:{ status:{ lastSeen:1447353509962 lastSeenState:START desiredState:START firstSeen:1447353503922 failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@26a4ce } 13:38:32.965 [lifecycleSupervisor-1-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Status check complete 13:38:33.882 [conf-file-poller-0] INFO o.a.flume.sink.DefaultSinkFactory - Creating instance of sink: couchbaseSink, type: org.apache.flume.sink.couchbase.CouchBaseSink 13:38:33.890 [conf-file-poller-0] DEBUG o.a.flume.sink.DefaultSinkFactory - Sink type org.apache.flume.sink.couchbase.CouchBaseSink is a custom type 13:38:34.024 [conf-file-poller-0] INFO o.a.f.n.AbstractConfigurationProvider - Channel ch1 connected to [sql-source, couchbaseSink] 13:38:34.048 [conf-file-poller-0] INFO org.apache.flume.node.Application - Starting new configuration:{ sourceRunners:{sql-source=PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } }} sinkRunners:{couchbaseSink=SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@849039 counterGroup:{ name:null counters:{} } }} channels:{ch1=org.apache.flume.channel.MemoryChannel{name: ch1}} } 13:38:34.066 [conf-file-poller-0] INFO org.apache.flume.node.Application - Starting Channel ch1 13:38:34.077 [conf-file-poller-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Supervising service:org.apache.flume.channel.MemoryChannel{name: ch1} policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@f8f6d0 desiredState:START 13:38:34.083 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:org.apache.flume.channel.MemoryChannel{name: ch1} supervisoree:{ status:{ lastSeen:null lastSeenState:null desiredState:START firstSeen:null failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@f8f6d0 } 13:38:34.083 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - first time seeing org.apache.flume.channel.MemoryChannel{name: ch1} 13:38:34.083 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Want to transition org.apache.flume.channel.MemoryChannel{name: ch1} from IDLE to START (failures:0) 13:38:34.117 [lifecycleSupervisor-1-1] INFO o.a.f.i.MonitoredCounterGroup - Monitored counter group for type: CHANNEL, name: ch1: Successfully registered new MBean. 13:38:34.118 [lifecycleSupervisor-1-1] INFO o.a.f.i.MonitoredCounterGroup - Component type: CHANNEL, name: ch1 started 13:38:34.119 [conf-file-poller-0] INFO org.apache.flume.node.Application - Starting Sink couchbaseSink 13:38:34.119 [conf-file-poller-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Supervising service:SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@849039 counterGroup:{ name:null counters:{} } } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@354005 desiredState:START 13:38:34.120 [lifecycleSupervisor-1-2] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@849039 counterGroup:{ name:null counters:{} } } supervisoree:{ status:{ lastSeen:null lastSeenState:null desiredState:START firstSeen:null failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@354005 } 13:38:34.130 [lifecycleSupervisor-1-2] DEBUG o.a.f.lifecycle.LifecycleSupervisor - first time seeing SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@849039 counterGroup:{ name:null counters:{} } } 13:38:34.130 [lifecycleSupervisor-1-2] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Want to transition SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@849039 counterGroup:{ name:null counters:{} } } from IDLE to START (failures:0) 13:38:34.142 [conf-file-poller-0] INFO org.apache.flume.node.Application - Starting Source sql-source 13:38:34.143 [conf-file-poller-0] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Supervising service:PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@119ddb8 desiredState:START 13:38:34.144 [lifecycleSupervisor-1-1] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Status check complete 13:38:34.174 [lifecycleSupervisor-1-2] INFO o.a.f.sink.couchbase.CouchBaseSink - Starting couchbaseSink... 13:38:34.144 [lifecycleSupervisor-1-4] DEBUG o.a.f.lifecycle.LifecycleSupervisor - checking process:PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } } supervisoree:{ status:{ lastSeen:null lastSeenState:null desiredState:START firstSeen:null failures:0 discard:false error:false } policy:org.apache.flume.lifecycle.LifecycleSupervisor$SupervisorPolicy$AlwaysRestartPolicy@119ddb8 } 13:38:34.174 [lifecycleSupervisor-1-4] DEBUG o.a.f.lifecycle.LifecycleSupervisor - first time seeing PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } } 13:38:34.175 [lifecycleSupervisor-1-4] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Want to transition PollableSourceRunner: { source:org.keedio.flume.source.SQLSource{name:sql-source,state:IDLE} counterGroup:{ name:null counters:{} } } from IDLE to START (failures:0) 13:38:34.178 [lifecycleSupervisor-1-4] INFO org.keedio.flume.source.SQLSource - Starting sql source sql-source ... 13:38:34.188 [lifecycleSupervisor-1-4] INFO o.a.f.i.MonitoredCounterGroup - Monitored counter group for type: SOURCE, name: SOURCESQL.sql-source: Successfully registered new MBean. 13:38:34.189 [lifecycleSupervisor-1-4] INFO o.a.f.i.MonitoredCounterGroup - Component type: SOURCE, name: SOURCESQL.sql-source started 13:38:34.190 [lifecycleSupervisor-1-2] INFO o.a.f.i.MonitoredCounterGroup - Monitored counter group for type: SINK, name: couchbaseSink: Successfully registered new MBean. 13:38:34.190 [lifecycleSupervisor-1-2] INFO o.a.f.i.MonitoredCounterGroup - Component type: SINK, name: couchbaseSink started 13:38:34.220 [lifecycleSupervisor-1-4] DEBUG o.a.f.lifecycle.LifecycleSupervisor - Status check complete 13:40:28.427 [agent-shutdown-hook] INFO o.a.f.sink.couchbase.CouchBaseSink - CouchBase sink {} stopping 2015-11-12 13:40:28.462 INFO com.couchbase.client.CouchbaseConnection: Shut down Couchbase client 2015-11-12 13:40:28.552 INFO com.couchbase.client.ViewConnection: I/O reactor terminated 13:40:28.567 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Component type: SINK, name: couchbaseSink stopped 13:40:28.571 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.start.time == 1447353514190 13:40:28.573 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.stop.time == 1447353628567 13:40:28.574 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.batch.complete == 0 13:40:28.577 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.batch.empty == 12 13:40:28.579 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.batch.underflow == 1 13:40:28.582 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.connection.closed.count == 1 13:40:28.583 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.connection.creation.count == 1 13:40:28.589 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.connection.failed.count == 0 13:40:28.590 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.event.drain.attempt == 7 13:40:28.590 [agent-shutdown-hook] INFO o.a.f.i.MonitoredCounterGroup - Shutdown Metric for type: SINK, name: couchbaseSink. sink.event.drain.sucess == 7 The row index for the incremental column gets updated to 7, which is the number of rows streamed. Displaying Streamed Data in Couchbase server Admin Console The data streamed to Couchbase server may be displayed in Couchbase Console at URL http://localhost:8091 . The Item Count is listed as 8 for the “default” bucket. Click on the Documents button to display the documents. The eight documents get listed. Only seven documents were streamed and the eighth document is docCounter for the maximum index in the document name. The document names start with log and are suffixed with _i in which i is the index from 0 to 6 for the 7 documents. In this tutorial we streamed Oracle Database table data to Couchbase server using Apache Flume.
↧