Hi,
I'm attempting to use Flume to send data using Avro between two servers. The process is running perfectly fine until I started to use SSL encryption. I'm sending data from server A to server B.
Say on server B I make a keystore using:
keytool -genkey -alias serverB -keyalg RSA -keystore keystore.jks -keysize 2048
Then get a certificate using
keytool -export -alias serverB -file serverB.crt -keystore keystore.jks
Then on server A I import this certificate to the truststore:
keytool -import -file serverB.crt -alias serverB -keystore trust.store
On server A I change the conf to include:
agent_foo.sinks.avro-forward-sink.ssl = true
agent_foo.sinks.avro-forward-sink.truststore = trust.store
agent_foo.sinks.avro-forward-sink.truststore-password = serverApassword
On server B I change the conf to include:
a1.sources.sr1.ssl = true
a1.sources.sr1.keystore = keystore.jks
a1.sources.sr1.keystore-password = serverBpassword
When I start the agents, I can see that a connection was opened between server A and server B.
When I try to send data between them, I get this exception on server B:
2014-06-13 12:12:28,301 (pool-7-thread-3) [WARN - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] Unexpected exception from downstream.
org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected: 1124139008 items! Connection closed.
at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167)
at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Can anyone help me with this exception? I expect that I may have created the certificates incorrectly.
Regards,
Guy Needham | Data Discovery
Virgin Media | Enterprise Data, Design & Management
Bartley Wood Business Park, Hook, Hampshire RG27 9UP
D 01256 75 3362
I welcome VSRE emails. Learn more at http://vsre.info/
I'm attempting to use Flume to send data using Avro between two servers. The process is running perfectly fine until I started to use SSL encryption. I'm sending data from server A to server B.
Say on server B I make a keystore using:
keytool -genkey -alias serverB -keyalg RSA -keystore keystore.jks -keysize 2048
Then get a certificate using
keytool -export -alias serverB -file serverB.crt -keystore keystore.jks
Then on server A I import this certificate to the truststore:
keytool -import -file serverB.crt -alias serverB -keystore trust.store
On server A I change the conf to include:
agent_foo.sinks.avro-forward-sink.ssl = true
agent_foo.sinks.avro-forward-sink.truststore = trust.store
agent_foo.sinks.avro-forward-sink.truststore-password = serverApassword
On server B I change the conf to include:
a1.sources.sr1.ssl = true
a1.sources.sr1.keystore = keystore.jks
a1.sources.sr1.keystore-password = serverBpassword
When I start the agents, I can see that a connection was opened between server A and server B.
When I try to send data between them, I get this exception on server B:
2014-06-13 12:12:28,301 (pool-7-thread-3) [WARN - org.apache.avro.ipc.NettyServer$NettyServerAvroHandler.exceptionCaught(NettyServer.java:201)] Unexpected exception from downstream.
org.apache.avro.AvroRuntimeException: Excessively large list allocation request detected: 1124139008 items! Connection closed.
at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decodePackHeader(NettyTransportCodec.java:167)
at org.apache.avro.ipc.NettyTransportCodec$NettyFrameDecoder.decode(NettyTransportCodec.java:139)
at org.jboss.netty.handler.codec.frame.FrameDecoder.callDecode(FrameDecoder.java:286)
at org.jboss.netty.handler.codec.frame.FrameDecoder.messageReceived(FrameDecoder.java:208)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:268)
at org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:255)
at org.jboss.netty.channel.socket.nio.NioWorker.read(NioWorker.java:94)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.processSelectedKeys(AbstractNioWorker.java:364)
at org.jboss.netty.channel.socket.nio.AbstractNioWorker.run(AbstractNioWorker.java:238)
at org.jboss.netty.channel.socket.nio.NioWorker.run(NioWorker.java:38)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:897)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:919)
at java.lang.Thread.run(Thread.java:738)
Can anyone help me with this exception? I expect that I may have created the certificates incorrectly.
Regards,
Guy Needham | Data Discovery
Virgin Media | Enterprise Data, Design & Management
Bartley Wood Business Park, Hook, Hampshire RG27 9UP
D 01256 75 3362
I welcome VSRE emails. Learn more at http://vsre.info/