Too Many Open Files Broker Error

Hi Guys,

I’m seeing the following errors from the 0.8.1.1 broker. This occurs most often on the Controller machine. Then the controller process crashes, and the controller bounces to other machines, which causes those machines to crash. Looking at the file descriptors being held by the process, it’s only around 4000 or so(looking at . There aren’t a whole lot of connections in TIME_WAIT states, and I’ve increased the ephemeral port range to “16000 – 64000” via "/proc/sys/net/ipv4/ip_local_port_range”. I’ve written a Java test program to see how many sockets and files I can open. The socket is definitely limited by the ephemeral port range, which was around 22K at the time. But I
can open tons of files, since the open file limit of the user is set to 100K.

So given that I can theoretically open 48K sockets and probably 90K files, and I only see around 4K total for the Kafka broker, I’m really confused as to why I’m seeing this error. Is there some internal Kafka limit that I don’t know about?

Paul Lung

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

at kafka.network.Acceptor.accept(SocketServer.scala:200)

at kafka.network.Acceptor.run(SocketServer.scala:154)

at java.lang.Thread.run(Thread.java:679)

[2014-07-08 13:07:21,534] ERROR Error in acceptor (kafka.network.Acceptor)

java.io.IOException: Too many open files

at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)

at sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:163)

at kafka.network.Acceptor.accept(SocketServer.scala:200)

at kafka.network.Acceptor.run(SocketServer.scala:154)

at java.lang.Thread.run(Thread.java:679)

[2014-07-08 13:07:21,563] ERROR [ReplicaFetcherThread-3-2124488], Error for partition [bom__021____active_80__32__mini____activeitem_lvs_qn,0] to broker 2124488:class kafka.common.NotLeaderForPartitionException (kafka.server.ReplicaFetcherThread)

[2014-07-08 13:07:21,558] FATAL [Replica Manager on Broker 2140112]: Error writing to highwatermark file: (kafka.server.ReplicaManager)

java.io.FileNotFoundException: /ebay/cronus/software/cronusapp_home/kafka/kafka-logs/replication-offset-checkpoint.tmp (Too many open files)

at java.io.FileOutputStream.open(Native Method)

at java.io.FileOutputStream.<init>(FileOutputStream.java:209)

at java.io.FileOutputStream.<init>(FileOutputStream.java:160)

at java.io.FileWriter.<init>(FileWriter.java:90)

at kafka.server.OffsetCheckpoint.write(OffsetCheckpoint.scala:37)

at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:447)

at kafka.server.ReplicaManager$$anonfun$checkpointHighWatermarks$2.apply(ReplicaManager.scala:444)

at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)

at scala.collection.immutable.Map$Map1.foreach(Map.scala:109)

at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771)

at kafka.server.ReplicaManager.checkpointHighWatermarks(ReplicaManager.scala:444)

at kafka.server.ReplicaManager$$anonfun$1.apply$mcV$sp(ReplicaManager.scala:94)

at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100)

at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)

at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)

at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)

at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)

at java.lang.Thread.run(Thread.java:679)