I see several similar others have much the same problem, but I have not found
any way of fixing it so far. We are running activemq 5.9.0 with medium load.
Our problem is that the number of kahadb log files is growing every day to
more than 700 files over a 3 months period.
From the kahadb debug log, we see that every queue is cleaned up normally
when messages are consumed, but the files are not cleaned up because of tx.
We also copied the production installation to a test environment with no
consumers. Still the files are not cleaned up because of tx.
If I read the log correct, there are pending transactions blocking the
cleanup. How can a situation like this occur even after copy to another
environment with no consumers, and what could possible resolutions be?
In the print from the log below, I have shortened down the list of files
since we now have around 700 files ...
2014-02-13 08:30:51,401 [eckpoint Worker] TRACE MessageDatabase
- Last update: 1080:24387211, full gc candidates set: [2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, ....]
2014-02-13 08:30:51,404 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after producerSequenceIdTrackerLocation:1067, [2, 3, 4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
45, 46, 47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, ....]
2014-02-13 08:30:51,406 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after ackMessageFileMapLocation:1080, [2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85....]
2014-02-13 08:30:51,411 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after tx range:[2:25196115, 977:16063730], [978]
....
....
any way of fixing it so far. We are running activemq 5.9.0 with medium load.
Our problem is that the number of kahadb log files is growing every day to
more than 700 files over a 3 months period.
From the kahadb debug log, we see that every queue is cleaned up normally
when messages are consumed, but the files are not cleaned up because of tx.
We also copied the production installation to a test environment with no
consumers. Still the files are not cleaned up because of tx.
If I read the log correct, there are pending transactions blocking the
cleanup. How can a situation like this occur even after copy to another
environment with no consumers, and what could possible resolutions be?
In the print from the log below, I have shortened down the list of files
since we now have around 700 files ...
2014-02-13 08:30:51,401 [eckpoint Worker] TRACE MessageDatabase
- Last update: 1080:24387211, full gc candidates set: [2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, ....]
2014-02-13 08:30:51,404 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after producerSequenceIdTrackerLocation:1067, [2, 3, 4, 5,
6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25,
26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44,
45, 46, 47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, ....]
2014-02-13 08:30:51,406 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after ackMessageFileMapLocation:1080, [2, 3, 4, 5, 6, 7, 8,
9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46,
47, 48, 49, 50, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85....]
2014-02-13 08:30:51,411 [eckpoint Worker] TRACE MessageDatabase
- gc candidates after tx range:[2:25196115, 977:16063730], [978]
....
....