MQTT Apollo Unable to publish when to remote client when client reconnects

I have a server application which is using embedded Jetty 9 and the Apollo
MQTT broker. I use the Paho MQTT client to talk to the broker from the
server. A remote client also uses the Paho MQTT client to talk to the
broker. The /remote/ connections use mutual TLS.

It works fine. I bring up the server, connect to the broker. The client
connects to the server and the broker. The server is able to publish
messages to the client and vice-versa.

The problem occurs when I shut down the remote client and then restart the
client. I save state so all the client does is reconnect and the server gets
signaled of the reconnect via the broker. However, I cannot publish messages
to the client anymore. Well, at least for a while. After about a minute, I
am suddenly able to publish. But all messages I attempted to publish before
I was suddenly able to send are lost. Since the QoS is 2 I would not expect
the loss. But mostly I would expect to be able to publish as soon as the
client reconnects.

I am not sure if this is a Paho client misconfiguration or an MQTT broker
misconfiguration. Anyone have any idea why the ability to publish is delayed
for about a minute on remote client reconnect?

(I have the same issue if it is a PC client or Android client ... well they
are using the same client code!)

Thanks for any ideas.