We are seeing a log of CHANNEL_ERROR - expected 'channel.open' 504

[FlaviuRadulescu]

Hi,

We are seeing a lot of Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504 in an production environment.

What could be the cause?

amqproxy-1  | 2025-10-16T17:24:19.643850Z  INFO amq_proxy.channel_pool[remote_address: "redacted:45872"] Adding upstream connection
amqproxy-1  | 2025-10-16T17:24:36.773968Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:36:48.494735Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:38:22.619527Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:47:07.802517Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:48:09.282512Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T17:48:31.041006Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:08:12.175095Z  INFO amq_proxy.channel_pool[remote_address: "redacted:46308"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:11:08.115664Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:17:25.878805Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:18:13.484713Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:41:09.148686Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:42:30.450506Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:42:30.662506Z  INFO amq_proxy.channel_pool[remote_address: "redacted:60778"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:45:22.076712Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:47:09.479162Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:47:09.875097Z  INFO amq_proxy.channel_pool[remote_address: "redacted:34560"] Adding upstream connection
amqproxy-1  | 2025-10-16T18:53:07.727178Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:53:46.052101Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T18:58:16.656650Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T19:01:13.753575Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504
amqproxy-1  | 2025-10-16T19:03:32.973562Z ERROR amq_proxy.upstream[remote_address: "redacted:5671"] Upstream closed connection: CHANNEL_ERROR - expected 'channel.open' 504

[kickster97]

When did this start? Can you give us some information about what version you are running, and your client set up?

[FlaviuRadulescu]

we are using the amqproxy:v2.0.4 version, we saw the first logs around a month ago, but we are seeing these more and more happening. not sure if is somehow related to more traffic on our side (it should not be the case) but everytime this is happening we lost that message that is pushed to the queue and in the app (php) we don't get any exception, so we are not aware that the mssage was lost.

we tried to restart the amqproxy and we see the same behavior after restart. what could be the cause?

[markusgust]

try using amqproxy:v3.0.2, it should now close downstream connections if upstream closes, and handle reconnections better. feel free to report back with result.

[FahdAlShalhoub]

Hi Amqproxy Contributors.

We're facing the same issue and tried upgrading to amqproxy:v3.0.2 but the issue is still occurring.

For context we're using CloudAmqp and running our Amqproxy as a daemonset in a GKE Autopilot Cluster.

Thank you so much for your generous contributions.