#### Pre-Submission Checklist - [x] Commit message has the format required by CONTRIBUTING guide - [x] Commits are split per component (core, individual modules, libs, utils, ...) - [x] Each component has a single commit (if not, squash them into one commit) - [x] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated)
#### Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality)
#### Checklist: - [ ] PR should be backported to stable branches - [x] Tested changes locally - [x] Related to issue #3667
#### Description Allow creating Record-Route and Via header using destination address and port in the haproxy protocol header.
PR created behalf @ivanuschak You can view, comment on, or merge this pull request online at:
https://github.com/kamailio/kamailio/pull/3731
-- Commit Summary --
* core: extented haproxy protocol parser * rr: extented haproxy protocol suppport * path: extented haproxy protocol suppport * websocket: extented haproxy protocol suppport * siptrace: extented haproxy protocol suppport
-- File Changes --
M src/core/forward.h (4) M src/core/msg_translator.c (137) M src/core/parser/msg_parser.h (1) M src/core/receive.c (6) M src/core/receive.h (3) M src/core/tcp_conn.h (1) M src/core/tcp_main.c (22) M src/core/tcp_read.c (20) M src/modules/path/path.c (2) M src/modules/rr/loose.c (4) M src/modules/rr/record.c (6) M src/modules/siptrace/siptrace.c (89) M src/modules/websocket/ws_frame.c (9) M src/modules/websocket/ws_handshake.c (3)
-- Patch Links --
https://github.com/kamailio/kamailio/pull/3731.patch https://github.com/kamailio/kamailio/pull/3731.diff
We work to clean warning ``` CC (gcc) [kamailio] core/daemonize.o CC (gcc) [kamailio] core/msg_translator.o core/msg_translator.c: In function ‘lumps_len’: core/msg_translator.c:558:6: warning: variable ‘recv_port_no’ set but not used [-Wunused-but-set-variable] int recv_port_no = 0; ^~~~~~~~~~~~ core/msg_translator.c: In function ‘process_lumps’: core/msg_translator.c:974:6: warning: variable ‘recv_port_no’ set but not used [-Wunused-but-set-variable] int recv_port_no = 0; ^~~~~~~~~~~~ CC (gcc) [kamailio] core/fmsg.o ```
@miconda commented on this pull request.
@@ -204,6 +204,10 @@ static inline int msg_send_buffer(
goto error; }
+ if(!dst->id) { + dst->id = con->id; + } +
Any more details about why this is needed? I cannot easy relate it to the commit message.
@@ -828,7 +829,22 @@ static inline int lumps_len(
send_proto_id = send_sock->proto; } /* init recv_address_str, recv_port_str & recv_port_no */ - if(msg->rcv.bind_address) { + if (ksr_tcp_accept_haproxy && msg->rcv.proto_reserved2) { + if (msg->rcv.bind_address->proto == PROTO_TLS && msg->rcv.bind_address != NULL && msg->rcv.bind_address->useinfo.address_str.len > 0) { + recv_address_str = &msg->rcv.bind_address->useinfo.address_str;
It should be formatted with clang-format in order to match the coding style.
@@ -2845,6 +2876,47 @@ int branch_builder(unsigned int hash_index,
return size; }
+static int is_haproxy(struct dest_info *send_info, struct receive_info *haproxy_rcv) {
Can you add some comment to describe the purpose of the function? The name is rather generic, from C code point of view it looks like it tries to figure out if the target via via haproxy...
@@ -396,6 +396,7 @@ typedef struct sip_msg
char *unparsed; /*!< here we stopped parsing*/
struct receive_info rcv; /*!< source & dest ip, ports, proto a.s.o*/ + struct receive_info haproxy_rcv; /*!< source & dest ip, ports, proto a.s.o for the message from haproxy*/
Traffic via haproxy is not very common, does it make sense to add a full value-structure for every message instead of a pointer that is allocated only when it is the case. Practically the size of message increases for common traffic over direct UDP/TCP/TLS/SCTP/WebSocket. On the other hand, cloning in shm has to be done if allocated. This is just to ponder on what would be better...
@@ -1843,13 +1843,17 @@ inline static int handle_io(struct fd_map *fm, short events, int idx)
* handle_io might decide to del. the new connection => * must be in the list */ tcpconn_listadd(tcp_conn_lst, con, c_next, c_prev); - t = get_ticks_raw(); - con->timeout = t + S_TO_TICKS(TCP_CHILD_TIMEOUT); - /* re-activate the timer */ - con->timer.f = tcpconn_read_timeout; - local_timer_reinit(&con->timer); - local_timer_add(&tcp_reader_ltimer, &con->timer, - S_TO_TICKS(TCP_CHILD_TIMEOUT), t); + if(con->rcv.proto_reserved2 && con->type == PROTO_WSS) { + //not setting up timers for haproxy WSS connections
The patch is quite consistent to important parts of the core. It has to be reviewed properly, so far I did a quick walk through, other developers are welcome to jump in.
@ivanuschak commented on this pull request.
@@ -204,6 +204,10 @@ static inline int msg_send_buffer(
goto error; }
+ if(!dst->id) { + dst->id = con->id; + } +
This change is needed for siptrace. Below in this file at the line 334 `dst` is passed as an argument to the `SREV_NET_DATA_SENT` related function call.
If the tcp connection ID is assigned to `dst` param then the tcp_connection search is correct and we have a proper siptrace with HAPROXY IP addresses (IP address of client and IP address of HAPROXY) , otherwise tcp connection search is based on real IP addresses (IP addresses of haproxy and IP address of kamailio) and in this case the siptrace contains not the correct IP-addresses.
@ivanuschak commented on this pull request.
@@ -2845,6 +2876,47 @@ int branch_builder(unsigned int hash_index,
return size; }
+static int is_haproxy(struct dest_info *send_info, struct receive_info *haproxy_rcv) {
The function goal is as follows: 1. it retrieves the tcp_connetion_id from the `dest_info` parameter 2. if this TCP connection is a haproxy connection, then the `haproxy_rcv` param is filled with the corresponding haproxy data and the function returns non-zero value 3. otherwise if the TCP connection is not a connection to haproxy then the function returns zero value.
We will add the comments for this function in the source code.
@ivanuschak commented on this pull request.
@@ -396,6 +396,7 @@ typedef struct sip_msg
char *unparsed; /*!< here we stopped parsing*/
struct receive_info rcv; /*!< source & dest ip, ports, proto a.s.o*/ + struct receive_info haproxy_rcv; /*!< source & dest ip, ports, proto a.s.o for the message from haproxy*/
Thanks for highlighting that. We will check what would be less expensive in terms of resource usage - to have a full value-structure, or to have a pointer that needs to be cloned in shm.
@ivanuschak commented on this pull request.
@@ -1843,13 +1843,17 @@ inline static int handle_io(struct fd_map *fm, short events, int idx)
* handle_io might decide to del. the new connection => * must be in the list */ tcpconn_listadd(tcp_conn_lst, con, c_next, c_prev); - t = get_ticks_raw(); - con->timeout = t + S_TO_TICKS(TCP_CHILD_TIMEOUT); - /* re-activate the timer */ - con->timer.f = tcpconn_read_timeout; - local_timer_reinit(&con->timer); - local_timer_add(&tcp_reader_ltimer, &con->timer, - S_TO_TICKS(TCP_CHILD_TIMEOUT), t); + if(con->rcv.proto_reserved2 && con->type == PROTO_WSS) { + //not setting up timers for haproxy WSS connections
Please pay attention at this particular change - from line 1846 to line 1856.
The purpose of this change is to keep the WSS connection connected. When testing this haproxy change without this particular piece of change (from line 1846 to line 1856) we observed that kamailio was dropping WSS connection by timer, and this drop looked strange, because it happened in about 5-7 seconds after the connection was established. One of the requirement for haproxy was to keep TLS/WSS connection established during all the dialog and not to drop it without any important reason. I tried to find how to make it not to be dropped by timer, but found the only way is to not to set the timer up for WSS. Also the drop cause is still not cleat for me. I don't think this is a good solution, but I was not able to find a better one. I would appreciate if you check this change attentively. If needed I can provide with the logs where kamailio drops WSS connection.
I was thinking. Could we have something generic that abstracts the rcv parameter of the message?
Indeed, it would be good to try to find a more generic way to approach this kind of needs to store intermediary hop address. The patch is rather consistent for the core part and couldn't assert the impact so far.
@sergey-safarov pushed 5 commits.
33087b06c578679d9c54c90dd5a871306fcc4036 core, rr, path: updated relaying logic when haproxy is used 8c4627555f5b6eed802238a6a24c0579d1ce48f0 fixed transport via haproxy 43452b75e45303d4a00078730256246ffeebbcd7 core: updated TCP connection selection for haproxy and Via header processing 7613b5171ec1d1c77bcd46a368937182cde3501f siptrace: fixed siptrace when tcp haproxy is used 96795acd315c423aa4b5186151be7cffdba71cbc siprec for haproxy fix
@sergey-safarov pushed 1 commit.
90e2cfc1eb90b881e021e85f60aaf5551d69f286 WIP: format-1
@sergey-safarov pushed 1 commit.
f8f0b7198badfac6741fbfdbd202cdf0043ad57c WIP: format-2
@sergey-safarov pushed 1 commit.
d81c1fed7233b8117998022bd5725e920078c753 WIP: format-3
@sergey-safarov pushed 1 commit.
361326d0827cd795a3d276e4b67a37214d78c87a WIP: format-4
@sergey-safarov pushed 5 commits.
bf71af1811710c06a425d3e45245960b25b87922 core, rr, path: updated relaying logic when haproxy is used 383015a2c6fcd23fb1ebac1e1ae44839a616d3d8 fixed transport via haproxy b1496c493208f62bc784a699eb3e9ac2be471691 core: updated TCP connection selection for haproxy and Via header processing be77bbafad54a509a7d0ed40fad507e9ecb2bd52 siptrace: fixed siptrace when tcp haproxy is used 4e70f50c8d644426e204ce533cff9f7c18d95740 siprec for haproxy fix
@sergey-safarov pushed 3 commits.
929854ef7cb194b138c2e9a28b98f920f17d6bc0 core: updated TCP connection selection for haproxy and Via header processing 25960ff1873ece98172c660c96c9f61140e044f8 siptrace: fixed siptrace when tcp haproxy is used 444e42229b2ffeeb34547894966708047b2b90fa siprec for haproxy fix
This PR is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.
Closed #3731.
Reopened #3731.
@sergey-safarov do you like to continue to work on this? I think there have been already several discussions about the PR.
Yes, we prefer to merge this. @ivanuschak continues to work on this to allow dynamic memory allocation, but this has not been implemented.
This PR is stale because it has been open 6 weeks with no activity. Remove stale label or comment or this will be closed in 2 weeks.
Closed #3731.