[sr-dev] [kamailio/kamailio] Kafka producer only tasks (PR #3951)

<!-- Kamailio Pull Request Template --> <!-- IMPORTANT: - for detailed contributing guidelines, read: https://github.com/kamailio/kamailio/blob/master/.github/CONTRIBUTING.md - pull requests must be done to master branch, unless they are backports of fixes from master branch to a stable branch - backports to stable branches must be done with 'git cherry-pick -x ...' - code is contributed under BSD for core and main components (tm, sl, auth, tls) - code is contributed GPLv2 or a compatible license for the other components - GPL code is contributed with OpenSSL licensing exception --> #### Pre-Submission Checklist <!-- Go over all points below, and after creating the PR, tick all the checkboxes that apply --> <!-- All points should be verified, otherwise, read the CONTRIBUTING guidelines from above--> <!-- If you're unsure about any of these, don't hesitate to ask on sr-dev mailing list --> - [X] Commit message has the format required by CONTRIBUTING guide - [X] Commits are split per component (core, individual modules, libs, utils, ...) - [X] Each component has a single commit (if not, squash them into one commit) - [X] No commits to README files for modules (changes must be done to docbook files in `doc/` subfolder, the README file is autogenerated) #### Type Of Change - [ ] Small bug fix (non-breaking change which fixes an issue) - [X] New feature (non-breaking change which adds new functionality) - [ ] Breaking change (fix or feature that would change existing functionality) #### Checklist: <!-- Go over all points below, and after creating the PR, tick the checkboxes that apply --> - [ ] PR should be backported to stable branches - [X] Tested changes locally - [ ] Related to issue #XXXX (replace XXXX with an open issue number) #### Description <!-- Describe your changes in detail --> Hello, I recently did some tests with kamailio's kafka module, with focus on what happens when kafka broker goes down. First thing I noticed is that kamailio won't start initially if kafka broker is down. Second thing I noticed syslog overflowing with err logs, when kafka broker is down. Third, I noticed that when it starts, kamailio queries topic metadata and waited up to 100 seconds for any answer. I've added modparams for all these three to change behavior. Updated doc for all the new modparams. I've also updated stats errors counting when exposed cfg functions failed for any reason. And some rework to stats function to accept str* instead of char* topic. Also moved librdkafka's poll() function at the begining of the cfg function logic, so polling happens even when some errors happen with the cfg function. You can view, comment on, or merge this pull request online at: https://github.com/kamailio/kamailio/pull/3951 -- Commit Summary -- * Makefile.defs: version set to 5.8.0-rc0 * pkg/kamailio/deb: version set 5.8.0~rc0 * github: set branch to 5.8 [skip ci] * pkg/docker: set branch 5.8 and refresh [skip ci] * tls: restore early init for other modules that use TLS * tls: fix restore early init * tls: fix OpenSSL 1.1.1 compatibility * OpenSSL integration: manage curl_global_init(...) used by modules * Revert ac4f1be039 - split into per module commits * core/rthreads.h: add thread executor for curl_global_init() * http_async_client: libssl thread executor for curl_global_init() * http_client: libssl thread executor for curl_global_init() * xcap_client: libssl thread executor for curl_global_init() * http_async_client: libssl refactor thread executors for curl * file_out: Add header guards * file_out: Fix leaks * tls: fix OpenSSL 1.1.1 engine keys * tls: make explicit ENGINE deprecation in OpenSSL 3 * tls: clean-up of ENGINE * tlsa: removed the map files used in the past for tls engine * tls: remove unused ENGINE define * tls: basic OpenSSL 3 support of provider keys (replaces ENGINE) * core/rthreads.h: add new option tls_threads_mode = 2 * tls: new option tls_threads_mode = 2 * etc/kamailio.cfg: document tls_threads_mode = 2 to use atfork handler * tls: NULL safety check * ndb_redis: reformat exported structures * evrexec: reformat exported structures * microhttpd: docs - note about variables available in the event route * ndb_redis: init enabled for tcp-main and postchildinit callbacks * tcpops: variables to get active connection attributes * tcpops: if connection not found, return attributes from sip msg rcv * ims_usrloc: make matching of username in contact conditional * ims_registrar_pcscf: update registered state to pending registration if contact exists * core: rthreads - init return variable * influxdbc: copy with buffer size limit * core: ppcfg - jump to end when defexp eval does not get a str val * core: select - handle ws and wss inside select_ip_port() * core: parser rr - check for rr value before serializing * acc: cdr - do not free static empty string in case of error * ctl: reset log prefix on reading ctl traffic * core: main - init local variables used for cli param parsing * presence: docs - remove extra section end tag * tls/docs: Update tls.reload docs * tls/docs: Update certificate generation docs * modules: readme files regenerated - modules ... [skip ci] * ChangeLog: added content from 5.8.0 developent * pkg/kamailio/deb: version set 5.8.0 [skip ci] * pkg/kamailio: version set 5.8.0 for rpms and alpine * Makefile.defs: version set to 5.8.0 * ChangeLog: content set for v5.8.0 * pkg/kamailio/obs: packaged README.file_out * core: set null-terminated char for modparam (backport from master) * Avoid 'qm_strnstr' defined but not used warning * topos: small rearangement by removing else after return in the true block * topos: properly handle cases of no user in contact for mode 1 * pkg/kamailio/obs: Allow builders to disable wolfssl module [skip ci] * file_out: Fix var position and initialize * pdb: Return negative carrierid for failures * pdb server: Return negative carrierid for failures. Clang format * ctl: handle int input for double storage * dispatcher: refine documentation * ims_ipsec_pcscf: TLS support in ipsec_forward() improved * acc: more log messages of failure arsing extra acc string * keepalive: proper storing of last up/down timestamps * keepalive: small formatting updates and comments to the ka dest structure * ndb_redis: clang-format for WITH_SSL code * ndb_redis: fix index on parsing sentinel address inside redisc_reconnect_server() * ndb_redis: break on finding a server via sentinel inside redisc_reconnect_server() * modules: readme files regenerated - modules ... [skip ci] * dmq: fix inter-lock on error * file_out: additional checks for silenting warnings * file_out: Cast to correct type of str * tls: Fix ‘for’ loop initial declarations error * file_out: FIx ‘for’ loop initial declarations error * gcrypt: docs - module name used for section ids * core: socket info - parse advertise ip part to get address family * core: resolve/create_srv_pref_list() - insert at correct position in list * corex: fix conditions for dns_cache modparam srv attributes * keepalive: reformat module exports structures * db_mysql: MYSQL_OPT_RECONNECT is deprecated by MySQL 8.0.34 * db_mysql: right position for define KSR_MYSQL_OPT_RECONNECT * smsops: reformat module exports structures * smsops: short reference section about variables * blst: reformat exported structures * modules: readme files regenerated - modules ... [skip ci] * pkg/kamailio: version set 5.8.1 for rpms and alpine * pkg: deb specs updated for v5.8.1 * core: xavp - added missing closing parenthesis in log messages * core: msg translator - duplicate reply via xavp params and handle errors * Makefile.defs: version set to 5.8.1 * ChangeLog: content updated for v5.8.1 * pdb server: allow carrier id 0 * db_redis: docs - refine docs regarding client certificates [skip ci] * ndb_redis: docs - refine docs regarding client certificates [skip ci] * sdpops: reformat the codecs map * core: parser/sdp - rename parameter to suggest better its role * siputils: tel2sip2() - memset to 0 pkg allocs and recompute len of new uri * ims_ipsec_pcscf: log error message made info * corex: Fix unknown af in list_sockets * rtpengine: fix pkg mem leak in send_rtpp_command() * tcp_main: Add protocol argument for searching tcp/tls connections * tcp_main: Add proto argument to tcpconn_exists function * tcp_main: Update comment docs * core/forward: Match protocol when forwarding * tcp_main: Match wss protocol * usrloc: check on db delete the return value of memchr * core: tcp - keep connection id on event route cb structure * tcpops: reworked some debug messages * core: tcp - set a global shortcut to tcp connection for event route * pv: use global tcp connection callback for event route on $conid * core: fmsg - parse loopback ip when initializing faked msg structure * regex: init globals for safer shutdown on failure * regex: allocate pcres array * regex: reformat exported structures * tcpops: removed unused variable introduced recently * modules: readme files regenerated - modules ... [skip ci] * tls: clear thread-local variables up to tls_pthreads_key_mark * core: spelling fix in comment and log message * uid_auth_db: spelling fix in documentation * ims_usrloc_scscf: reformat exported structures * ims_usrloc_scscf: enclose examples in dotted lines for better visibility * modules/statsd: ensure statsd failures do not stop execution (#3819) * ims_registrar_scscf: init variable to fix analyzer warning * ims_ipsec_pcscf: decouple IPSEC_FORWARD_USEVIA and IPSEC_TCPPORT_UEC for requests * ims_ipsec_pcscf: docs updated for ipsec_forward() flags * ims_ipsec_pcscf: updated comments for ipsec_forward() flags * ims_ipsec_pcscf: handle tls for replies * ims_registrar_scscf: fix uninitialized arguments in save() * ims_registrar_scscf: refactor save_fixup4 * ims_registrar_scscf: fix typo in function name uint_fixup * ims_registrar_scscf: free_uint_fixup: fix memleak/freeing the wrong parameter * ims_ipsec_pcscf: cascade on options to get the uri for fill_contact() * sipcapture: docs - fix default value for parameter * tm: free new buffer on drop for local requests * tm: cb flag to mark local uac on request drop * tm: cb_flags made unsigned for uac_req * pv_headers: print also the new TMCB_LOCAL_REQUEST_DROP in debug function * presence: check if the local notify was dropped in event route * uac: check if local requests are dropped and free the param * pua: check if local requests are dropped and free the param * core: parser/contact - add limit for max number of contacts * core: parser/contact - basic validation tests for contact uri * core: parse/contact: jump to error on too many contacts * ctl: removed level and function names from log messages * ctl: info log on ECONNRESET (connection reset by peer) * modules: readme files regenerated - modules ... [skip ci] * tcpops: note about execution of event routes by TCP main process * modules: readme files regenerated - modules ... [skip ci] * sipcapture: fix default DB URL docs * modules: readme files regenerated - modules ... [skip ci] * etc/kamailio.cfg: remove executable flag * tm: add mandatory Max-Forwards header to local AC * db_cluster: Update log levels * dialplan: don't call free on empty pcre pointer * tcpops: reformat exported structures * sipcapture: use local static buffers for ip address and port * db_mysql: fixed build on CentOS 7 * usrloc: delete location_attrs in db_only mode on delete_urecord * htable: error on not finding htable in pv set * htable: error on not finding htable in pv get * rtpengine: Fix extra process in pkg.stats * siputils: free head of params list for get_uri_param() * siputils: free head of params list for get_uri_param() * regex: fix pcre2 migration * geoip2: init MMDB_s global pointer and check on destroy * geoip2: do not close global MMDB_s on destroy * geoip2: rename global lock variable * geoip2: keep global structure per process * geoip2: clang format module file * geoip2: Reload database before accessing it * iims_registrar_scscf: include header for common c functions * core/resolve: Check dns_cache_init and choose appropriate functions * nathelper: fixed handle_ruri_alias_mode(1) cannot proper handle multi alias * nathelper: remove magic constant * dmq_usrloc: reformat exported structures * ims_ipsec_pcscf: Fix typo * pua_dialoginfo: Fix ruri length * modules: readme files regenerated - modules ... [skip ci] * pkg/kamailio: version set 5.8.2 for rpms and alpine * pkg: deb specs updated for v5.8.2 * Makefile.defs: version set to 5.8.2 * ChangeLog: content updated for v5.8.2 * kafka: add modparam init_without_kafka * kafka: add modparam log_without_overflow * kafka: add modparam metadata_timeout * kafka: export statistics for total/err messages * kafka: Update kfk_stats_add() to use str topic * kafka: Update doc with the new modparams -- File Changes -- M .devcontainer/devcontainer.json (2) M .github/workflows/devcontainer.yaml (2) M .github/workflows/main.yml (2) M .gitmodules (1) M ChangeLog (16562) M etc/kamailio.cfg (5) M pkg/docker (2) M pkg/kamailio/alpine/APKBUILD (2) M pkg/kamailio/deb/bionic/changelog (24) M pkg/kamailio/deb/bookworm/changelog (24) M pkg/kamailio/deb/bullseye/changelog (24) M pkg/kamailio/deb/buster/changelog (24) M pkg/kamailio/deb/debian/changelog (24) M pkg/kamailio/deb/focal/changelog (24) M pkg/kamailio/deb/jammy/changelog (24) M pkg/kamailio/deb/jessie/changelog (24) M pkg/kamailio/deb/precise/changelog (24) M pkg/kamailio/deb/sid/changelog (24) M pkg/kamailio/deb/stretch/changelog (24) M pkg/kamailio/deb/trusty/changelog (24) M pkg/kamailio/deb/wheezy/changelog (24) M pkg/kamailio/deb/xenial/changelog (24) M pkg/kamailio/obs/kamailio.spec (24) M src/Makefile.defs (4) M src/core/fmsg.c (11) M src/core/forward.h (3) M src/core/mem/q_malloc.c (2) M src/core/modparam.c (1) M src/core/msg_translator.c (18) M src/core/parser/contact/contact.c (29) M src/core/parser/parse_rr.c (24) M src/core/parser/sdp/sdp_helpr_funcs.c (14) M src/core/ppcfg.c (2) M src/core/resolve.c (60) M src/core/resolve.h (14) M src/core/rthreads.h (58) M src/core/select_core.c (12) M src/core/socket_info.c (5) M src/core/switch.c (2) M src/core/tcp_conn.h (6) M src/core/tcp_main.c (52) M src/core/ut.h (2) M src/core/xavp.c (6) M src/main.c (6) M src/modules/acc/acc_cdr.c (4) M src/modules/acc/acc_extra.c (16) M src/modules/blst/blst.c (49) M src/modules/corex/corex_mod.c (6) M src/modules/corex/corex_rpc.c (4) M src/modules/ctl/binrpc.h (15) M src/modules/ctl/fifo_server.c (1) M src/modules/ctl/io_listener.c (90) M src/modules/db_cluster/dbcl_api.c (83) M src/modules/db_mysql/km_my_con.c (21) M src/modules/db_redis/README (17) M src/modules/db_redis/doc/db_redis_admin.xml (10) M src/modules/dialplan/dp_repl.c (3) M src/modules/dispatcher/README (13) M src/modules/dispatcher/doc/dispatcher_admin.xml (9) M src/modules/dmq/notification_peer.c (6) M src/modules/dmq_usrloc/dmq_usrloc.c (44) M src/modules/evrexec/evrexec_mod.c (40) M src/modules/file_out/file_out.c (69) M src/modules/file_out/types.c (14) M src/modules/file_out/types.h (5) M src/modules/gcrypt/doc/gcrypt_admin.xml (4) M src/modules/geoip2/geoip2_mod.c (25) M src/modules/geoip2/geoip2_pv.c (81) M src/modules/geoip2/geoip2_pv.h (1) M src/modules/htable/ht_var.c (10) M src/modules/http_async_client/http_async_client_mod.c (115) M src/modules/http_client/http_client.c (23) M src/modules/ims_ipsec_pcscf/README (13) M src/modules/ims_ipsec_pcscf/cmd.c (130) M src/modules/ims_ipsec_pcscf/doc/ims_ipsec_pcscf_admin.xml (12) M src/modules/ims_registrar_pcscf/save.c (11) M src/modules/ims_registrar_scscf/ims_registrar_scscf_mod.c (55) M src/modules/ims_registrar_scscf/lookup.c (2) M src/modules/ims_registrar_scscf/userdata_parser.c (5) M src/modules/ims_usrloc_pcscf/udomain.c (58) M src/modules/ims_usrloc_scscf/README (29) M src/modules/ims_usrloc_scscf/doc/ims_usrloc_scscf_admin.xml (129) M src/modules/ims_usrloc_scscf/ims_usrloc_scscf_mod.c (78) M src/modules/influxdbc/ic.c (16) M src/modules/kafka/doc/kafka_admin.xml (64) M src/modules/kafka/kafka_mod.c (33) M src/modules/kafka/kfk.c (117) M src/modules/keepalive/keepalive.h (12) M src/modules/keepalive/keepalive_core.c (4) M src/modules/keepalive/keepalive_mod.c (56) M src/modules/microhttpd/doc/microhttpd_admin.xml (5) M src/modules/nathelper/nathelper.c (4) M src/modules/ndb_redis/README (17) M src/modules/ndb_redis/doc/ndb_redis_admin.xml (10) M src/modules/ndb_redis/ndb_redis_mod.c (128) M src/modules/ndb_redis/redis_client.c (34) M src/modules/pdb/pdb.c (14) M src/modules/presence/doc/presence_admin.xml (1) M src/modules/presence/notify.c (5) M src/modules/pua/pua.c (5) M src/modules/pua/send_publish.c (3) M src/modules/pua_dialoginfo/pua_dialoginfo.c (1) M src/modules/pv/pv_core.c (7) M src/modules/pv_headers/pv_headers.c (2) M src/modules/regex/regex_mod.c (78) M src/modules/rtpengine/rtpengine.c (7) M src/modules/sdpops/sdpops_data.c (28) M src/modules/sipcapture/README (5) M src/modules/sipcapture/doc/sipcapture_admin.xml (4) M src/modules/sipcapture/hep.c (13) M src/modules/siputils/checks.c (14) M src/modules/smsops/README (13) M src/modules/smsops/doc/smsops_admin.xml (14) M src/modules/smsops/smsops.c (47) M src/modules/statsd/statsd.c (51) M src/modules/tcpops/README (8) M src/modules/tcpops/doc/eventroutes.xml (12) M src/modules/tcpops/tcpops.c (6) M src/modules/tcpops/tcpops_mod.c (155) M src/modules/tls/README (12) M src/modules/tls/doc/rpc.xml (4) M src/modules/tls/doc/tls.xml (10) M src/modules/tls/tls_domain.c (94) D src/modules/tls/tls_map.c (213) D src/modules/tls/tls_map.h (77) M src/modules/tls/tls_mod.c (223) M src/modules/tls/tls_rand.h (2) M src/modules/tls/tls_server.c (24) D src/modules/tlsa/tls_map.c (27) D src/modules/tlsa/tls_map.h (27) M src/modules/tm/t_hooks.h (4) M src/modules/tm/t_msgbuilder.c (5) M src/modules/tm/uac.c (3) M src/modules/tm/uac.h (2) M src/modules/topos/tps_storage.c (71) M src/modules/uac/uac_reg.c (7) M src/modules/uac/uac_send.c (9) M src/modules/uid_auth_db/auth_db.xml (2) M src/modules/usrloc/ucontact.c (36) M src/modules/usrloc/urecord.c (18) M src/modules/xcap_client/xcap_client.c (5) M utils/pdbt/common.h (2) M utils/pdbt/pdb_server.c (330) M utils/pdbt/pdb_server_backend.c (21) -- Patch Links -- https://github.com/kamailio/kamailio/pull/3951.patch https://github.com/kamailio/kamailio/pull/3951.diff -- Reply to this email directly or view it on GitHub: https://github.com/kamailio/kamailio/pull/3951 You are receiving this because you are subscribed to this thread. Message ID: <kamailio/kamailio/pull/3951(a)github.com>