"It would be good to just use libwolfssl from the OS distro." - another point to take note is that wolfSSL currently does not support semantic versioning and will bump the soname even for a patch release. E.g., 5.6.4 soname version is 41, 5.6.6 soname version is 42. For distro packagers this is unexpected.
Debian has package names with the soname tacked on to handle this.