Is there any particular reason for why we always fetch docbook style sheets over HTTP when running "make README" ?
I can tell you that it doesn't work well with lousy connections. The result is an empty README file - just a few blank lines.
Maybe we should consider storing the versions we use in the docbook directory.
/O
On Thu, Oct 15, 2009 at 4:29 PM, Juha Heinanen jh@tutpro.com wrote:
I just tried to build a couple READMEs in modules and both files with docbook 4.2 and 4.4 seem to build fine on Debian with docbook-xsl package installed.
If I uninstall the package then the build system fetches the XSL stylesheets from the net.
Olle, if you have the XSL stylesheets in a non-standard location then modify the file docbook/catalog.xml so that the build system can find them. There is a small example (commented out) which points to a directory in my home directory.
If you are no debian then just install package debian-xsl.
-- Jan
On Fri, Oct 16, 2009 at 4:30 PM, Olle E. Johansson oej@edvina.net wrote:
In that case simply download the stylesheets somewhere and add the location to the catalog file in sip-router/docbook/catalog.xml (but do not commit, this should be your local change). Then it should work.
Had very poor connections during Astricon, so I could not fix this. Thanks for the explanation, we need to store it somewhere...
By the way, the docbook build system should not overwrite READMEs if the stylesheets are missing or if you terminate the build process, that's a bug. I will fix it as soon as I am finished with some other debugging I am doing right now.
-- Jan
On Donnerstag, 15. Oktober 2009, Olle E. Johansson wrote:
Hello Olle,
it should be possible to to specify a catalog.xml file that points to the stylesheets on the localhost host, i think. Perhaps there is an error in the existing catalog.xml file?
Yes, i noticed the missing READMEs, for tm and dialplan it still missing. ;)
Regards,
Henning
On Thu, Oct 15, 2009 at 4:26 PM, Olle E. Johansson oej@edvina.net wrote:
This should not happen if you have a local copy of docbook xsl stylesheets on your computer and those stylesheets can be located using the catalog file in sip-router/docbook/catalog.xml.
The catalog file contains references for Docbook 4.3, have you changed the docbook version number in documentation files perhaps? If yes then this could be the reason why the build system tries to download XSL stylesheets from the internet.
The whole system works as follows:
1) All documentation files refer to docbook using public identifiers. Public identifiers use HTTP URIs as identifiers: http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd
2) If you try to compile a docbook file into README, the toochain needs to first convert the docbook file to HTML and then convert it to a plaintext file with lynx. To convert the docbook file into HTML, it needs Docbook XSLT stylesheets.
3) The processor tool consults sip-router/docbook/catalog.xml file for instructions where to find the XSLT stylesheets. The catalog file refers to /etc/xml/catalog which is a system wide catalog file. If you have the XSLT stylesheets installed on your computer then they are registered in /etc/xml/catalog and thus the tool will use the local copy and will not download it from the internet.
4) If a local copy of XSLT stylesheets cannot be found in the catalog then it will simply try to download it using the HTTP uri present in the documentation file.
If you have a local copy of XSLT stylesheets for docbook (in debian they are in the package docbook-xslt, docbook DTD files are in docbook-xml) then there is either problem in the catalog file in sip-router/docbook/catalog.xml or the source documentation files are referring to a wrong version of docbook.
Which documentation files are causing problems?
-- Jan
Checking modules directory I see that we have both 4.2 and 4.4 used. Maybe we should standardize and make sure that we have the 4.4 files referenced.
/O
agave:modules olle$ grep 'OASIS//DTD' */doc/*.xml auth_identity/doc/auth_identity.xml:<!DOCTYPE section PUBLIC "-// OASIS//DTD DocBook XML V4.2//EN" auth_radius/doc/auth_radius.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" auth_radius/doc/auth_radius_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS// DTD DocBook XML V4.4//EN" avpops/doc/avpops.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" avpops/doc/avpops_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" carrierroute/doc/carrierroute.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" carrierroute/doc/carrierroute_admin.xml:<!DOCTYPE book PUBLIC "-// OASIS//DTD DocBook XML V4.4//EN" carrierroute/doc/carrierroute_db.xml:<!DOCTYPE book PUBLIC "-//OASIS// DTD DocBook XML V4.4//EN" cfg_db/doc/cfg_db.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" cfg_rpc/doc/cfg_rpc.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" cfg_rpc/doc/rpc.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" ctl/doc/ctl.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4// EN" ctl/doc/params.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" ctl/doc/rpc.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4// EN" db_berkeley/doc/db_berkeley.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" db_berkeley/doc/db_berkeley_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS// DTD DocBook XML V4.4//EN" db_flatstore/doc/flatstore.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" db_flatstore/doc/functions.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" db_flatstore/doc/params.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" dialplan/doc/dialplan.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" dialplan/doc/dialplan_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" dialplan/doc/dialplan_devel.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" enum/doc/enum.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" enum/doc/enum_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" iptrtpproxy/doc/iptrtpproxy.xml:<!DOCTYPE section PUBLIC "-//OASIS// DTD DocBook XML V4.2//EN" lcr/doc/lcr.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4// EN" lcr/doc/lcr_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" mediaproxy/doc/mediaproxy.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" mediaproxy/doc/mediaproxy_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS// DTD DocBook XML V4.4//EN" mi_rpc/doc/mi_rpc.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" mi_rpc/doc/mi_rpc_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" misc_radius/doc/misc_radius.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" misc_radius/doc/misc_radius_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS// DTD DocBook XML V4.4//EN" pdb/doc/pdb.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4// EN" pdb/doc/pdb_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" peering/doc/peering.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" peering/doc/peering_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" tls/doc/certs_howto.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tls/doc/functions.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tls/doc/history.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tls/doc/params.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tls/doc/tls.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tm/doc/api.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tm/doc/functions.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tm/doc/params.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" tm/doc/tm.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" topoh/doc/topoh.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" topoh/doc/topoh_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" utils/doc/utils.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" utils/doc/utils_admin.xml:<!DOCTYPE book PUBLIC "-//OASIS//DTD DocBook XML V4.4//EN" xmlrpc/doc/functions.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" xmlrpc/doc/params.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" xmlrpc/doc/xmlrpc.xml:<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN"
On Freitag, 16. Oktober 2009, Olle E. Johansson wrote:
Hi Olle,
i doubt that any documentation source for the modules use syntax that would break if we update the 4.2 definitions to 4.4. So i think its save to change it.
BTW, could you please restore the missing READMEs for tm and dialplan? ;)
Regards,
Henning
On Fri, Oct 16, 2009 at 4:32 PM, Olle E. Johansson oej@edvina.net wrote:
Changing docbook version only makes sense if you need to use any of the bug-fixes or improvements done between 4.2 and 4.4 in XML source files, that means in docbook sources. Those changes fix minor bugs in the schema and sometimes allow some elements to be used in new contexts. In other words, you only need to change docbook version if you want to use something that has been specified after 4.2 and it makes your docbook documents invalid.
Changing Docbook version does not change the version of XSL stylesheets that are used to generate various output formats, such as HTML or READMEs. These are configured in catalog files and in *.xsl files in sip-router/docbook.
If you want to change docbook versions anyway then the latest compatible version is 4.5. There is also version 5.0, but that would require changes in source XML files because this version is not backwards compatible anymore.
Also, if you do the change, please wait until the master branch is unfrozen.
-- Jan
On Fri, Oct 16, 2009 at 5:18 PM, Jan Janak jan@ryngle.com wrote:
After second reading of emails in this thread I realized that my earlier explanation of the Docbook system is not only confusing, but also incorrect. I apologize, please forget what I wrote in the email sent yesterday and let me try again.
The usual text at the beginning of Docbook source documents
<!DOCTYPE section PUBLIC "-//OASIS//DTD DocBook XML V4.2//EN" "http://www.oasis-open.org/docbook/xml/4.2/docbookx.dtd">
determines the particular version of the DTD (Document Type Definition) the Docbook document tries stay compatible with. This information is only used if you try to validate your Docbook document against the schema, for example by running make check in sip-router or if you use an editor which performs document validation on the fly as you edit the document.
The HTTP URL in the example above refers to the DTD file online, this is the recommended way of writing DTD declarations in Docbook XML files. If you actually try to download the document then you get the DTD file for Docbook version 4.2.
If you try to validate a Docbook document in sip-router by typing "make check" in a doc directory then the makefile system runs xmllint to validate the document. Xmllint uses the document type declaration presented above to locate the DTD file needed for the validation. But before it tries to download it using the HTTP URL above, the tool consults the file sip-router/docbook/catalog.xml to see if there is perhaps a local copy of the file already on a local disk.
The file catalog.xml is a XML catalog file. XML catalog files are used to map document type declarations (like the one above) to local files. XML catalogs can map both type of identifiers, the textual one "//OASIS//DTD.." and the HTTP URL as well. If the catalog file has a reference to a local DTD file then Xmllint uses the local file and does not try to download it from the HTTP URL. If there is no local copy of the DTD file then Xmllint downloads it using the HTTP URL above.
The XML catalog file in sip-router/docbook/catalog.xml refers to system-wide "standard" catalog files by default. On Debian (and perhaps other Linux/GNU based systems) the system-wide catalog is /etc/xml/catalog.
On Debian systems the system-wide catalog file is updated by the package manager. That means if you install the package docbook-xml, which contains Docbook DTD files for all 4.x versions, the DTD files from that package become automatically available to the Docbook build system in sip-router and it never has to download anything from the internet.
On systems that do not have automatically updated system-wide catalog files you can edit the file sip-router/docbook/catalog.xml and map the document type declaration presented above to some local file that you downloaded manually.
XML catalog files conveniently collect all the information about locally available DTD and XSL files in one place, so that you do not have to edit multiple Docbook source files when the configuration of your system changes.
That was validation which we normally do not use much, only when someone edits a Docbook file and wants to make sure that the document structure is valid. Having a valid Docbook document is good before you try to convert it to other formats, but not strictly required.
The Docbook build system works slightly differently if you try to convert a Docbook source file to other formats, such as module READMEs, plain-text or HTML files. To convert Docbook documents to other formats you need Docbook XSL stylesheets. This is a set of XML files that describe how individual elements in Docbook sources can be converted to HTML, XHTML, and FO (Formatting Objects--used to generate PDFs).
Because Docbook source files contain no information about version and whereabouts of Docbook XSL stylesheets that can be used to convert them to other formats, we need to put this information somewhere else in our Docbook build system. That "somewhere else" is in files with xsl extension in sip-router/docbook. There is one xsl file for each output format we support in the build system. When you type make README then the build system looks for file sip-router/docbook/readme.xsl, when you type make html then it looks for file sip-router/docbook/html.xsl, and so on.
If you actually look at the content of the file sip-router/docbook/readme.xsl, you will see that it is very simple. The file only contains a bunch of configuration options (Docbook XSL stylesheets have configuration options that can be used to fine-tune the output) customized to make README files look nice. The file also imports other XSL stylesheets and one of them is identified by the following HTTP URL:
http://docbook.sourceforge.net/release/xsl/current/xhtml/docbook.xsl
This is the canonical URL of the current version of Docbook XSL stylesheets. And here again, we use the XML catalog system to map that HTTP URL to a local file. XSL stylesheets are processed by the tool xsltproc and when the tool sees a URL like that, it is configured to look into the catalog file sip-router/docbook/catalog.xml to see if it refers to a local copy of the XSL stylesheet.
On Debian systems Docbook XSL stylesheets are available in the package docbook-xsl. The package manager again updates the system-wide catalog file if you install the package, so it becomes instantly available to our Docbook build system. On other systems you can download the latest copy of the XSL stylesheets manually and enter its location into the sip-router catalog file.
Note that Docbook XSL stylesheets are big, the whole package comprises many files and that's why it takes such a long time to download them from the internet if you do not have a local copy.
Also note that there is no strict relationship between the version of Docbook schema and the version of Docbook XSL stylesheets. There is no such thing as XSL stylesheets for Docbook 4.2, XSL stylesheets for Docbook 4.5, and so on. They are the same for all Docbook schema versions. You should always use the latest version of Docbook XSL stylesheets available, regardless of the Docbook schema version your documents use.
-- Jan
16 okt 2009 kl. 18.54 skrev Jan Janak:
On Mac, I just found a package named docbook-xml in MacPorts.
[....]
Now, is the catalog.xml protected by .gitignore? If I have to change it and later commits, I see another oej mistake coming along...
Thanks for a very good text. We do need to save it so we can reuse it later. Wiki or doxygen.
Thanks!
/O
On Sat, Oct 17, 2009 at 4:58 PM, Olle E. Johansson oej@edvina.net wrote:
Not it is not. .gitignore only works on files that are not tracked in the repository.
If you want make git ignore your local changes to docbook/catalog.xml then you can use the following command:
git update-index --assume-unchanged docbook/catalog.xml
This sets a special bit for the file and git will ignore any changes you do in your copy of the file, until you cancel it again with git update-index --no-assume-unchanged.
-- Jan
-
Henning Westerholt -
Jan Janak -
jh＠tutpro.com -
Olle E. Johansson