Not included here because they are kept current in ../:

acl2-manual.lisp
render-doc.lisp

--------------------------------------------------------------------------------

Hi Matt,

You probably haven't talked to J yet, but here is a preliminary tool for
converting ACL2's :doc documentation into a single file of defxdoc forms, which
may be of use for making things concrete.

Writing this tool was fun, but trickier than I expected.  When I just jammed
the :doc topics through the ordinary xdoc converter (as I'd originally
planned), the output was horribly ugly.

To correct for a lot of this, I am using a mixture of ACL2 and ELISP.  On the
ACL2 side, the exporting tool now does some crazy things, e.g.,

  - (where possible) it converts the <code> sections introduced by
    ~bv[]...~ev[] into @({...}) stuff so that we get autolinking and more
    readable markup without &lt; style escaping everywhere.

  - (where possible) it converts <tt> sections introduced by ~c[...] into
    @('...'), for better readability and to avoid escaping.

  - It injects a space at the start of each @({...}) line, to avoid having
    ( characters on column 0 that screw up emacs color highlighting.

  - It converts the extremely verbose <see topic="@(url FOO)">foo</see> into
    @(see foo), when it's the right thing to do.

  - It produces nicely indented defxdoc topics without a bunch of unnecessary
    package names, and with more sensible newline handling.

On the ELISP side, I do some additional fixup to normalize whitepsace,
eliminate empty paragraphs, and generally try to hit as much as I can safely
hit with fill-paragraph.

--------------------------------------------------------------------------------

Instructions (from Jared but edited by Matt):

Note: To start over inside the tool/ directory:

rm -rf *.cert *.out Makefile-tmp *.dx64fsl manual *~ rendered-doc.lisp acl2-doc.lisp*

1. Update your acl2-books (at least to svn revision 2106, which fixes
   an xdoc autolink bug that was exposed by the tool).  Then issue the
   following command in the books/ directory, with a suitable
   replacement for the arguments, including -j and the value of ACL2.
   time make -j 4 ACL2=/Users/kaufmann/acl2/devel/saved_acl2 xdoc/all.cert

2. Extract tool.tar.gz somewhere.  It should create a new "tool" directory.
   Go into this directory.

3. The following should create acl2-doc.lisp (edit the value of --acl2
   and give suitable absolute pathname for cert.pl);
   don't be surprised if it makes some books under the books/
   subdirectory of your ACL2 distribution.  Of course, feel free to
   change or eliminate the -j option.
     /Users/kaufmann/acl2/devel/books/build/cert.pl -j 4 --acl2 /Users/kaufmann/acl2/devel/saved_acl2 export-acl2doc
   Perhaps also do this, just for completeness:
     cp -p acl2-doc.lisp acl2-doc.lisp.orig

4. Open fix.el in emacs 23 or emacs 24 (not emacs 22)
   Execute the definitions there in emacs
   Open acl2-doc.lisp in emacs
   Run M-x fix (this may take a good minute)
   Save the modified acl2-doc.lisp

[NOTE: Now that we have created books/system/doc/, we should stop
here.  To do the rest, one just updates books/system/doc/acl2-doc.lisp
and rebuilds.]

5. Create manual/* (again, edit the value of --acl2 and give suitable
   absolute pathname for cert.pl)
   /Users/kaufmann/acl2/devel/books/build/cert.pl --acl2 /Users/kaufmann/acl2/devel/saved_acl2 acl2-manual
   Look for warnings, e.g.:
   fgrep ";;; WARNING" acl2-manual.cert.out

At this point, you should have:

   acl2-doc.lisp   -- the single Lisp file with all the defxdoc topics,
                      with formatting somewhat fixed-up.

   manual/         -- an ordinary "fancy" xdoc manual with just the ACL2
                      topics [*]

     [*] I think there is exactly one non-ACL2 topic in the resulting manual,
          namely BROKEN-LINK.  This topic can probably be safely removed, since
          there are no broken links in the documentation.  (It's mainly there
          for defdoc forms in books that might try to reference missing
          topics.)  But this clearly seems like fair use.

--------------------------------------------------------------------------------

With a file this large, I can't really just inspect everything and make sure
it's okay.  The file is looking pretty good to me now, but maybe you will
notice something that I haven't -- if so, please let me know!

Known issues:

1. The fix.el script doesn't necessarily fill-paragraph everything it should.
It's hard to do this correctly because you don't want to run it in any <code>
or @({...}) sections, but you do want to run it everywhere else.

(I think we can live with this.  I've at least gotten fill-paragraph to run on
an awful lot of stuff.  The file seems at least pretty reasonably well
word-wrapped in general.)


2. There are a handful of links (24 of them) that get broken by the automatic
use of fill-paragraph.  These are generally things like:

-----
<p><see topic='@(url |Using the Associativity of App to Prove a Trivial
Consequence|)'><img src='walking.gif'></img></see></p>
-----

   The problem is, XDOC thinks we're trying to link to a symbol that literally
   has a newline character in it.

(Well, this is rare enough that we can just fix it by hand.  And the output
in acl2-manual.cert.out says where all the problems are, so it should be easy
to fix).


3. We have never imported the cited-by stuff into xdoc.  I don't really know
much about it.  It seems likely that those links are valuable and that it'd be
a shame to just throw them away.  Maybe we can come up with a more principled
way to integrate them.


4. The organization of the file is somewhat lacking.  I just put the topics in
alphabetical order, except that the ACL2 topics come first and then the acl2-pc
topics.  Maybe this is good enough.


Please feel free to call if anything seems not to work, or if there's anything
to discuss.

Cheers,
Jared

============================================================

NOTE: The following section is obsolete, in the sense that we have
incorporated what we need to into ACL2 and the books.

[Below is an email from Jared for building plain strings for :doc,
slightly edited.
Note that cert.pl should be given a suitable full pathname
and needs a --acl2 option, say,
/Users/kaufmann/acl2/devel/saved_acl2.
Run this from the tool/ directory.
]

   # prerequisite: build acl2-doc.lisp
   #   (e.g., run cert.pl export-acl2doc.lisp)

   $ /Users/kaufmann/acl2/devel/books/build/cert.pl render-doc.lisp   # creates rendered-doc.lisp
   $ /Users/kaufmann/acl2/devel/books/build/cert.pl rendered-doc.lisp # or load it in at ACL2 build time

   # now all the rendered docs are just in *acl2-system-documentation*
   # for instance:

   $ acl2
   ACL2 !> (include-book "rendered-doc")
   ACL2 !> (princ$ (cadr (assoc 'rewrite *acl2-system-documentation*)) *standard-co* state)
   Parent topic: Rule-classes

     Make some :rewrite rules (possibly conditional ones)

     See [rule-classes] for a general discussion of rule classes,
     including how they are used to build rules from formulas and a
     discussion of the various keywords in a rule class description.
   [...]

Eventually we can just distribute render-doc.lisp alongside the
main acl2-doc.lisp file.  (There's presumably no reason to distribute
the rest of the tool that creates acl2-doc.lisp, since ideally this
conversion is a one-time process.)
