Abimkdocs: the new infrastructure for the ABINIT documentation

M. Giantomassi

9th international ABINIT developer workshop
20-22nd May 2019 - Louvain-la-Neuve, Belgium

Use the Space key to navigate through all slides.

Why a new infrastructure for docs?

Problems

  • High-quality documentation is extremely important but find me someone who loves writing docs (especially in HTML) and I'll pay 🍺🍺🍺
  • Websites use lot of HTML5/Javascript/CSS technology but we are more familiar with plain text files and $\LaTeX$
  • Modern websites are interactive and responsive (layout adjusts to screen size)
  • Documentation should be kept in synch with new developments
  • Need git-friendly format to facilitate code-merge and code-reviewing

Solution

  • Use Markdown( MD) for documentation files (lightweight markup language)
  • Use python objects to represent input variables (including metadata)
  • Employ python framework to generate static website from MD files
  • Extend the framework with plugins based on the code developed for TestSuite/InputVariables

Software stack

Python

  • Python Markdown to parse the Markdown documentation
  • pymdown-extensions to extend Markdown syntax
  • pybtex: BibTeX-compatible bibliography processor written in Python
  • PyYaml: Full-featured YAML framework for the Python programming language
  • MkDocs: Static site generator geared towards project documentation

Dependencies listed in ~abinit/requirements.txt. Must be installed manually with pip

Javascript (JS), CSS, HTML ...

JS/CSS libraries are downloaded by the browser at runtime. Well, we are building a website so I assume you have an internet connection 🤦‍♂️ 🤦‍♀️

  • Static site generator that combines MD files and Jinja2 templates to produce websites
  • Python project hosted on GitHub: 7617 stars, 1137 forks, BSD2 license
  • The built-in server allows you to preview the HTML page as you're writing it
  • Plugin infrastructure to extend and customize the framework 🚀
  • Easy configuration via single YAML file 🎉

Order, title, and nesting of each page in the navigation header specified in mkdocs.yml:

site_name: abinit

pages:
  - Home: index.md
  - About: about.md

Installation

cd ~abinit
pip install -r requirements.txt [--user]

How to build the website

./mksite.py serve --dirtyreload

Regenerating database...
Saving database to /Users/gmatteo/git_repos/abidocs/doc/tests/test_suite.cpkl
Initial website generation completed in 9.17 [s]
Generating markdown files with input variables of code: `abinit`...
...
...
INFO    -  Building documentation...
INFO    -  Cleaning site directory
[I 170826 03:37:05 server:283] Serving on http://127.0.0.1:8000
[I 170826 03:37:05 handlers:60] Start watching changes
[I 170826 03:37:05 handlers:62] Start detecting changes
  • Open http://127.0.0.1:8000 in the browser and start editing MD files
  • With --dirtyreload, mkdocs automatically rebuilds modified files

How to add a new tutorial

  • Create a new MD file in doc/tutorial
  • Register the new page in mkdocs.yml
  • Build with mksite.py

Excerpt of mkdocs.yml

pages:
- Tutorial:
    - Overview: tutorial/index.md
    - Base Tutorials: 
        - Base1: tutorial/base1.md
        - Base2: tutorial/base2.md

MD files start with a Yaml section providing metavariables used by the framework:

$ head ~abinit/doc/developers/markdown.md

---
authors: MG, XG
plotly: true
---

Metavariables are stored in page.meta

For our markdown dialect, see https://docs.abinit.org/developers/markdown/

HTML page:

Jinja2 Template (Warning: 🤓 slide)

Excerpt of ~abinit_theme/main.html

{% extends "base.html" %}  <!-- Reuse base template from mkdocs-material -->

{% block libs %}
  {{ super() }} <!-- Include libs defined in the super class -->

  <!-- Add MathJax support -->
  <script type="text/javascript" async 
    src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_SVG">
  </script> 

  <!-- Add plotly support if "plotly" in page metavariables -->
  {% if page.meta and page.meta.plotly %} 
     <script type="text/javascript" src="https://cdn.plot.ly/plotly-latest.min.js"></script> 
  {% endif %}
{% endblock %}

{% block content %}
    {{ super() }} <!-- HTML content generated from MD file goes here -->

    <!-- Add dropdown button with toolbar and return to Top button -->
    <div class="md-container"> HERE BE HTML 🐲🐲🐲 </div>
{% endblock %}
  • Creating links to external resources is easy: [See here](http://bfy.tw/P9e)
  • Inserting links to internal resources (vars, bibtex, input files) is boring and error-prone

Solution for internal links:

  • Extend the MD parser to support [[token]] regex
  • The extension will interpret token to generate the link

  • Syntax: [[namespace:name#fragment|text]]

    • namespace defines the context e.g. cite, test, tutorial, pdf, gitsha...
    • name is interpreted depending on namespace e.g. [[cite:Amadon2008]]
    • fragment specifies a location within a page (advanced option)
    • text gives the name of the HTML link

Full documentation available here

It seems complicated but it's very flexible and syntax for common cases is really easy...

MathJax integration

  • $\LaTeX$ formulas are interpreted at visualization time thanks to the MathJax processor
  • Extension for Python-Markdown provided by python-markdown-math
  • MathJax configuration defined in the Jinja template

Latex equations can be used everywhere including:

  • the description of the input variables (text entry)
  • the description of the tests in TEST_INFO

Conventions:

  • $...$ for inlined LaTeX formula.
  • $$...$$ for display mode: formula rendered on one line without label.
  • To have numbered equations , use:
\begin{equation}\label{eq:my_awesome_result}
    1 + 1 = 3  
\end{equation}

Using Eq.(\ref{eq:my_awesome_result}), one obtains...

List of macros for common symbols defined in jinja template...

MathJax configuration in ~abinit/abinit_theme/main.html

<script type="text/javascript" async 
        src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.1/MathJax.js?config=TeX-AMS-MML_SVG">
    </script> 
    <!--
    Configure MathJax to produce automatic equation numbers 
    See http://docs.mathjax.org/en/latest/tex.html
    -->
    <script type="text/x-mathjax-config">
      MathJax.Hub.Config({
        TeX: {
            equationNumbers: { autoNumber: "AMS" },
            extensions: ["AMSmath.js"],
            // Hic sunt Macros
            Macros: {
                GG: "{\\bf G}",
                kk: "{\\bf k}",
                qq: "{\\bf q}",
                kq: "{\\kk + \\qq}",
                // PAW
                tpsi: "{\\tilde\\psi}",
                tphi: "{\\tilde\\phi}",
                tprj: "{\\tilde p}"
            }
          }
      });
    </script>

Topic Files

  • Topic files are written in MD and can be found in ~abinit/doc/topics
  • Filename starts with an underscore e.g. _AbiPy.md.
  • These are template files containing the MD content and two {{ variables }}:
---
description: How to to perform a Δ-SCF calculation of neutral excitations
authors: FJ
---
Hints on how to to perform a Δ-SCF calculation of neutral excitations.

## Introduction

Although formally not justified, difference in total energy using constrained
occupation numbers sometimes results in surprisingly good agreement with
experimental neutral excitations. See e.g. [[cite:Jia2017]].

## Related Input Variables
 {{ related_variables }}

## Selected Input Files
 {{ selected_input_files }}
  • The template will be filled by mksite.py by inspecting the Variables and the TestSuite.
  • A new MD file without underscore will be generated and included in mkdocs.yml

Input variables

  • List of Variable objects declared in python modules
  • One module per main exec (e.g. ~abinit/abimkdocs/variables_abinit.py)
  • Python API to access variables (used for generating links and HTML):

# Build database with all input variables indexed by code name.
codevars = get_codevars()
ecut_var = codevars["abinit"]["ecut"] 

for var in codevars.iter_allvars():
    print("Metadata:", var.name, var.varset, var.executable)
    print("Documentation in MD format:", var.text)
    url = "/variables/%s#%s" % (var.varset, var.name)

Correspondence between vars and topics in TEST_INFO is computed dynamically:

for test in all_abinit_tests:
    vd = codevars[test.executable]
    for vname in test.get_varnames(list(vd.keys())):
        var = vd[vname]
        var.tests.append(test)

An example of Abinit variable containing a table in MD format

Variable(
    abivarname="accuracy",
    varset="basic",
    vartype="integer",
    topics=['Planewaves_basic', 'SCFControl_basic'],
    dimensions="scalar",
    defaultval=0,
    mnemonics="ACCURACY",
    text=r"""
Allows to tune the accuracy of a calculation according to the following table:

accuracy         | 1         | 2          | 3            | 4         
---              |---        |---         |---           |---        
[[ecut]]         | E_min     | E_med      | E_med        | E_max       
[[pawecutdg]]    | ecut      | ecut       | 1.2 * ecut   | 1.5 * ecut   
[[fband]]        | 0.5       | 0.5        | 0.5          | 0.5          

*accuracy* = 4 corresponds to the default tuning of ABINIT. It is already a very accurate tuning.
""",
),
  • varset ➜ each variable belongs to a set
  • topics ➜ this variable is associated to two topics. Convention: topicname_level
  • text ➜ python raw string that will be passed to our customized MD parser

All the gory details at https://docs.abinit.org/developers/abimkdocs/#variable-object

Bibliographic References

  • Bibliographic references are in bibtex format and stored in ~abinit/doc/abiref.bib
  • Entries should provide DOI and URL (if available) so that python can generate links
  • Use FirstAuthorYear for the key.
  • A letter might be added in case of conflicts: e.g. Amadon2008a
  • Please use bibtex data from the publisher or betterbib:

$ betterbib-doi2bibtex 10.1103/PhysRevLett.96.066402

@article{Amadon2008,
  author = {Amadon, B. and Biermann, S. and Georges, A. and Aryasetiawan, F.},
  doi = {10.1103/physrevlett.96.066402},
  issn = {0031-9007, 1079-7114},
  journal = {Physical Review Letters},
  month = feb,
  number = {6},
  publisher = {American Physical Society (APS)},
  source = {Crossref},
  title = {{The α−γ Transition} of Cerium Is Entropy Driven},
  url = {http://dx.doi.org/10.1103/physrevlett.96.066402},
  volume = {96},
  year = {2006}
}

[[cite:Amadon2008]] produces a link to one of the entries in the bibliography page...

In [2]:
%embed https://docs.abinit.org/theory/bibliography/#amadon2008
Out[2]:

ROOT/theory/bibliography/#amadon2008 is an example of permalink ...

A permalink is a URL that is intended to remain unchanged for many years into the future, yielding a hyperlink that is less susceptible to link rot (wikipedia)

  • tutorials and sections in pages
  • input variables, input files and talks like this one
  • (varset, varname) if variable
  • Filepath and section name if MD doc
  • Location of input file for tests
  • Bibtex key

What's next?

  • Migration to Mkdocs > 1.0 and official plugin-API
  • Drop support for py2 in abimkdocs
  • Enable search bar in website (require new mkdocs)
  • Support multiple versions of docs (v9.0, v9.2)
  • Move more content from wiki to website:
    • stable doc goes to Mkdocs
    • wiki is for dynamic documentation

Any suggestions to improve the website?

  • Contents ⟾ Markdown files
  • Template ⟾ Jinja + Mkdocs-Material theme and lot of HTML/JS/CSS techology
  • Compile ⟾ Python Mkdocs + our customized logic (plugins)
  • Static website means that the server sends pre-built HTML pages:
    • easier to implement than dynamic websites
    • safer (no code running on the server)
  • Static website does not mean that we cannot have interactivity or responsive design:
    • Interactivity provided by JS/CSS/HTML code generated by python plugins
    • Responsive design provided by the Mkdocs-Material theme

Good news: we don't need to master all this stuff, we mainly interact with MD extensions 🎉

Testing infrastructure

  • Typos detected at runtime by mksite.py (check for WARNINGs in terminal)
  • Website built by abiref buildbot worker
  • Run linkchecker to detect broken links in website (@Jean-Michel)

Still, we need human supervision to be sure the HTML looks OK so, please, build the website on your machine before git push!

Tests to be activated

Pull request won't be accepted if:

  • New variable is not documented and is not present in, at least, one input file
  • New executables shall provide documentation and py-module with vars
  • Find values of scalar variables that are not tested by TestSuite to improve coverage

## Abinit extensions

To create a button that opens a dialog containing the input file, use:

    {% dialog tests/v1/Input/t01.in %}
In [3]:
%embed https://docs.abinit.org/developers/markdown#abinit-extensions
Out[3]:

Other examples with namespace

Full documentation available here