The new features of AbiPy v0.9.1

M. Giantomassi and the AbiPy group

10th international ABINIT developer workshop
May 31 - June 4, 2021 - Smart Working, Lockdown@BE

---

• These slides have been generated using jupyter, nbconvert and revealjs
• The notebook can be downloaded from this github repo
• To install and configure the software, follow these installation instructions

Use the Space key to navigate through all slides and SHIFT + Space key to go back one slide.

What is AbiPy?

Python package for:

• Generating ABINIT input files automatically
• Post-processing results extracted from netcdf or text files
• Interfacing ABINIT with external tools such as vesta, wannier90, lobster, etc.
• Running ABINIT-specific workflows on laptops as well as on HPC clusters

Dependencies:

• Hard deps: pymatgen, netcdf4, matplotlib, plotly, numpy, scipy, pandas, ipython …
• Soft deps: ASE, phonopy, jupyter …

NB: AbiPy can be interfaced with other packages (e.g ASE, phonopy) via converters.

How to install AbiPy

Using conda and the conda forge channel (recommended):

    conda install abipy --channel conda-forge

Since conda is not limited to python packages, one can install ABINIT in the same env with:

    conda install abinit -c conda-forge

NB: conda packages for AbiPy and ABINIT are now provided by conda-forge therefore:

• new pkgs are automatically built when new releases are pushed to github
• the abiconda channel is deprecated. Please use conda-forge.

---

Using pip and python wheels:

    pip install abipy --user

---

For further info see http://abinit.github.io/abipy/installation.html

What's new at the level of the documentation?

• New website based on sphinx-rtd-theme
• More examples:
  • Gallery of plots based on matplotlib and plotly (59 scripts)
  • New examples for Flows (38 scripts)
  • Each script can now be executed on mybinder.org inside an automatically generated Docker container. Just click the LaunchBinder badge.
• New notebook tutorial for elastic properties with AbiPy
• Post-processing tools for EPH results are also documented in the new eph4zpr tutorial

#%embed https://abinit.github.io/abipy/gallery/index.html
#%embed https://abinit.github.io/abipy/flow_gallery/index.html

What's new at the level of workflow infrastructure?

In addition to workflows/tools for GS, DFPT, GW, IPA optics, BSE, we now have:

• Python converters: DDB $\;\rightleftarrows \;$ phonopy / tdep

• New Flows and improved post-processing tools for:

  • elastic and piezoelectric tensors with DFPT (clamped/relaxed atoms)
  • non-linear optical properties (SHG) with DFPT
  • Phonopy with AbiPy flows
  • Gruneisen parameters with DFPT + finite differences
  • effective masses with DFPT or finite differences
  • e-ph self-energy, e-ph matrix elements and scattering potentials
  • phonon-limited transport properties

Quick intro to AbiPy Flows and selected examples

⚠️ There are two different workflow infrastructures:

Internal AbiPy implementation (abipy.flowtk modules):

• ✅ Lightweigth, no database required
• ✅ Designed for rapid prototyping and/or for supporting advanced ABINIT capabilities
• ❌ No explicit support for high-throughput (HT) applications.

AbiFlows package (requires Fireworks and MongoDB database)

• ✅ HT-oriented: use MongoDB to store workflow status and results for further analysis
• ✅ High-level API designed for HT applications (e.g. phonon calculations for the materials project)
• ❌ Not all the flowtk workflows are immediately available in AbiFlows.

NB: There's an ongoing effort to reimplement AbiFlows in terms of the atomate framework. In this talk, we will mainly discuss the new features available in abipy.flowtk.

Effective masses with DFPT

This flow performs:

• GS-SCF run followed by NSCF run with k-path to locate band edges automatically

• Compute $\epsilon^{\alpha\beta}_{n\bf{k}}$ and the effective mass tensor at the band edges using $|u_{n\mathbf{k}}\rangle$, and the k-derivatives $|u_{n\mathbf{k}}^\alpha\rangle$ $H^\alpha_{\mathbf{k}}$, $H^{\alpha\beta}_{\mathbf{k}}$

• For the formalism, see J. Laflamme Janssen, et. al. Phys. Rev. B 93, 205147

• More that 1k systems already computed by J. C. Abreu.

Python API:

flow = flowtk.Flow("flow_effmass_dfpt")

# Build input for GS SCF calculation.
scf_input = make_scf_input()

# This object implements all the worflow logic
from abipy.flowtk.effmass_works import EffMassAutoDFPTWork
work = EffMassAutoDFPTWork.from_scf_input(scf_input) 

flow.register_work(work)

Dependency Graph

---

• To run the flow with the high-level interface, use: run_effmass_dfpt.py --sheduler
• autoparal algorithm:
  • select "optimal" Ncpus for each task
  • set MPI-related variables in the input
  • submit tasks and start monitoring
  • indipendent tasks are executed in parallel
• Support for bluegene, moab, pbspro, sge, shell, slurm, torque
• Restart capabilities (e.g. timeout limit) and error handlers

To generate a GS-SCF input, one can use factory functions (HT-oriented)

from abipy.abio.factories import gs_input

gs_input(structure="si.cif", pseudos="14si.pspnc", ecut=8)

##############################################
#### SECTION: basic
##############################################
ecut 8
ngkpt 8 8 8
shiftk 0.5 0.5 0.5
nshiftk 1
kptopt 1
nsppol 2
nband 16
occopt 3
tolvrs 1e-08
##############################################
#### SECTION: gstate
##############################################
spinat
0.0 0.0 0.6
0.0 0.0 0.6
chksymbreak 0
nspinor 1
nspden 2
charge 0.0
tsmear 0.0036749322175655 Ha
##############################################
#### STRUCTURE
##############################################
natom 2
ntypat 1
typat 1 1
znucl 14
xred
0.0000000000 0.0000000000 0.0000000000
0.2500000000 0.2500000000 0.2500000000
acell 1.0 1.0 1.0
rprim
6.3285005244 0.0000000000 3.6537614813
2.1095001748 5.9665675141 3.6537614813
0.0000000000 0.0000000000 7.3075229627

or build it from scratch if full controll is wanted:

other_inp = abilab.AbinitInput(structure="si.cif", pseudos="14si.pspnc")

other_inp.set_vars(ecut=8, nsppol=2, toldfe=1e-8)
other_inp.set_autokmesh(8)

other_inp

##############################################
#### SECTION: basic
##############################################
ecut 8
nsppol 2
toldfe 1e-08
ngkpt 8 8 8
kptopt 1
nshiftk 4
shiftk
0.5 0.5 0.5
0.5 0.0 0.0
0.0 0.5 0.0
0.0 0.0 0.5
##############################################
#### STRUCTURE
##############################################
natom 2
ntypat 1
typat 1 1
znucl 14
xred
0.0000000000 0.0000000000 0.0000000000
0.2500000000 0.2500000000 0.2500000000
acell 1.0 1.0 1.0
rprim
6.3285005244 0.0000000000 3.6537614813
2.1095001748 5.9665675141 3.6537614813
0.0000000000 0.0000000000 7.3075229627

Workflows do not necessarily imply HT applications

• Sometimes, we need to address rather technical/fundamental questions
• For instance, one may ask whether the treatment of the dipole interaction is enough to obtain an accurate interpolation of the e-ph scattering potentials
• Answering this question required hundreds of DFPT + WFQ + EPH calculations along a very dense q-path (300 points):

---

• See G. Brunin's talk and PRL 125, 136601 (2020), PRB 102, 094308 (2020)

Obviously, we used an AbiPy script to automate most of the steps:

---

NB: This is a simplified version with just 2 q-points in the path. In our work, we used 278 points.

Other AbiPy applications are presented in the following talks:

• Electron-phonon beyond Fröhlich: dynamical quadrupoles in polar and covalent solids by G. Brunin
• Phonon-limited conductivity in 2D and 3D metals by O. Nadeau
• Absorption spectrum calculations using cumulant expansion in electron-phonon interactions by J. C. Abreu
• Automating ΔSCF computations of point defects using AbiPy workflows by J. Bouquiaux

What's new at the level of the post-processing tools?

• New plotting tools based on plotly
• GUIs and dashboards based on panel and bokeh

---

Why plotly?

• ✅ Publication quality figures.
• ✅ Flexible python API able to produce rather advanced plots
• ❌ Plots are difficult to customize without changing the python code
• ❌ Plots lacks interactivity and integration with HTML/JS

• ✅ Interactive plots + chart editor GUI to customize the figure
• ✅ Plays well with HTML (plotly is written in js with python bindings)
• ❌ Open source project but not all the features are available in the free plan
• ❌ Requires browser (this may represent an issue on some HPC centers)

AbiPy plots with matplotlib

gsr = abiopen("si_nscf_GSR.nc")
gsr.ebands.plot(with_gaps=True);

AbiPy plots with plotly

(contributed by Y. He)

gsr.ebands.plotly(with_gaps=True);  # obj.plot becomes obj.plotly

To upload the plotly figure to the chart studio server, use:

gsr.ebands.plotly(with_gaps=True, chart_studio=True);

---

Users can finally customize the AbiPy plot without changing the python code 🎉

Interactive 3d plots with plotly:

gsr.ebands.kpoints.plotly(title="k-path in 3d with plotly");

and, ça va sans dire, phonons with plotly:

znse_ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")

phbst_file, phdos_file = znse_ddb.anaget_phbst_and_phdos_files()
phbands, phdos = phbst_file.phbands, phdos_file.phdos

phbands.plotly_with_phdos(phdos, units="cm-1", title="ZnSe Phonon bands + DOS in cm-1");

Integrating AbiPy with web-based technologies via Panel

• AbiPy GUIs inside jupyter notebooks
• Dashboards and web apps
• Integration with the AbiPy command line interface
• Web apps for ABINIT users (🚧)

Pros and cons of the client-server model

Advantages:

• The client does not need to install the scientific software stack (when running on different machines)
• Can implement web apps that allows the user to upload data and analyze the results e.g DDB GUI.

Disavantages

• Round trip delay if client != host and slow connection
• Upoloadng a 1Gb file to the remote server just because you don't want to install software on the localhost is a very bad idea.
• OK for relatively small files (< 1Gb) but this approach is not designed to handle big data.
• Not all the HPC centers provide specialized nodes to post-process the results inside a web browser/notebook.

How to use AbiPy panels inside jupyter notebooks

To build a panel GUI inside the notebook, call the get_panel method:

ddb = abilab.abiopen("ZnSe_hex_qpt_DDB")

abilab.abipanel(); # Important
ddb.get_panel()

Don't be surprised if you start to click buttons and nothing happens in the GUI! One needs a python backend to execute the callbacks triggered by the widgets.

---

To open a dashboard for the DDB file from the shell, use:

abiopen.py out_DDB --panel  # or -pn

Integration with the AbiPy scripts

"Old" approach to produce matplotlib figures from FILE.

abiopen.py FILE --expose

To generate plotly figures and show them in the browser, use:

abiopen.py FILE --plotly # -ply

For matplotlib figures in the browser, use:

abiopen.py FILE --expose-web # -ew

To generate a jupter-lab notebook for FILE, use:

abiopen.py FILE --notebook # or --notebook-classic

To automatically compute phonons from DDB and show results in the brower, use:

abiview.py ddb out_qpt_DDB --panel  # -pn

Documentation for these new features available at:

%embed https://abinit.github.io/abipy/graphical_interface.html

How to run AbiPy Flows

In order to run a Flow, we need two configurations files:

1. scheduler.yml providing:

   • scheduler parameters such as days, hours, minutes, max_njobs_inqueue, max_ncores_used, …

1. manager.yml providing:

   • list of shell commands to be executed before running ABINIT
   • list of modules to load
   • options for the queue manager (bluegene, moab, pbspro, sge, shell, slurm, torque)

See this page for examples or use the abidoc.py script and the syntax:

• abidoc.py scheduler
• abidoc.py manager
• abidoc.py manager slurm

How to run calculations?

The simplest way to start the scheduler from the shell is via the syntax:

run_elastic.py --scheduler # -s for the short option

For non-trivial Flows, we suggest to put the scheduler in background and use nohup so that we can disconnect from the shell session without killing the scheduler.

nohup run_elastic.py --s > log 2> err &

Obviously, it is possible to submit a Slurm script that executes the script on the compute note with 1 core.

To interact with the Flow, one can also use the abirun.py script, e.g:

abirun.py FLOWDIR status

DDB converters

(contributed by G. Petretto)

• Based on previous work by H. Xu, E. Bousquet and A. Romero

• Can be used to:

  • connect the ABINT DPT part with other packages requiring phonopy files (e.g. anharmonic calculations with hiphive)
  • interface AbiPy with phonopy tools e.g. irreps

• Algorithm:

  • Runs anaddb to get the interatomic force constants IFC($\bf{R}$), BECs and $\epsilon^\infty$
  • Save results in nc format (anaddb.nc)
  • Convert IFC($\bf{R}$) and tensors from ABINIT to phonopy conventions
  • Optionally, save phonopy files in output_dir_path.

In python, everything boils down to:

ddb = abilab.abiopen("mp-149_DDB")
phonopy_obj = ddb.anaget_phonopy_ifc(output_dir_path="output_dir")

• Example: obtain phonon irreps from DDB using phonopy

ddb = abilab.abiopen("mp-149_DDB")
ph = ddb.anaget_phonopy_ifc()
ph.set_irreps([0, 0, 0])
ph.get_irreps().show()