Allow user defined forward_common_files and backward_files (#482)

* Add Cu's init_surf examples.

* Delete personal information

* Update examples/init/cu.surf.hcp.111.json

Co-authored-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>

* Add fp.log in backward_files

* Add original methane INCAR

* Refactor decide_machine; Allow user_defined input and output;

* Debug user_defined_forward_common_files

* Fix bug in user_forward_common_files

* Allow define input and output files

* Fix bugs

* Fix bugs

* Fix bug

* Fix bug

* Fix bug

* Add documentation

* Fix bug

* Add file for unittest

* Fix backward_files and examples.

* Fix user_forward_files

* Change comments; add support for init; fix cvasp problems

* Fix bugs in cvasp of init_bulk

* Fix bug in user_forward_files in init_bulk

* Fix bug in init's user_forward_files

* Fix cvasp bugs in init_bulk

* Fix bugs in cvasp of init_bulk

* Fix bug in INCAR_md, fixed unit test

* Update context.py

* Update context.py

* Update test_convert_mdata.py

* Update context.py

* Update test_convert_mdata.py

Co-authored-by: Han Wang <amcadmus@gmail.com>
Co-authored-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>

Author: 3 people

README.md

@@ -25,7 +25,7 @@
 [![conda install](https://img.shields.io/conda/dn/conda-forge/dpgen?label=conda%20install)](https://anaconda.org/conda-forge/dpgen)
 [![pip install](https://img.shields.io/pypi/dm/dpgen?label=pip%20install)](https://pypi.org/project/dpgen)
 
-DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is depedent on DeepMD-kit (https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
+DP-GEN (Deep Generator)  is a software written in Python, delicately designed to generate a deep learning based model of interatomic potential energy and force field. DP-GEN is dependent on [DeepMD-kit](https://github.com/deepmodeling/deepmd-kit/blob/master/README.md). With highly scalable interface with common softwares for molecular simulation, DP-GEN is capable to  automatically prepare scripts and maintain job queues on HPC machines (High Performance Cluster) and analyze results.
 
 If you use this software in any publication, please cite:
 
@@ -34,7 +34,7 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
 ### Highlighted features
 + **Accurate and efficient**: DP-GEN is capable to sample more than tens of million structures and select only a few for first principles calculation. DP-GEN will finally obtain a uniformly accurate model.
 + **User-friendly and automatic**: Users may install and run DP-GEN easily. Once succusefully running, DP-GEN can dispatch and handle all jobs on HPCs, and thus there's no need for any personal effort.
-+ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with LAMMPS  and *ab-initio* calculation interface with VASP, PWSCF,SIESTA and Gaussian. We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
++ **Highly scalable**: With modularized code structures, users and developers can easily extend DP-GEN for their most relevant needs. DP-GEN currently supports for HPC systems (Slurm, PBS, LSF and cloud machines ), Deep Potential interface with DeePMD-kit, MD interface with [LAMMPS](https://www.lammps.org/), [Gromacs](http://www.gromacs.org/)   and *ab-initio* calculation interface with VASP, PWSCF, CP2K, SIESTA and Gaussian, Abacus, PWMAT, etc . We're sincerely welcome and embraced to users' contributions, with more possibilities and cases to use DP-GEN.
 
 ### Code structure and interface
 + dpgen:
@@ -43,7 +43,8 @@ Yuzhi Zhang, Haidi Wang, Weijie Chen, Jinzhe Zeng, Linfeng Zhang, Han Wang, and
     * generator: source codes for main process of deep generator.
 
     * auto_test : source code for undertaking materials property analysis.
-    * remote : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    * remote and dispatcher : source code for automatically submiting scripts,maintaining job queues and collecting results.
+    Notice this part hase been integrated into [dpdispatcher](https://github.com/deepmodeling/dpdispatcher) 
     * database : source code for collecting data generated by DP-GEN and interface with database.
 + examples : providing example JSON files.
 
@@ -63,6 +64,15 @@ Options for TASK:
 * `test`: Auto-test for Deep Potential.
 * `db`: Collecting data from DP-GEN.
 
+
+[Here](examples) are examples you can refer to. You should make sure that provide a correct [JSON](https://docs.python.org/3/library/json.html) file.
+You can use following command to check your JSON file.
+```python
+import json
+#Specify machine parameters in machine.json
+json.load(open("machine.json"))
+```
+
 ## Download and Install
 One can download the source code of dpgen by
 ```bash
@@ -1322,7 +1332,9 @@ mem_limit | Interger | 16 | Maximal memory permitted to apply for the job.
 | # End of resources
 | command | String | "lmp_serial" | Executable path of software, such as `lmp_serial`, `lmp_mpi` and `vasp_gpu`, `vasp_std`, etc.
 | group_size | Integer | 5 | DP-GEN will put these jobs together in one submitting script.
-
+| user_forward_files | List of str | ["/path_to/vdw_kernel.bindat"] | These files will be uploaded in each calculation task. You should make sure provide the path exists. 
+| user_backward_files | List of str | ["HILLS"] | Besides DP-GEN's normal output, these files will be downloaded after each calculation. You should make sure these files can be generated.
+  
 ## Troubleshooting
 1. The most common problem is whether two settings correspond with each other, including:
     - The order of elements in `type_map` and `mass_map` and **`fp_pp_files`**.

doc/CONTRIBUTING.md

@@ -0,0 +1,10 @@
+# DP-GEN Contributing Guide
+Welcome to [DP-GEN](https://github.com/deepmodeling/dpgen/tree/master/dpgen) !
+
+
+## How to contribute
+DP-GEN adopts the same convention as other softwares in DeepModeling Community. 
+You can first refer to DeePMD-kit's
+[Contributing guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/CONTRIBUTING.md)
+and [Developer guide](https://github.com/deepmodeling/deepmd-kit/edit/devel/doc/development/index.md).
+

dpgen/auto_test/common_equi.py

@@ -9,10 +9,9 @@
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.auto_test.mpdb import get_structure
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from distutils.version import LooseVersion
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -133,9 +132,9 @@ def run_equi(confs,
     inter_type = inter_param['type']
     # vasp
     if inter_type == "vasp":
-        mdata = decide_fp_machine(mdata)
+        mdata = convert_mdata(mdata, ["fp"])
     elif inter_type in lammps_task_type:
-        mdata = decide_model_devi_machine(mdata)
+        mdata = convert_mdata(mdata, ["model_devi"])
     else:
         raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/common_prop.py

@@ -13,9 +13,8 @@
 from dpgen.auto_test.Vacancy import Vacancy
 from dpgen.auto_test.calculator import make_calculator
 from dpgen.dispatcher.Dispatcher import make_dispatcher
-from dpgen.remote.decide_machine import decide_fp_machine, decide_model_devi_machine
 from dpgen.dispatcher.Dispatcher import make_submission
-
+from dpgen.remote.decide_machine import convert_mdata
 lammps_task_type = ['deepmd', 'meam', 'eam_fs', 'eam_alloy']
 
 
@@ -150,9 +149,9 @@ def run_property(confs,
             inter_type = inter_param_prop['type']
             # vasp
             if inter_type == "vasp":
-                mdata = decide_fp_machine(mdata)
+                mdata = convert_mdata(mdata, ["fp"])
             elif inter_type in lammps_task_type:
-                mdata = decide_model_devi_machine(mdata)
+                mdata = convert_mdata(mdata, ["model_devi"])
             else:
                 raise RuntimeError("unknown task %s, something wrong" % inter_type)
 

dpgen/auto_test/lib/util.py

@@ -77,11 +77,11 @@ def get_machine_info(mdata,task_type):
         command = vasp_exec
         command = cmd_append_log(command, "log")
     elif task_type in lammps_task_type:
-        lmp_exec = mdata['lmp_command']
+        model_devi_exec = mdata['model_devi_command']
         group_size = mdata['model_devi_group_size']
         resources = mdata['model_devi_resources']
         machine=mdata['model_devi_machine']
-        command = lmp_exec + " -i in.lammps"
+        command = model_devi_exec + " -i in.lammps"
         command = cmd_append_log(command, "model_devi.log")
     return machine, resources, command, group_size
 

dpgen/data/gen.py

@@ -22,14 +22,16 @@
 import dpgen.data.tools.sc as sc
 from distutils.version import LooseVersion
 from dpgen.generator.lib.vasp import incar_upper
+from dpgen.generator.lib.utils import symlink_user_forward_files
 from pymatgen.core import Structure
 from pymatgen.io.vasp import Incar
-from dpgen.remote.decide_machine import  decide_fp_machine
+from dpgen.remote.decide_machine import  convert_mdata
 from dpgen import ROOT_PATH
 from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher, make_submission
 
 
 
+
 def create_path (path,back=False) :
     if  path[-1] != "/":
         path += '/'
@@ -311,12 +313,7 @@ def make_vasp_relax (jdata, mdata) :
         os.remove(os.path.join(work_dir, 'POTCAR'))
     shutil.copy2( jdata['relax_incar'], 
                  os.path.join(work_dir, 'INCAR'))
-    is_cvasp = False
-    if 'cvasp' in mdata['fp_resources'].keys():
-        is_cvasp = mdata['fp_resources']['cvasp']
-    if is_cvasp:
-        cvasp_file=os.path.join(ROOT_PATH,'generator/lib/cvasp.py')
-        shutil.copyfile(cvasp_file, os.path.join(work_dir, 'cvasp.py'))
+    
     out_potcar = os.path.join(work_dir, 'POTCAR')
     with open(out_potcar, 'w') as outfile:
         for fname in potcars:
@@ -338,8 +335,17 @@ def make_vasp_relax (jdata, mdata) :
            os.symlink(ln_src, 'POTCAR')
         except FileExistsError:
            pass
+        is_cvasp = False
+        if 'cvasp' in mdata['fp_resources'].keys():
+            is_cvasp = mdata['fp_resources']['cvasp']
+        if is_cvasp:
+            cvasp_file = os.path.join(ROOT_PATH, 'generator/lib/cvasp.py')
+            shutil.copyfile(cvasp_file, 'cvasp.py')
         os.chdir(work_dir)
     os.chdir(cwd)
+    symlink_user_forward_files(mdata=mdata, task_type="fp",
+                               work_path=os.path.join(os.path.basename(out_dir),global_dirname_02),
+                               task_format= {"fp" : "sys-*"})
 
 def make_scale(jdata):
     out_dir = jdata['out_dir']
@@ -373,6 +379,7 @@ def make_scale(jdata):
             os.chdir(scale_path) 
             poscar_scale(pos_src, 'POSCAR', jj)
             os.chdir(cwd)
+    
 
 def pert_scaled(jdata) :
     out_dir = jdata['out_dir']
@@ -425,7 +432,7 @@ def pert_scaled(jdata) :
                 shutil.copy2(pos_in, pos_out)
             os.chdir(cwd)
 
-def make_vasp_md(jdata) :
+def make_vasp_md(jdata, mdata) :
     out_dir = jdata['out_dir']
     potcars = jdata['potcars']
     scale = jdata['scale']   
@@ -451,7 +458,9 @@ def make_vasp_md(jdata) :
             with open(fname) as infile:
                 outfile.write(infile.read())
     os.chdir(path_md)
-    os.chdir(cwd)    
+    os.chdir(cwd)
+    
+    
 
     for ii in sys_ps :
         for jj in scale :
@@ -478,8 +487,20 @@ def make_vasp_md(jdata) :
                     os.symlink(os.path.relpath(file_potcar), 'POTCAR')
                 except FileExistsError:
                     pass
+
+                is_cvasp = False
+                if 'cvasp' in mdata['fp_resources'].keys():
+                    is_cvasp = mdata['fp_resources']['cvasp']
+                if is_cvasp:
+                    cvasp_file = os.path.join(ROOT_PATH, 'generator/lib/cvasp.py')
+                    shutil.copyfile(cvasp_file,  'cvasp.py')
 
-                os.chdir(cwd)                
+                os.chdir(cwd)
+                
+    symlink_user_forward_files(mdata=mdata, task_type="fp",
+                               work_path=os.path.join(os.path.basename(out_dir),global_dirname_04),
+                               task_format= {"fp" :"sys-*/scale*/00*"})
+                            
 
 def coll_vasp_md(jdata) :
     out_dir = jdata['out_dir']
@@ -565,11 +586,14 @@ def run_vasp_relax(jdata, mdata):
     work_dir = os.path.join(jdata['out_dir'], global_dirname_02)
 
     forward_files = ["POSCAR", "INCAR", "POTCAR"]
+    user_forward_files = mdata.get("fp" + "_user_forward_files", [])
+    forward_files += [os.path.basename(file) for file in user_forward_files]
     backward_files = ["OUTCAR","CONTCAR"]
+    backward_files += mdata.get("fp" + "_user_backward_files", [])
     forward_common_files = []
     if 'cvasp' in mdata['fp_resources']:
         if mdata['fp_resources']['cvasp']:
-            forward_common_files=['cvasp.py']
+            forward_files +=['cvasp.py']
     relax_tasks = glob.glob(os.path.join(work_dir, "sys-*"))
     relax_tasks.sort()
     #dlog.info("work_dir",work_dir)
@@ -624,11 +648,14 @@ def run_vasp_md(jdata, mdata):
     md_nstep = jdata['md_nstep']
 
     forward_files = ["POSCAR", "INCAR", "POTCAR"]
+    user_forward_files = mdata.get("fp" + "_user_forward_files", [])
+    forward_files += [os.path.basename(file) for file in user_forward_files]
     backward_files = ["OUTCAR"]
+    backward_files += mdata.get("fp" + "_user_backward_files", [])
     forward_common_files = []
     if 'cvasp' in mdata['fp_resources']:
         if mdata['fp_resources']['cvasp']:
-            forward_common_files=['cvasp.py']
+            forward_files +=['cvasp.py']
 
     path_md = work_dir
     path_md = os.path.abspath(path_md)
@@ -694,7 +721,7 @@ def gen_init_bulk(args) :
 
     if args.MACHINE is not None:
        # Selecting a proper machine
-       mdata = decide_fp_machine(mdata)
+       mdata = convert_mdata(mdata, ["fp"])
        #disp = make_dispatcher(mdata["fp_machine"])
 
     # Decide work path
@@ -757,9 +784,12 @@ def gen_init_bulk(args) :
             pert_scaled(jdata)
         elif stage == 3 :
             dlog.info("Current stage is 3, run a short md")
-            make_vasp_md(jdata)
             if args.MACHINE is not None:
+               make_vasp_md(jdata, mdata)
                run_vasp_md(jdata, mdata)
+            else:
+               make_vasp_md(jdata, {"fp_resources":{}})
+               
         elif stage == 4 :
             dlog.info("Current stage is 4, collect data")
             coll_vasp_md(jdata)

dpgen/data/surf.py

@@ -11,7 +11,7 @@
 import dpgen.data.tools.bcc as bcc
 from dpgen import dlog
 from dpgen import ROOT_PATH
-from dpgen.remote.decide_machine import  decide_fp_machine
+from dpgen.remote.decide_machine import  convert_mdata
 from dpgen.dispatcher.Dispatcher import Dispatcher, make_dispatcher
 #-----PMG---------
 from pymatgen.io.vasp import Poscar
@@ -596,7 +596,7 @@ def gen_init_surf(args):
 
     if args.MACHINE is not None:
        # Decide a proper machine
-       mdata = decide_fp_machine(mdata)
+       mdata = convert_mdata(mdata, ["fp"])
        # disp = make_dispatcher(mdata["fp_machine"])
 
     #stage = args.STAGE

dpgen/generator/ch4/machine.json

@@ -21,7 +21,7 @@
 	"_comment":	"that's all"
     },
 
-    "lmp_command":	"/sharedext4/softwares/lammps/bin/lmp_serial",
+    "model_devi_command":	"/sharedext4/softwares/lammps/bin/lmp_serial",
     "model_devi_group_size":	1,
     "_comment":		"model_devi on localhost",
     "model_devi_machine":	{

dpgen/generator/lib/utils.py

@@ -1,6 +1,7 @@
 #!/usr/bin/env python3
 
 import os, re, shutil, logging
+import glob
 
 iter_format = "%06d"
 task_format = "%02d"
@@ -60,4 +61,37 @@ def log_task (message) :
 
 def record_iter (record, ii, jj) :
     with open (record, "a") as frec :
-        frec.write ("%d %d\n" % (ii, jj)) 
+        frec.write ("%d %d\n" % (ii, jj))
+
+def symlink_user_forward_files(mdata, task_type, work_path, task_format = None):
+    '''
+    Symlink user-defined forward_common_files
+    Current path should be work_path, such as 00.train
+    
+    Parameters
+    ---------
+    mdata : dict
+        machine parameters
+    task_type: str
+        task_type, such as "train"
+    work_path : str
+        work_path, such as "iter.000001/00.train"
+    Returns
+    -------
+    None
+    '''
+    user_forward_files = mdata.get(task_type + "_" + "user_forward_files", [])
+    #Angus: In the future, we may unify the task format.
+    if task_format is None:
+        task_format = {"train" : "0*", "model_devi" : "task.*", "fp": "task.*"}
+        #"init_relax" : "sys-*", "init_md" : "sys-*/scale*/00*"
+    for file in user_forward_files:
+        assert os.path.isfile(file)  ,\
+            "user_forward_file %s of %s stage doesn't exist. " % (file, task_type)
+        tasks = glob.glob(os.path.join(work_path, task_format[task_type]))
+        for task in tasks:
+            if os.path.isfile(os.path.join(task, os.path.basename(file))):
+                os.remove(os.path.join(task, os.path.basename(file)))
+            os.symlink(file, os.path.join(task, os.path.basename(file)))
+    return
+