Skip to content

Backward incompatible changes in 1.0.0

Jump to bottom
ConradJohnston edited this page Feb 15, 2019 · 15 revisions

As there will be a few backward-incompatible changes in AiiDA 1.0.0, let's use this page to list them so we don't forget them. Before the release we will convert them to a nice doc.

Some TODOs

  • New class names, link names, link connection rules (provenance redesign). Probably mention main changes and point to new docs.
  • New way of defining CalcJobs
  • conversion of is_alloy and has_vacancies to properties, change of order of parameters in arrays, change of type of symbols (and where they are stored). Check carefully also other changes in #2310 and put in a linter also the changes that are only deprecated. See also #2374
  • Most of the imports have been moved. See what is the simplest way of explaining this (probably explain the new structure of the aiida code and the fact that one should use only functions from aiida and aiida.xxx and not deeper down). See e.g. #2357
  • Most of the command line -> mention renames, changes, ...
  • Dropped legacy workflows

Conrad's Notes: The target content for the docs should live between Content below and Content above tags. The final section is just a list of all PRs that didn't look relevant. In principle, all PRs from after v.1.0.0a4 are on this page, either within the content section, because I though they looked possibly relevant, or in the end section, because I though they looked benign.

Content below

Backwards incompatible changes in AiiDA v1.0

TODO: This section should probably cover any kind of change which changes the behaviour of soemthing from 0.12.2, even it is not strictly backwards incompatible.

General

  • Restore the correct plugin type string based on the module path of the base Data types [#1192] [Advanced]

  • Do not allow the copy or deepcopy of Node, except for Data [#1705](sphuber) [Users - new clone method]

  • Fix 2181 node incoming outgoing [#2236](waychal) Fixes #2181

    Two new methods are implemented:

    • get_incoming(): to get all input nodes of the given node
    • get_outgoing(): to get all output nodes of the given node

    Deprecated:

    • get_inputs()
    • get_inputs_dict()
    • get_outputs()
    • get_outputs_dict()

    Also all the occurrences of get_inputs() and get_inputs_dict() are replaced by get_incoming() method and for get_outputs() and get_outputs_dict() by get_outgoing().

  • Prevent to store base Nodes in the DB but only their subclasses [#2301](giovannipizzi) This PR fixes #2300, that explains the reasoning behind it

  • Reorganization of some top level modules [#2357](sphuber) Fixes #2311

    There are a few top-level modules whose purpose was not fully clear and partially as well as partially overlapping, which caused the content to be quite unorganized and scattered. Here we reorganize these modules after having clearly defined their purpose:

    • aiida.common: for generic utility functions and data structures
    • aiida.manage: for code that manages an AiiDA instance
    • aiida.tools: for code that interfaces with third-party libraries

    The module aiida.control has been removed. Code that interacted with internal components have been moved into cmdline or manage where applicable. The aiida.control.postgres has been moved to the aiida.manage.external module which is reserved for code that have to interact with system wide components, such as the postgres database.

    The module aiida.utils has been removed. Generic functionality has been placed in aiida.common and more specific functionality has been placed in the appropriate sub folders.

ORM

In order to define a clearer interface for users to the AiiDA entities, and allow (in the future) to swap between different profiles, the underlying hierarchy of nodes and links has been reworked (and a new paradigm has been implemented, with a "frontend" class that all users will interact with, and a "backend" class that users should never use directly, that implements the interaction to the database with a common interface, independent of the underlying ORM). The reorganisation of nodes and linktypes is mostly invisible to the user, but existing import statements will need to be updated.

  • Orm design refactor [#2190](muhrin) Ported Computer, AuthInfo and User to new backend design where the user interacts with concrete classes which themselves contain the abstract class corresponding to the backend type for that class.

    Having the user interact with concrete classes means that they can, amongst other things, instantiate them using the constructor and have the correct backend type be crated underneath (rather than using import magic as before).

    There is also a corresponding Collection class for each object type to interact with the collection of those things eg. User.objects resolves to the collection of users and can be used to do things like delete users.

  • Rename node class for workfunctions [#2189](sphuber) The node class used by workfunction changed from FunctionCalculation to WorkFunctionNode.

  • All calculations now go through the Process layer, homogenizing the state of work and job calculations [#1125]

  • Implement the new base ORM classes for process nodes [#2184](sphuber) Fixes #2167

    This commit defines the new basic hierarcy of the ORM classes for process nodes. In addition, it already moves over the functionality of the current abstract calculation classes to their new corresponding counterpart.

    • AbstractCalculation -> ProcessNode and CalcJobNode
    • AbstractJobCalculation -> CalcJobNode

    This should allow the refactoring of the code according to the renaming of the node ORM classes to begin.

  • Add temporary support for loading of new process node ORM classes [#2188](sphuber) Fixes #2187

    The loading of the correct ORM class for a given entry in the DbNode table is based on the type column, but its interpretation is still in a half-way state. The plugin loader will first attempt to reverse engineer it into an entry point and if that works load the correspondingly class. If that fails, it is interpreted as an internal class and will attempt to load it directly as a module path.

    Adding the new process node ORM classes, will need to be supported by the plugin loader, so here we add their entry points. We do this in the aiida:node group but note that in the future the entire system will be changed to solely rely on entry points, at which the format of the type column can be changed to be a fully qualified entry point string, just as the process_type column already is.

  • Rename node class for WorkChains [#2192](sphuber) Fixes #2168

    The node class used by WorkChains changed from WorkCalculation to WorkChainNode.

  • Rename node class for inline calculations [#2195](sphuber) Fixes #2170

    The node class changed from InlineCalculation to CalcFunctionNode.

  • Make Code a real sub class of Data [#2193](ConradJohnston) Fixes #2173 . Subclasses Code from Data rather than Node, and simplifies places in the code where code had to be treated as a 'special' data-like node.

  • Rename node class for JobCalculation [#2201](sphuber) Fixes #2199

    The node class changed from JobCalculation to CalcJobNode.

    Since the JobCalculation class is heavily used by users, as it is the class that is sub classed to define a job calculation, we cannot quite get rid of it. Instead we moved all the logic from the old Calculation and JobCalculation classes to their new equivalents ProcessNode and CalcJobNode, and put an alias to CalcJobNode for JobCalculation.

    This last class can now still be imported from aiida.orm.calculation and aiida.orm.calculation.job. Note that the paths that import directly from the implementation.general sub module have been removed, because the job calculation classes are now fully concrete.

    Furthermore, the old entry points for the base calculation classes inline, job work and function have been removed as they have been replaced with the entry points in the aiida.node group for the new node classes.

  • Converted Group to new backend system [#2210](sphuber) Fixes #2080

  • Implement new link types [#2220](sphuber) Fixes #2177

    With the new ORM hierarchy in place, the corresponding link types can be defined:

    CREATE: CalculationNode -> Data RETURN: WorkflowNode -> Data INPUT_CALC: Data -> CalculationNode INPUT_WORK: Data -> WorkflowNode CALL_CALC: WorkflowNode -> CalculationNode CALL_WORK: WorkflowNode -> WorkflowNode

    These rules are encoded in the validate_link function in aiida.orm.utils.links. This function is called from add_link_from, which has been renamed to add_incoming, and validates the triple (source, link, target) representing the addition of a link from a source node to a target node.

  • Converted Log to new backend system [#2227](muhrin)

    • Removed custom Log find methods. These are piped through to the QueryBuilder now
    • Temporarily adding dblog dummy model (can be removed once spyros finishes with the aldjemy work)

  • Move the Comment class to the new backend interface [#2225](sphuber) Fixes #2223

  • Make Groups great again [#2329](yakutovicha) Fixes #2075 and fixed #160

    • aiida.orm.importexport

      • While importing data from the export file allow to specify user-defined group and to put all the imported data in this group

    • aiida.common.utils

      • Remove get_group_type_mapping function which was mapping machine-specific group names with the user-friendly ones
      • Add escape_for_sql_like that escapes % or _ symbols provided by user

    • aiida.orm.groups

      • Add GroupTypeString enum which contains all allowed group types: data.upf (was data.upf.family) auto.import (was aiida.import) auto.run (was autogroup.run) user (was empty string)
      • Remove Group.query and Group.group_query methods, as they are redundant

    • aiida.orm.data.upf:

      • Set UPFGROUP_TYPE to GroupTypeString.UPFGROUP_TYPE
      • Replace the usage of Group.query by QueryBuilder in get_upf_groups and get_upf_family_names methods

    • aiida.orm.autogroup:

      • set VERDIAUTOGROUP_TYPE to GroupTypeString.VERDIAUTOGROUP_TYPE

    • aiida.cmdline.commands.cmd_group

      • Add verdi group copy
      • Add option to show all available group types
      • Add defaulf for group_type option
      • Replace Group.query with QueryBuilder in verdi group list
      • Remove usage of the get_group_type_mapping() function

    • aiida.cmdline.commands.cmd_import

      • Add the possibility to define a group that will contain all imported nodes.

    • aiida.cmdline.params.types.group

      • Add the possibility to the GroupParamType to create groups if they don't exist

    • aiida.backend*:

      • Rename type and name to type_string and label for the database models

    • Improve documentation for django and sqla backends migration

  • Add CalculationTools base and entry point aiida.tools.calculations [#2331](sphuber) Fixes #2322

    With the migration to the new provenance design, the type string of calculation nodes, no longer refer to the actual sub class of the JobCalculation that was run, but just the base CalcJobNode. The actual class of the calculation process was stored in the process_type if it could be mapped onto a known entry point.

    However, this change means that when a user now loads a node of a completed calculation node, let's say PwCalculation, the loaded node will be an instance of CalcJobNode and not PwCalculation, which means that any utility methods defined on the PwCalculation class are inaccessible.

    We can return this functionality through the concept of calculation tools, which will get exposed through the CalcJobNode class and will load a specifically registered entry point.

  • Define correct type string for CalcJobNode instances [#2376](sphuber) The type string of a CalcJobNode should now be just that of the node class and not the sub class. Since the users are currently still sub classing the node class and not a process class, we have to make a special exception when generating the type string for the node.

    Conversely, this type string, stored in the type column of the node, should be used to load the CalcJobNode class when loading from the database. The only classes that can legally exist in a database, and therefore be loaded, are defined in aiida-core. Therefore, using the entry point system to map the type string onto an actual ORM class is no longer necessary. We rename the aiida.plugins.loader.load_plugin function to the more correct load_node_class, which given a type string, will return the corresponding ORM node sub class.

    Note that the whole machinery around generating type and query strings and loading the nodes based on them is still somewhat convoluted and contains hacks for two reasons:

    1. Data is not yet moved within the aiida.orm.node sub module and as a result gets the data.Data. type string, which will not match the node.Node. type when sub classing in queries.

    2. CalcJobProcesses are defined by sub classing JobCalculation Until the user directly define a Process sub class that uses the CalcJobNode as its node class, exceptions will have to be made.

    If these two issues are addressed, a lot of the code around type strings can be simplified and cleaned up.

  • Further simplification of node type string definition and loading [#2401](sphuber) Fixes #1603

    With the recent introduction of the CalcJob process, which no ensures that also for job calculations, it is the CalcJobNode that gets stored in the database, which has a type string just like all the other node types. This made a lot of the old logic aiida.plugins.loader to determine the correct type string for a Node (sub) class obsolete as now all node classes are per definition internal to aiida-core and have a type string that is based directly on their module path.

    The few functions that remain, get_type_string_from_class that formats the correct type string for a given Node class and the load_node_class which can reload that class from the string have been moved to aiida.orm.utils.node since it has no longer anything to do with a "plugin".

    Finally, by moving the WorkflowNode, CalculationNode and ProcessNode classes in the __init__ files of their respective modules allows us to add further simplification to load_node_class as well as making all these classes and their subclasses importable from the aiida.orm.node module. Eventually, when aiida.orm.data is also moved there, all classes can even be exposed directly on aiida.orm and that is the deepest a user should ever go for they should need or be allowed to import.

  • Move aiida.orm.data within the aiida.orm.node module [#2402](sphuber) Fixes #2200

    This final move makes the hierarchy of all node modules and its sub classes consistent. The Data and ProcessNode classes who are both sub classes of the Node base class and are direct siblings live on the same level in the hierarchy, direct within the aiida.orm.node module.

    Note that because the type string of all Data nodes now will start with node. instead of data., a data migration had to be added.

Workflows

Upgrading workchains from pre v1.0.0

One of the most significant changes in version 1.0.0 is the new workflow engine. The new engine aims to improve its robustness and userfriendliness. In doing so a few minor changes were introduced that break workchains that were written before 1.0.0. However, these workchains can be updated with just a few minor updates that we will list here:

  • The free function submit in any WorkChain should be replaced with self.submit.
  • The _options input for CalcJob is now options, simply removed the leading underscore.
  • The label and description inputs for CalcJob or a WorkChain have also lost the underscore.
  • The free functions from aiida.work.run have been moved to aiida.work.launch, even though for the time being the old import will still work.
  • The future returned by submit no longer has the pid attribute but rather pk.
  • The get_inputs_template class method has been replaced by get_builder. See the section on the :ref:process builder<process_builder> on how to use it.
  • The import aiida.work.workfunction.workfunction has been moved to aiida.work.process_function.workfunction.
  • The input_group has been deprecated and been replaced by namespaces. See the section on :ref:port namespaces<ports_portnamespaces> on how to use them.
  • The use of a . (period) in output keys is not supported in Process.out because that is now reserved to indicate namespaces.
  • The method ArrayData.iterarrays() has been renamed to ArrayData.get_iterarrays().
  • The method TrajectoryData._get_cif() has been renamed to TrajectoryData.get_cif().
  • The method TrajectoryData._get_aiida_structure() has been renamed to TrajectoryData.get_structure().
  • The method StructureData._get_cif() has been renamed to StructureData.get_cif().
  • The method Code.full_text_info() has been renamed to Code.get_full_text_info().
  • The method Code.is_hidden() has been changed and is now accessed through the Code.hidden property.
  • The method RemoteData.is_empty() has been changes and is now accessed through the RemoteData.is_empty.
  • The method .is_alloy() for classes StructureData and Kind is now accessed through the .is_alloy property.
  • The method .has_vacancies() for classes StructureData and Kind is now accessed through the .has_vacancies property.
  • The arguments stepids and cells of the :meth:TrajectoryData.set_trajectory()<aiida.orm.node.data.array.trajectory.TrajectoryData.set_trajectory> method are made optional which has implications on the ordering of the arguments passed to this method.
  • The list of atomic symbols for trajectories is no longer stored as array data but is now accessible through the TrajectoryData.symbols attribute.

  • Convention of leading underscores for non-storable inputs has been replaced with a proper non_db attribute of the Port class [#1105] (sphuber)

  • @sphuber we should replace this with the sub-namespace where we now have store_provenance etc.

  • TODO should we move the 'updating workchains from pre-v1.0.0' from readthedocs to here, to keep only one version?

  • TODO add dropping of LegacyWorkflows

Maybe

  • Implemented the ProcessBuilder which simplifies the definition of Process inputs and the launching of a Process [#1116](sphuber) - Probably remove?

  • Namespaces have been added to the port containers of the ProcessSpec class [#1099](sphuber)

  • Remove implementation of legacy workflows [#2379](sphuber) Fixes #2378

    Note that the database models are kept in place as they will be removed later, because it will require a migration that will drop the tables. However, we want to include some functionality that allows the user to dump the content for storage, before having the migration executed.

Job calculations

In the workflow system, it is not possible to launch calculations by first creating an instance of the Calculation and then calling the calculation.use_xxx methods, as it was common in early versions of AiiDA. Instead, you need to pass the correct Calculation class to the run or submit function, passing the nodes to link as input as kwargs. For the past few versions, we have kept back-compatibility by supporting both ways of submitting. In version 1.0 we have decided to keep only one single way of submitting calculations for simplicity.

  • TODO add PRs with the fix above

  • Implement the new calcfunction decorator. [#2203](sphuber) Fixes #2172

    This calcfunction decorator replaces the make_inline and optional_inline decorators, which are deprecated. The calcfunction operates identical to the workfunction except the former gets a CalcFunctionNode to represent itself in the provenance graph as opposed to the WorkFunctionNode of the latter.

    The behavior of the optional_inline decorator is already intrinsically contained within the calcfunction and workfunction decorators. Since under the hood a FunctionProcess will be generated, and each process has an input port store_provenace that can be toggled to False to prevent the provenance from being stored, each calcfunction can be run without storing the provenance by simply passing store_provenance=False when calling the function.

  • Refactor the function process decorator [#2246](sphuber) Fixes #2171

    Note: I have not updated the documentation as that will require more work in any case and I leave it for the coding week.

    The calcfunction and workfunction decorator are essentially running the same code, with the only difference being the type of node they use to represent their execution in the provenance graph. The code has been refactored to a generic process_function decorator which takes a node class as an argument. The calcfunction and workfunction decorators are then simply defined by wrapping the process_function with the correct node class, CalcFunctionNode and WorkFunctionNode respectively.

    The difference in link rules between the two concrete decorators is also completely defined by those node classes, so the logic of the FunctionProcess class that is dynamically generated by the process function decorator can be implemented once. The common functionality of defining and overriding inputs is tested in test_process_functions. A process_function should never actually be run, but for testing purposes we defined a dummy ORM class.

    Functionality that is specific to the two concrete decorators, such as which type of links can be created, is tested individually.

  • Put in an explicit check raising when calculations call another process [#2250](sphuber) Fixes #2248

    By definition, calculation type processes cannot call other processes. This was already indirectly guarded against by the link validation process. If one attempted to add a CALL link from a CalulationNode, the link validation would raise a ValueError. However, it is more instructive for the user to catch this problem before attempting to add the link. Therefore, when setting up the database record for a process, if a parent process is available, we ensure that it is not a calculation type process, or we raise an InvalidOperation exception. A test is added for a calcfunction calling another calcfunction.

  • Implement CalcJob process class [#2389](sphuber) Fixes #2377, fixed #2280, fixes #2219 and fixed #2381

This commit can be summarized in three steps:

  • Reimplementation of a job calculation as Process called CalcJob
  • Changing job calculation to be purely informational
  • Remove the old job calculation mechanics and business logic

The old way of creating a job calculation, was to subclass the JobCalculation class and override the _use_methods class method to define the input nodes and the _prepare_for_submission to setup the input files for the calculation. The problem was that these methods were implemented on the Node class, thus mixing the responsabilities of running and introspecting the results of a completed calculation.

Here we define the CalcJob class, a subclass of Process. This class replaces the old JobCalculation and allows a user to defined the inputs and outputs through the ProcessSpec, just as they would do for a WorkChain. Except, instead of defining an outline, one should implement the prepare_for_submission, which fulfills the exact same function as before, only it is now a public method of the CalcJob process class.

Finally, the role of the job calculation state, stored as an attribute with the key state on the CalcJobNode has changed significantly. The original job calculations had a calculation state that controlled the logic during its lifetime. This was already superceded a long time ago by the process wrapper that now fully governs the progression of the calculation. Despite the calculation state no longer being authoritative during the calculation's lifetime, it was still present. Here we finally fully remove and only leave a stripped down version. The remaining state is stored as an attribute and is a sub state while the CalcJob process is in an active state and serves as a more granual state that can be queried for. This is useful, because the process status, which also keeps similar information is human readable and doesn't allow for easy querying.

  • Redesign the Parser class [#2397](sphuber) Fixes #2390

    Now that job calculations are directly implemented as processes, by sub classing the CalcJob process, a lot of functionality has been removed from the CalcJobNode that did not belong there. However, quite a bit of attributes and methods remain there that are used by the Parser that parses results attached to the node, such as names for retrieved links. These should, just like the input and output nodes, be specified on the Process class.

    We redesign the Parser class to get this the information it needs from the Process class which can be gotten through the CalcJobNode as a proxy. Sub sequently, it can directly get information like default output node labels as well as exit codes from the process class. The base method that will be called by the engine to trigger parsing of a retrieved job calculation is the parse method. The engine will pass the nodes that the parser needs for parsing as keyword arguments. The engine knows which nodes to pass as they have been marked as such by the CalcJob process spec.

    The Parser class exposes a convenience method parse_from_node that can be called outside of the engine, passing in a CalcJobNode. This will trigger the parse method to be called again, but it will be wrapped in a calcfunction ensuring that the produced outputs will be connected to a ProcessNode in the provenance graph.

    Finally, what was the CalculationResultManager has been renamed to CalcJobResultManager and has been moved to aiida.orm.utils. The interface has also been adapted and it now no longer goes through the parser class to get the information it needs but rather through the CalcJob class that can be retrieved from the CalcJobNode for which it is constructed.

Not-back-comp

  • Implement the concept of an "exit status" for all calculations, allowing a programmatic definition of success or failure for all processes [#1189]

  • Implement the get_options method for JobCalculation [#1961](sphuber)

QueryBuilder

  • Fix 2183 Update QB Relationship Indicators [#2224](ConradJohnston) Fixes #2183 .

    Changes the existing QueryBuilder join types frominput_ofand output_of to with_outgoing and with_incoming respectively. For now, ancestor_of, descendant_of remain in use. The remaining join types are changed to with_{entity} where the type of join performed depends contextually on the class passed in the qb.append() method. For example: QueryBuilder().append(User, tag='u').append(Group, with_user='u').all() Throughout the code, the old relationships have been replaced.

    The old methods, if used, now print a deprecation warning, provided by the new AiidaDeprecationWarning class, from aiida.common.warnings. This warning has the advantage of working with pycharm and not being swallowed like DeprecationWarning. All deprecation warnings in the code have now been replaced by this new warning and the documentation has been updated.

  • Change 'ancestor_of'/'descendant_of' to 'with_descendants'/'with_ancestors' [#2278](ConradJohnston) Fixes #2209 .

    Deprecates the Querybuilder 'ancestor_of'/'descendant of' join type specifications and replaces them with 'with_descendants'/'with_ancestors' respectively. All deprecated uses are updated and the docs changed to reflect this new convention.

  • Enforced the behavior of queries to ressemble backend, by disallowing deduplication of results by SQLAlchemy [#2281](lekah) This resolves #1600, by forcing SQLAlchemy to not de-duplicate rows. Therefore, the backend (psql) behavior is more closely reproduced, and we have the following equality:

    qb.count() == len(qb.all())

    A test for this was added.

Verdi

The verdi command line interface has been migrated over to a new system (called click), making the interface of all verdi commands consistent: now the way to specify a node (via a PK, a UUID or a label) is the same for all commands, and command-line options that have the same meaning use the same flags in all commands. To make this possible, the interface of various verdi commands has been changed to ensure consistency. Also the output of most commands has been homogenised (e.g. to print errors or warnings always in the same style). Moreover, some of the commands have been renamed to be consistent with the new names of the classes in AiiDA.

  • Migrate verdi to the click infrastructure [#1795]

  • The output of verdi calculation list and verdi work list has been homogenized [#1197]

  • Improve the grouping and ordering of the output of verdi calculation show [#1212]

  • verdi code show no longer shows number of calculations by default to improve performance, with --verbose flag to restore old behavior [#1428]

  • verdi work tree has been removed in favor of verdi work status [#1299]

  • Homogenize the interface of verdi quicksetup and verdi setup [#1797]

  • Synchronize heuristics of verdi work list and verdi calculation list [#1819]

  • ZipFile: set allowZip64=True as the default. [#1619](yakutovicha) Solves the issue #1617 (exporting large databases)

    According to my tests enabling the flag allowZip64=True does not change the time to create a zip archive. For more details see #1617

  • Renaming and reorganizing some verdi commands [#2204](sphuber) Fixes #2175

  • Implement verdi config [#2354](sphuber) Fixes #2208

    The new command verdi config replaces the various commands under verdi devel that were used to list, get, set and unset configuration options, that were used to be called "properties". Since the term "property" is a reserved keyword, it was decided to rename them to "options", which is also the term used by "git". The interface of verdi config also mirrors that of git config. That is to say to get the value of an option simply call git config <option_name> and to set it git config <option_name> <option_value>. To unset it, the --unset flag can be used. Finally, the getting, setting or unsetting can be applied to a certain "scope", meaning to be configuration wide or profile specific.

    To make the implementation of this command simple and clean, the bulk of the work was in refactoring the definition, construction and operation on the configuration of an AiiDA instance. This is now represented by the Config class, through which these configuration options can be set or unset as well as retrieved.

Probably not

  • Add utility functions based on the click library to simplify writing command line interface scripts [#1194]
  • verdi export create set allowZip64=True as the default when exporting as zip file [#1619]
  • Fix the formatting of verdi calculation logshow [#1850]

Importers

As part of a number of new features and improvements, the underlying library used in the CifData object has been updated to use pymatgen rather than ase.

  • Default library used in _get_aiida_structure to convert the CifData to StructureData has been changed from ase to pymatgen [#1257]

Developers

  • fix plugin test fixtures for unittest [#1622](ltalirz)
    • unittest fixtures now use a custom testrunner
    • testrunner allows to run multiple testcases (fix #1425)
    • testrunner will set up aiida environment only once per run (fix #1451)

Non-breaking? but important (Python3)

  • Verdi run: don't inject future statements into the target script [#2218](borellim) Fix #2217

Content above

Suspected to be non-breaking

From v1.0.0a4 ChangeLog

General

  • Code has been made python 3 compatible. [#804][#2136][#2125][#2117][#2110][#2100][#2094][#2092]
  • AiiDA now enforces UTF-8 encoding for text output in its files and databases. [#2107]
  • Implementation of the AuthInfo class which will allow custom configuration per configured computer [#1184]
  • Implemented the DbImporter for the Materials Platform of Data Science API, which exposed the content of the Pauling file [#1238]
  • Implement the has_atomic_sites and has_unknown_species properties for the CifData class [#1257]
  • Added element X to the elements list in order to support unknown species [#1613]
  • Enable use of tuple in QueryBuilder.append for all ORM classes [#1608], [#1607]
  • Bump version of Django to 1.8.19 for py3 support [#1915]

Engine and daemon

  • Each profile now has its own daemon that can be run completely independently in parallel [#1217]
  • Polling based daemon has been replaced with a much faster event-based daemon [#1067]
  • Replaced Celery with Circus as the daemonizer of the daemon [#1213]
  • The daemon can now be stopped without loading the database, making it possible to stop it even if the database version does not match the code [#1231]
  • Ported job calculation to use coroutines for tasks [#1827]
  • Implement exponential backoff retry mechanism for transport tasks [#1837]
  • Pause JobProcess when transport task falls through exponential backoff [#1903]
  • Each daemon worker now respects an optional minimum scheduler polling interval [#1929]

Workflows

  • InlineCalculations have been ported to use the new Process infrastructure, while maintaining full backwards compatibility [#1124]
  • Implemented a Sphinx extension for the WorkChain class to automatically generate documentation from the workchain definition [#1155]
  • Added new feature for a WorkChain to expose the inputs and outputs of another WorkChain, which is perfect for writing modular workflows [#1170]
  • Add built-in support and API for exit codes in WorkChains [#1640], [#1704], [#1681]
  • Overload PortNamespace mutable properties upon exposing [#1635]

Job calculations

  • Implement method for JobCalculation to create a restart builder [#1962]

Verdi

  • Added new command verdi process to interact with running processes [#1855]
  • Added new command verdi group rename [#1224]
  • Added new command verdi code duplicate [#1737]
  • Added new command verdi computer duplicate [#1937]
  • Added new command verdi profile show [#2028]
  • Added new command verdi work show [#1816]
  • Added new command verdi export inspect [#2128]
  • Determine active nodes in verdi calculation list based on process state [#1873]
  • Add the option --version to verdi to display current version [#1811]
  • Improve error message for verdi import when archive version is incompatible [#1960]

Database

  • Allow PostgreSQL connections via unix sockets [#1721]
  • Creating unique constraint and indexes at the db_dbgroup_dbnodes table for SqlAlchemy [#1680]
  • Performance improvement for adding nodes to group [#1677]

Schedulers

  • Renamed aiida.daemon.execmanager.job_states to JOB_STATES, conforming to python conventions [#1799]
  • Abstract method aiida.scheduler.Scheduler._get_detailed_jobinfo_command() raises aiida.common.exceptions.FeatureNotAvailable (was NotImplemented).

Importers

  • Added an importer class for the Materials Project API [#2097]

Documentation

  • Big reorganization of the documentation structure [#1299]
  • Added section on the basics of workchains and workfunctions [#1384]
  • Added section on how to launch workchains and workfunctions [#1385]
  • Added section on how to monitor workchains and workfunctions[#1387]
  • Added section on the concept of the Process [#1395]
  • Added section on advance concepts of the WorkChain class, as well as best-practices on designing/writing workchains [#1459]
  • Remove outdated or duplicated docs for legacy and new workflow system [#1718]
  • Added entry of potential issues for transports that enter bash login shells that write spurious output [#2132]

Bug fixes

  • Fix leaking of SSH processes when using a proxy command for a computer using SSH transport [#2019]

  • Fixed a problem with the temporary folder containing the files of the retrieve_temporary_list that could be cleaned before parsing finished [#1168]

  • Fixed a bug in the store method of CifData which would raise and exception when called more than once [#1136]

  • Restored a proper implementation of mutability for Node attributes [#1181]

  • Fix bug in verdi export create when only exporting computers [#1448]

  • Fix copying of the calculation raw input folder in caching [#1745]

  • Fix sphinxext command by allowing whitespace in argument [#1644]

  • Check for test_ keyword in repo path only on last directory during profile setup [#1812]

  • Fix bug in the RemoteData._clean method [#1847]

  • Differentiate quicksetup profile settings based on project folder [#1901]

  • Ensure WorkChain does not exit unless stepper returns non-zero value [#1945]

  • Fix variable virtual_memory_kb in direct scheduler. [#2050]

Developers

  • Enable tab-completion for verdi devel tests [#1809]
  • Add -v/--verbose flag to verdi devel tests [#1807]

From provenance_redesign, not in develop or v.1.0.0a4

  • Remove unserializable data from metadata in Log records [#2469](sphuber) Fixes #2363

    If unserializable data, often found in the exc_info or args keys, is not removed from the metadata of a log record, a database exception will be thrown.

  • Add the verdi data upf content command [#2468](sphuber) Fixes #2467

    This will print the bare contents of the file behind CifData nodes

  • Implement support for process functions in verdi process report [#2466](sphuber) Fixes #2465

    The nodes used by process functions, i.e. CalcFunctionNode and WorkFunctionNode also can have log messages attached that should be displayable through verdi process report. This is implemented in a helper function get_process_function_report.

  • Ensure nested calcfunctions work as long store_provenance=False [#2464](sphuber) Fixes #2444

    In principle calcfunctions are not allowed to call other calcfunctions because a calculation is not allowed to call other calculations. However, calcfunctions can be called as normal functions by setting the metadata variable store_provenance to False. Therefore, as long as store_provenance is turned off for nested functions, a calcfunction is allowed to call other calcfunctions.

    To make this work, the Process._setup_db_record method had to be adapted to only add a call link to the parent process, if and only if store_provenance for the child node is enabled. Otherwise, the link validation would fail since a CALL link outgoing of a CalculationNode is forbidden.

  • Pin the version of dependency pika==1.0.0b1 [#2463](sphuber) The second beta was just released which breaks our implementation. This will have to be adapted later.

  • Disable some SSH transport tests until Travis issue is solved [#2461](sphuber) Problems described in issue #2460

  • Fix upper limit for dependency to pg8000<1.13.0 [#2450](sphuber) Fixes #2449

    The library pg8000 which is used by pgtest to interface with Postgres and is used by us for testing purposed released a new minor version v1.13.0 on Feb 1 2019, in which it dropped py2 support causing our tests to fail miserably. For the time being, we put an upper bound on that dependency.

  • Corrects the Log migrations, improves import/export [#2447](szoupanos) This PR improves the DbLog migrations for Django & SQLA that were created for issues #1102 and #1759 with PR #2393.

    More specifically the changes are described in issue #2423. Also the objuuid was removed and objpk is used. Import/export was adapted by @CasperWA and log is exported as Node records (with their pk which is recreated during export/import to avoid pk collisions).

    We have to see what happens when a node is re-imported with more log entries - We should import the new log entries based on their UUID. A new ticket should be created for this.

  • Catch UnroutableError for orphaned processes in verdi process calls [#2445](sphuber) Fixes #2443

    If the RabbitMQ task of a running process is lost before it is completed, it will never complete, but any verdi process commands that try to reach it such as pause, play and kill will throw an UnroutableError. This exception should be caught and an error message should be printed instead.

  • Fix incorrect call in StructuresCollection.find of MpdsImporter [#2442](sphuber) Fixes #2441

    The find method requires a query as a first argument which was not being passed.

  • Fix bug in delete_nodes when passing pks of non-existing nodes [#2440](sphuber) Fixes #2439

    The aiida.manage.database.delete.nodes.delete_nodes function was not checking whether the pks that were passed correspond to existing nodes which would cause the line that fetches the repository folders, after the graph traversal queries had been completed, to except. To fix this we first verify that the passed pks exist and if not print a warning and discard them.

  • Fix import error of md5_file in CifData and UpfData [#2438](sphuber) Fixes #2437

    The recent move of md5_file from aiida.common.utils to aiida.common.files was not caught in some of the methods of these classes because it imported the whole module aiida.common.utils and used the function as aiida.common.utils.md5_file.

  • Fix description of 'contains' filter in QB docs [#2429](borellim) The 'contains' filter operator in the QueryBuilder takes a list of values, not an individual value. This is confirmed by in-source documentation in aiida/orm/implementation/sqlalchemy/querybuilder.py.

  • Add exception for calculation.job.JobCalculation process type inference [#2428](sphuber) Fixes #2406

    In the migration following the provenance redesign, the process type had to be inferred from the existing type string for JobCalculation nodes. The function infer_calculation_entry_point did not account for literal calculation.job.JobCalculation. type strings being present and would return .JobCalculation as the fallback process type. However, it makes more sense to just have an empty process type string in this case, as would be the case for a base CalcJobNode instance.

  • Prevent verdi from failing on empty profile list [#2425](astamminger) Resolves #2424

    Added a simple check for an empty profile list to the verdi base command, so that it only attempts to load a profile if the configs profile list is not empty (profile will be set to None otherwise, such as it is already done for the config if get_config() throws an exception).

    I added a small testcase as well to check for this issue in the future.

  • Changes to Data class attributes and TrajectoryData data storage [#2422](asle85) Fixes #201

    Added migrations and tests for TrajectoryData symbols (from numpy array to attribute). In sqlalchemy migrations and test I used load_node; to discuss if we want to rewrite it to avoid using aiida functionality (e.g. to get and delete numpy arrays).

  • Export/import of extras: [#2416](yakutovicha) fixes #1761

    1. Extras are now exported together with other data
    2. Extras are now fully imported if the corresponding node did not exist
    3. In case the imported node already exists in a database the following logic may be chosen to import the extras:
      • keep_existing (default): keep extras with different names (old and imported ones), keep old value in case of name collision
      • update_existing: -/-/-, keep new value in case of name collision
      • ask: -/-/-, ask what to do in case of the name collision
      • mirror: completely overwrite the extras by the imported ones
      • none: keep old extras

  • Freeze pip version to 18.1 in Travis [#2415](CasperWA) Hack-fix for issue #2414.

    Also add .vscode to .gitignore.

  • Export/import of comments (#1760) [#2413](CasperWA) This fixes #1760.

    Comments can now be exported/imported using export-version 0.4.

    At this stage of the implementation, all comments pertaining to a node chosen for export will be exported. Comments cannot be exported by themselves, nor is a Comment a valid entity to export directly, i.e. comments can only be exported indirectly through nodes.

    There is a suggestion to add a flag to ex-/include comments when exporting.

    If multiple users add comments to a node, all users will be exported recursively.

    Examples of how data.json and metadata.json might look have been updated with Comment in the documentation.

    An additional pair of joins has been added to QueryBuilder:

    • with_comment to join a Comment to a User
    • with_user to join a User to a Comment

    These, along with a similar pair of joins for Node, has been added to the table of Joining entities in the documentation.

    Lastly, Comment can now be imported directly from aiida.orm.

  • update rmq docs for macports [#2411](ltalirz)

  • Fix blocking bug on new SQLAlchemy migration tests [#2405](giovannipizzi) The problem was that the session was not properly closed. I am also moving some logic around and having a couple of useful methods that are reused in all tests.

    This fixes #2404

  • Fix bug in the setup of a profile through verdi setup/quicksetup [#2395](sphuber) Fixes #2394

    When setting up a profile that already exists, verdi is supposed prompt the user whether they want to reuse the information of the existing profile or if they want to change it. However, instead of asking to confirm, the command was prompting, causing the user to always have to reenter the information.

  • Necessary changes to properly export and import log records. [#2393](szoupanos) This PR adresses issue #1759

  • Add framework for migration tests in SQLAlchemy [#2392](giovannipizzi) This fixes #2391. In particular it adds a framework for testing migrations in SQLAlchemy/Alembic, and ports one test from Django, also to act as an example and to verify that the migration testing is properly working.

    I've also removed old files that were running some tests but actually were just running on a set of example migrations and not on those of AiiDA.

  • Add a transaction context manager to backend [#2387](sphuber) This allows to group operations that will be rolled back if the context is exited with an exception. This is laying the groundwork for implementing Node as part of the new backend system as links, caches, etc will have to be done in a transaction.

  • Ensure that configuration options are considered for logging config [#2375](sphuber) Fixes #2372

    The configuration knows various options to change the logging configuration, however, these were not respected for two reasons:

    • Logging configuration was not lazily evaluated
    • Globally configured options were not agglomerated

    The first problem was caused by the fact that the logging configuration is evaluated upon loading the aiida module, at which point the profile is not necessarily loaded yet, causing the get_config_option functions to return the option defaults. The solution is to have the dictionary lazily evaluated by using lambdas, which are resolved when configure_logging is called. Finally, we make sure this function is called each time the profile is set.

    The second problem arose from the fact that if a profile is defined, the get_config_option only returned the config value if explicitly set for that profile and otherwise it would return the option default. This means that if the option was defined globally for the configuration it was ignored. This is now corrected where if the current profile does not explicitly define a value for the option but it is globally defined, the global value is returned.

  • Various bug and consistency fixes for CifData and StructureData [#2374](sphuber) Fixes #2373

    Bug fixes:

    • CifData.has_partial_occupancies
    • CifData.has_unknown_species

    Added properties:

    • CifData.has_unknown_atomic_sites

    Consistency changes:

    • Kind.is_alloy -> property
    • Kind.has_vacancies -> property
    • StructureData.is_alloy -> property
    • StructureData.has_vacancies -> property

  • Add support for indexing and slicing in orm.Group.nodes iterator [#2371](sphuber) Fixes #2370

    This allows one to directly get just a single or a slice of all the nodes contained within a Group instead of having to iterate over it to get the nodes of interest.

  • Add functionality to delete individual Log entries to collection [#2369](sphuber) Fixes #2368

    Also fixed a bug in the delete method of the AuthInfo and Comment collections that were not caught because there were no tests. If the entry that was asked to be deleted did not exist, no exception was raised, however, that was expected by the implementation.

  • Add support for compound projection state in verdi process list [#2367](sphuber) Fix #2366

    Also set it as one of the default projections.

  • Merge develop into provenance_redesign and update pre-commit dependencies [#2365](sphuber) The original plan was to just merge develop into provenance_redesign since it contains some crucial bug fixes. However, tests stopped passing due to various issues with incompatible version of pre-commit tools and python versions. I added these changes on top of the merge commit and fixed a whole slew of new linter warnings that came as a result of upgrading pylint to version 2.0 which only supports python 3.

  • Reenable the Travis linter pre-commit test [#2364](sphuber) Fixes #2362

    The release of ruby gem==3.0.0 in December 2018 broke the pre-commit package because it used the --no-ri / --no-rdoc flags that were deprecated in that gem release. These flags were replaced by the new --no-document flag in pre-commit==1.13.0, which allows us to reenable the Travis linter that had been temporarily disabled.

  • Replace call to deprecated set_ methods with set_option [#2361](sphuber) Fixes #2360

    The Node._set_internal method was still calling deprecated set_ methods for options of the CalcJobNode sub class. These should now be set through the explicit set_option method. The JobProcess was likewise violating this rule, when setting the options of the underlying CalcJobNode it is using as a storage record.

  • Fix issue 1439 remove wrong path [#2359](zhubonan) Fix issue #1439, remove reference or implication of putting plugin files into the sub-packages on aiida_core. In addition, for both two tutorials I included a brief example of the plugin project directory structure. @giovannipizzi Is there any other reference you know are left out?

  • Fix bug in CalcJobNode.get_desc [#2353](sphuber) Fixes #2352

    The function still called CalcJobNode.get_state with the argument from_attribute, which has recently been removed. Originally, the state was stored in a separate database table and had a proxy as an attribute in the attribute table. The calc state table has been removed, leaving the attribute as the only source of the state.

  • Fixed issue with wrong file mode for python3 [#2351](muhrin) The log file to report an integrity violation was not correct and would complain about expecting a bytes type instead of str. Fixed by setting the mode to w+.

  • Docs: update the example provenance graph images in concepts section [#2350](sphuber) The original .svg files are also included. They were generated in Inkspace but saved as plain SVG files. After that they were exported as PNG files. With the convert Unix utility, the images were rescaled to 800 px as follows:

    convert filename.png -resize 800 filename.png

    This will rescale the image to 800px wide, maintaining the aspect ratio and writing it to the same file.

  • Refactor the management of the configuration file [#2349](sphuber) Fixes #2348

    The configuration file is now abstracted by the Config class. Loading the configuration through aiida.manage.load_config will now return a Config instance which internally keep the contents of the configuration file of the AiiDA instace. Altering the config will go through this object.

    All configuration related code, such as settings and methods to setup the configuration folder for a new AiiDA instance, are organized in the aiida.manage.configuration module.

  • Fixed 1714 expose_inputs deepcopy default value [#2347](muhrin) Allowing deepcopy of a stored node to support this use case. See the bug report https://github.com/aiidateam/aiida_core/issues/1741 for default but simply put if you expose the inputs of a workchain with a stored default plumpy will try to deepcopy that default value which was expecting because of a ban on deepcopying stored nodes.

  • Reorganised some SQL related commands [#2346](muhrin) Moved from QueryManager to the corresponding backend classes. The longer term view will probably be that QueryManager gets dropped completely and more backend specific methods will be moved into the Backend class itself. This way we have a single point where we know to find such methods.

  • Use ProfileParamType in verdi and expose in ctx.obj [#2345](sphuber) Fixes #2344

    The verdi entry point now uses the ProfileParamType for its -p option, which enables tab-completion for the profile option. In addition, we add the loaded profile to the obj attribute of the click context ctx. This way, when a verdi sub commands needs the active profile, one can simply use the @click.pass_context decorator to access the profile through ctx.obj.profile.

  • Implement verdi database integrity endpoints for links and nodes [#2343](sphuber) Fixes #2326

    These two new end points will scan the database for invalid links and nodes, by running prepared SQL statements. If any violations are found, they will be printed to the terminal. For now there are no options implemented to apply a patch to remove the violations.

  • Update export schema version and temporarily disable export [#2341](sphuber) Fixes #2340

    The migration after the provenance redesign has been merged into provenance_redesign. To avoid people from importing "dirty" export files into a cleanly migrated database, the export schema version is upped. Additionally, creating new export archives with the latest version is disabled, until the export procedure has been verified to abide by the new rules in place after the provenance redesign.

  • Add migration after the provenance redesign [#2336](sphuber) Fixes #2178

    Tested on 5 production databases:

    • COFS (MaterialsCloud)
    • SSSP (MaterialsCloud)
    • 2D (MaterialsCloud)
    • 3DD (personal)
    • SDB (personal)

  • Implement verdi database migrate [#2334](sphuber) Fixes #2332

    Up till now, the triggering of a database migration happened through separate interfaces for the two different backends. For Django, if the database schema did not match the code's, a warning message would be printed asking the user to execute a stand alone python script that was a modified Django manage.py script. For SqlAlchemy, the user was automatically prompted with an Alembic migration.

    Here we unify the migration operation by making it go through verdi through the endpoint verdi database migrate. The error message upon an outdated database schema and the to be executed commands are now identical for both backends.

  • Add test for availability of sudo in quicksetup [#2333](ConradJohnston) Partially addresses #1382

    Quicksetup needs to access the Postgres database, but does not know anything about the system setup. It initially tries to see if it is the postgres superuser, but if it is not, it tries to become the Postgres superuser via sudo. This is the "backstop" position and relies on the user having a modern Linux-like system. If, however, the sudo command is not available to this user at all (for example, perhaps the user is on an HPC cluser with command hidden), a system error will be raised and quicksetup will fail ungracefully.

    This commit adds a test to see if sudo can be found before attempting to use it, prints a warning, and then falls back to the 'Postgres not detected' behaviour.

  • Docs reorganization - step 2 (installation section) [#2330](ltalirz) This is my go at reorganizing the installation section of the docs to make it more intuitive and efficient to use.

    • quick install now contains all necessary information to actually set up AiiDA
    • foldable sections with instructions for different operating systems keep quick install page short
    • detailed instructions for different OS's retained under "prerequisites" section and linked directly from quick install
    • OS-specific sections on postgres and rabbitmq merged into corresponding "prerequisites" sections

    Note: I added the provenance_redesign branch to RTD, i.e. you'll be able to inspect the docs there (after the PR is merged).

  • Ensure QueryBuilder return frontend entity of DbLog and DbComment [#2325](muhrin) Fixes #2321

  • Process queues are now removed when a process terminates [#2324](muhrin) Fixes #2269

    Change the way that validation of ports is done which now also shows which port a validation error came from

    Also changed the process tests to use close() because it is illegal to have two active process instances that refer to the same underlying process instance (i.e. with the same id)

  • Document the Windows Subsystem for Linux [#2319](zhubonan) Fixes #2256

    Added guide for installing AiiDA on WSL in the documentation. I have tested the installation and there are a 2 failures (file modes with ssh transport and profile detetion) and 7 error out of 1051 tests. Nevertheless, I think overall it is OK to install on WSL if the user just want to try out on a windows machine especially when they don't have enough RAM to support a virtual machine.

  • upgrade prospector [#2317](ltalirz) prospector 1.1.5 finally works with pylint <2

  • document how to enable tab completion in conda [#2316](ltalirz)

  • Various speedups for verdi cmdline [#2315](giovannipizzi) Fixes #376 and fixes #2261

    In particular when completing tests. Thanks to @ltalirz This fixes #376 (again...). Fixes include:

    • using a new instance of the reentry manager that prevents automatic reentry scans
    • removed unuseful with_dbenv decorators from 'verdi devel tests' and from the TestModuleParamType of click
    • prevent unnecessary imports of plumpy that is a bit slow to import

    Also, we fixed the place where AIIDADB_PROFILE is set in the main verdi click command, to set it also when a default profile is used (this was probably the reason of the unneeded with_dbenv calls)

  • Add documentation for plugin fixtures [#2314](zhubonan) Added documentation about writing pytest styled tests using the aiida.utils.fixtures module (#822). I also included a guide with examples for migrating existing tests that are sub-class of AiidaTestCase to run with pytest.

  • Fixes #1249 Removed pika log level [#2312](muhrin) This was added before, see #1249, but this no longer happens as we're using separate a thread for the pika event loop and what's more the issue has, I believe, been resolved with pika 1.0 which we now use.

  • Remove get_aiida_class [#2309](muhrin) Fixes #2237

    This is part of the procedure that @szoupanos is leading to convert to automatically generated dummy models for the QueryBuilder

    Replaced all instances with either get_backend_entity or get_orm_entity (as appropriate).

    Also changed backend group to be independent of the ORM. Because the iterator was getting the orm class and returning that (which required it to access up to the ORM level). Now it just returns backend entities which are in turn converted to ORM entities by the orm.Group. This also means you can't call len(group.nodes) because this is not allowed for iterators but the same can be achieved with len(list(group.nodes)).

    Removal of get_aiida_class from QueryBuilder and addition of DbModel-to-BackendEntity and BackendEntity-to-OrmEntity convertors.

  • Migrate the type and process type string of built in calculation plugins [#2308](sphuber) Fixes #2306

    The built in calculation entry points were recently moved and renamed but no migration was put in.

    This requires renaming the node types, process types and the input plugin attribute of code instances.

  • Move requirements from setup_requirements.py to setup.json [#2307](ltalirz) fix #2241

    This moves all dependency information from the setup_requirements.py python file to a setup.json.

    • this makes it easier for plugin developers to process dependencies of aiida-core
    • once merged, aiida-core can easily be registered on the aiida-plugin-registry. This will allow us to implement automatic checks on collisions of entry points
    • add pre-commit hook to update version number in setup.json (necessary to keep it there for the registration in the registry)
    • remove pylint blacklist from .pylintrc. the only blacklist is now in .pre-commit-config.yaml
    • remove setup.py from blacklist
    • remove dev_spinxext extra (contained only pytest, made part of testing)

  • Fixes structure visualisation end point [#2305](waychal) Fixes #1763

  • Add provenance docs [#2302](asle85) added provenance documentation file and figures

  • Move practical developer docs to AiiDA wiki [#2297](ltalirz)

  • Improve robustness of parsing versions and element names from UPF files [#2296](astamminger) Fixes #2228

    Changed parsing behavior such that regular expressions for elements and versions are now matched against the whole file instead of matching line by line. This makes the parsing process more robust and allows for parsing desired quantities independent of the actual position of the UPF file (See for instance the additional header line mentioned in issue #2228). Since re.search() returns the first match found in the string, there is no major performance impact expected compared to the previous implementation.

  • Document the purpose and use of clean_value [#2295](ConradJohnston) Fixes #2038 and #1498

    • Update the docs to explain the use of clean_value and its behaviour.
    • Update clean_value doc string to better explain its purpose.
    • Change collection imports from, for example, "import collections.Iterable" to "from collections import Iterable" to comply with a forthcoming deprecation that will break these imports in Python 3.8.

  • removed wsgi file/configuration/documentation for django application [#2293](waychal) Fixes #415

  • Ensure correct types of LinkTriples returned by Node.get_stored_link_triples [#2292](sphuber) Fixes #2291

    The Node.get_stored_link_triples was using LinkTriple instances where the type of the link_type element was not of LinkType but of string, as it was directly using the string value returned by the query builder for the type column of the DbLink table. By wrapping this value in LinkType the returned LinkTriples once again have the correct type for all its elements.

  • Temporarily lock versions of kiwipy and pika [#2290](sphuber) Fixes #2284

    A new patch version v0.3.11 of kiwipy was released that breaks functionality, so until that is fixed, we lock the version. We do the same for pika which requires v1.0.0b1.

  • Documented optional graphviz dependency and removed use of os.system() in draw_graph [#2287](ConradJohnston) Fixes #835 and #2285.

    The use of verdi graph to generate a plot of part of the provenance graph requires the graphviz package, which was not documented other than in the source code. This pull request increases the visibility of this dependency in the docs and adds a more human-parsable error message suggest when it could be the case that graphviz is not installed.

    Also replaces the use of os.system() in aiida.common.graph with subprocess.call to prevent shell script from being injected via verdi graph generate.

  • Skip test for unicode folder names if file system encoding is not set to unicode [#2283](astamminger) Fixes #294

    Since all tests run as intended even tough file system encoding is not necessarily set to unicode (ran tests for locales de_DE.UTF-8 and de_DE.iso88591) it should be safe to just skip this test if the file system encoding differs from unicode.

  • Docs reorganization - step 1 [#2279](ltalirz) Reorganization of documentation top level as discussed with @sphuber

  • specify versions of python for aiida [#2277](ltalirz) specify exactly which versions of python aiida is supposed to run on (see p.6 of presentation by @dev-zero on python3 compatibility in aiida from Sep 13th 2018)

  • Updated documentation about the usage of CodeInfo [#2276](zhubonan) Fixes #1000

    Added notes in the plugin development documentation that multiple Code and hence CodeInfos may be used for a single JobCalculation to allow pre/post-processing or packing multiple jobs in a single submission if necessary. We should be able to close issue #1000 now.

  • Escape newline chars in strings used to label nodes in visual provenance graphs [#2275](astamminger) Fixes #1874

  • Raise exception when exporting a node without path sub folder in repository [#2274](yakutovicha) Fixes #2247

    Raise an exception when exporting a node which does not contain a 'path' folder in the file repository.

  • Remove preceeding '$' chars from code-blocks. [#2273](astamminger) Fixes #1561

  • improved setup of ipython line magic %aiida [#2272](zhubonan) Fixes #1562

    Instead of telling users to edit the 'ipython_config.py', which does not exist by default. We now suggest simply add a python file in the startup folder of ipython to register the line magic %aiida. A new file is added serving as an example.

  • Check of existence of groups on import [#2270](lekah) This resolves #2074. Now on import, the existence of groups with same name is checked

  • Remove work around for postgres on Travis [#2266](sphuber) Fixes #1739

  • require recent pip version for pyproject.toml [#2264](ltalirz) fix #2262

  • Changed type of uuids returned by the QueryBuilder to be unicode [#2259](lekah) Fixes #231 and fixes #1862

  • Add --group option to verdi process list [#2254](sphuber) Fixes #2253

    This option allows a user to narrow down the query set to only those nodes that are within the specified group.

  • Retrieve computers through front end collection in _presubmit [#2252](sphuber) Fixes #2251

    The implementation was using the backend collection to get a given computer for the remote copy lists, but this method is not defined. Instead this operation should go through the front end collection.

  • Allow existing profile to be overriden in verdi setup [#2244](sphuber) Fixes #2243

    With the --force flag a user can suppress the exception that is raised when one tries to create an existing profile in non-interactive mode. With the flag the existing profile will be overridden with the specified values except for the profile UUID which will be kept the same as it is auto generate and can also not be specified by the user on the command line.

  • Make verdi callable as sub command [#2240](sphuber) Fixes #2239

    Because the verdi group was not marked as callable without subcommand, the --version option was broken as it would throw a Missing command error. Adding invoke_without_command=True to the verdi group declaration and making the --version an eager option will ensure it works without specifying a sub command.

    A test has been added to test this functionality.

  • Return correct ORM AuthInfo class for JobCalculation._get_authinfo [#2235](sphuber) Fixes #2234

    The method returned the backend AuthInfo class, which does not have the get_transport method that would be called for example in the _get_transport method of the JobCalculation class. The fix is to wrap the backend entity in the ORM class by calling

    `AuthInfo.from_backend_entity`

    Added two regression tests for this bug.

  • Fix get_metadata name of AuthInfo class (#2221) [#2222](PhilippRue) Fixes #2221 Fixes this error:

    Error: * The test raised an exception!
** Full traceback:
   Traceback (most recent call last):
     File ".../aiida_core/aiida/cmdline/commands/cmd_computer.py", line 561, in computer_test
       succeeded = test(transport=trans, scheduler=sched, authinfo=authinfo)
     File ".../aiida_core/aiida/cmdline/commands/cmd_computer.py", line 145, in _computer_create_temp_file
       workdir = authinfo.get_workdir().format(username=remote_user)
     File ".../aiida_core/aiida/orm/authinfos.py", line 136, in get_workdir
       metadata = self.get_metadata()
     File ".../aiida_core/aiida/orm/authinfos.py", line 122, in get_metadata
       return self._backend_entity.get_metadata()
   AttributeError: 'DjangoAuthInfo' object has no attribute 'get_metadata'
Some tests failed! (1 out of 4 failed)

    when running verdi computer test ...

  • Speed up creation of Nodes in the AiiDA ORM [#2214](ltalirz)

    • move AiiDAManager from aiida.work to aiida.manage
    • introduce caching of .objects in aiida.orm.entities.Collection
    • cache default user in aiida.orm.user.User.Collection (fix #2216)

    Benchmark on MacBook Pro, python3

    • before: 58.5s to create 10k nodes (57.2 spent inside Node.__init__)
    • after: 2.8s to create 10k nodes (1.7s spent inside Node.__init__)

    The timing of 58.5s is significantly worse than what we wrote down during the tests with @szoupanos and @giovannipizzi (which was ~10s for 10k nodes), i.e. probably someone broke the caching.

    Anyhow, the target we set then was: 2s for 10k nodes, which we have reached now on python3 (it's the 1.7s that should scale linearly with the number of nodes).

  • fix django 1.11 migrations [#2212](ltalirz)

    • update django settings (cleanup + adaptation to django 1.11) together with @giovannipizzi
    • fix migration of UUID column for databases created with previous schema versions
    • make sure we give unicode strings to django (not byte strings)
    • add test to check that uuid migration does not change uuids
    • add test to check that no django schema migrations remain

    Will look into adding sqlalchemy unique constraints on uuid column in a separate PR

  • Django 1.11 [#2206](ltalirz)

    • update django from 1.8.19 to 1.11.16
    • remove django-extensions dependency
    • move to UUIDField of django 1.11 + add migration
      Note: for the moment, I've replaced the uuid fields also in old migrations. One could try to use a CharField(editable=False, blank=True, max_length=36) there to be safe
    • add unique constraints on all uuid fields
    • move get_filter_expr_from_column from django/sqla to interface
    • delete django-specific get_all_parents function (was only used in tests)

  • psycopg2-binary 2.7.4 => psycopg2 2.7.6 [#2202](ltalirz) "The binary package is a practical choice for development and testing but in production it is advised to use the package built from sources." [1] https://pypi.org/project/psycopg2-binary/

  • Drop the DbCalcState table [#2198](sphuber) Fix #2197

    The DbCalcState table was introduced to both keep the current job state of JobCalculations but also to function as a sort of locking mechanism to ensure only one process was operating on a job at a time, by enforcing progressive state changes on the database level. In the new system, this responsibility is taken care of by the Runner class that is running the task corresponding to the job calculation. The calculation state is now merely set as on the node for informational purposes to the user.

    Therefore, the simpler solution is to set the job state as a simple attribute and remove the DbCalcState. This also means that the IMPORTED calc state is now obsolete, as that was needed to prevent imported calculations from being picked up by the daemon, which used to determine what calculations to run based on its calculation state.

  • Merging develop into provenance_design [#2191](sphuber)

  • Fix bug in parse_formula for formula's with leading or trailing whitespace [#2186](sphuber) FIxes #2185

    If a formula contains leading or trailing whitespace, one of the parts over which the function loops will be empty, and so the regex match m will be None. To prevent an AttributeError from being thrown when group is called on None we check for it and continue.

  • Compile the scripts before calling exec in verdi run [#2166](sphuber) Fixes #2165

    This extra step is necessary for python modules like inspect to function normally. Without this change, running a script with verdi run that defines a workfunction and then runs it would fail, because inspect would fail to determine the path of the source file of the workfunction, i.e. the script that was passed to verdi run. This change fixes that.

  • Centralize the management of entities central to the loaded profile [#2164](sphuber) Fixes #2154 and fixes #2145

    This PR contains a few commits that address the previous scattering of creation and managing of entities whose configuration depends on the loaded profile, which is now gathered in the AiiDAManager class. This ensures that entities like the RabbitMQ controllers and communicators as well as runners, get the correct configuration based on the loaded profile, as for example during testing.

  • Revert current database revision after migration unittests [#2162](sphuber) Fixes #2161

    After running a migration unittest that changes the revision of the test database one should ensure that after the tests finishes, the current revision is applied.

  • Set default polling interval for Runners to 1 second [#2160](sphuber) In a recent change this parameter was corrected from 0 to 30 seconds to prevent the daemon workers from spinning the processors too much, but this really slows down the tests. Until resources like runners and communicators are created in a central place, that allows settings to be determined based on the active profile, e.g. with special settings for test profiles, we need to revert this to 1 second. This will allow the tests to run reasonably fast, without overloading processors too much.

  • Engine stability and communications improvements [#2158](sphuber) This PR will update plumpy to version 0.11.4 which underneath runs on kiwipy version 0.3.*. The biggest change is that the communications of a Runner now happen on a separate thread, while all the real operations and callbacks are scheduled on the main thread. This should ensure that even under heavy load, a (daemon) runner remains responsive to heartbeats and remote procedure calls (RPCs) from RabbitMQ. The missing of heartbeats and the subsequent cutting of the connection by RabbitMQ spawned a whole host of other problems that will now no longer surface.

    Fixes #1994 Fixes #2088 Fixes #1748 Fixes #1426 Fixes #1821 Fixes #1822 Fixes #2157

  • Fix incorrect model attribute deref in SqlAlchemy implementation of Log [#2156](sphuber) Fixes #2155

    The id property was piping through to the objpk of the underlying model instead of the id.

  • Modularize verdi profile delete [#2151](ltalirz) fix #2121

    • add 3 options to verdi profile delete:
      • --include-db/--skip-db
      • --include-repository/--skip-repository
      • --include-config/--skip-config
    • split into delete_repository, delete_db, delete_from_config
    • move functions into aiida.control.profile

    To do:

    • skip deletion of DB in test (problem: cannot reproduce #2121 on my mac)

  • Upgrade the dependency requirement of SqlAlchemy [#2113](lekah) Fixes #465

    This fixes the long-standing issue of upgrading SQLAlchemy. The way attributes stored in a JSON were queried had to be chained. Additional tests were added to see this kind of errors before. In essence, the issue was that the type check and value check were in arbitrary order. Now a case statements enforces type check before value check.

Clone this wiki locally