Suggestion: Use tox and pytest (#178)

* Run tests via tox and py.test

* Use tox in Travis

* Use tox in AppVeyor

* Add PYJULIA_TEST_REBUILD to recompile PyCall

* Disable capturing for now

* Pass JULIA_EXE through tox.ini

* Travis style versioning in ci/install-julia.sh

* Use a script to rebuild PyCall.jl

* More debugging info about Python interpreter

* Use "python -m pytest" in tox

* Move with_rebuilt script to a module

so that it's runnable in Windows as well.

* Pass around Julia runtime path

* Don't show stack-trace from julia.with_rebuilt

* Flush stderr in Julia._debug

* Avoid pytest hiding debug message

* Smoke test with IPython; fix a bug in JuliaImporter

* Fix exception printing

The Julia exception objects don't have a `.message` attribute, we can just print the plain object.

* Fix AppVeyor configuration

* Fix comparison of the interpreter paths in Windows

* Fix env for Python 2.7 + Windows

* Flush stdout in julia.with_rebuilt

* Test success paths of IPython magics

* Test failure paths of IPython magics

* Fix exe_differs check

* Add Testing section in README

Author: tkf

.gitignore

@@ -24,6 +24,7 @@ pip-log.txt
 
 # Unit test / coverage reports
 .coverage
+.pytest_cache
 .tox
 nosetests.xml
 

.travis.yml

@@ -1,54 +1,43 @@
+language: python
+os:
+  - linux
+python:
+  - "2.7"
+  - "3.6"
+env:
+  matrix:
+    - JULIA_VERSION=0.6.4 CROSS_VERSION=1
+    - JULIA_VERSION=0.7.0-rc2
+    # - JULIA_VERSION=nightly
+  global:
+    - TOXENV=py
 matrix:
   # Python environment is not functional on OS X
   include:
-      - language: python
-        python: 2.7
-        env: JULIA_VERSION=juliareleases
-        os: linux
-      - language: python
-        python: 2.7
-        env: JULIA_VERSION=julianightlies
-        os: linux
-      - language: python
-        python: 3.5
-        env:
-          - JULIA_VERSION=juliareleases
-          - CROSS_VERSION=1
-        os: linux
-      - language: python
-        python: 3.5
-        env: JULIA_VERSION=julianightlies
-        os: linux
-      - language: python
-        python: 3.5
-        env:
-          - JULIA_VERSION=julianightlies
-          - CROSS_VERSION=1
-        os: linux
       - language: generic
         env:
           - PYTHON=python2
-          - JULIA_VERSION=julianightlies
+          - JULIA_VERSION=0.7.0-rc2
+          # - JULIA_VERSION=nightly
         os: osx
       - language: generic
         env:
           - PYTHON=python2
-          - JULIA_VERSION=juliareleases
+          - JULIA_VERSION=0.6.4
+          - CROSS_VERSION=1
         os: osx
       - language: generic
         env:
           - PYTHON=python3
-          - JULIA_VERSION=julianightlies
+          - JULIA_VERSION=0.7.0-rc2
+          # - JULIA_VERSION=nightly
         os: osx
       - language: generic
         env:
           - PYTHON=python3
-          - JULIA_VERSION=juliareleases
-        os: osx
-  allow_failures:
-      - env:
-          - JULIA_VERSION=julianightlies
+          - JULIA_VERSION=0.6.4
           - CROSS_VERSION=1
+        os: osx
 branches:
   only:
     - master
@@ -57,19 +46,21 @@ notifications:
 before_script:
   - echo ./ci/install-julia.sh "$JULIA_VERSION"
   - ./ci/install-julia.sh "$JULIA_VERSION"
-  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get update -qq -y; fi
-  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install libpcre3-dev -y; fi
-  - if [ "$TRAVIS_OS_NAME" = "linux" ]; then sudo apt-get install python-numpy python3-numpy -y; fi
   - if [ "$TRAVIS_OS_NAME" = "osx" -a "$PYTHON" = "python3" ]; then brew update; brew upgrade python || echo "Ignoring errors..."; fi
   - if [ "$TRAVIS_OS_NAME" = "osx" -a "$PYTHON" = "python2" ]; then brew update; brew list python@2 &>/dev/null || brew install python@2 || echo "Ignoring errors..."; fi
   # Ignoring errors from brew since it may actually be OK to do so.
   # Following which command will catch installation failure:
-  - which ${PYTHON:-python}
-script:
-  - julia -e 'Pkg.add("PyCall")'
-  - /usr/bin/python --version
   - PYTHON=${PYTHON:-python}
-  - echo $PYTHON
-  - $PYTHON --version
-  - if [ "$CROSS_VERSION" = "1" ]; then /usr/bin/python -m unittest discover; fi
-  - $PYTHON -m unittest discover
+  - which $PYTHON
+  - $PYTHON -m pip --version
+  - $PYTHON -m pip install --quiet tox
+  - julia -e 'Pkg.add("PyCall")'
+script:
+
+  # "py,py27" below would be redundant when the main interpreter is
+  # Python 2.7 but it simplifies the CI setup.
+  - if [ "$CROSS_VERSION" = "1" ]; then
+    $PYTHON -m tox -e py,py27 -- -s;
+    fi
+
+  - PYJULIA_TEST_REBUILD=yes $PYTHON -m tox -- -s

README.md

@@ -9,12 +9,14 @@ Experimenting with developing a better interface to [Julia language](https://jul
 to run the tests, execute from the toplevel directory
 
 ```shell
-python -m unittest discover
+tox
 ```
 
+See [Testing](#testing) below for details.
+
 **Note** You need to explicitly add julia to your `PATH`, an alias will not work.
 
-`pyjulia` is tested against Python versions 2.7 and 3.5.  Older versions of Python (than 2.7)  are not supported.
+`pyjulia` is tested against Python versions 2.7, 3.6, and 3.7.  Older versions of Python (than 2.7)  are not supported.
 
 Installation
 ------------
@@ -138,3 +140,38 @@ Limitations
 ------------
 
 Not all valid Julia identifiers are valid Python identifiers.  Unicode identifiers are invalid in Python 2.7 and so `pyjulia` cannot call or access Julia methods/variables with names that are not ASCII only.  Additionally, it is a common idiom in Julia to append a `!` character to methods which mutate their arguments.  These method names are invalid Python identifers.  `pyjulia` renames these methods by subsituting `!` with `_b`.  For example, the Julia method `sum!` can be called in `pyjulia` using `sum_b(...)`.
+
+
+Testing
+-------
+
+The full syntax for invoking `tox` is
+
+```shell
+[PYJULIA_TEST_REBUILD=yes] [JULIA_EXE=<julia>] tox [options] [-- pytest options]
+```
+
+* `PYJULIA_TEST_REBUILD`: *Be careful using this environment
+  variable!* When it is set to `yes`, your `PyCall.jl` installation
+  will be rebuilt using the Python interpreter used for testing.  The
+  test suite tries to build back to the original configuration but the
+  precompilation would be in the stale state after the test.  Note
+  also that it does not work if you unconditionally set `PYTHON`
+  environment variable in your Julia startup file.
+
+* `JULIA_EXE`: `julia` executable to be used for testing.
+
+* Positional arguments after `--` are passed to `pytest`.
+
+For example,
+
+```shell
+PYJULIA_TEST_REBUILD=yes JULIA_EXE=~/julia/julia tox -e py37 -- -s
+```
+
+means to execute tests with
+
+* `pyjulia` in shared-cache mode
+* `julia` executable at `~/julia/julia`
+* Python 3.7
+* `pytest`'s capturing mode turned off

appveyor.yml

@@ -1,45 +1,38 @@
 environment:
+
+  TOXENV: py,py27
+  TOX_TESTENV_PASSENV: DISTUTILS_USE_SDK MSSdk INCLUDE LIB
+  # https://packaging.python.org/guides/supporting-windows-using-appveyor/#testing-with-tox
+
   # for more python versions have a look at
   # https://github.com/ogrisel/python-appveyor-demo/blob/master/appveyor.yml
   matrix:
-  # 64 julia-0.6 Python-27
-  - JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x64/0.6/julia-0.6-latest-win64.exe"
-    PYTHONDIR: "C:\\Python27-x64"
-
-  # 32 julia-0.6 Python-27
+  # 32 julia-0.6 Python-35
   - JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x86/0.6/julia-0.6-latest-win32.exe"
-    PYTHONDIR: "C:\\Python27"
+    PYTHONDIR: "C:\\Python35"
+    BATDIR: ci\appveyor\win32
+    CROSS_VERSION: 1
+
+  # 32 julia latest Python-35
+  - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
+    PYTHONDIR: "C:\\Python35"
+    BATDIR: ci\appveyor\win32
 
   # 64 julia-0.6 Python-35
   - JULIA_URL: "https://julialang-s3.julialang.org/bin/winnt/x64/0.6/julia-0.6-latest-win64.exe"
     PYTHONDIR: "C:\\Python35-x64"
-
-  # 32 julia-latest Python-27
-  - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
-    PYTHONDIR: "C:\\Python27"
-
-  # 64 julia-latest Python-27
-  - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
-    PYTHONDIR: "C:\\Python27-x64"
+    BATDIR: ci\appveyor\win64
+    CROSS_VERSION: 1
 
   # 64 julia latest Python-35
   - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
     PYTHONDIR: "C:\\Python35-x64"
-
-  # 64 julia latest Cross Version
-  - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
-    PYTHONDIR: "C:\\Python35-x64"
-    CROSS_VERSION_PATH: "C:\\Python27-x64"
-
-  # 32 julia latest Cross Version
-  - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
-    PYTHONDIR: "C:\\Python35"
-    CROSS_VERSION_PATH: "C:\\Python27"
+    BATDIR: ci\appveyor\win64
 
 matrix:
- allow_failures:
-  - CROSS_VERSION_PATH: "C:\\Python27-x64"
-  - CROSS_VERSION_PATH: "C:\\Python27"
+  allow_failures:
+    - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x86/julia-latest-win32.exe"
+    - JULIA_URL: "https://julialangnightlies-s3.julialang.org/bin/winnt/x64/julia-latest-win64.exe"
 
 branches:
   only:
@@ -71,11 +64,18 @@ build_script:
   # - C:\projects\julia\bin\julia -e "using PyCall; @assert isdefined(:PyCall); @assert typeof(PyCall) === Module"
   - "SET PYTHON=%PYTHONDIR%\\python.exe"
   - C:\projects\julia\bin\julia -e "versioninfo(); Pkg.add(\"PyCall\")"
+  - "%PYTHONDIR%\\python.exe -m pip install --quiet tox"
 
 test_script:
-  - "SET PATH=%PYTHONDIR%;%PYTHONDIR%\\Scripts;C:\\projects\\julia\\bin;%PATH%"
-  - python --version
+  - "SET PATH=%cd%\\%BATDIR%;%PYTHONDIR%;%PYTHONDIR%\\Scripts;C:\\projects\\julia\\bin;%PATH%"
   - dir
-  - ps: if (Test-Path Env:\CROSS_VERSION_PATH) { Invoke-Expression "$env:CROSS_VERSION_PATH -m unittest discover" }
-  # - python -c "import julia; julia.Julia(debug=True)"
-  - python -m unittest discover
+
+  # Run cross-version tests but ignore the failures (from Python 2).
+  # Once cross-version in Windows is fmixed, stop using
+  # Invoke-Expression (which ignores the exit status).
+  - ps: if ($env:CROSS_VERSION -eq 1) { Invoke-Expression "tox -- -s" }
+  # - ps: if ($env:CROSS_VERSION -eq 1) { tox -- -s }
+
+  # Rebuild PyCall.ji for each Python interpreter before testing:
+  - "SET PYJULIA_TEST_REBUILD=yes"
+  - tox -- -s

ci/appveyor/win32/python2.7.bat

@@ -0,0 +1 @@
+@C:\Python27\python.exe %*

ci/appveyor/win32/python3.5.bat

@@ -0,0 +1 @@
+@C:\Python35\python.exe %*

ci/appveyor/win64/python2.7.bat

@@ -0,0 +1 @@
+@C:\Python27-x64\python.exe %*

ci/appveyor/win64/python3.5.bat

@@ -0,0 +1 @@
+@C:\Python35-x64\python.exe %*

ci/install-julia.sh

@@ -1,30 +1,23 @@
-#!/bin/sh
-# install julia release: ./install-julia.sh juliareleases
-# install julia nightly: ./install-julia.sh julianightlies
-
-VERSION="0.6.2"
-SHORTVERSION="0.6"
+#!/bin/bash
+# install julia vX.Y.Z:  ./install-julia.sh X.Y.Z
+# install julia nightly: ./install-julia.sh nightly
 
 # stop on error
 set -e
-# default to juliareleases
-if [ $# -ge 1 ]; then
-  JULIAVERSION=$1
-elif [ -z "$JULIAVERSION" ]; then
-  JULIAVERSION=juliareleases
-fi
+VERSION="$1"
 
-case "$JULIAVERSION" in
-  julianightlies)
+case "$VERSION" in
+  nightly)
     BASEURL="https://julialangnightlies-s3.julialang.org/bin"
     JULIANAME="julia-latest"
     ;;
-  juliareleases)
+  [0-9]*)
     BASEURL="https://julialang-s3.julialang.org/bin"
+    SHORTVERSION="$(echo "$VERSION" | grep -Eo '^[0-9]+\.[0-9]+')"
     JULIANAME="$SHORTVERSION/julia-$VERSION"
     ;;
   *)
-    echo "Unrecognized JULIAVERSION=$JULIAVERSION, exiting"
+    echo "Unrecognized VERSION=$VERSION, exiting"
     exit 1
     ;;
 esac
@@ -34,22 +27,22 @@ case $(uname) in
     case $(uname -m) in
       x86_64)
         ARCH="x64"
-        case "$JULIAVERSION" in
-          julianightlies)
+        case "$JULIANAME" in
+          julia-latest)
             SUFFIX="linux64"
             ;;
-          juliareleases)
+          *)
             SUFFIX="linux-x86_64"
             ;;
         esac
         ;;
       i386 | i486 | i586 | i686)
         ARCH="x86"
-        case "$JULIAVERSION" in
-          julianightlies)
+        case "$JULIANAME" in
+          julia-latest)
             SUFFIX="linux32"
             ;;
-          juliareleases)
+          *)
             SUFFIX="linux-i686"
             ;;
         esac