Merge pull request #81 from holgerteichgraeber/master

YoungFaithful · web-flow · commit 4e5895089a87 · 2019-05-02T11:26:28.000-07:00
Bring dev branch up to date with master
diff --git a/.travis.yml b/.travis.yml
@@ -13,18 +13,19 @@ after_success:
 
 jobs:
   include:
-    - stage: "Documentation"
+    - stage: "Testing"
       julia: 1.0
       os: linux
       script:
-        - julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
-        - julia --project=docs/ docs/make.jl
+        - julia --project=test/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
+        - julia --project=test/ test/clustering.jl
+        - julia --project=test/ test/capacityexpansion.jl
       after_success:
-    - stage: "Testing"
+    - stage: "Documentation"
       julia: 1.0
       os: linux
       script:
-        - julia --project=test/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
-        - julia --project=test/ test/clustering.jl
+        - julia --project=docs/ -e 'using Pkg; Pkg.develop(PackageSpec(path=pwd())); Pkg.instantiate()'
+        - julia --project=docs/ docs/make.jl
       after_success:
 services: docker
diff --git a/Project.toml b/Project.toml
@@ -4,7 +4,7 @@ keywords = ["clustering", "JuMP", "optimization"]
 license = "MIT"
 desc = "julia implementation of using different clustering methods for finding representative periods for the optimization of energy systems"
 author = ["Holger Teichgraeber"]
-version = "0.2.3"
+version = "0.3.4"
 
 [deps]
 CSV = "336ed68f-0bac-5ca0-87d4-7b16caf5d00b"
@@ -18,3 +18,5 @@ LinearAlgebra = "37e2e46d-f89d-539d-b4ee-838fcccc9c8e"
 Reexport = "189a3867-3050-52da-a836-e630ba90ab69"
 StatsBase = "2913bbd2-ae8a-5f71-8c99-4fb6c76f3a91"
 
+[compat]
+julia = "^1.0"
diff --git a/README.md b/README.md
@@ -1,4 +1,4 @@
-![ClustForOpt](docs/src/assets/clust_for_opt_text.svg)
+﻿![ClustForOpt](docs/src/assets/clust_for_opt_text.svg)
 ===
 [![](https://img.shields.io/badge/docs-stable-blue.svg)](https://holgerteichgraeber.github.io/ClustForOpt.jl/stable)
 [![](https://img.shields.io/badge/docs-dev-blue.svg)](https://holgerteichgraeber.github.io/ClustForOpt.jl/dev)
@@ -7,7 +7,7 @@
 
 ClustForOpt is a [julia](www.juliaopt.com) implementation of clustering methods for finding representative periods for the optimization of energy systems. The package furthermore provides a multi-node capacity expansion model.
 
-The package has three main purposes: 1) Provide a simple process of clustering time-series input data, with clustered data output in a generalized type system 2) provide an interface between clustered data and optimization problem 3) provide a generalizable capacity expansion problem formulation and data to test clustering on this problem.
+The package has two main purposes: 1) Provide a simple process of clustering time-series input data, with clustered data output in a generalized type system 2) provide an interface between clustered data and optimization problem.
 
 The package follows the clustering framework presented in [Teichgraeber and Brandt, 2019](https://doi.org/10.1016/j.apenergy.2019.02.012).
 The package is actively developed, and new features are continuously added. For a reproducible version of the methods and data of the original paper by [Teichgraeber and Brandt, 2019](https://doi.org/10.1016/j.apenergy.2019.02.012), please refer to release [v0.1](https://github.com/holgerteichgraeber/ClustForOpt.jl/tree/v0.1).
@@ -34,7 +34,7 @@ Install using:
 
 ```julia
 ]
-add https://github.com/holgerteichgraeber/ClustForOpt.jl.git
+add ClustForOpt
 ```
 where `]` opens the julia package manager.
 
@@ -102,4 +102,4 @@ For use of DTW barycenter averaging (DBA) and k-shape clustering on single-attri
 ### Optimization
 The function `run_opt()` runs the optimization problem and gives as an output a struct that contains optimal objective function value, decision variables, and additional info. The `run_opt()` function infers the optimization problem type from the input data. See the examples folder for further details.
 
-More detailed documentation on the Capacity Expansion Problem can be found in the documentation.
+A Capacity Expansion Optimization Problem that utilizes `ClustForOpt` can be found in the package [CEP](https://github.com/YoungFaithful/CEP.jl). 
diff --git a/docs/src/clust.md b/docs/src/clust.md
@@ -36,7 +36,7 @@ ClustResultSimple
 ## Example running clustering
 ```@example
 using ClustForOpt
-# laod ts-input-data
+# load ts-input-data
 ts_input_data = load_timeseries_data(normpath(joinpath(@__DIR__,"..","..","data","TS_GER_1")); T=24, years=[2016])
 ts_clust_data = run_clust(ts_input_data).best_results
 using Plots
diff --git a/docs/src/index.md b/docs/src/index.md
@@ -4,7 +4,7 @@
 [![](https://img.shields.io/badge/docs-dev-blue.svg)](https://holgerteichgraeber.github.io/ClustForOpt.jl/dev)
 [![Build Status](https://travis-ci.com/holgerteichgraeber/ClustForOpt.jl.svg?token=HRFemjSxM1NBCsbHGNDG&branch=master)](https://travis-ci.com/holgerteichgraeber/ClustForOpt.jl)
 
-[ClustForOpt](https://github.com/holgerteichgraeber/ClustForOpt.jl) is a [julia](https://www.juliaopt.com) implementation of clustering methods for finding representative periods for optimization problems. A utilization in a scalable capacity expansion problem can be found in the package [CEP](https://github.com/YoungFaithful/CEP.jl).
+[ClustForOpt](https://github.com/holgerteichgraeber/ClustForOpt.jl) is a [julia](https://www.juliaopt.com) implementation of clustering methods for finding representative periods for optimization problems. A utilization in a scalable capacity expansion problem can be found in the package [CEP](https://github.com/YoungFaithful/CapacityExpansion.jl).
 
 The package has two main purposes: 1) Provide a simple process of clustering time-series input data, with clustered data output in a generalized type system 2) provide an interface between clustered data and optimization problem.
 
@@ -33,6 +33,6 @@ Install using:
 
 ```julia
 ]
-add https://github.com/holgerteichgraeber/ClustForOpt.jl.git
+add ClustForOpt
 ```
 where `]` opens the julia package manager.
diff --git a/src/clustering/extreme_vals.jl b/src/clustering/extreme_vals.jl
@@ -78,7 +78,7 @@ function simple_extr_val_ident(clust_data::ClustData,
                                data_type::String;
                                extremum::String="max",
                                peak_def::String="absolute",
-                               consecutive_periods::Int64=1)
+                               consecutive_periods::Int=1)
   data=clust_data.data[data_type]
   delta_period=consecutive_periods-1
   # set data to be compared
@@ -168,7 +168,7 @@ function extreme_val_output(data::ClustData,
   end
   weights_ed=[]
   #initiate new k-ids-ed that don't represent any original time-period
-  k_ids_ed=zeros(Int64,size(data.k_ids))
+  k_ids_ed=zeros(Int,size(data.k_ids))
   if rep_mod_method == "feasibility"
     weights_ed = zeros(length(unique_extr_val_idcs))
     #no representation is done of the original time-period, it's just for feasibility
diff --git a/src/clustering/intraperiod_segmentation.jl b/src/clustering/intraperiod_segmentation.jl
@@ -38,7 +38,7 @@ function run_clust_segmentation(period::Array{Float64,2};
   norm_period, typely_mean, typely_sdv=z_normalize(period;scope=norm_scope)
   #x,weights,clustids,x,iter= run_clust_hierarchical(norm_period,n_seg,iterations)
   data=norm_period
-  clustids=run_clust_hierarchical_partitional(data::Array, n_seg::Int64)
+  clustids=run_clust_hierarchical_partitional(data::Array, n_seg::Int)
   weights = calc_weights(clustids,n_seg)
 
 
@@ -48,7 +48,7 @@ function run_clust_segmentation(period::Array{Float64,2};
   return centers,weights,clustids,cost,1
 end
 
-function get_clustids(ends::Array{Int64,1})
+function get_clustids(ends::Array{Int,1})
   clustids=collect(1:size(data,2))
   j=1
   for i in 1:size(data,2)
@@ -61,12 +61,12 @@ function get_clustids(ends::Array{Int64,1})
 end
 
 """
-      run_clust_hierarchical_partitional(data::Array, n_seg::Int64)
+      run_clust_hierarchical_partitional(data::Array, n_seg::Int)
 !!! Not yet proven
 Usees provided data and number of segments to aggregate them together
 """
 function run_clust_hierarchical_partitional(data::Array,
-                                            n_seg::Int64)
+                                            n_seg::Int)
   _dist= SqEuclidean()
   #Assign each timeperiod it's own cluster
   clustids=collect(1:size(data,2))
@@ -99,20 +99,20 @@ function run_clust_hierarchical_partitional(data::Array,
 end
 
 """
-      merge_clustids!(clustids::Array{Int64,1},index::Int64)
+      merge_clustids!(clustids::Array{Int,1},index::Int)
 Calculate the new clustids by merging the cluster of the index provided with the cluster of index+1
 """
-function merge_clustids!(clustids::Array{Int64,1},index::Int64)
+function merge_clustids!(clustids::Array{Int,1},index::Int)
   clustids[index+1]=clustids[index]
   clustids[index+2:end].-=1
 end
 
 """
-      get_mean_data(data::Array, clustids::Array{Int64,1})
+      get_mean_data(data::Array, clustids::Array{Int,1})
 Calculate mean of data: The number of columns is kept the same, mean is calculated for aggregated columns and the same in all with same clustid
 """
 function get_mean_data(data::Array,
-                    clustids::Array{Int64,1})
+                    clustids::Array{Int,1})
   mean_data=zeros(size(data))
   for i in 1:size(data,2)
     mean_data[:,i]=mean(data[:,findall(clustids.==clustids[i])], dims=2)
diff --git a/src/clustering/run_clust.jl b/src/clustering/run_clust.jl
@@ -62,7 +62,7 @@ function run_clust(data_norm_merged::ClustDataMerged;
                   n_clust::Int=5,
                   n_init::Int=100,
                   iterations::Int=300,
-                  orig_k_ids::Array{Int64,1}=Array{Int64,1}(),
+                  orig_k_ids::Array{Int,1}=Array{Int,1}(),
                   kwargs...)
 
 method: "kmeans","kmedoids","kmedoids_exact","hierarchical"
@@ -75,7 +75,7 @@ function run_clust(data_norm_merged::ClustDataMerged,
                   n_clust::Int=5,
                   n_init::Int=100,
                   iterations::Int=300,
-                  orig_k_ids::Array{Int64,1}=Array{Int64,1}(),
+                  orig_k_ids::Array{Int,1}=Array{Int,1}(),
                   kwargs...)
     # initialize data arrays
     centers = Array{Array{Float64},1}(undef,n_init)
diff --git a/src/utils/datastructs.jl b/src/utils/datastructs.jl
@@ -7,13 +7,13 @@ abstract type ClustResult end
 "FullInputData"
 struct FullInputData <: TSData
  region::String
- years::Array{Int64}
+ years::Array{Int,1}
  N::Int
  data::Dict{String,Array}
 end
 
 """
-      ClustData{region::String,K::Int,T::Int,data::Dict{String,Array},weights::Array{Float64},mean::Dict{String,Array},sdv::Dict{String,Array}} <: TSData
+      ClustData{region::String,K::Int,T::Int,data::Dict{String,Array},weights::Array{Float64,2},mean::Dict{String,Array},sdv::Dict{String,Array},delta_t::Array{Float64,2},k_ids::Array{Int}} <: TSData
 - region: optional information to specify the region data belongs to
 - K: number of periods
 - T: time steps per period
@@ -26,21 +26,21 @@ end
 """
 struct ClustData <: TSData
  region::String
- years::Array{Int64}
+ years::Array{Int}
  K::Int
  T::Int
  data::Dict{String,Array}
  weights::Array{Float64}
  mean::Dict{String,Array}
  sdv::Dict{String,Array}
  delta_t::Array{Float64,2}
- k_ids::Array{Int64}
+ k_ids::Array{Int}
 end
 
 "ClustDataMerged"
 struct ClustDataMerged <: TSData
  region::String
- years::Array{Int64}
+ years::Array{Int}
  K::Int
  T::Int
  data::Array
@@ -49,7 +49,7 @@ struct ClustDataMerged <: TSData
  mean::Dict{String,Array}
  sdv::Dict{String,Array}
  delta_t::Array{Float64,2}
- k_ids::Array{Int64}
+ k_ids::Array{Int}
 end
 
 "ClustResultAll"
@@ -88,12 +88,12 @@ struct SimpleExtremeValueDescr
    data_type::String
    extremum::String
    peak_def::String
-   consecutive_periods::Int64
+   consecutive_periods::Int
    "Replace default constructor to only allow certain entries"
    function SimpleExtremeValueDescr(data_type::String,
                                     extremum::String,
                                     peak_def::String,
-                                    consecutive_periods::Int64)
+                                    consecutive_periods::Int)
        # only allow certain entries
        if !(extremum in ["min","max"])
          @error("extremum - "*extremum*" - not defined")
@@ -142,7 +142,7 @@ end
 
 """
   ClustData(region::String,
-                         years::Array{Int64,1},
+                         years::Array{Int,1},
                          K::Int,
                          T::Int;
                          el_price::Array=[],
@@ -153,12 +153,12 @@ end
                          mean::Dict{String,Array}=Dict{String,Array}(),
                          sdv::Dict{String,Array}=Dict{String,Array}(),
                          delta_t::Array{Float64,2}=ones(T,K),
-                         k_ids::Array{Int64,1}=collect(1:K)
+                         k_ids::Array{Int,1}=collect(1:K)
                          )
 constructor 1 for ClustData: provide data individually
 """
 function ClustData(region::String,
-                         years::Array{Int64,1},
+                         years::Array{Int,1},
                          K::Int,
                          T::Int;
                          el_price::Array=[],
@@ -169,7 +169,7 @@ function ClustData(region::String,
                          mean::Dict{String,Array}=Dict{String,Array}(),
                          sdv::Dict{String,Array}=Dict{String,Array}(),
                          delta_t::Array{Float64,2}=ones(T,K),
-                         k_ids::Array{Int64,1}=collect(1:K)
+                         k_ids::Array{Int,1}=collect(1:K)
                          )
    dt = Dict{String,Array}()
    mean_sdv_provided = ( !isempty(mean) && !isempty(sdv))
@@ -208,26 +208,26 @@ end
 
 """
     ClustData(region::String,
-                      years::Array{Int64,1},
+                      years::Array{Int,1},
                       K::Int,
                       T::Int,
                       data::Dict{String,Array},
                       weights::Array{Float64},
                       delta_t::Array{Float64,2},
-                      k_ids::Array{Int64,1};
+                      k_ids::Array{Int,1};
                       mean::Dict{String,Array}=Dict{String,Array}(),
                       sdv::Dict{String,Array}=Dict{String,Array}()
                       )
 constructor 2 for ClustData: provide data as dict
 """
 function ClustData(region::String,
-                       years::Array{Int64,1},
+                       years::Array{Int,1},
                        K::Int,
                        T::Int,
                        data::Dict{String,Array},
                        weights::Array{Float64},
                        delta_t::Array{Float64,2},
-                       k_ids::Array{Int64,1};
+                       k_ids::Array{Int,1};
                        mean::Dict{String,Array}=Dict{String,Array}(),
                        sdv::Dict{String,Array}=Dict{String,Array}()
                        )
@@ -273,27 +273,28 @@ end
 
 """
     ClustDataMerged(region::String,
-                        years::Array{Int64,1},
+                        years::Array{Int,1},
                         K::Int,
                         T::Int,
                         data::Array,
                         data_type::Array{String},
                         weights::Array{Float64},
-                        k_ids::Array{Int64,1};
+                        k_ids::Array{Int,1};
+                        delta_t::Array{Float64,2}=ones(T,K),
                         mean::Dict{String,Array}=Dict{String,Array}(),
                         sdv::Dict{String,Array}=Dict{String,Array}()
                         )
 constructor 1: construct ClustDataMerged
 """
 function ClustDataMerged(region::String,
-                       years::Array{Int64,1},
+                       years::Array{Int,1},
                        K::Int,
                        T::Int,
                        data::Array,
                        data_type::Array{String},
                        weights::Array{Float64},
-                       k_ids::Array{Int64,1};
-                       delta_t::Array{Float64}=ones(T,K),
+                       k_ids::Array{Int,1};
+                       delta_t::Array{Float64,2}=ones(T,K),
                        mean::Dict{String,Array}=Dict{String,Array}(),
                        sdv::Dict{String,Array}=Dict{String,Array}()
                        )
diff --git a/src/utils/load_data.jl b/src/utils/load_data.jl
diff --git a/test/Project.toml b/test/Project.toml
diff --git a/test/capacityexpansion.jl b/test/capacityexpansion.jl
diff --git a/test/cep_exact_data.jld2 b/test/cep_exact_data.jld2
