Update docs and bump to 0.2.0-dev (#16)

Author: cigrainger

RELEASE.md

@@ -15,7 +15,7 @@ is a bit more involved than it would be otherwise.
 3. This will kick off the "Build precompiled NIFs" GitHub Action. Wait for this to complete. It
    usually takes around 40-60 minutes.
 4. While the NIFs are compiling, ensure you have the latest version of `main` and don't have any
-   intermediate builds by running `rm -rf native/explorer/target`.
-5. Once the NIFs are built, use `mix rustler_precompiled.download Explorer.PolarsBackend.Native --all --print` to download generate the checksum file.
+   intermediate builds by running `rm -rf native/ex_tokenizers/target`.
+5. Once the NIFs are built, use `mix rustler_precompiled.download Tokenizers.Native --all --print` to download generate the checksum file.
 6. Run `mix hex.publish`.
 7. Bump the version in the `mix.exs` and add the `-dev` flag to it.

flake.lock

@@ -25,7 +25,7 @@
               cc
               clang
               clippy
-              elixir
+              elixir_1_14
               erlang
               gdb
               gcc
@@ -38,16 +38,16 @@
               darwin.apple_sdk.frameworks.Carbon
               darwin.apple_sdk.frameworks.AppKit
             ];
-          shellHook = ''
-            mkdir -p .nix-mix
-            mkdir -p .nix-hex
-            export MIX_HOME=$PWD/.nix-mix
-            export HEX_HOME=$PWD/.nix-hex
-            export PATH=$MIX_HOME/bin:$PATH
-            export PATH=$HEX_HOME/bin:$PATH
-            export PATH=$MIX_HOME/escripts:$PATH
-            export ERL_AFLAGS="-kernel shell_history enabled"
-          '';
-        };
-      });
+            shellHook = ''
+              mkdir -p .nix-mix
+              mkdir -p .nix-hex
+              export MIX_HOME=$PWD/.nix-mix
+              export HEX_HOME=$PWD/.nix-hex
+              export PATH=$MIX_HOME/bin:$PATH
+              export PATH=$HEX_HOME/bin:$PATH
+              export PATH=$MIX_HOME/escripts:$PATH
+              export ERL_AFLAGS="-kernel shell_history enabled"
+            '';
+          };
+        });
 }

flake.nix

@@ -2,7 +2,7 @@ defmodule Tokenizers.MixProject do
   use Mix.Project
 
   @source_url "https://github.com/elixir-nx/tokenizers"
-  @version "0.1.0"
+  @version "0.2.0-dev"
 
   def project do
     [

mix.exs

@@ -122,7 +122,7 @@ Tokenizer.decode(tokenizer, list_of_ids)
 
 Typically the reason we're tokenizing text is to use it as an input in a machine learning model. For that, we'll need tensors.
 
-In order to get a tensor, we need sequences that are all of the same length. We'll get some data from `Scidata` and use `Tokenizers.pad/3` and `Tokenizers.truncate/3` to yield a tensor.
+In order to get a tensor, we need sequences that are all of the same length. We'll get some data from `Scidata` and use `Tokenizers.Encoding.pad/3` and `Tokenizers.Encoding.truncate/3` to yield a tensor.
 
 ```elixir
 %{review: reviews} = Scidata.YelpPolarityReviews.download_test()