Single command Scala setup

2020-09-21

scala
coursier

You can now set up your machine for Scala development with a single command via the coursier CLI:

$ cs setup

This command ensures that both a JVM and the most common Scala CLI tools are installed on your machine.

cs is distributed as a native application without any external dependencies. As such, it can be run immediately after download and thus provides a very simple and convenient way to set up one's machine for Scala development.

In this post, we're going to have a closer look at:

what cs setup does in more detail
the lower level commands that back cs setup , that are also useful on their own:
- cs java
- cs launch
- cs bootstrap
- cs install

We'll see that these lower level commands can do what cs setup does, but in a more fine-grained fashion. We'll also see that their features can be used independently of each other, allowing to

install some applications with coursier, but keeping managing JVMs however you'd like, or
manage some JVMs with coursier, while installing applications another way.

We'll also see that these commands can be used without actually installing or setting up anything: getting JVMs or application details only via the coursier CLI, without touching any profile file or global environment variable. This allows to get a quick peek at the coursier CLI features while not installing anything proper — everything stays in the coursier cache, that you can remove entirely any time you'd like.

setup command

Getting the coursier launcher and run setup

On Linux and macOS, download the coursier CLI, and run it with:

$ curl -fLo cs https://git.io/coursier-cli-"$(uname | tr LD ld)"
$ chmod +x cs
$ ./cs setup
$ rm -f cs

On Windows, you can download cs.exe from your browser, and double-click it to open a terminal, where cs setup will run.

Upon a successful run of cs setup, cs is available in the PATH (see below). The downloaded launcher can be safely removed.

What happens when setup runs

cs setup asks for confirmation prior to do anything. If neither a JVM nor the usual Scala applications are installed, and you press Enter every time it asks for confirmation, it should print something like this:

$ cs setup
Checking if a JVM is installed
  No JVM found, should we try to install one? [Y/n]
Extracting
  /home/user/.cache/coursier/v1/https/github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz
in
  /home/user/.cache/coursier/jvm/adopt@1.8.0-252
Done
  Should we update ~/.profile? [Y/n]
Some shell configuration files were updated. It is recommended to close this terminal once the setup command is done, and open a new one for the changes to be taken into account.

Checking if ~/.local/share/coursier/bin is in PATH
  Should we add ~/.local/share/coursier/bin to your PATH via ~/.profile? [Y/n]

Checking if the standard Scala applications are installed
  Installed ammonite
  Installed cs
  Installed coursier
  Installed scala
  Installed scalac
  Installed sbt
  Installed scalafmt

On Linux and macOS, it edits profile files in your home directory.

On Windows, it updates the so-called User environment variables, that you can also access by via Control Panel > Advanced system settings > Environment Variables.

The JVM archive is unpacked in a subdirectory of the managed JVM directory, which depends on your OS:

OS	Managed JVM directory
Linux	`~/.cache/coursier/jvm`
macOS	`~/Library/Caches/Coursier/jvm`
Windows	`C:\Users\_username_\AppData\Local\Coursier\Cache\jvm` (`%LOCALAPPDATA%\Coursier\Cache\jvm` in the general case)

If cs setup installs a JVM, it sets JAVA_HOME and appends $JAVA_HOME/bin to the PATH.

Applications are installed in the installation directory, which is:

OS	Application directory
Linux	`~/.local/share/coursier/bin`
macOS	`~/Library/Application Support/Coursier/bin`
Windows	`C:\Users\_username_\AppData\Local\Coursier\data\bin` (`%LOCALAPPDATA%\Coursier\data\bin` in the general case)

If cs setup installs applications, it appends this directory to the PATH.

Overall, cs setup wraps together new JVM and application handling capabilities of the coursier CLI. These can be used in a more fine grained fashion, or assembled together differently if you wish. Let's dive into those!

Managing JVMs

The java command of coursier can be called with cs java. It backs how cs setup handles JVMs.

The cs java command itself aims at being a drop-in and more featureful replacement for java. cs java simply runs the java executable:

$ java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)
$ cs java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)

If a JVM is already available on your system, calling cs java is equivalent to just calling the already installed java.

If no JVM is found, cs java transparently downloads and unpacks the latest Adopt OpenJDK 8, and calls its java executable. That way, cs java works even when no JVM is installed and a java executable is not in your PATH:

$ java -version
bash: java: command not found

$ cs java -version
Downloading https://github.com/shyiko/jabba/raw/master/index.json
Downloaded https://github.com/shyiko/jabba/raw/master/index.json
Downloading https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz
Downloaded https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz
Extracting
  ~/.cache/coursier/v1/https/github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz
in
  ~/.cache/coursier/jvm/adopt@1.8.0-252
Done
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)

$ cs java -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)

See explicit JVM below to download and use a different JVM.

The java command parses its arguments the following way: it processes arguments as long as it recognizes options it handles itself, and stops at the first unrecognized argument. All arguments from the first non-recognized one are passed as is to the java executable. cs java --help prints a list of the arguments accepted by the cs java command. In the command above, -version is not recognized as an option of cs java. -version, and any argument that would be passed after it, is thus passed as is to the java executable. If cs java encounters -- as argument, all the remaining arguments are passed to java:

$ cs java -- -version
openjdk version "1.8.0_252"
OpenJDK Runtime Environment (AdoptOpenJDK)(build 1.8.0_252-b09)
OpenJDK 64-Bit Server VM (AdoptOpenJDK)(build 25.252-b09, mixed mode)

Compared to other JVM managers, cs java works equally well on Linux, macOS, and Windows. It doesn't need any other CLI tool to be installed. On Windows, it does not require WSL or Git BASH to work well. It can also be used alongside other JVM managers. We'll see that cs java doesn't require a JVM installation step, unlike other JVM managers – JVMs are transparently downloaded and unpacked. It also accepts more user-friendly JVM names, allowing to pick JVMs without even listing its JVM index.

Managed JVM directory

JVM archives are unpacked in subdirectories of the managed JVM directory, mentioned above too, which is

OS	Managed JVM directory
Linux	`~/.cache/coursier/jvm`
macOS	`~/Library/Caches/Coursier/jvm`
Windows	`C:\Users\_username_\AppData\Local\Coursier\Cache\jvm` (`%LOCALAPPDATA%\Coursier\Cache\jvm` in the general case)

In the example above, Adopt OpenJDK 1.8.0-252 on Linux would be unpacked under

~/.cache/coursier/jvm/adopt@1.8.0-252

Explicit JVM

You can pass a custom JVM name with --jvm, like

$ cs java --jvm graalvm-ce-java11:19 -version
openjdk version "11.0.7" 2020-04-14
OpenJDK Runtime Environment GraalVM CE 19.3.2 (build 11.0.7+10-jvmci-19.3-b10)
OpenJDK 64-Bit Server VM GraalVM CE 19.3.2 (build 11.0.7+10-jvmci-19.3-b10, mixed mode, sharing)
$ cs java --jvm 14 -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

To know which JVMs are available for the current system, pass --available to cs java, like

$ cs java --available
adopt:1.8.0-172
adopt:1.8.0-181
adopt:1.8.0-192
adopt:1.8.0-202
adopt:1.8.0-212
…

To list the JVMs already unpacked on your system, pass --installed, like

$ cs java --installed
adopt:1.8.0-252
adopt:1.9.0-0
adopt:1.10.0-2
adopt:1.11.0-7
adopt:1.13.0-2
adopt:1.14.0-1
graalvm:19.3.2
graalvm:20.1.0
graalvm-ce-java11:19.3.2
openjdk:1.14.0
zulu:1.7.262

Note that if no exact match is found for the version passed to --jvm, a + is appended to the version, so that we get a version interval, and the highest version in the interval is picked. That way, graalvm:19 is accepted, and matches graalvm:19.3.2 as of writing this.

Also note that 1. is prepended to the version when it makes sense, so that the 1. in the output of --available and --installed can be ignored most of the time. For example, openjdk:14 is equivalent to openjdk:1.14.

Lastly, note that if no JVM type is passed, only a version, like --jvm 1.14.0-1, Adopt OpenJDK is used (JVM type: adopt). This can be used in conjunction with the automatic version interval expansion above, as well as the automatic handling of the 1. prefix, so that things like --jvm 14 are valid (and expanded to adopt:1.14.0-1 as of writing this), like

$ cs java --jvm 14 -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

Set up a JVM for the current shell session

(Only for Linux and macOS.)

Pass --env to cs java to print a shell snippet setting up a JVM:

$ cs java --jvm 14 --env
export CS_FORMER_JAVA_HOME="$JAVA_HOME"
export JAVA_HOME="/home/user/.cache/coursier/jvm/adopt@1.14.0-1"
export PATH="/home/user/.cache/coursier/jvm/adopt@1.14.0-1/bin:$PATH"

Eval it to set up a JVM in the current session:

$ eval "$(cs java --jvm 14 --env)"
$ java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

Set up a JVM for the current user

Pass --setup to cs java to update profile files (on Linux and macOS) or user environment variables (on Windows):

$ cs java --jvm 14 --setup
Checking if ~/.profile need(s) updating.
Some shell configuration files were updated. It is recommended to close this terminal once the setup command is done, and open a new one for the changes to be taken into account.
$ source ~/.profile
$ java -version
openjdk version "14.0.1" 2020-04-14
OpenJDK Runtime Environment AdoptOpenJDK (build 14.0.1+7)
OpenJDK 64-Bit Server VM AdoptOpenJDK (build 14.0.1+7, mixed mode, sharing)

This installs a JVM on your system just like cs setup would have.

JVM index

To know where to download JVMs, the java command currently uses the JVM index of jabba, another JVM manager. For each OS / CPU, this index lists available JVMs, their available versions, and where to download them. For example, for Linux on AMD64, Adopt OpenJDK 1.8.0-252 can be downloaded at

https://github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz

The coursier CLI downloads the JVM archives via its cache, so that on Linux, this archive would be downloaded in

~/.cache/coursier/v1/https/github.com/AdoptOpenJDK/openjdk8-binaries/releases/download/jdk8u252-b09/OpenJDK8U-jdk_x64_linux_hotspot_8u252b09.tar.gz

On Linux, it would then be unpacked under

~/.cache/coursier/jvm/adopt@1.8.0-252

The JVM index of jabba lists numerous JVMs, like AdoptOpenJDK, OpenJDK, Zulu, etc. It does so for Linux / macOS / Windows, targeting various architectures (amd64, aarch64, etc.). See cs java --available to list all currently available JVMs on your system.

Managing applications

Launching applications

The launch command can start some applications with just their name, like

$ cs launch scala
Welcome to Scala 2.13.3 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121).
Type in expressions for evaluation. Or try :help.

scala>

You can optionally pass a version, like

$ cs launch scala:2.12.11
Welcome to Scala 2.12.11 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_121).
Type in expressions for evaluation. Or try :help.

scala>

Note that nothing is actually "installed" when launching applications this way: the relevant application index and JARs are fetched via the coursier cache if needed, and the right set of JARs from the coursier cache is loaded and run, to start the application.

Channels

Applications are defined in "channels". Channels define lists of applications that can be launched. Some channel types are versioned, while the content of others is simply overridden and newer data gets pulled upon updates.

The default channel sources live at this address. It pushes artifacts here, so that its Maven coordinates are io.get-coursier:apps. As a Maven-based channel, the content of this channel is versioned.

When looking for an application, the coursier CLI downloads the latest version of io.get-coursier:apps. It looks inside its JAR for "application descriptors". Application descriptors are JSON files that look like this:

{
  "repositories": [
    "central"
  ],
  "dependencies": [
    "org.scala-lang:scala-compiler:latest.stable"
  ],
  "mainClass": "scala.tools.nsc.MainGenericRunner",
  "properties": {
    "scala.usejavacp": "true"
  }
}

Application descriptors specify which repositories should be used (here, central, for Maven Central), which dependencies should be pulled (here, org.scala-lang:scala-compiler:latest.stable). Some other optional parameters can be specified, like the main class to launch, or Java properties to set before launching the app, etc.

The README of the default application channel contains a list of applications available in this channel.

Adding applications to the contrib channel

The contrib channel, whose sources live here, accepts any application available from public repositories. Feel free to submit applications there by opening a pull request, adding new application descriptors.

Users can then use your application by specifying the --contrib option, like

$ cs launch --contrib proguard
$ cs install --contrib proguard

Creating your own channels

The coursier website briefly describes how to create and use your own channels. There is also a giter8 model to create a repository for such a channel.

Creating launchers for applications

Unlike cs launch which starts an application straightaway, cs bootstrap creates launchers starting applications, like

$ cs bootstrap scala -o scala
$ ./scala

By default, the generated launchers are very small (around 28 kB as of writing this). These launchers are tiny Java applications, with a minimal coursier cache logic. Upon start-up, this Java application ensures that all the relevant JARs are available in the coursier cache, and starts the actual application (here, scala) from those JARs. The cs bootstrap documentation describes these launchers in more detail, and lists the other kinds of launcher that cs bootstrap can generate, such as "uber JARs", or GraalVM native images, among others.

Installing launchers for applications

The install command creates and installs launchers for applications. Use it like

$ cs install scala
$ scala

Launchers are installed in the installation directory mentioned above, which is

OS	Application directory
Linux	`~/.local/share/coursier/bin`
macOS	`~/Library/Application Support/Coursier/bin`
Windows	`C:\Users\_username_\AppData\Local\Coursier\data\bin` (`%LOCALAPPDATA%\Coursier\data\bin` in the general case)

You can pass a custom installation directory with --install-dir, or by setting COURSIER_INSTALL_DIR in the environment, like

$ cs install --install-dir /usr/local/bin scala
$ which scala
/usr/local/bin/scala
$ export COURSIER_INSTALL_DIR=/usr/local/bin
$ cs install ammonite
$ which amm
/usr/local/bin/amm

The launchers installed by cs install are very similar to the ones created by cs bootstrap. These are JARs too. cs install writes additional metadata in those JARs, mentioning the channel the application originates from, including the channel version, and the list of URLs the JARs of the application are downloaded from, along with their checksums. This allows to update applications later on.

Updating launchers

The update command can update launchers, checking for new / updated

channel data,
application artifacts.

$ cs update scala
Updated scala

If you installed applications in a custom directory, pass that directory with --install-dir, or via COURSIER_INSTALL_DIR in the environment, like for install above.

Removing launchers

The uninstall command removes an installed launcher, like

$ cs uninstall scala
Uninstalled scala

If you installed applications in a custom directory, pass that directory with --install-dir, or via COURSIER_INSTALL_DIR in the environment, like for install above.

Wrapping things together

Manual setup

A few invocations of the cs java and cs install commands allow to effectively do the same as the cs setup command.

You can ensure a JVM is available, just like the cs setup command, with

$ cs java --setup

If a JVM is already installed on your system, this command does nothing. If no JVM is found, the following happens:

the latest AdoptOpenJDK 8 is unpacked in the managed JVM directory (mentioned above),
profile files or user environment variables are updated so that JAVA_HOME is set, and PATH contains $JAVA_HOME/bin.

You can then ensure the application directory is in your PATH with

$ cs install --setup

Lastly, you can install a few scala applications in the installation directory with

$ cs install scala sbt ammonite scalafmt cs

Using applications and JVMs without installing them

The launch command accepts a --jvm option, specifying a JVM to use. Use it like

$ cs launch ammonite --jvm 14
Loading...
Welcome to the Ammonite Repl 2.1.4-12-f697522 (Scala 2.13.3 Java 14.0.1)
@

This transparently downloads and unpacks a JVM in the managed JVM directory, and starts the application with it.

The --scala option can come in handy too:

$ cs launch ammonite --scala 2.12 --jvm 14
Loading...
Welcome to the Ammonite Repl 2.1.4-11-307f3d8 (Scala 2.12.12 Java 14.0.1)
@

Alternatively to --jvm, you can just set up a JVM for the current shell session, then call launch:

$ eval "$(cs java --env --jvm 14)"
$ cs launch ammonite
Loading...
Welcome to the Ammonite Repl 2.1.4-12-f697522 (Scala 2.13.3 Java 14.0.1)
@

Final words

cs setup, alongside cs java and cs install, makes it easier to setup one's machine for Scala development, taking care of managing both JVMs and applications, and works on all major OSs (Linux, macOS, Windows). It also offers a way for JVM application developers to distribute their applications.

On top of that, it can install, or build if necessary, native applications, like GraalVM native-image launchers, or Scala Native-based ones, which are beyond the scope of this post.

Don't hesitate to ask questions on its Gitter channel, open issues on its GitHub repository, or contribute to it or ask for guidance for it.

Allow me to finish with giving a big thanks to the Scala Center without whose support most of the features described in this post would still be unimplemented! The Scala Center made the development of the Coursier CLI possible through a combination of direct employment and contracting of myself (Alexandre Archambault).

Many thanks to the numerous contributors that contributed code to coursier, but also bug reports, or simply support to its users or authors.

I'd also like to give a big thanks to Martijn Hoekstra, Jamal CHAQOURI, Przemek Pokrywka, nachinius, Ghislain Antony Vaillant, Eric Loots, and Mark T. Kennedy, who proof-read this post and suggested many improvements.

Addendum

Other `cs` commands

cs has other commands, such as cs complete, cs resolve, and cs fetch, that come in handy when working with Maven / Ivy dependencies.

cs complete allows to complete Maven coordinates:

$ cs complete org.apache.pdfbox:
…
pdfbox-examples
pdfbox-lucene
pdfbox-parent
…
$ cs complete org.apache.pdfbox:pdfbox-lucene:
…
1.8.14
1.8.15
1.8.16

cs resolve lists transitive dependencies:

$ cs resolve org.apache.pdfbox:pdfbox-lucene:1.8.16
commons-logging:commons-logging:1.1.1:default
org.apache.lucene:lucene-core:2.4.1:default
org.apache.lucene:lucene-demos:2.4.1:default
org.apache.pdfbox:fontbox:1.8.16:default
org.apache.pdfbox:jempbox:1.8.16:default
org.apache.pdfbox:pdfbox:1.8.16:default
org.apache.pdfbox:pdfbox-lucene:1.8.16:default

These dependencies can optionally be printed as a tree, with the --tree or -t option:

$ cs resolve -t org.apache.pdfbox:pdfbox-lucene:1.8.16
  Result:
└─ org.apache.pdfbox:pdfbox-lucene:1.8.16
   ├─ org.apache.lucene:lucene-core:2.4.1
   ├─ org.apache.lucene:lucene-demos:2.4.1
   │  └─ org.apache.lucene:lucene-core:2.4.1
   └─ org.apache.pdfbox:pdfbox:1.8.16
      ├─ commons-logging:commons-logging:1.1.1
      ├─ org.apache.pdfbox:fontbox:1.8.16
      │  └─ commons-logging:commons-logging:1.1.1
      └─ org.apache.pdfbox:jempbox:1.8.16

cs fetch fetches artifacts, JARs in most cases:

$ cs fetch org.apache.pdfbox:pdfbox-lucene:1.8.16
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/pdfbox-lucene/1.8.16/pdfbox-lucene-1.8.16.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/pdfbox/1.8.16/pdfbox-1.8.16.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/lucene/lucene-core/2.4.1/lucene-core-2.4.1.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/lucene/lucene-demos/2.4.1/lucene-demos-2.4.1.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/fontbox/1.8.16/fontbox-1.8.16.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/jempbox/1.8.16/jempbox-1.8.16.jar
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar

The --classpath or -p option makes it print artifacts in a way that can be passed to java -cp:

$ cs fetch -p org.apache.pdfbox:pdfbox-lucene:1.8.16
~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/pdfbox-lucene/1.8.16/pdfbox-lucene-1.8.16.jar:~/Library/Caches/Coursier/v1/https/repo1.maven.org/maven2/org/apache/pdfbox/pdfbox/1.8.16/pdfbox-1.8.16.jar:…

Things that are not applications

The launch and bootstrap commands were originally written to launch applications from their Maven coordinates, like

$ cs launch com.lihaoyi:::ammonite:2.0.4 -M ammonite.Main
$ cs bootstrap org.scalameta::scalafmt-cli:2.6.3 -o scalafmt

Application descriptors allow to gather the dependencies and parameters passed to these commands.

Note that you can add dependencies alongside an application. The following starts sqlline with the PostgreSQL JDBC driver, allowing to connect to a PostgreSQL database:

$ cs launch sqlline org.postgresql:postgresql:42.2.14 -- \
    -u jdbc:postgresql://localhost/postgres -n postgres -p mysecretpassword
sqlline version 1.9.0
0: jdbc:postgresql://localhost/postgres> select 1;
+----------+
| ?column? |
+----------+
| 1        |
+----------+
1 row selected (0.007 seconds)

Reverting `cs setup`

Pass --try-revert to cs setup to revert what it installed:

$ cs setup --try-revert
  Warning: the --try-revert option is experimental. Keep going only if you know what you are doing. [y/N] y
JVM system|adopt@1.8+ was not installed
JVM system|adopt@1.8+ not setup
Removed /Users/alexandre/Library/Application Support/Coursier/bin from PATH in ~/.profile, ~/.zprofile, ~/.bash_profile
Uninstalled amm
Uninstalled cs
Uninstalled coursier
Uninstalled scala
Uninstalled scalac
Uninstalled sbt
Uninstalled scalafmt

Beware that --try-revert also removes things that were manually set up, not only what was install by a cs setup run.