Use Sbt Subproject for Better Scala Compilation Times

At VNGRS we are using Scala for our customers. Unlike many other scala developers, who are having trouble with long scala compilation time, we are very happy with our development cycle. Our code base is around 80k in size and the whole compilation takes arround 140~150 seconds. Thanks to sbt, we never reach such long compilation times. However, we couldn’t directly figure this out, we had to overcome difficulties through each step and in this blog post I will try to explain those steps. TL;DR use subprojects in large Scala projects

"I can't compile this sh*t"

After we started working in a product we were developing, we found ourselves in a messy development process. Codebases we were responsible for were not compiling at all. There were 5 different Scala repositories and some of those repositories were depending on each other. You had to have one of the repository’s jar file in your classpath, to compile the other repository. Since the former team had the jar files for each repository from the beginning, they could compile projects, but we had only one project as source code. In order to solve this problem, we moved cyclical dependencies to a common project, and published them as an internal artifact. Although we solved compilation issue , it was still possible to add cyclical dependencies between artifacts. We had to manually track the dependency adding process or we had to clear local ivy cache and hit compilation errors. There were also some mysterious cases where some tests were passing for one developer while for others they weren’t. It was due to local caches again.

One project to rule them all

Improving compilation time is not enough, you have to improve your debug cycles, as well. Publishing to local repositories were time consuming. We had to write some scripts to automate this processes, but it was still taking too much time. Thus, we decided to combine all of the projects in one single repository and use sbt’s subproject feature. Initially, we thought that we could use git submodules in order to keep git histories. Then we figured out that we could also use git subtree merge to merge different repositories by keeping git histories, so we decided to take this approach and completed this phase successfully.

Current structure

Right now we have 17 subprojects, some of them depend on each other, but there are no circular dependencies, thanks to sbt. If we add mistakenly a cyclical dependency, sbt raises an error regarding this issue. In such cases, we refactor our codebase and add some traits or we decide that feature we are adding is not in the correct subproject. You might ask how to distinguish between a subproject and a feature. Although, there might be several approaches, we follow this principle: “If it is self deployable, make it a subproject, if not add it to a relevant project”.