Testing With TypeSafe Config
Testing With TypeSafe Config
TL/DR - I found a workable compromise for testing with different configurations using typesafe config without starting a new JVM or turning the code inside out. See this gist.
Simple Structures
TypeSafe Config code loads the config from a hierarchy of sources - primarily files on the classpath and java.lang.System properties. I can access my system’s configuration from anywhere by calling
val config = ConfigFactory.load();
val dbUrl = config.getString("database.url")
I want to keep that simplicity. However, I also needed a good way to test with different configurations. My day-job project, Shrine’s Data Steward Web App, has three different major configurations, maybe a dozen variations.
Specifically, I want the code to use simple structures that pull their configurations from a typesafe Config object. I want to be able to replicate the technique shamelessly without sharing code between subprojects. Further, I do not want that Config exposed in my system’s Scala API; I should be able to use Scala’s singleton [objects] (https://raseshmori.wordpress.com/2013/06/20/scala-part-4-classes-objects/) where I need singletons. At the day job, the configuration is unchanging once set - except for during tests. Having it front-and-center distracts from more important details. I should not have to couple my code via constructor parameters that make me wish for an aspect-oriented style. After all it’s just config.
Three Compromises
I thought of three options. None are perfect, but one is good enough.
Start a New Process
The most brute force approach is to use a different JVM process for testing each configuration. That’s how I’d have handled this back in the age of ant. However, Shrine uses maven. Creating a new JVM process for tests is out-of-model in Shrine and possibly in any maven project. I skipped from ant to sbt. During the age of maven I was at MathWorks where we used make. Maven feels as alien as a three-fingered glove to me. I don’t know a tasteful way to spin up several new processes for running tests in the same maven subproject.
Config Parameter Objects
The older code in Shrine uses config parameter objects, skipping typesafe config, to test parts of the system in isolation. A maven subproject uses typesafe config to drive various abstract-factory-pattern-inspired parts to construct parts for Shrine. Testing different configurations means constructing specialized configuration helpers instead of using a simple Config. The approach definitely solves the problem, but it adds a long, twisty maze between the config files where a value is defined and the code that actually uses it. Following that path adds about five minutes to each task involving a config key-value pair. The approach seems particularly invasive because singletons that could have been Scala objects have to be constructed class instances. The existing pattern tightly couples the Shrine system around a single instance of a ShrineConfig class to keep the whole works from becoming a furball. Testing more than a single subsystem is very difficult. I found myself repeating my criticisms of Spring mixed with profane mutterings about GLOOP. Someday I want to clean up that part of the code, not add to the confusion.
Higher-Order Config Hack
I decided to wrap typesafe config with just a little mutability and use defs where I need configurable values. Using defs for configurable values forces them to be reevaluated each time the owning code accesses them; there will always be a little in-memory overhead. However, any part of the system can access the Config when needed, with a key’s name right next to the def that supplies the value.
def dbUrl = ExampleConfigSource.config.getString("database.url")
My first hack at the solution was to set and clean up system properties in a try/finally block, and use ConfigFactory’s resetCache method. The project is a web app showing a database; I can afford a little compute overhead, but it just seemed sloppy. I won’t share some of the uglier code, but the progression to something clean went fine. The second hack was to put the try/finally into a higher-order function. The third step started to look less hacky. I replaced the cache flush and system properties with API to use Config’s withFallback() to get the default (cached and unchanging) Config. I put the changeable Config inside an AtomicReference for minimal concurrent safety. Finally I dressed it up in a Scala-style Try/Success/Failure . It’s not fool-proof, but should be fine for running one test at a time.
Here’s what the code looks like:
import java.util.concurrent.atomic.AtomicReference
import scala.util.{Failure, Success, Try}
import com.typesafe.config.{Config, ConfigFactory}
/**
* Use to tweak a Config without clearing and reloading a new config (for testing).
*
* @author dwalend
*/
class AtomicConfigSource(baseConfig:Config) {
val atomicConfigRef = new AtomicReference[Config](ConfigFactory.empty())
/**
* Get the atomic Config. Be sure to use defs for all
* config values that might be changed.
*/
def config:Config = atomicConfigRef.get().withFallback(baseConfig)
/**
* Use the config in a block of code with just one key/value replaced.
*/
def configForBlock[T](key:String,value:AnyRef,origin:String)(block: => T):T = {
val configPairs = Map(key -> value)
configForBlock(configPairs,origin)(block)
}
/**
* Use the config in a block of code.
*/
def configForBlock[T](configPairs:Map[String, _ <: AnyRef],origin:String)(block: => T):T = {
import scala.collection.JavaConverters.mapAsJavaMapConverter
val configPairsJava:java.util.Map[String, _ <: AnyRef] = configPairs.asJava
val blockConfig:Config = ConfigFactory.parseMap(configPairsJava,origin)
val originalConfig:Config = atomicConfigRef.getAndSet(blockConfig)
val tryT:Try[T] = Try(block)
val ok = atomicConfigRef.compareAndSet(blockConfig,originalConfig)
tryT match {
case Success(t) => {
if(ok) t
else throw new IllegalStateException(
s"Expected config from ${blockConfig.origin()} to be from ${atomicConfigRef.get().origin()} instead.")
}
case Failure(x) => {
if(ok) throw x
else throw new IllegalStateException(
s"Throwable in block and expected config from ${blockConfig.origin()} to be from ${atomicConfigRef.get().origin()} instead.",x)
}
}
}
}
To use it, I create a Scala object to hold the config:
/**
* A little object to let you reach your config from anywhere.
*
* @author dwalend
*/
object ExampleConfigSource {
//load from application.conf and the usual typesafe config sources
val atomicConfig = new AtomicConfigSource(ConfigFactory.load())
def config:Config = atomicConfig.config
def configForBlock[T](key:String,value:AnyRef,origin:String)(block: => T):T =
atomicConfig.configForBlock(key,value,origin)(block)
}
To change config in a test, wrap the test code in a configForBlock:
"Steward" should " accept query requests with no topic in 'just log and approve everything' mode " in {
ExampleConfigSource.configForBlock("shrine.steward.createTopicsMode",
CreateTopicsMode.TopicsIgnoredJustLog.name){
Post(s"/qep/requestQueryAccess/user/${researcherUserName}",InboundShrineQuery(5,"test query","Not even using a topic")) ~>
addCredentials(qepCredentials) ~>
route ~> check {
assertResult(OK)(status)
}
}
}