Kontextfrei: A new approach to testable Spark applications

Scalar 2017

08. April 2017 | -

Apache Spark has become the de-facto standard for writing big data processing pipelines. While the business logic of Spark applications is often at least as complex as what we have been dealing with in a pre-big data world, enabling developers to write comprehensive, fast unit test suites has not been a priority in the design of Spark. The main problem is that you cannot test your code without at least running a local SparkContext. These tests are not really unit tests, and they are too slow for pursuing a test-driven development approach. In this talk, I will introduce thekontextfrei library, which aims to liberate you from the chains of the SparkContext. I will show how it helps restoring the fast feedback loop we are taking for granted. In addition, I will explain how kontextfrei is implemented and discuss some of the design decisions made and look at alternative approaches and current limitations.

Museum of the History of Polish Jews, Warsaw
Thumb 1kddbugv

Daniel Westheide is a senior consultant at innoQ Deutschland GmbH and has been developing server applications on the JVM for 7 years. He is particularly interested in functional programming and distributed systems and published the e-book “The Neophyte’s Guide to Scala”.

More content

Comments

Please accept our cookie agreement to see full comments functionality. Read more