Two Styles of Continue and Break in Scala

1. Introduction

In this article, I note how to write 'continue' and 'break' by using Scala in two styles. One is a classic Java style and the other is a elegant style like Japanese calligraphy drawn unicursally. I loved the latter style :)

2. Continue and Break in Scala

2.1. Continue

First, I note the 'continue' in Scala. Test data is the simple Map object.

・ test data

val data = Map[String, Double]( "libor1M" -> 0.1, "libor3M" -> 0.2, "tibor1M" -> 0.3, "libor6M" -> 0.4)

If you write a progam by using Scala in classic Java stile, it may be the following. It's so redundant and the readability is low and not beautiful.

・ 'continue' in Scala (classic Java style)

val exitBlock = new Breaks()
import exitBlock.{breakable, break}

for (one <- data) {
  breakable {
    if (!one._1.startsWith("libor")) {
      break
    }
    println(one._1 + ":" + one._2)
  }
}

But, you can write so simple and elegant code as follows, using Scala. Compared to the above classic Java style, the amount of code is decreased. It's like the beautiful Japanese calligraphy drawn unicursally and so readable :)

・ 'continue' in Scala (elegant style :))

data.withFilter(p => p._1.startsWith("libor")).foreach(f => println(f._1 + ":" + f._2))

The full source code is as follows.

・ full source code

import scala.util.control.Breaks

object test {
  def main(args: Array[String])={
    val data = Map[String, Double]( "libor1M" -> 0.1, "libor3M" -> 0.2,
                                                             "tibor1M" -> 0.3, "libor6M" -> 0.4)
                                       
   
    // I don't like the following code. It's not elegant ...
    val exitBlock = new Breaks()
    import exitBlock.{breakable, break}

    for (one <- data) {
      breakable {
        if (!one._1.startsWith("libor")) {
          break
        }
        println(one._1 + ":" + one._2)
      }
    }
   
   
    // I like the following code :)
    data.withFilter(p => p._1.startsWith("libor")).foreach(f => println(f._1 + ":" + f._2))
  }
}

2.2. Break

Next, I note the 'break' in Scala. Test data is the simple List object.

・ test data

val data = List[String]("libor1M", "libor3M", "tibor1M", "libor6M")

If you write a progam by using Scala in classic Java style, it may the following redundant one in common with 'continue'.

・ 'break' in Scala (classic Java style)

val exitBlock = new Breaks()
import exitBlock.{breakable, break}

breakable {
  for (one <- data) {
    if (one == "tibor1M") {
      println(one)
      break
    }
  }
}

But, as with the case of 'continue', you can write simple and elegant code. It's so readable :)

・ 'break' in Scala (elegant style :))

val result = data.find(p => p == "tibor1M").toString()
println(result)

The full source code is as follows.

・full source code

import scala.util.control.Breaks

object test {
  def main(args: Array[String])={  
    val data = List[String]("libor1M", "libor3M", "tibor1M", "libor6M")
   
   
    // I don't like the following code. It's not elegant ...
    val exitBlock = new Breaks()
    import exitBlock.{breakable, break}

    breakable {
      for (one <- data) {
        if (one == "tibor1M") {
          println(one)
          break
        }
      }
    }
   
   
    // I like the following code :)
    val result = data.find(p => p == "tibor1M").toString()
    println(result)

  }
}

3. Conclusion

In Scala, you can write 'continue' and 'break' in the classic Java Style and the beautiful style like Japanese calligraphy drawn unicursally. Writing in the latter style will reduce the amount of code and make the readability so high.  

Architecture for Overnight Batch Processing and Data Scientists' Work with Apache Spark

I'm on vacation and consider architecture for overnight batch processing and data scientists' work with Apache Spark :)

I. Business flow

The following is a simple business flow I assumed. After overnight batch processing, many data scientists start to work :)

II. Architecture with the use of Parquet

They also request the usage of DataFrames created by overnight batch processing and the performance of queries. In this case, I think architecture with the use of Parquet is suitable.

An example of this architecture is shown below.

DataFrame 1 is converted to Parquet 1. When data scientists want to use DataFrame 1, they load it from Parquet 1. DataFrame 4 is also the same. But not all of columns are loaded. Parquet is a columnar storage format so that it allows loading only selective records.

Therefore, the performance of queries against a Parquet table is higher than against a Text table.

III. Usage of Parquet

How to use Parquet is so easy. We can convert a DataFrame to a Parquet file like the following.

・convert a DataFrame to a Parquet file

dataFrame1.write.format("parquet").save("parquetFilePath")

We can also load a DataFrame from a Parquet file like the following.

・load a DataFrame from a Parquet file

val dataFrame1' = sqlContext.read.parquet("parquetFilePath")
                                              // Not all of columns are loaded. Only selective columns are loaded.

val dataFrame5 = dataFrame1'.join(dataFrame3, dataFrame1'("id") === dataFrame3("id"))
.groupBy(dataFrame3("category1"), dataFrame3("category2"))
.agg(avg(dataFrame1'(value)))