Showing posts with label Java. Show all posts
Showing posts with label Java. Show all posts

Thursday, February 18, 2016

Happy OpenTSDB Users With JVM Garbage Collection Tuning

Until last December, I considered Java garbage collection as something to be relegated to the JVM. I believed that the only impactable control we had over it was how much garbage we generated - and would strive to reduce the number of object creation to a minimum. I thought that garbage collection optimization and tuning did not have any significant or noticable impact.

However I was wrong - and had the opportunity to correct that misconception. The holiday season in December gave me some time to attend to a nagging latency problem with an application called OpenTSDB (or simply TSDB). Its a simple, yet very effective, scalable and highly performant timeseries datastore used primarily for monitoring and metrics data. Our usecase was nontrivial with 550+ million metrics per day being injected into the datastore and 20-30 concurrent users monitoring and displaying graphs that would fetch upto a few million metrics per graph and refreshed periodically. With so much activity going on in the system, it was not realistic to get sub-second graph refreshes - and I deemed it acceptable to have some latency to refresh a graph that fetches say, 2 million data points at each refresh. In TSDB once a graph is loaded on the browser, it does not disappear when the next refresh is scheduled. In fact, it is replaced only when the next graph is ready. Thus the user does not see a blank screen, but a graph that is replaced immediately.

Anyway, I decided to give this issue some attention. This required pouring over the TSDB logs and turning on garbage collection logging. As a result, I found that users were experiencing two kinds of latencies from TSDB. The first kind were long GC pauses that occurred quite frequently within TSDB. The second were delays from HBase, the underlying NoSQL datastore. This article focuses on the former.

With the frequent long gc pauses, it was obvious that the TSDB daemon JVM was working very hard to manage memory reclamation. Memory churn (object creation and collection) was probably more a measure of the activity in the system and not something that I could easily and directly influence. Since reclamation was triggered when the fractions of free heap memory falls below a threshold, it seemed that the JVM was crossing that used memory threshold very frequently. There were 2 easy options to help dampen it - increase heap memory or lower the free heap memory threshold(i.e. increase the occupancy threshold). I increased the TSDB memory limit from 6GB to 12GB in 2GB increements and saw noticable reduction in the frequency and duration of garbage collection.

By this time I had given gc a good look over and realized there were still a number of tuning knobs. But first, given the parallel, CMS and G1 garbage collectors, I wanted to find one that would go a long way - one that would give a decent improvement without any explicit tuning. And that work led to the revelations in this presentation. 

For the TSDB usecase, I discovered that G1 performed best out of the box. However, I did see occasional multi-second GC pauses. While 90% of the GC pauses were reasonable (within 200ms), the 10% of the GC pauses were long. However G1 did have the lowest gc frequency and the least amount of total gc time. The only further tuning that I had to do was to add the following JVM flags and things became smooth.

-XX:InitiatingHeapOccupancyPercent=70 
-XX:MaxGCPauseMillis=100 
-XX:+ParallelRefProcEnabled

InitiatingHeapOccupancyPercent changes the value at heap occupancy level (in %) at which the G1 garbage collection will start a concurrent garbage collection cycle. The default value of 45 is a little too early and does not (or did not) result in much garbage collection.

As the name indicates, MaxGCPauseMillis is a goal for the garbage collector to try to adhere to. Essentially G1 collector will use heuristics from previous cycles to estimate the amount of space it will try to collect to meet the GC pause goal.

And ParallelRefProcEnabled enables multiple/parallel threads during the concurrent collection cycle of the Old generation.

These changes and few changes at the HBase level (related to compaction and hstore tuning) got us back happy TSDB users again.

Thursday, September 17, 2015

I am a Java Geek - Why Should I Learn Scala?

Introduction

Java is undoubtedly the most common enterprise application programming language. However, its not usually the language of choice when it comes to writing small scripts, single-use informal code or code to glue together other scripts. In such cases, many resort to Python.

Thus there often seems to be a dichotomy within enterprises when it comes to programming languages - with Java being the language of choice for application development and Python being popular among data analysts, data scientists, sysadmins and devops. The later groups of people seem to like the interactive shell (REPL), the rapid prototyping, built-in support for high level constructs like file access and manipulation, high level internet protocol support, JSON parsing and processing, etc. 

Well, Scala helps resolve not only this dichotomy but also unite object-oriented and functional programming paradigms:

Scala = Java + Python + functional programming = OOP + FP + Rapid prototyping

Scala is an object-oriented language similar to Java and Python that comes with a REPL-based interactive shell for quick prototyping and has a lot of functionality and features bundled with the core language. And better still, its a language that marries OOP and FP as equal partners.

Isn't this argument reason enough for you to try out Scala? 
I figured you would need more convincing. So here are more reasons.


Scala's DNA and Origins

Martin Odersky, the developer of Scala has distilled into Scala a lot of object-oriented and functional programming features and practical experience, along with lessons learnt while working on Java Generics (introduced in Java 1.5) and the Java compiler. Odersky studied under Niklaus Wirth (inventor of Pascal) and also experimented with several pure OOP and FP languages before embarking on Scala. All this has given a solid foundation to the language's features and capabilities.


Scala Is Not Learning All Over Again

The fusion of OOP and FP is seamless, transparent and each paradigm can be exploited independently as well as jointly (in fact its hard to not use them together seamlessly once you discover their combined power). You can thus approach Scala from either the OOP or the FP camp, get comfortable with the language using familiar paradigms and then incorporate idioms from the other. Furthermore, Scala integrates seamlessly with Java itself - having the ability to use Java libraries and API from Scala and vice-versa. 


Increased Productivity

In the old days (1980s and 19990s), there used to be competitions on writing terse and obfuscated C code. People would try to write code that was often very succinct, hard to read and understand what it was trying to do. These competitions would be hard in Scala - because although the language is succinct, its not hard to to read or understand! As pointed out by Venkat Subramaniam, Scala does away with a lot of ceremony and allows writing code that focuses on the "what needs to be done" rather than decorators and syntactic appendage. Lets look at the example below:

In Scala, you can combine defining a class and its singleton instance by using object as shown above. And just like in Java, when the object is executed by the JVM, it looks for a main method that accepts an array of String. Note the use of Array genenric class typed with String. In Scala, type specification for a generic is done using "[type]" and array indexing is done using "(index)". And you can try out the above code in the Scala REPL shell as shown below:

And if you prefer, you can save it and run as a script as shown below:

Or run the compiled code as shown below:

Ofcourse, most developers will use an IDE like Eclipse or IDEA, but the above examples show the ease of development and prototyping at the most fundamental level. There are many more productivity helpers as discussed next.


Type Inference

Like Java, Scala is a statically typed language. However, it can infer the type of a variable or the return type of a function by analyzing the right-hand-side of an assignment. E.g. the two assignments below are equivalent.

And the concept can be applied when defining functions also (BTW, Scala has "standalone functions" too)

The function myUpperCase above returns a String - although the return type of String is not explicitly specified.


A Powerful match...case

Scala's case expression is very powerful as can be seen from the example below. It forms the basis of several idioms and best-practices in Scala.


Constructors and Built-in Setters and Getters

Scala allows skipping the explicit definition of class constructors by allowing you to define it as part of defining the class signature. The parameters defined in the class signature become the constructor parameters and the statements in the body becomes the body of the constructor. Furthermore, you can skip mundane setters and getters as shown in the example below. Note that the classes below have no "code" - just the signature itself.
Class parameters can be defined as private or as public. If parameters are prefixed with "val" or "var", then they are public.
Class parameters that are defined as "val" (immutable parameter variables) come with "setters" as shown below.

Classes with "var" (mutable parameter variables) come with both "setter" and "getter" functions as shown below:

Under the cover, "ex2.i = 5" is not an assignment, but invocation of a method (but its all transparent).


An Even Better Case With case

As see above, setters and getters come for free when you define public class parameters. But wait, there's more. If you prefix the class definition with "case", you get a "toString" method for free as can be seen here 


Anonymous Functions and Functional Programming

Scala being also a functional programming language, allows using functions and methods to be used in assignments, as function parameters, as return values etc. In many situations, you just want to plug a few lines directly where a function is expected. Scala makes this task easy in the form of anonymous functions. Here's an example



In the example above, "foreach" is an iterator method that requires a function parameter taking one argument. The argument type depends upon the type of the iterated collection  which in this case is of type Int. "x" above is the name for each value each time while iterating from 1 to 10. This "x" is then passed to the explicit function provided as parameter to foreach's function parameter.


More Cool Stuff

There are many more labor-saving, elegant and efficient features that I love - but they need a little more background to really appreciate them. But let me enumerate some of them:
  • Traits (these are the critical foundation for collection classes)
  • Collection classes and concurrent collection classes
  • advanced match...case with pattern matching 
  • Nesting within classes, functions and objects 
  • Companion objects and object's apply methods as default "action"
  • Function currying
  • Ability to use special characters in function names and omission of "()" brackets (this facilitates writing DSLs  or domain specific languages)
  • Ability to define multiple classes in a single file

Finally.....

As you can see, there's so much to Scala - and its best if you peel the layers off yourself at your own pace. The route I took was to use Twitter's crash course and the Neophyte's Guide to get me started and then referring to the Scala Tour, Scana Cheatsheets and the API whenever I wanted to dig a little deeper. And to keep you inspired and motivated, do watch some of Martin Odersky's videos. Then to challenge yourself, try out the 99 problems. Here's a list of some of the online resources that I used and found useful are:

Twitter's crash course on Scala 
The Neophyte's Guide to Scala
The Scala Tutorial, Tour and FAQ
Scala Style Guide
Scala Cheatsheets
Collection article on Scala website
A Concise Introduction to Scala
Scala API Documentation and Source Code Links
Interesting Scala Teasers
Scala Puzzlers
Scala with Style (Video)
Martin Odersky's Video - Working Hard to Keep It Simple
Martin Odersky's Video - The Simple Parts

A note for posterity - this blog article was written in Sept 2015 during the days of Scala 2.10 and 2.11. I am sure there will be many newer and better things in the future!