Tuesday, November 24, 2015

Building configurable applications

As it usually is the case recently I've been reviewing options to build web applications that can be easily transferred between environments. I found some very interesting examples on what can be done to achieve some quite interesting results. But let's start at the beginning..

The project has been released to the public as com.github.testdriven:cfgagent:1.0.0

The problem

The problem usually is that the application we develop tend to have too many configuration options. Database connection string, SMTP server settings, file locations (more than one), configuration of connection pool... just to name a few. As the number of options grows (the project I'm working on now has about 200 of those) passing them all in the command line using JAVA_OPTS is just not possible. And I don't mean like "not practical" but plain and simple not possible due to the limits of what a command-line can take. There has to be a better way to do it.

There actually is a mechanism that one can use with Tomcat called catalina.properties (and I'm sure one would find similar ones in other containers) but that is not going to fly if we want to select which set of options we want to use for this particular run or if the values should come from environmental variables (for example from Docker or Heroku).

JVM Agent to the rescue

I've been looking for a while for a nice, pluggable solution to this problem. My idea was roaming about something that could be fixed but parameterized at the same time, could be specified at execution time (just like JAVA_OPTS are) and in general to be just easy to use regardless of the execution environment.

Looking at it from different angles I turned my attention to what is executed before the main method. As it turns out there is such a thing and it is the concepts of Java Agents.

Without further due let's see the solution:

The code is pretty much self explanatory. First we load the properties from a file name given to the agent (default: system.properties), then we replace all the {placeholders} with environment variable values.

Example usage

At first we create a Docker (instantly cool) container with Oracle. Then we start Tomcat with system properties configured based on the system.properties.

Conclusion

This small utility can easily be used with any type of process where the _OPTS part is getting way too long to be maintained. I've tried it with JBoss, Tomcat, command-line and it worked great every single time :) It also has this nice property that you compose behaviors instead of hard coding them in every application and your products depend only on the concept of system properties (System.getProperty()) that is already present in Java

Happy coding!

It feels great to be blogging again :)

Monday, November 23, 2015

Running SQL queries - the groovy way

Recently I've been tasked with the creation of a utility that executes predefined queries against a database via a command-line interface. A maintenance-sort of utility. I thought this is a a perfect opportunity to freshen up my Groovy DSL skills and see what I can put together to make the code readable.

The idea

The generic idea was to create an object that would describe what the query is all about, with placeholders, then fill in those placeholders and finally execute the statement. It's a one-off operation so we won't be concerning ourselves with connection pooling and the like. Let's KISS.

The implementation

What I'm about to show you here is an over simplified version of a implementation that might be useful. Actually it is quite a bit of overhead for the simplicity Groovy already has in its toolset (the groovy.sql.Sql class) but it opens up a way of thinking of interoperability with the database.

So here we go. First we create a class to house the main property (the query to be executed) and allows for switching context to one where only execution-time properties play a role. In that switched context we execute the query and complete the execution.

Imagine now that instead of just using the Object[] array one would allow for a map to be passed on, like so:

It really doesn't get much more complex than this in regard to predefined queries :) If you'd extract the engine that fills in templates from placeholders and a map of keys (placeholders) and values you could apply the same to pretty much everything that takes a string and parameters and executes it, for example the execution of external commands :)

Happy coding

Sunday, November 22, 2015

Algorithms, data structures, accidental and essential complexity, inheritance, composition and functional programming

Today we're going to go through the tools a programmer has in order to see how and when we can use them to solve programming problems.

Let's start with some of the good old slogans that we have been fed for years:


Every loop can be replaced with functional programming
Every decision can be replaced with inheritance
Every inheritance can be replaced with composition

Does it mean that all our inheritance structures should be 2 level (Object -> MyClass) and that inheritance for itself is bad? Does it mean that if we use a switch/case or an if we're committing a crime against purity? Are those just relics of an era that's come an gone? What do you think?

I strongly believe if there is a tool (even like the goto instruction) and even if the fashion for it has been mentally superseded with a new construct (or even an older one!) then it is still valid to use it in certain contexts. For example goto in the context of text parsers is still very much valid and in use for years to come.

So how about loops and conditions? When is the right place to use them?

First things first - inheritance

Let's tackle the inheritance first because it is the simplest one. You inherited genes from your parents because you're a man. If you'd be a dog you'd not inherit a thing from your human owners as they are not your parents (although your mrs. loves you very much!). An employee is most probably a human being (although there are examples where that would not be true) and a car is a vehicle (when it works otherwise it's a problem).

To put it bluntly: whenever what you inherit is a thing you inherit from then you're doing the right thing. When you're inheriting because you'd like to have that same functionality but extended or twister then you're committing a felony and you should burn in hell.

Complexity

Let's take a look at the types of complexity we're working with on a daily basis. As we've been already told there is essential complexity (driven by the complexity of the domain we're working with) and accidental complexity (driven by the technical details of framework/language you're working in). A good example everyone can relate to is sorting so let's take a look at 2 quicksort implementations:

It is pretty straightforward, right? At first we're slicing the input in half, figuring out the elements that are lesser than the middle point and swapping them out with those that are to the right. Basically we're grouping elements smaller, equal and greater than the pivot and sorting those groups further until there is anything to sort. Efficiency aside the algorithm presented here is pretty clear. Can it get any clearer?

At first when I described the algorithm to you in the above paragraph I told you not how we're grouping things but that we do group them. I even gave you the recipe on assigning elements their destination group. This in turn describes the essential complexity of the algorithm. Can we remove all the accidental complexity? Is that even possible? Let's try it out with Groovy:

Try to read the above code and explain to yourself what it does and how it does it. Immediately you'll notice that it lacks something. I mean it must lack something, right? The Java implementation is 56 lines long and this is just 6 so there surely is a huge gap in those two implementations. But is there really a gap? First we define the exit criteria (list size less than 2), then we figure out the middle point, then we group the elements by their comparative ratio (if there are no elements then return an empty list), then we take all the smaller, sort them, add the equal to the pivot and then sort the bigger ones and add them to. I mean it takes more words to describe what is taking place than really reading the algorithm!

By the way... Take note of the if statement. It's there because the algorithm dictates it.

I think we should go over one more example, maybe less algorithmic but definitely very useful: running external applications from Java.

It is again, very straight forward: get the hold of the runtime, execute the command, read line-by-line the output from a buffered reader. Done. Can it be any simpler?

Wooow! Now that is really the salt itself: just execute the command I gave you and print out the text that came out of it. It's almost like writing a bash script but with the indescribable potential you get when using a full-fledged, mainstream JVM-based programming language.

Personal preference aside I think those two examples demonstrate pretty clearly that software can be created much faster and in a readable fashion. Obviously I'm far from suggesting you write your own quicksort implementation. Those types of lego pieces are already in place and you don't really need to worry about them. But you do need to worry about your algorithms.

Algorithms and data

Remember when you have been introduced to the SOLID principles? My God is that a bizarre harness for a developer when he hears about this the first time! Hands get soaking wet, head shakes in disbelief, and immediately questions like "why" and "how" start to raise but what it really is are means to implement extendable, easy to understand programs.

Single responsibility principle tells you that a piece of code shall only have one reason to be changed. That reason would be for example change in the algorithm but a you probably already know those change very infrequently.

Open/Closed principle tells you to write your software in such way that you can extend it without recompiling.

Liskov substitution principle tells you that no matter which implementation you're going to use the outcome of the algorithm shall stay the same and there should be no difference in the outcome when that algorithm ends. That don't mean it there can only be a single implementation because the sole purpose of doing a different implementation is to inflict change. But the fact is that if your algorithm is generic enough it will always work the same and the changing parts will be properly externalized. A fantastic example here is the template method pattern and composition over inheritance is JDBCTemplate in Spring JDBC. It allows you to tell "in place" what the behavior you'd like to inflict on the result of your query is and takes aside all that nagging boiler-plate away. No matter what you do the algorithm there will always do the same which is iterate over the ResultSet and call your method when appropriate.

Interface segregation principle tells you to only expose as much of functionality as the user of your code would. So for example the JDBCTemplate could theoretically give you the option to close the ResultSet but it doesn't do it because you'll never need it (and if you do you're doing something wrong)

Dependency injection principle modeled after one of the the Hollywood's most famous quotes, Don't call us - we'll call you, is also very well observed in the JDBCTemplate where the algorithm in that implementation takes complete control over the flow of your program and calls you back when your intervention is required.

In addition to this Niklaus Wirth came out with this book of his, Algorithms + Data Structures = Programs. So what's that all about? Has it not been superseded by SOLID principles?

The thruth is that the SOLID principles back up that book's messgage which is to take a point in dividing the essential and the accidental complexity. To make sure you don't include data in your algorithms. What do I mean by this?

Suppose you're writing a very simple program to tell the driver that the current environmental conditions are somehow dangerous. My car has that kind of feature and when it is too cold (below 4°C a subtle alarm will go off and the temperature display will blink a few times. What my car lacks however is that if it is too hot (and that'd be for me > 30°C) then I should be notified by this as well. Otherwise I could fall into the same danger as a boiled frog. I suspect that the program they have has some fixed boundaries, maybe something like this:

What is wrong with this type of programming is not that the temperature is not extracted to some constant or even taken from some function that would combine it with different sensor readings. The bad part of it is that if we would extend it we definitely need to change it and that is not kosher.

That's a whole different story! You can even read the settings from a database or mock the input in test! Cool, ain't it?

Conclusion

Every time you write a program think if what you are doing comes from the essential complexity of the domain you're working with or if it is just because the tools you have are not powerful enough to allow you to express it more elegantly. If you find yourself in the latter position extend the tools (use meta-programming, use reflections, use monkey-patching, use whatever you can) to make sure that if you read the algorithm a year from now it will still read nicely and clearly.

Happy coding!

Friday, August 28, 2015

Maven, Groovy, Appassember...

Today I wanted to show you how to structure a Maven project so that the end result looks like Maven or Ant distribution. Let's start with the basics.

The problem

When you download Maven or Ant you'll notice that the structure has some very well defined folders. There is a bin folder holding a script that you execute, then there is a lib folder where you store all the jars and then there is some kind of configuration folder (conf, etc) that contains the configuration.

Achieving this kind of final project structure with Maven isn't all that hard but it is a combination of at least 2 plugins.

The solution

First of all you need to make sure you are able to package the application properly. I use for that both the appassembler plugin and the assembly plugin. The first one just brings together the necessary project artifacts, introduces platform-specific startup scripts and finishes the final project structure by copying the production configuration files to their proper location. In our case the final structure will look like this:


bin   <-- platform-dependent scripts that start your application are stored
conf  <-- production configuration files are stored
lib   <-- all the jars live here

Next we use the assembly plugin to package everything nicely into a zip file for distribution. It makes sure the unix/linux scripts have the executable bit set too :)

The second part is enabling Groovy language in the project. There are many ways to use Groovy with Maven but my favorite is the eclipse-groovy-compiler. What it does is it does an automatic joint compilation of both .groovy and .java files so you can interchangeably use Groovy from Java and Java from Groovy. I mean it! You see absolutely no difference there (besides the obviously cleaner Groovy code). You put all files into the obvious /src/main/java folder and that's it!

Since we already have Groovy on the classpath and the Groovy compiler processes all the sources wouldn't it be nice to use it in some tests? How about Spock? There is a catch: there are 2 types of tests you would normally write: unit tests/specs and integration tests/specs. To address both needs we'll use 2 Maven plugins: the standard maven-surefire-plugin for everything unit-related and the maven-failsafe-plugin for all them integration tests. Easy!

And last but not least: I like being able to execute the tool right from Maven's command line. To do that I use the exec-maven-plugin. It's very straightforward. You just define the main class, give it some id, done.

Summary

It may seem like a lot of work to set it all up and I certainly didn't do it all at once. But now that I have this in one place I am finding myself using it more and more often. And it just works! And it makes it dead easy to strip parts out to use it for other projects, for example web applications, integration-tests-only module - you name it!

You can explore a sample project that has all that configured on my GitHub page: https://github.com/padcom/groovy-example. It's all nicely documented to make it easier to follow and customize. All the Java and Groovy source code is obviously trash but is only there to illustrate the idea. You can make use of all the configuration like so:


mvn clean install exec:java

You'll see the unit tests/specs executed as part of the build, then appassembler will kick in and build the final structure, then assembler will package it all nicely into a zip file. Next the integration tests will be executed using the failsafe plugins and both artifacts (the jar and the zip) will be installed in your local maven cache (~/.m2/repository by default). Of course you can also find it all in the target folder where it is all initially built.

Enjoy!

Monday, June 22, 2015

Building runC on Ubuntu

I just got really excited with runC after watching the intro presentation on dockercon and wanted to try it out. It seems my knowledge of go programming language and its ecosystem was just not enough to just go ahead and do it and a little bit of googling was required. To ease the pain for the newcomers here's how I did it.

First you need to clone the repository, obviously:
git clone https://github.com/opencontainers/runc
Then you need to make it. To do so you obviously need to have go installed. I installed mine from the following PPA
sudo apt-add-repository -y ppa:evarlast/golang1.4
sudo apt-get update
sudo apt-get install golang
Then you need to do the building
GOPATH="$(pwd)" PATH="$PATH:$GOPATH/bin" make
The last step is installing it which needs to be done as root
sudo make install
That's it! Then you can go an play with runc - it's AWESOME!

Sunday, February 8, 2015

Reading encoders with Arduino

I know that the encoders topic has been done to the death already (it seems it's next in line with the blink example) but I've nowhere seen it done the way I'll present here.

1. We know that 0b10 == 2, 0b11 == 3, 0b00 == 0 and 0b01 == 1 - those are just basic 2-bit values
2. We can put 2 values like that in a nibble (half-byte) so a uint8_t is more than able to house that value
3. We'll assume pins 2&3 (PC1 and PC0)

So the reading could be like so:

volatile uint8_t state = 0;
volatile int32_t counter = 0, oldCounter = 0;
// for full-stop encoder reads only use this state changes
volatile int8_t QEM[16]  = {
 0, 0,  0, 0, -1, 0, 0, 1, 1, 0, 0, -1, 0,  0, 0, 0
};
// for full quadrature decoding use this state changes
// volatile int8_t QEM[16]  = {
// 0, 1, -1, 0, -1, 0, 0, 1, 1, 0, 0, -1, 0, -1, 1, 0
// };

void setup() {
 // configure pin direction
 DDRC &= ~(1<<PC1);
 DDRC &= ~(1<<PC0);

 // enable pullups
 PORTC |= (1<<PC1) || (1<<PC0);

 // load initial encoder values
 if ((PINC & (1 << PC1)) != 0) state |= 0b00000010;
 if ((PINC & (1 << PC0)) != 0) state |= 0b00000001;
}

void loop() {
 // make space for current state; we just want the lower nibble
 // mask everything else out
 state = (state << 2) & 0x0F;

  // read the current state into the lower part of the nibble
 // is half a nibble a nib? ;)
 if ((PINC & (1 << PC1)) != 0) state |= 0b00000010;
 if ((PINC & (1 << PC0)) != 0) state |= 0b00000001;

 // At this stage the state variable contains 4 bits of information
 // containing the previous state and the new state that can be
 // directly used as an index - just the array needs to be a little
 // bit different.
 // 0b0010 = -1
 // 0b0001 =  1
 // 0b1000 =  1
 // 0b1011 = -1
 // 0b1110 =  1
 // 0b1101 = -1
 // 0b0100 = -1
 // 0b0111 =  1
 // which results in the array defined above

  // next we use the state value as an index to the array
 // we do this only on full encoder stops
 counter += QEM[state];

  // react on counter change
 if (oldCounter != counter) {
  oldCounter = counter;

   // counter changed - process
 }
}

Now this is all nice if your loops are quick but if they aren't you're going to loose precision. To get it back we need to change to interrupts - it's not that difficult!

volatile uint8_t state = 0;
volatile int32_t counter = 0, oldCounter = 0;
// for full-stop encoder reads only use this state changes
volatile int8_t QEM[16]  = {
 0, 0,  0, 0, -1, 0, 0, 1, 1, 0, 0, -1, 0,  0, 0, 0
};
// for full quadrature decoding use this state changes
// volatile int8_t QEM[16]  = {
// 0, 1, -1, 0, -1, 0, 0, 1, 1, 0, 0, -1, 0, -1, 1, 0
// };

ISR(PCINT1_vect) {
 // make space for current state; we just want the lower nibble
 // mask everything else out
 state = (state << 2) & 0x0F;

 // read the current state into the lower part of the nibble
 // is half a nibble a nib? ;)
 if ((PINC & (1 << PC1)) != 0) state |= 0b00000010;
 if ((PINC & (1 << PC0)) != 0) state |= 0b00000001;

 // At this stage the state variable contains 4 bits of information
 // containing the previous state and the new state that can be
 // directly used as an index - just the array needs to be a little
 // bit different.
 // 0b0010 = -1
 // 0b0001 =  1
 // 0b1000 =  1
 // 0b1011 = -1
 // 0b1110 =  1
 // 0b1101 = -1
 // 0b0100 = -1
 // 0b0111 =  1
 // which results in the array defined above

 // next we use the state value as an index to the array
 // we do this only on full encoder stops
 counter += QEM[state];
}

void setup() {
 // configure pin direction
 DDRD &= ~(1<<PC1);
 DDRD &= ~(1<<PC0);

 // enable pullups
 PORTC |= (1<<PC1) || (1<<PC0);

 // enable pin-change interrupts for PC1 and PC0
 PCICR |= (1<<PCIE2);
 PCMSK2 |= (1<<PCINT18) | (1<<PCINT19);

 // load initial encoder values
 if ((PINC & (1<<PC1)) != 0) state |= 0b00000010;
 if ((PINC & (1<<PC0)) != 0) state |= 0b00000001;

 // enable interrupts
 sei();
}

void loop() {
 // react on counter change
 if (oldCounter != counter) {
  oldCounter = counter;

  // counter changed
 }
}

As you can see the ISR-driven method isn't all that different from the polling one and gives a significant advantage in terms of flexibility and reliability.

Many thanks for dr Robert Paz for his series of lectures on Arduino programming!

Have fun!

Using the Arduino environment with Eclipse

I've recently fallen in love with the AVR MCUs especially because of the Arduino and it's hugely successful Arduino IDE. It seems that if there is a piece of hardware, a sensor perhaps, then Arduino has a library for it that you can use. It's just great!

There's however one small problem: you need the IDE to build it, the IDE is very small in capabilities (there's no code insight help like in Eclipse for example) and everything seems to be a bitt ... I don't know how to call it.. "amateur" is the word I guess... There's nothing wrong with doing things this way - unless you're used to using something more effective than the notepad for coding - like I am.

So.. You have the Arduino IDE, you have UNO board (possibly a cheep chinese clone) and you have done the Blink example to the death. Now it's time to do some serious damage!

We're going to need the following:

- a couple of packages (sudo apt-get install avrdude avr-libc gcc-avr make openjdk-7-jdk)
- Eclipse for C/C++ developers
- AVR plugin for Eclipse CDT
- Arduino 1.0.6 IDE (may work for a newer one, haven't checked it yet)

I'm going to assume you have downloaded Eclipse, installed the AVR plugin as instructed on the
AVR plugin page and extracted the Arduino IDE into the /tmp folder so you have the /tmp/arduino-1.0.6 location with all the things in it).

First we need to create a programmer to make Eclipse happy. To do so click Window -> Preferences, expand the AVR node and select AVRDude. In the list of programmers click Add, enter "Arduino" as the name of the programmer, select Arduino from the Programmer Hardware list, optionally if you're using a Nano or Mini enter the proper port (I for example have had to enter /dev/ttyUSB0 and I'm using an Arduino Nano with Optiboot) and click OK.

Next we need a project that will use the Arduino environment. Let's create one:

File -> New -> C++ Project
Enter project name, select "Makefile project" and then "AVR GCC Toolchain" and click "Finish".

Next bit is a bit tricky so you need to follow it to the letter. In your freshly created project create a folder called "arduino" and copy there the entire content of the following directories:

/tmp/arduino-1.0.6/hardware/cores/arduino
/tmp/arduino-1.0.6/hardware/variants/standard (that's actually just one file)
/tmp/arduino-1.0.6/libraries/Wire
/tmp/arduino-1.0.6/libraries/Wire/utility
(and any other library you'd like to use)

That'll give you a mini version of the environment to use. Next we need a way to build it. That's quite easy - you just drop in this Makefile into the arduino folder and type "make clean all" and you're done. When you want to use more libraries just drop their files in there and make clean all again.

One last thing to enable Eclipse to understand Arduino libraries. We need to define the symbol ARDUINO with value 106. To do this select Project -> Properties, then C/C++ Build -> Build Variables, click Add, enter "ARDUINO" into "Variable name" and "106" into "Value", click OK, then OK the properties dialog and you're all set!

Now for the fun part - let's create a blinker!

For that we need to create a new file, let's call it sketch.cpp (File -> New -> File, enter the name, press enter, done). In that file we'll enter the following:

#include

void setup() {
pinMode(13, OUTPUT);
}

void loop() {
delay(500);
digitalWrite(13, LOW);
delay(500);
digitalWrite(13, HIGH);
}

This is basic Arduino thing with the addition of one extra include file at the beginning of the file. Nothing more. Now to build that we'll need this Makefile in your project's folder.

Now all you need to do is select "Project -> Make target -> Build..." which will open a list of known targets. Click "Add" and enter "clean load" (that's the set of commands to build and upload the sketch to your board). Unfortunately due to a bug you'll have to close this window and re-open it to see the target you just added - do that and then select it and press "Build". And presto! your Arduino blinks!

There's an added bonus: you can now build your project without any IDE, for example using some sort of continuous integration or something - and ride like a pro! :)