Kubernetes Ingress, Nginx Controller, Blue-Green Deployment

This post describes how we use Kubernetes Ingress to create multiple deployments that allow blue-green deployments.We run Kubernetes on AWS.

Initially we took the approach of creating our services with the type LoadBalancer that create AWS ELBs for each service. With the multiplication of services, the number and cost of ELB grew quickly. We also had to make sure that our Route53 records match their respective ELBs, it was error prone and costly.

Enter Kubernetes Ingress, here is what we have now:
- Services of type ClusterIP for each of our deployments
- Nginx Ingress Controller
- Service of type LoadBalancer for Nginx Ingress Controller
- Ingress resources to define the routing of the requests by the nginx ingress controller to the right service

In our case the routing is based on host names. With that solution, our Route53 records all refer to the same ELB which delegates the requests to the nginx ingress controller service.

Blue Green Deployment

To do a blue/green deployment:

• First update the deployment (green) that is inactive/not used with the new version
• Wait until the green deployment is ready 
• Update the service selector to make it use the green deployment
• Scale down the previous deployment "blue" to zero replicas

Here you go, the service is now sending client requests to the new version of the application.

References

https://kubernetes.io/docs/concepts/services-networking/ingress/
https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md
https://github.com/kubernetes/ingress/tree/master/examples/aws/nginx

Kubernetes Flash Cards

While learning to use Kubernetes, I figured I could make flash cards of the concepts. Here are the first ones. What do you think?

Pushing to docker registry running in Kubernetes cluster from Docker Mac

Goal

Push local images from Docker Mac to a remote Docker registry running in a Kubernetes cluster on AWS

Solution

Get ip of your machine (thats the one that docker engine can reach)
$ local_ip=$(ipconfig getifaddr en0)

Define registry.example.com as  in /etc/hosts

  local_ip registry.example.com

Alias lo0 with registration.example.com defined as local_ip in hosts

https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds

I WANT TO CONNECT FROM A CONTAINER TO A SERVICE ON THE HOSTThe Mac has a changing IP address (or none if you have no network access). Our current recommendation is to attach an unused IP to the lo0 interface on the Mac; for example: sudo ifconfig lo0 alias 10.200.10.1/24, and make sure that your service is listening on this address or 0.0.0.0 (ie not 127.0.0.1). Then containers can connect to this address.

$ sudo ifconfig lo0 alias registration.example.com/24

Tunnel :5000 to registry DNS 

$ ssh -N -p 22 user@bastion -L local_ip:5000:registry.example.com:5000

Add local_ip:5000 to docker daemon config insecure registries;

save and restart docker daemon

Push to registration.example.com

$ docker tag example-base registration.example.com:5000/example-base
$ docker push registry.example.com:5000/example-base

References

https://github.com/moby/moby/issues/29608
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds

Download and Socket timeout on EC2

I recently encountered an interesting problem on EC2:
- 2 machines with the same configuration, running the same software
- one is in a private subnet
- one in a public subnet
- none has a public IP
- private subnet use AWS NAT Gateway  (created in VPC panel) to access internet
- public subnet uses AWS Internet Gateway

Problem: on the machine in the private subnet, the download of a big file stalls after downloading a big part of the file and provokes a socket timeout.

Observations:
- the machine on the public subnet has no problem downloading the big files
- both machine can access internet without any issue

After finding that NAT gateway can slow down connection and given the obvious subnet difference between the 2 machines. I recreated the first machine in a public subnet and... now it's able to download big files without issue.

Hypothesis:
After a while the AWS NAT gateway throttle the bandwidth up to choking the connection which creates a time out.

My experience with iOS automated testing frameworks

On my current iOS project, the focus on quality naturally pushed me to go further in automated testing.

They run locally, on simulator, on device and in CI.

Unit tests

GHUnit: http://gabriel.github.com/gh-unit/
OCMockito: https://github.com/jonreid/OCMockito
OCHamcrest: https://github.com/hamcrest/OCHamcrest

Pros

• GHUnit support for asynchronous operations
• Ease of use of OCMockito and OCHamcrest

Cons

?

KIF

https://github.com/square/KIF

Pros

• Run fast and consistently
• Easy access to the application objects since the tests are written in Objective C
• Extensible

Cons

• Tests language is not QA/Client friendly

Extension for long tap / press

As a new method of UIView+KIFAdditions (thanks to: http://blog.dimaj.net/content/howto-long-press-kif ) :

- (void)longTapAtPoint:(CGPoint)point withDelay:(NSInteger)delay andCompletion:(void(^)(void)) completion

{

    // Handle touches in the normal way for other views

    UITouch *touch = [[UITouch alloc] initAtPoint:point inView:self];

    [touch setPhase:UITouchPhaseBegan];

    

    // Create the touch event and send it to the application

    UIEvent *event = [self _eventWithTouch:touch];

    [[UIApplication sharedApplication] sendEvent:event];

    

    // Perform long touch

    dispatch_after( dispatch_time( DISPATCH_TIME_NOW, NSEC_PER_SEC * delay), dispatch_get_current_queue(), ^(void){

        [touch setPhase:UITouchPhaseEnded];

        [[UIApplication sharedApplication] sendEvent:event];

        

        // Dispatching the event doesn't actually update the first responder, so fake it

        if ([touch.view isDescendantOfView:self] && [self canBecomeFirstResponder]) {

            [self becomeFirstResponder];

        }

        

        if (completion) completion();

    });

    

    // Release the touch

    [touch release];

}

Calaba.sh

http://calaba.sh

Pros

• Tests in Gherkin langage, BDD Cucumber
• Powerful CSS like selectors
• Gesture recording and playback : you can record complex gesture
• iOS and Android support

Cons

• Issues with the execution of tests where the interaction with the UI stopped working, did not find why (Button tap not working in some scenarios only)
• Execution of test action not very reliable: gestures stop working, maybe issues with the embedded calaba.sh server.

Embedding a Core Data model from a static library project

In one of my project, I want to share a Core Data model between 2 projects. I have a common static library project where the model and classes are defined.

Situation

• a common project that builds a static library which contains my Core Data model (to be shared with an iOS and an OSX project)
• my main iOS project

Problem

• static library cannot include resources
• how can I give make my main project use the Core Data model ?

Solution

• Create a new target of type Bundle (in OSX/Frameworks and Library)
• Change the build settings of this target:
• Base SDK:  Latest iOS 
• Supported platform: iOS
• Valid architecture: arm architecture
• Add your model file (.xcdatamodeld) to the Build phases / Compile Sources section of the bundle target
• Link your bundle with Core Data (add it to Build phase / Link Binary section)
• Build the bundle target

Now in your main project:

• Select the project to show Build Phases / Copy Bundle Resources
• Drag the Bundle target product (from Products group in the project navigator) to the Copy Bundle Resources section
• If it s not already there, add your static library: in Build Phases / Link binary section, use the "+" button to add your static library file
• Load the model from the bundle:

NSString *staticLibraryBundlePath = [[NSBundle mainBundle] pathForResource:@"ModelBundle" ofType:@"bundle"];

NSURL *staticLibraryMOMURL = [[NSBundle bundleWithPath:staticLibraryBundlePath] URLForResource:@"MyDataModel" withExtension:@"momd"];

model = [[NSManagedObjectModel alloc] initWithContentsOfURL:staticLibraryMOMURL];

Mobile Application Testing

iOS

For iOS, I use the excellent KIF from Square inc. : https://github.com/square/KIF

Although it uses private APIs, it is still very valuable and run fast. It can also run headless on the simulator (think CI server)

Android

Here Robolectric from Pivotal Labs is your friend : https://github.com/pivotal/robolectric

This framework is great because it does not run on the Dalvik VM but as a regular JUnit test suite, that means speed, speed and easy integration in CI.

REST API Testing with Cucumber

To test an API we are working on, I wanted to document/test the API so that client developers would be able to use theses specification to learn how to use the API and the same documentation would be used to test our API.

After some research, I settled (for now) to use Cucumber, json_spec and cucumber-api-steps.

These allow me to write this kind of test against the API:

Feature: User API

Background:

Given the following users exist:

| id | name | password | email |

| 1 | ben | abcdef | ben@email.com |

| 2 | jon | bcdefg | jon@email.com |

When I sign in as "ben@email.com/abcdef"

And I send and accept JSON




Scenario: GET /users

When I send a GET request for "/users"

Then the response status should be "200"

And the JSON should be:

"""

[

{

"email": "ben@email.com",

"name": "ben"

},

{

"email": "jon@email.com",

"name": "jon"

}

]

"""

And the JSON at "0/name" should be "ben"




Scenario: GET users/1

When I send a GET request for "/users/1"

Then the response status should be "200"

And the JSON should be:

"""

{

"email": "ben@email.com",

"name": "ben"

}

"""

And the JSON at "name" should be "ben"




Sproutcore 2.0, Templates and Datastore

With Sproutcore 2.0, there are different ways to use the template views :

- Define a view template in the header of a page, define its View class and instantiate it from the body of the page

- Define a template in the body of the page and have it refer to its View class

Template in the <head> of the page

In the HTML:

<head>

<script type="text/html" data-template-name="account">
MODE:{{mode}}
<h1>{{username}}</h1>
<h1>{{city}}</h1>
</script>
</head>

<body>
<script type="text/html" data-view="App.MyView">
<h1>Hello world! {{title}}</h1>
</script>
<script type="text/html">
{{view App.AccountView}}
</script>

</body>

In the javascript:

App.AccountView = SC.View.extend({
templateName: 'account',
usernameBinding: 'App.accountController.content.username',
cityBinding: 'App.accountController.content.city',
mode: 'DEV'
});

Template in the <body> of the page

In the HTML:

<body>
 <script type="text/html" data-view="App.MyView">
   <h1>Hello world! {{title}}</h1>
 </script>

In the javascript:

App.MyView = SC.View.extend({
 mouseDown: function() {
   window.alert("hello world!");
 },
 title: 'HELLO',
});

Adding the datastore

TBD

Source code

app.js

var App = SC.Application.create({
store: SC.Store.create().from(SC.Record.fixtures)
});

App.MyView = SC.View.extend({
mouseDown: function() {
window.alert("hello world!");
},
title: 'HELLO',
});

App.Account = SC.Record.extend({
username: SC.Record.attr(String),
city: SC.Record.attr(String)
});

App.Account.FIXTURES = [ { username: 'bilou', city: 'new york' } ]

App.accountController = SC.Object.create({
content: SC.Object.create({ username: 'TEST', city: 'San Francisco' }),
city: 'SF'
});

App.AccountView = SC.View.extend({
templateName: 'account',
usernameBinding: 'App.accountController.content.username',
cityBinding: 'App.accountController.content.city',
mode: 'DEV'
});

SC.$(document).ready(function(){
var account = App.store.find(App.Account);
App.accountController.content = account.objectAt(0);
});

Mastery Autonomy Purpose - 1

These three pillars of motivation as defined by D. Pink seem to be a perfect canvas for evaluating one's motivation status.

Let's apply it to my previous jobs:

Mastery

• HTTP, REST API, Restlet framework (+ implemented security customizations)
• Spring, JMS
• OAuth, Web security
• AndroMDA : model driven application and code generation UML -> DB-Java-Hibernate
• Flex / Actionscript : implemented a video player and composable widgets, integration with browser plugin
• Messaging : ActiveMQ, RabbitMQ
• Cloud computing: Amazon Web Services, Rightscale, Engine yard
• Javascript client side: ajax, jQuery
• Video : playback using streaming, pseudo-streaming, download / Wowza Media Server
• Ruby on Rails : proof of concept projects, selection and integration of all plugins needed to cover our needs

Autonomy

• Agile methodology : the team is at the core and drives the project, user stories, sprint
• Scrum : everyone involved/committed as a team (product, qa, dev), daily standup
• Collective design of architecture and technical challenges
• Collaborative management of issues, removal of blockers, celebration
• Involved in the hiring process when building the team

Purpose

• Technically innovative and challenging
• Video is fun
• Engineering was core to the business
• Excellent team, common goal and investment

Personal Kanban and Pomodoro technique

For a few weeks now, I've been using both a personal Kanban board and the Pomodoro technique.
I've been using the Pomodoro technique for a little while now and I really appreciate the gains in term of productivity and focus.
The satisfaction of getting the things done more efficiently feels great. The breaks are fully appreciated: they allow me to rest my shoulders and back as well as to deal with the 'noise' : emails, twitter, etc.

The personal kanban board helps me and my coworkers to visualize what I'm working on. I limit my work in progress and see at any moment what I've accomplished, what's left for the current timeframe.

I invite you to try these tools to enhance your day to day work and feel.

Lean Software Development and Features Injection

I recently discovered 'Feature injection' and started to apply it in my daily work for part of the project. So far I really enjoyed it and did not find a major flaw in the process.
Lean Software Development resonate as common sense, organized way of getting the work done efficiently.

Here is what I retained of it:
1- WHY? : Find the value that you want your product to generate for your user, generally this value allows the user to save time or money, gain productivity, have fun, or event better: be really good at what he is doing using your product ("kick ass user of Y").

2- WHAT? : What are the outputs that represent or give this value to the user ?

3- HOW? : How do you process to get these outputs ? which features produce them ? what are the inputs needed? Build the model during analysis, list all questions and answers, some questions may get an answer later in the process.

At this point we have :
OUTPUTs <- Feature(Process + Model) <- INPUTs

4- Write Feature Tests : for each feature injected we can implement the tests that will pass when the feature is done

5- Analysis and Design of features allow us to Write Unit Tests

6- Implement code to green our Unit Tests, then our Feature Tests : at this point the features that are DONE are ready for final review and release in production

The implementation followed the inverse direction of the analysis so we implemented only what was really needed to produce the outputs that have a value for the user.

Only the necessary features were injected into the system!

The quest of beauty out of the box

In my software engineering work as in all my creations, I want to create beauty. It's the common quest of all creative human beings, I guess. This quest gives a meaning to my actions. I appreciate to create and invent things that I can be proud of and to achieve that I have to let my passion for beautiful creation express itself. Every day I try to remember that no matter what I'm doing I can do it in a way that fullfill my needs to create, innovate, build, enhance. I truly believe that the more beauty you create, the more you receive/see.

It goes the same with happiness: the more happy you make others, the more you are happy yourself !
Ok, easy to say harder to actually do. But if I remind to stay out of the box (when will the french translation of "Anatomy of peace" - Arbinger, will come out ?), life is clearer and much more enjoyable.
At work, with my family and friends, if I manage to stay out of the box, I feel that I can be really myself and feel good about what I am and what I do. Never betray yourself and you'll be happy. Understand your feelings and you'll be able to go out of the box and stay out !
Sounds simple too, I will read and reread again "Anatomy of peace", this book was really a keystone for my life and how I see the world of human beings.

Read you later, I have to create some new nice things today to keep my creative mind happy !

Argumentation or Collaborative thinking

Something I'm not missing at all since I work in California is the apparent need for french workers to argue and win the argument.
For example, in our US company, when we are in a technical meeting, the common goal for everyone is to come up collectively with the best solution for the product we are building. In France the individual goal for each of the participant would be to expose its arguments and argue until they are accepted by the others or until they surrender : the goal is to win the argument ! The product interest is almost completely out of the picture !

The collaborative approach has obviously a very good impact on the quality of the product, in the team spirit, in the sustainability of the efforts that will be deployed to achieve the common goal.
Another great side effect of the collaborative approach vs the arguing one, is that the focus is on the ideas and their suitability to the project/goal, therefore the 'ego' of each participant remains out of the target of the argumentation. If I'm not theatened in my self-estim/ego, I can feel free to suggest new ideas to feed the discussion thus the discussion is richer and more productive.

This leads me to one of the best discovery I've made while working at Veodia : it is actually possible to be recognized and appreciated for your skills and work by your colleague ! Coming after 10 years working in french companies, it sounds revolutionnary. The first time I received an appreciation email from one of my colleague, it completely blew my mind. Since then, I've felt free to express my appreciation of my colleagues work and free as well to express my concerns about it without being afraid of their reaction.
This is an enormous liberation and way of progress. Seeing thing positively, expressing your appreciation of one's work made me feel that I could grow professionnaly in a very happy way.

That spirit made me feel eager to join the team every morning to continue the adventure of building our platform : the joy to go to work !

Update: just read great article on pair programming http://misko.hevery.com/2009/06/12/what-pair-programing-is-not/

Generalizing specialist objectives

New objectives to grow my skills in the art of software development :
- Implement executable tests first at every level : acceptance. functional, unit
cf. "Changing roles"
cf. "Functional TDD"
- Executable tests of UI layer as well as underlying layers
- Continue to become a better generalizing specialist : cf. "Generalizing specialist"

Implementing executable tests first, leads to :
- better understanding of requirements (acceptance tests)
- better design of functionalities (functional tests)
- better design and implementation (unit tests)
Making them executable brings :
- easier regression testing
- better maintainability

That makes a great program for an enthusiast software engineer
eager to become a better generalizing specialist !

UI Model and separation of concern

While reading the excellent "Beautiful Architecture", I found the concepts used by LPS Creator Studio around UI and UI Model very interesting. So I started the implementation of a sample actionscript framework to validate my feelings about this architecture.
So far, I've created Properties (StringProperty, BooleanProperty, CommandProperty) and one form (RegistrationForm). I also started a RegistrationCanvas which represents the 'screen'part : pure UI, no behavior.

The UI elements on the screen are driven through binding to the form and its properties so the UI can stay 'dumb'.
The form gathers the properties and is in charge of the UI business logic : validation, flow, command launching.

I like the separation of concerns that this architecture brings and the fact that it will allow to write unit tests easily for the UI model.