Kubernetes Ingress, Nginx Controller, Blue-Green Deployment

This post describes how we use Kubernetes Ingress to create multiple deployments that allow blue-green deployments.We run Kubernetes on AWS.

Initially we took the approach of creating our services with the type LoadBalancer that create AWS ELBs for each service. With the multiplication of services, the number and cost of ELB grew quickly. We also had to make sure that our Route53 records match their respective ELBs, it was error prone and costly.

Enter Kubernetes Ingress, here is what we have now:
- Services of type ClusterIP for each of our deployments
- Nginx Ingress Controller
- Service of type LoadBalancer for Nginx Ingress Controller
- Ingress resources to define the routing of the requests by the nginx ingress controller to the right service

In our case the routing is based on host names. With that solution, our Route53 records all refer to the same ELB which delegates the requests to the nginx ingress controller service.

Blue Green Deployment

To do a blue/green deployment:

• First update the deployment (green) that is inactive/not used with the new version
• Wait until the green deployment is ready 
• Update the service selector to make it use the green deployment
• Scale down the previous deployment "blue" to zero replicas

Here you go, the service is now sending client requests to the new version of the application.

References

https://kubernetes.io/docs/concepts/services-networking/ingress/
https://github.com/kubernetes/ingress/blob/master/controllers/nginx/README.md
https://github.com/kubernetes/ingress/tree/master/examples/aws/nginx

Kubernetes Flash Cards

While learning to use Kubernetes, I figured I could make flash cards of the concepts. Here are the first ones. What do you think?

Pushing to docker registry running in Kubernetes cluster from Docker Mac

Goal

Push local images from Docker Mac to a remote Docker registry running in a Kubernetes cluster on AWS

Solution

Get ip of your machine (thats the one that docker engine can reach)
$ local_ip=$(ipconfig getifaddr en0)

Define registry.example.com as  in /etc/hosts

  local_ip registry.example.com

Alias lo0 with registration.example.com defined as local_ip in hosts

https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds

I WANT TO CONNECT FROM A CONTAINER TO A SERVICE ON THE HOSTThe Mac has a changing IP address (or none if you have no network access). Our current recommendation is to attach an unused IP to the lo0 interface on the Mac; for example: sudo ifconfig lo0 alias 10.200.10.1/24, and make sure that your service is listening on this address or 0.0.0.0 (ie not 127.0.0.1). Then containers can connect to this address.

$ sudo ifconfig lo0 alias registration.example.com/24

Tunnel :5000 to registry DNS 

$ ssh -N -p 22 user@bastion -L local_ip:5000:registry.example.com:5000

Add local_ip:5000 to docker daemon config insecure registries;

save and restart docker daemon

Push to registration.example.com

$ docker tag example-base registration.example.com:5000/example-base
$ docker push registry.example.com:5000/example-base

References

https://github.com/moby/moby/issues/29608
https://docs.docker.com/docker-for-mac/networking/#use-cases-and-workarounds

Download and Socket timeout on EC2

I recently encountered an interesting problem on EC2:
- 2 machines with the same configuration, running the same software
- one is in a private subnet
- one in a public subnet
- none has a public IP
- private subnet use AWS NAT Gateway  (created in VPC panel) to access internet
- public subnet uses AWS Internet Gateway

Problem: on the machine in the private subnet, the download of a big file stalls after downloading a big part of the file and provokes a socket timeout.

Observations:
- the machine on the public subnet has no problem downloading the big files
- both machine can access internet without any issue

After finding that NAT gateway can slow down connection and given the obvious subnet difference between the 2 machines. I recreated the first machine in a public subnet and... now it's able to download big files without issue.

Hypothesis:
After a while the AWS NAT gateway throttle the bandwidth up to choking the connection which creates a time out.

My experience with iOS automated testing frameworks

On my current iOS project, the focus on quality naturally pushed me to go further in automated testing.

They run locally, on simulator, on device and in CI.

Unit tests

GHUnit: http://gabriel.github.com/gh-unit/
OCMockito: https://github.com/jonreid/OCMockito
OCHamcrest: https://github.com/hamcrest/OCHamcrest

Pros

• GHUnit support for asynchronous operations
• Ease of use of OCMockito and OCHamcrest

Cons

?

KIF

https://github.com/square/KIF

Pros

• Run fast and consistently
• Easy access to the application objects since the tests are written in Objective C
• Extensible

Cons

• Tests language is not QA/Client friendly

Extension for long tap / press

As a new method of UIView+KIFAdditions (thanks to: http://blog.dimaj.net/content/howto-long-press-kif ) :

- (void)longTapAtPoint:(CGPoint)point withDelay:(NSInteger)delay andCompletion:(void(^)(void)) completion

{

    // Handle touches in the normal way for other views

    UITouch *touch = [[UITouch alloc] initAtPoint:point inView:self];

    [touch setPhase:UITouchPhaseBegan];

    

    // Create the touch event and send it to the application

    UIEvent *event = [self _eventWithTouch:touch];

    [[UIApplication sharedApplication] sendEvent:event];

    

    // Perform long touch

    dispatch_after( dispatch_time( DISPATCH_TIME_NOW, NSEC_PER_SEC * delay), dispatch_get_current_queue(), ^(void){

        [touch setPhase:UITouchPhaseEnded];

        [[UIApplication sharedApplication] sendEvent:event];

        

        // Dispatching the event doesn't actually update the first responder, so fake it

        if ([touch.view isDescendantOfView:self] && [self canBecomeFirstResponder]) {

            [self becomeFirstResponder];

        }

        

        if (completion) completion();

    });

    

    // Release the touch

    [touch release];

}

Calaba.sh

http://calaba.sh

Pros

• Tests in Gherkin langage, BDD Cucumber
• Powerful CSS like selectors
• Gesture recording and playback : you can record complex gesture
• iOS and Android support

Cons

• Issues with the execution of tests where the interaction with the UI stopped working, did not find why (Button tap not working in some scenarios only)
• Execution of test action not very reliable: gestures stop working, maybe issues with the embedded calaba.sh server.

Embedding a Core Data model from a static library project

In one of my project, I want to share a Core Data model between 2 projects. I have a common static library project where the model and classes are defined.

Situation

• a common project that builds a static library which contains my Core Data model (to be shared with an iOS and an OSX project)
• my main iOS project

Problem

• static library cannot include resources
• how can I give make my main project use the Core Data model ?

Solution

• Create a new target of type Bundle (in OSX/Frameworks and Library)
• Change the build settings of this target:
• Base SDK:  Latest iOS 
• Supported platform: iOS
• Valid architecture: arm architecture
• Add your model file (.xcdatamodeld) to the Build phases / Compile Sources section of the bundle target
• Link your bundle with Core Data (add it to Build phase / Link Binary section)
• Build the bundle target

Now in your main project:

• Select the project to show Build Phases / Copy Bundle Resources
• Drag the Bundle target product (from Products group in the project navigator) to the Copy Bundle Resources section
• If it s not already there, add your static library: in Build Phases / Link binary section, use the "+" button to add your static library file
• Load the model from the bundle:

NSString *staticLibraryBundlePath = [[NSBundle mainBundle] pathForResource:@"ModelBundle" ofType:@"bundle"];

NSURL *staticLibraryMOMURL = [[NSBundle bundleWithPath:staticLibraryBundlePath] URLForResource:@"MyDataModel" withExtension:@"momd"];

model = [[NSManagedObjectModel alloc] initWithContentsOfURL:staticLibraryMOMURL];

Mobile Application Testing

iOS

For iOS, I use the excellent KIF from Square inc. : https://github.com/square/KIF

Although it uses private APIs, it is still very valuable and run fast. It can also run headless on the simulator (think CI server)

Android

Here Robolectric from Pivotal Labs is your friend : https://github.com/pivotal/robolectric

This framework is great because it does not run on the Dalvik VM but as a regular JUnit test suite, that means speed, speed and easy integration in CI.